Repolode
Submit
TinyLLaVA

TinyLLaVA/TinyLLaVA_Factory

A modular PyTorch/Hugging Face codebase for building, training and evaluating small-scale large multimodal (vision-language) models.

View on GitHubWebsite
Stars
1,011
Forks
100
Open issues
114
Last commit
19 days ago

What is it?

TinyLLaVA Factory is a modular codebase for small-scale large multimodal models that pair a vision encoder with a language model. It is organized so that the LLM, vision tower, connector and training recipe can be swapped independently, which makes it easier to build custom vision-language models without rewriting training code.

The framework supports LLMs such as OpenELM, TinyLlama, StableLM, Qwen, Gemma and Phi, vision towers including CLIP, SigLIP and DINO, and connectors such as MLP, Q-Former and Resampler. Training recipes cover frozen, full and partial tuning, as well as LoRA and QLoRA. The project also publishes several pretrained models, including TinyLLaVA-Phi-2-SigLIP-3.1B, with benchmark results on VQA-v2, GQA, TextVQA, MME and others.

The repository includes a Gradio web demo, a CLI for inference, example training scripts, and documentation for preparing datasets and running evaluations.

Key features

  • 01Swappable LLM, vision tower and connector components for building custom multimodal models.
  • 02Training recipes for frozen, full, partial, LoRA and QLoRA tuning.
  • 03Pretrained models based on Phi-2, Gemma, Qwen and TinyLlama with SigLIP or CLIP vision encoders.
  • 04Gradio web demo and command-line inference tools.
  • 05Documented data preparation and evaluation workflows.

Use cases

  • Training a small vision-language model on a single GPU for image question answering.
  • Experimenting with different language models and vision encoders in a modular setup.
  • Fine-tuning a multimodal model on a custom image-text dataset with LoRA.
  • Reproducing small LMM training and evaluation results.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/TinyLLaVA/TinyLLaVA_Factory.git
  2. 2

    Enter the project folder

    cd TinyLLaVA_Factory
  3. 3

    Create and activate a conda environment, then install the package

    conda create -n tinyllava_factory python=3.10 -y && conda activate tinyllava_factory && pip install --upgrade pip && pip install -e .
  4. 4

    Launch the local Gradio web demo with a pretrained model

    python tinyllava/serve/app.py --model-path tinyllava/TinyLLaVA-Phi-2-SigLIP-3.1B

Things to know

  • Requires a conda environment and a CUDA GPU for training and most inference, and flash-attn must be built separately.
  • Environment requirements differ from LLaVA's, so existing LLaVA setups may not be reused directly.

In collections

Natural language processing tools & libraries

Topics

#large-multimodal-models#vision-language#pytorch#llava#transformers#fine-tuning#llama#nlp#tinyllama

FAQ

What is TinyLLaVA_Factory?
A modular PyTorch/Hugging Face codebase for building, training and evaluating small-scale large multimodal (vision-language) models. TinyLLaVA Factory is a modular codebase for small-scale large multimodal models that pair a vision encoder with a language model. It is organized so that the LLM, vision tower, connector and training recipe can be swapped independently, which makes it easier to build custom vision-language models without rewriting training code.
Is TinyLLaVA_Factory free to use? What license does it have?
TinyLLaVA_Factory is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is TinyLLaVA_Factory still maintained?
Actively maintained. The last commit to TinyLLaVA_Factory was on Oct 19, 2026.
What can I use TinyLLaVA_Factory for?
Training a small vision-language model on a single GPU for image question answering. Experimenting with different language models and vision encoders in a modular setup. Fine-tuning a multimodal model on a custom image-text dataset with LoRA. Reproducing small LMM training and evaluation results.
How do I install TinyLLaVA_Factory?
Clone the repository: git clone https://github.com/TinyLLaVA/TinyLLaVA_Factory.git Enter the project folder: cd TinyLLaVA_Factory Create and activate a conda environment, then install the package: conda create -n tinyllava_factory python=3.10 -y && conda activate tinyllava_factory && pip install --upgrade pip && pip install -e . Launch the local Gradio web demo with a pretrained model: python tinyllava/serve/app.py --model-path tinyllava/TinyLLaVA-Phi-2-SigLIP-3.1B
Where can I get TinyLLaVA_Factory?
The source code and installation instructions are on GitHub: https://github.com/TinyLLaVA/TinyLLaVA_Factory

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects