Repolode
Submit
google-research

google-research/vision_transformer

JAX/Flax code and pretrained checkpoints for Vision Transformer (ViT) and MLP-Mixer image classification models from Google Research papers.

View on GitHub
Stars
12,770
Forks
1,478
Open issues
137
Last commit
last month

What is it?

This repository releases the code and pretrained checkpoints for Vision Transformer (ViT) and MLP-Mixer, two architectures for image recognition that process images as sequences of fixed-size patches or as all-MLP networks. It accompanies several Google Research papers, including the original ViT paper, MLP-Mixer, the AugReg study on training ViTs, and later work on LiT zero-shot image-text models and sharpness-aware training.

The models are pretrained on ImageNet and ImageNet-21k, and the code provides fine-tuning scripts in JAX and Flax for datasets such as CIFAR-10/100 and any dataset available through TensorFlow Datasets. Many checkpoints are hosted in Google Cloud Storage buckets, including over 50k ViT and hybrid variants with different augmentation and regularization settings. Colab notebooks cover model exploration, fine-tuning and LiT inference, and the timm PyTorch library can load several of the checkpoints.

The project is not an official Google product, and its author notes that more advanced training code, such as multi-host training, lives in the separate big_vision repository.

Key features

  • 01Implements ViT and MLP-Mixer models in JAX and Flax.
  • 02Provides pretrained checkpoints on ImageNet and ImageNet-21k.
  • 03Includes a fine-tuning command-line interface for custom datasets via TensorFlow Datasets.
  • 04Offers over 50k ViT and hybrid checkpoints with varied augmentation settings.
  • 05Includes Colab notebooks for exploration, fine-tuning and LiT inference.

Use cases

  • Fine-tuning a pretrained ViT or Mixer on a custom image classification dataset.
  • Comparing ViT and MLP-Mixer architectures and their training recipes in research.
  • Running zero-shot image-text inference with the released LiT models.

Quick start

  1. 1

    Make sure Python 3.10 or newer is installed

  2. 2

    Install JAX and dependencies for GPU

    pip install -r vit_jax/requirements.txt
  3. 3

    Fine-tune a ViT-B/16 model (pre-trained on ImageNet-21k) on CIFAR-10

    python -m vit_jax.main --workdir=/tmp/vit-$(date +%s) --config=$(pwd)/vit_jax/configs/vit.py:b16,cifar10 --config.pretrained_dir='gs://vit_models/imagenet21k'

Things to know

  • Fine-tuning needs a GPU or TPU, and the README says it uses all available accelerators.
  • The Colab TPU setup is slow, so a dedicated machine is recommended for larger datasets.
  • Installing JAX and Flaxformer requires following separate instructions from their repositories.

In collections

Computer vision libraries & tools

Topics

#computer-vision#vision-transformer#mlp-mixer#jax#flax#image-classification#pretrained-models

FAQ

What is vision_transformer?
JAX/Flax code and pretrained checkpoints for Vision Transformer (ViT) and MLP-Mixer image classification models from Google Research papers. This repository releases the code and pretrained checkpoints for Vision Transformer (ViT) and MLP-Mixer, two architectures for image recognition that process images as sequences of fixed-size patches or as all-MLP networks. It accompanies several Google Research papers, including the original ViT paper, MLP-Mixer, the AugReg study on training ViTs, and later work on LiT zero-shot image-text models and sharpness-aware training.
Is vision_transformer free to use? What license does it have?
vision_transformer is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is vision_transformer still maintained?
Updated within 6 months. The last commit to vision_transformer was on Oct 1, 2026.
What can I use vision_transformer for?
Fine-tuning a pretrained ViT or Mixer on a custom image classification dataset. Comparing ViT and MLP-Mixer architectures and their training recipes in research. Running zero-shot image-text inference with the released LiT models.
How do I install vision_transformer?
Make sure Python 3.10 or newer is installed Install JAX and dependencies for GPU: pip install -r vit_jax/requirements.txt Fine-tune a ViT-B/16 model (pre-trained on ImageNet-21k) on CIFAR-10: python -m vit_jax.main --workdir=/tmp/vit-$(date +%s) --config=$(pwd)/vit_jax/configs/vit.py:b16,cifar10 --config.pretrained_dir='gs://vit_models/imagenet21k'
Where can I get vision_transformer?
The source code and installation instructions are on GitHub: https://github.com/google-research/vision_transformer

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects