Repolode
Submit
lucidrains

lucidrains/vector-quantize-pytorch

A PyTorch library implementing vector, residual, scalar and lookup-free quantization layers for training discrete-latent models such as VQ-VAEs.

View on GitHub
Stars
4,019
Forks
342
Open issues
74
Last commit
29 days ago

What is it?

vector-quantize-pytorch is a Python package that provides ready-made quantization layers for PyTorch. Vector quantization maps continuous neural network features onto a learned set of discrete codes, a technique used in models such as VQ-VAE for image, audio and video generation. The library packages these layers so they can be dropped into a model with a few lines of code instead of being reimplemented from papers.

The core VectorQuantize class updates its codebook with exponential moving averages and returns quantized outputs, code indices and a commitment loss. Around it the library offers ResidualVQ and GroupedResidualVQ for stacking quantizers, along with options such as k-means codebook initialization, cosine-similarity distance, dead-code expiration, and the rotation trick or DiVeQ for gradient flow. Alternative schemes are also included: SimVQ, Finite Scalar Quantization (FSQ), Lookup-Free Quantization (LFQ), Latent Quantization, and a random projection quantizer for speech self-supervised learning.

Each quantizer exposes methods to convert indices back to codes, which is useful when decoding generated tokens. The package is installable with pip and is MIT licensed.

Key features

  • 01Vector quantization layer with EMA codebook updates and commitment loss.
  • 02Residual and grouped residual quantizers that stack multiple codebooks.
  • 03Options for k-means initialization, cosine-similarity distance and dead-code expiration.
  • 04Gradient strategies including the straight-through estimator, rotation trick and DiVeQ.
  • 05Scalar-style alternatives: FSQ, LFQ, Latent Quantization and SimVQ.

Use cases

  • Training a VQ-VAE or VQ-GAN style image or audio tokenizer in PyTorch.
  • Building residual quantization for neural audio codecs.
  • Experimenting with codebook collapse fixes in discrete representation research.
  • Creating a random projection quantizer for masked speech pretraining.

Quick start

  1. 1

    Install the package from PyPI

    pip install vector-quantize-pytorch
  2. 2

    Import VectorQuantize and run it on a random tensor

Things to know

  • Only the VQ layers are provided; you must build and train the surrounding model yourself.
  • Requires PyTorch, and the README examples assume a working PyTorch install.

Topics

#vector-quantization#pytorch#deep-learning#vq-vae#discrete-representation#neural-codec#artificial-intelligence#scalar-quantization

FAQ

What is vector-quantize-pytorch?
A PyTorch library implementing vector, residual, scalar and lookup-free quantization layers for training discrete-latent models such as VQ-VAEs. vector-quantize-pytorch is a Python package that provides ready-made quantization layers for PyTorch. Vector quantization maps continuous neural network features onto a learned set of discrete codes, a technique used in models such as VQ-VAE for image, audio and video generation. The library packages these layers so they can be dropped into a model with a few lines of code instead of being reimplemented from papers.
Is vector-quantize-pytorch free to use? What license does it have?
vector-quantize-pytorch is open source under the MIT license. Check the license terms before using it in commercial products.
Is vector-quantize-pytorch still maintained?
Actively maintained. The last commit to vector-quantize-pytorch was on Sep 24, 2026.
What can I use vector-quantize-pytorch for?
Training a VQ-VAE or VQ-GAN style image or audio tokenizer in PyTorch. Building residual quantization for neural audio codecs. Experimenting with codebook collapse fixes in discrete representation research. Creating a random projection quantizer for masked speech pretraining.
How do I install vector-quantize-pytorch?
Install the package from PyPI: pip install vector-quantize-pytorch Import VectorQuantize and run it on a random tensor
Where can I get vector-quantize-pytorch?
The source code and installation instructions are on GitHub: https://github.com/lucidrains/vector-quantize-pytorch

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects