Repolode
Submit
ModelCloud

ModelCloud/GPTQModel

A toolkit for quantizing large language models (GPTQ, AWQ, GGUF, FP8, EXL3 and more) and serving them on NVIDIA, AMD, Intel, Apple and CPU hardware.

View on GitHubWebsite
Stars
1,287
Forks
214
Open issues
11
Last commit
yesterday

What is it?

GPTQModel is a Python platform for compressing large language models through quantization. It provides one API for calibration, quantization, quality evaluation, saving, reloading and accelerated inference, covering methods such as GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, VPTQ and GPTAQ.

Quantized models can be loaded through Hugging Face Transformers, Optimum and PEFT, and served through vLLM or SGLang. The project lists kernel support for NVIDIA CUDA, AMD ROCm, Huawei Ascend NPU, Intel XPU, Apple Silicon (MPS and MLX) and Intel/AMD/Apple CPUs. Users can assign different quantization settings to individual layers, exclude modules, and resume interrupted quantization jobs from layer-level checkpoints.

The package installs with pip and exposes an OpenAI-compatible serving endpoint. Model coverage spans a long list of Llama, Qwen, Gemma, Mistral, DeepSeek and vision-language architectures.

Key features

  • 01Supports GPTQ, AWQ, GGUF, FP8, EXL3, QQQ, VPTQ and other quantization methods under one API.
  • 02Loads quantized checkpoints in Transformers and serves them through vLLM or SGLang.
  • 03Runs on NVIDIA, AMD, Huawei Ascend, Intel XPU, Apple Silicon and CPU backends.
  • 04Allows per-module quantization settings and excluding specific layers.
  • 05Resumes long quantization jobs from layer-level checkpoints.
  • 06Exposes an OpenAI-compatible API endpoint for a loaded model.

Use cases

  • Compressing a Llama or Qwen model so it fits in less GPU memory.
  • Producing quantized checkpoints for deployment with vLLM or SGLang.
  • Quantizing models on Apple Silicon with the MLX format.
  • Experimenting with mixed-precision or per-layer quantization recipes.

Things to know

  • The license metadata is NOASSERTION, so the license should be checked directly in the repository before use.
  • Some features need specific hardware, such as NVIDIA SM89+ or GB10 for the experimental W4A path.
  • Recent releases retired or changed several APIs, so existing code may need migration.

Topics

#llm#quantization#gptq#awq#model-compression#inference#gguf#pytorch#optimum#peft

FAQ

What is GPTQModel?
A toolkit for quantizing large language models (GPTQ, AWQ, GGUF, FP8, EXL3 and more) and serving them on NVIDIA, AMD, Intel, Apple and CPU hardware. GPTQModel is a Python platform for compressing large language models through quantization. It provides one API for calibration, quantization, quality evaluation, saving, reloading and accelerated inference, covering methods such as GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, VPTQ and GPTAQ.
Is GPTQModel free to use? What license does it have?
GPTQModel is open source under the Other license. Check the license terms before using it in commercial products.
Is GPTQModel still maintained?
Actively maintained. The last commit to GPTQModel was on Nov 5, 2026.
What can I use GPTQModel for?
Compressing a Llama or Qwen model so it fits in less GPU memory. Producing quantized checkpoints for deployment with vLLM or SGLang. Quantizing models on Apple Silicon with the MLX format. Experimenting with mixed-precision or per-layer quantization recipes.
Where can I get GPTQModel?
The source code and installation instructions are on GitHub: https://github.com/ModelCloud/GPTQModel

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects