Repolode
Submit
OpenNMT

OpenNMT/CTranslate2

C++ and Python library that runs Transformer models such as Whisper, Llama and NLLB faster and with less memory via quantization and optimized kernels.

View on GitHubWebsite
Stars
4,716
Forks
547
Open issues
281
Last commit
3 hours ago

What is it?

CTranslate2 is a C++ and Python library for running inference with Transformer models. It uses a custom runtime that applies optimizations such as weight quantization, layer fusion and batch reordering to reduce latency and memory use on both CPU and GPU.

It supports encoder-decoder models (for example NLLB, M2M-100, T5 and Whisper), decoder-only models (such as GPT-2, GPT-J, Llama, Mistral, Gemma and Qwen2) and encoder-only models (BERT, DistilBERT and XLM-RoBERTa). Models must first be converted into CTranslate2's optimized format, and converters are provided for OpenNMT-py, OpenNMT-tf, Fairseq, Marian, OPUS-MT and Hugging Face Transformers.

The library supports x86-64 and ARM64 processors with backends such as Intel MKL, oneDNN, OpenBLAS, Ruy and Apple Accelerate, and it offers Python and C++ APIs with few dependencies.

Key features

  • 01Supports weight quantization to FP16, BF16, INT8, INT16 and AWQ INT4.
  • 02Runs on CPU and GPU, including parallel and asynchronous batch processing across multiple GPUs or cores.
  • 03Automatically selects CPU instruction sets such as AVX and AVX2 at runtime.
  • 04Supports tensor parallelism to split very large models across multiple GPUs.
  • 05Offers Python and C++ APIs with advanced decoding such as prefix completion and alternative outputs.

Use cases

  • Serving machine translation models with lower latency and memory use than general-purpose frameworks.
  • Running Whisper speech recognition or Llama-family LLMs on CPU or GPU servers.
  • Deploying quantized Transformer models on ARM or Apple hardware.
  • Pairing with an OpenAI-compatible web server to expose CTranslate2 models through a familiar API.

Quick start

  1. 1

    Install the Python package with pip

    pip install ctranslate2
  2. 2

    Load a converted model and run translation or generation in Python

Things to know

  • Models must first be converted from a supported framework into CTranslate2's format before use.
  • AMD ROCm GPU support requires specific wheels from the releases page.

Topics

#inference#transformer-models#quantization#machine-translation#deep-learning#cuda#cpu-inference#llm#avx#avx2

FAQ

What is CTranslate2?
C++ and Python library that runs Transformer models such as Whisper, Llama and NLLB faster and with less memory via quantization and optimized kernels. CTranslate2 is a C++ and Python library for running inference with Transformer models. It uses a custom runtime that applies optimizations such as weight quantization, layer fusion and batch reordering to reduce latency and memory use on both CPU and GPU.
Is CTranslate2 free to use? What license does it have?
CTranslate2 is open source under the MIT license. Check the license terms before using it in commercial products.
Is CTranslate2 still maintained?
Actively maintained. The last commit to CTranslate2 was on Oct 22, 2026.
What can I use CTranslate2 for?
Serving machine translation models with lower latency and memory use than general-purpose frameworks. Running Whisper speech recognition or Llama-family LLMs on CPU or GPU servers. Deploying quantized Transformer models on ARM or Apple hardware. Pairing with an OpenAI-compatible web server to expose CTranslate2 models through a familiar API.
How do I install CTranslate2?
Install the Python package with pip: pip install ctranslate2 Load a converted model and run translation or generation in Python
Where can I get CTranslate2?
The source code and installation instructions are on GitHub: https://github.com/OpenNMT/CTranslate2

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects