Repolode
Submit
huggingface

huggingface/optimum-nvidia

Hugging Face library that accelerates transformers text generation on NVIDIA GPUs using TensorRT-LLM with a one-line code change.

View on GitHub
Stars
1,039
Forks
106
Open issues
59
Last commit
last month

What is it?

Optimum-NVIDIA is a Hugging Face library for running LLM inference faster on NVIDIA hardware. It wraps the transformers pipeline and AutoModelForCausalLM interfaces, so existing transformers code can switch to the optimized backend by changing the import. The project builds on TensorRT-LLM and exposes options such as FP8 quantization, maximum prompt and output lengths, and batch size.

The project currently accelerates text generation for LLaMA-architecture causal language models. Pip installation is validated on Ubuntu and requires Python 3.10, OpenMPI, and a compatible NVIDIA GPU; a Docker image is also published. FP8 requires Hopper or Ada-Lovelace GPUs, and the project is tested on RTX 4090, L40S, and H100.

Optimum-NVIDIA is aimed at teams serving Llama-family models on NVIDIA GPUs who want higher throughput without rewriting their transformers code. Support for other model architectures and tasks is listed as in progress.

Key features

  • 01Drop-in replacement for transformers pipeline and AutoModelForCausalLM via a one-line import change.
  • 02FP8 quantization on Hopper and Ada-Lovelace GPUs.
  • 03Configurable maximum prompt length, output length, and batch size.
  • 04Pip package and Docker image, with a source build option based on TensorRT-LLM.
  • 05Supports Hugging Face generate() options such as top_k, top_p, and repetition_penalty.

Use cases

  • Serving LLaMA-family chat models on NVIDIA GPUs with higher throughput.
  • Speeding up an existing transformers text-generation pipeline without rewriting application code.
  • Running FP8-quantized inference on H100 or L40S servers.

Quick start

  1. 1

    Install system packages (Ubuntu)

    apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev
  2. 2

    Install optimum-nvidia via pip

    python -m pip install --pre --extra-index-url https://pypi.nvidia.com optimum-nvidia

Things to know

  • Pip installation has only been validated on Ubuntu.
  • Requires an NVIDIA GPU from the Ampere, Hopper or Ada-Lovelace architectures.
  • FP8 is only available on Hopper and Ada-Lovelace GPUs.
  • Currently only accelerates text generation with LlamaForCausalLM.

In collections

Run LLMs locallyRun and train AI on GPUs

Topics

#llm-inference#nvidia#tensorrt-llm#transformers#gpu#fp8#text-generation

FAQ

What is optimum-nvidia?
Hugging Face library that accelerates transformers text generation on NVIDIA GPUs using TensorRT-LLM with a one-line code change. Optimum-NVIDIA is a Hugging Face library for running LLM inference faster on NVIDIA hardware. It wraps the transformers pipeline and AutoModelForCausalLM interfaces, so existing transformers code can switch to the optimized backend by changing the import. The project builds on TensorRT-LLM and exposes options such as FP8 quantization, maximum prompt and output lengths, and batch size.
Is optimum-nvidia free to use? What license does it have?
optimum-nvidia is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is optimum-nvidia still maintained?
Updated within 6 months. The last commit to optimum-nvidia was on Sep 24, 2026.
What can I use optimum-nvidia for?
Serving LLaMA-family chat models on NVIDIA GPUs with higher throughput. Speeding up an existing transformers text-generation pipeline without rewriting application code. Running FP8-quantized inference on H100 or L40S servers.
How do I install optimum-nvidia?
Install system packages (Ubuntu): apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev Install optimum-nvidia via pip: python -m pip install --pre --extra-index-url https://pypi.nvidia.com optimum-nvidia
Where can I get optimum-nvidia?
The source code and installation instructions are on GitHub: https://github.com/huggingface/optimum-nvidia

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects