openvinotoolkit/openvino.genai

What is it?

Key features

• 01Python, C++ and Node.js APIs for running generative AI pipelines
• 02Supports LLM, vision-language, image, video, speech and embedding models
• 03Runs on CPU, GPU and NPU through OpenVINO Runtime
• 04Includes speculative decoding, KV cache eviction and sparse attention
• 05Provides prefix caching and continuous batching for LLM serving
• 06Supports loading and blending multiple LoRA adapters

Use cases

• Running a local chatbot with an open LLM on an Intel laptop or desktop
• Generating images or short videos with Stable Diffusion or Flux models on local hardware
• Building a RAG pipeline with local text embedding and reranking models
• Adding offline speech-to-text with Whisper to a desktop or C++ application

Quick start

1. 1

   Install OpenVINO GenAI from PyPI

   pip install openvino-genai
2. 2

   Export a model from Hugging Face to OpenVINO IR format

   optimum-cli export openvino --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --weight-format int4 --trust-remote-code TinyLlama_1_1b_v1_ov
3. 3

   Run inference with LLMPipeline in Python

   import openvino_genai as ov_genai pipe = ov_genai.LLMPipeline("TinyLlama_1_1b_v1_ov", "CPU") # Use CPU or GPU as devices without any other code change print(pipe.generate("What is OpenVINO?", max_new_tokens=100))

Things to know

• !Runs on top of OpenVINO Runtime, so hardware support is limited to its CPU, GPU and NPU platform matrix.
• !Models must be converted to OpenVINO IR format before use, which requires an extra export step.

In collections

Topics

FAQ

What is openvino.genai?

A C++/Python/Node.js library for running generative AI models such as LLMs, image, speech and embedding models on OpenVINO Runtime. OpenVINO GenAI is a library of ready-made pipelines for popular generative AI models, built on top of the OpenVINO Runtime. It exposes simple C++, Python and Node.js APIs so developers can run text generation, vision-language, image and video generation, Whisper speech recognition, SpeechT5 text-to-speech, and text embedding and reranking workflows without assembling the model pipeline themselves.

Is openvino.genai free to use? What license does it have?

openvino.genai is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.

Is openvino.genai still maintained?

Actively maintained. The last commit to openvino.genai was on Oct 22, 2026.

What can I use openvino.genai for?

Running a local chatbot with an open LLM on an Intel laptop or desktop Generating images or short videos with Stable Diffusion or Flux models on local hardware Building a RAG pipeline with local text embedding and reranking models Adding offline speech-to-text with Whisper to a desktop or C++ application

How do I install openvino.genai?

Install OpenVINO GenAI from PyPI: pip install openvino-genai Export a model from Hugging Face to OpenVINO IR format: optimum-cli export openvino --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --weight-format int4 --trust-remote-code TinyLlama_1_1b_v1_ov Run inference with LLMPipeline in Python: import openvino_genai as ov_genai pipe = ov_genai.LLMPipeline("TinyLlama_1_1b_v1_ov", "CPU") # Use CPU or GPU as devices without any other code change print(pipe.generate("What is OpenVINO?", max_new_tokens=100))

Where can I get openvino.genai?

The source code and installation instructions are on GitHub: https://github.com/openvinotoolkit/openvino.genai