Repolode
Submit
kvcache-ai

kvcache-ai/ktransformers

A research framework for CPU-GPU heterogeneous LLM inference and LoRA/full fine-tuning of large MoE models on limited GPU memory.

View on GitHubWebsite
Stars
19,621
Forks
1,602
Open issues
526
Last commit
2 days ago

What is it?

KTransformers is a research project for running and fine-tuning large language models by splitting work between CPU and GPU. Its inference side provides CPU-optimized kernels for INT4/INT8 quantized weights, with AMX and AVX512/AVX2 acceleration and NUMA-aware memory management for Mixture-of-Experts models. Hot experts can be placed on the GPU while cold experts stay on the CPU, and the kernels offer a Python API for integration with SGLang.

The fine-tuning side integrates with LLaMA-Factory to support LoRA and full-parameter training of ultra-large MoE models such as DeepSeek-V3/R1, Qwen3 MoE and Kimi K2 variants on limited GPU memory. The project also documents BF16, FP8 and INT8 configurations, resource planning and troubleshooting, and publishes a research paper on its heterogeneous MoE fine-tuning approach.

Key features

  • 01Intel AMX and AVX512/AVX2 optimized kernels for INT4/INT8 quantized inference
  • 02CPU-GPU expert scheduling for MoE models, with hot experts on GPU and cold experts on CPU
  • 03Python API for integration with SGLang
  • 04LoRA and full-parameter fine-tuning integrated with LLaMA-Factory
  • 05Support for BF16, FP8 and INT8 training configurations

Use cases

  • Running large MoE models on a single GPU paired with a large-memory CPU server
  • Serving MoE models through SGLang with CPU-GPU hybrid expert placement
  • Fine-tuning ultra-large MoE models on a few consumer GPUs with LLaMA-Factory

Quick start

  1. 1

    Go to the kt-kernel source directory

    cd kt-kernel
  2. 2

    Install kt-kernel from source

    pip install .

Things to know

  • Built for specialized hardware, relying on Intel AMX or AVX512/AVX2 CPUs and GPUs for heterogeneous inference.
  • Positioned as a research project, and the original integrated framework has been archived.

Topics

#llm-inference#fine-tuning#mixture-of-experts#cpu-offload#quantization#lora#sglang

FAQ

What is ktransformers?
A research framework for CPU-GPU heterogeneous LLM inference and LoRA/full fine-tuning of large MoE models on limited GPU memory. KTransformers is a research project for running and fine-tuning large language models by splitting work between CPU and GPU. Its inference side provides CPU-optimized kernels for INT4/INT8 quantized weights, with AMX and AVX512/AVX2 acceleration and NUMA-aware memory management for Mixture-of-Experts models. Hot experts can be placed on the GPU while cold experts stay on the CPU, and the kernels offer a Python API for integration with SGLang.
Is ktransformers free to use? What license does it have?
ktransformers is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is ktransformers still maintained?
Actively maintained. The last commit to ktransformers was on Oct 20, 2026.
What can I use ktransformers for?
Running large MoE models on a single GPU paired with a large-memory CPU server Serving MoE models through SGLang with CPU-GPU hybrid expert placement Fine-tuning ultra-large MoE models on a few consumer GPUs with LLaMA-Factory
How do I install ktransformers?
Go to the kt-kernel source directory: cd kt-kernel Install kt-kernel from source: pip install .
Where can I get ktransformers?
The source code and installation instructions are on GitHub: https://github.com/kvcache-ai/ktransformers

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects