Repolode
Submit
Wallawalla47

Wallawalla47/Infernix

C++/CUDA inference engine that serves Qwen3.8 and Qwen3.8-27B models on a single NVIDIA RTX 5090 GPU, forked from NInfer.

View on GitHub
Stars
275
Forks
38
Open issues
7
Last commit
yesterday

What is it?

Infernix is an inference server for Qwen language models, built to run on one NVIDIA RTX 5090. It serves OpenAI Chat Completions, Responses and Anthropic Messages-compatible APIs, so coding agents and other clients can connect without changes. The Qwen3.8-Flash-Next models keep routed experts in pinned system RAM with hot experts cached in VRAM, and can read overflow experts from an SSD tier when RAM is limited.

The project supports speculative decoding with MTP, DFlash, DFlash2 and n-gram drafters, along with KV cache formats such as int8, fp8, nvfp4 and vq2. It also covers tool calling, constrained decoding with JSON Schema and GBNF grammars, a prefix cache that persists across restarts, and vision input. A beta web launcher manages models and configurations and exposes a single endpoint for several models.

It builds natively on Windows with MSVC and on Linux with GCC, and includes a Dockerfile. The author notes that much of the code and documentation was written with AI assistance and that this is hobby development, so completeness and accuracy are not guaranteed.

Key features

  • 01OpenAI and Anthropic-compatible APIs for coding agents and other clients
  • 02Expert offloading to pinned RAM and VRAM cache, with an optional SSD tier
  • 03Speculative decoding with MTP, DFlash, DFlash2 and n-gram drafters
  • 04Multiple KV cache formats including int8, fp8, nvfp4 and vq2
  • 05Tool calling, constrained decoding with JSON Schema and GBNF grammars
  • 06Web launcher for managing models, configurations and a single endpoint

Use cases

  • Running Qwen3.8 models locally on a single RTX 5090 for coding agents
  • Serving a local OpenAI or Anthropic-compatible endpoint for tools like Claude Code or Codex
  • Comparing inference speed and quality of different Qwen quantizations on one GPU

Things to know

  • Targets a single NVIDIA RTX 5090 GPU, so other hardware is not supported by its design.
  • The README says it is hobby development and that most of it was written with AI, so it may be incomplete or inaccurate.
  • Its model setup relies on large model files and roughly 84 GiB of RAM, so it is heavy to run.

In collections

Run LLMs locallyRun and train AI on GPUs

Topics

#llm-inference#cuda#qwen#gpu#local-llm#speculative-decoding#openai-compatible#cpp

FAQ

What is Infernix?
C++/CUDA inference engine that serves Qwen3.8 and Qwen3.8-27B models on a single NVIDIA RTX 5090 GPU, forked from NInfer. Infernix is an inference server for Qwen language models, built to run on one NVIDIA RTX 5090. It serves OpenAI Chat Completions, Responses and Anthropic Messages-compatible APIs, so coding agents and other clients can connect without changes. The Qwen3.8-Flash-Next models keep routed experts in pinned system RAM with hot experts cached in VRAM, and can read overflow experts from an SSD tier when RAM is limited.
Is Infernix free to use? What license does it have?
Infernix is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is Infernix still maintained?
Actively maintained. The last commit to Infernix was on Nov 6, 2026.
What can I use Infernix for?
Running Qwen3.8 models locally on a single RTX 5090 for coding agents Serving a local OpenAI or Anthropic-compatible endpoint for tools like Claude Code or Codex Comparing inference speed and quality of different Qwen quantizations on one GPU
Where can I get Infernix?
The source code and installation instructions are on GitHub: https://github.com/Wallawalla47/Infernix

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects