Open-source projects tagged “quantization”

A Rust CLI and TUI that detects your CPU, RAM and GPU, then ranks open LLMs by how well they will run on your machine.

Python library that runs large language models layer by layer so 70B-class models can infer on a single 4GB GPU, without quantization.

A CLI that detects your GPU, CPU and RAM and ranks the HuggingFace local LLMs that actually run on your hardware, by benchmarks rather than parameter count.

Run-and-benchmark record for serving DeepSeek-V4.1-Flash (552B MoE) across four NVIDIA DGX Sparks using TensorFold and vLLM, with launch scripts and measured results.

Local inference engine that runs the large Qwen3.8-Flash-Next model on consumer NVIDIA or AMD GPUs under Windows or Linux, with OpenAI and Anthropic compatible APIs.

An edge-native MoE serving engine that runs large open-weight models like DeepSeek-V4 and Qwen3.6 on consumer GPUs and CPUs.

A minimal C11 LLM inference engine that runs 1B-class LLaMA-architecture GGUF models on low-cost boards with about 45MB RAM, with no dependencies.

Open-weight 4B and 1.7B general-purpose language models from XHToken, aimed at on-device use and agentic tool-calling workflows.