Repolode
Submit
ashhart

ashhart/TensorFold

A local LLM inference engine written in Zig that serves models on Apple Silicon Metal and NVIDIA CUDA GPUs without Python or MLX.

View on GitHubWebsite
Stars
1,688
Forks
362
Open issues
145
Last commit
3 hours ago

What is it?

TensorFold is an LLM inference server built as a single Zig binary. It reads model checkpoints, tokenizes requests and runs Metal or CUDA kernels directly, so serving does not require a Python or MLX installation. It targets Apple Silicon Macs and NVIDIA GPUs, including GB10 systems, and exposes OpenAI-compatible chat and completions endpoints plus Responses and Anthropic Messages APIs.

The project maintains a list of qualified models with pinned Hugging Face revisions, covering families such as Qwen, Nemotron, GLM, Gemma and LFM. Each model has its own recipe describing hardware scope, memory needs and limits. Features include request admission limits, context compaction, optional speculative drafting with exact-output guarantees, prompt caching, and reasoning controls.

The repository also includes a desktop workspace for macOS, a 'Living Weights' experiment that writes taught facts into model files, and a contributor workflow built around agent coordination.

Key features

  • 01Runs as a single Zig binary on Metal (macOS) and CUDA (Linux) without Python or MLX.
  • 02Serves OpenAI-compatible chat and completions APIs, plus Responses and Anthropic Messages endpoints.
  • 03Pulls checkpoints from Hugging Face with pinned revisions and checksum verification.
  • 04Limits concurrent requests with a --parallel setting and queues the rest.
  • 05Supports context compaction that keeps recent turns verbatim and summarizes older ones.
  • 06Offers optional speculative drafting, where accepted tokens match the plain decoder output.

Use cases

  • Running open-weight LLMs locally on a Mac with Apple Silicon.
  • Hosting a self-managed OpenAI-compatible endpoint on an NVIDIA GPU server.
  • Serving large checkpoints across two GB10 or two Mac systems.
  • Evaluating model quality and speed on specific hardware with the built-in benchmark command.

Quick start

  1. 1

    Install TensorFold with Homebrew on a Mac with Apple Silicon

    brew install ashhart/tensorfold/tensorfold
  2. 2

    Download the pinned Qwen3.5-2B model weights

    tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
  3. 3

    Start the local model server

    tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking

Things to know

  • Native serving targets Apple Silicon Metal and NVIDIA CUDA GPUs only.
  • Several model paths are experimental, and the README notes known issues for some checkpoints.
  • Model weights carry their own licenses, separate from the Apache-2.0 engine license.

In collections

Run LLMs locallyOpen-source Mac apps

Topics

#llm-inference#metal#cuda#zig#local-llm#openai-compatible#apple-silicon#ai#ai-tools#llm

FAQ

What is TensorFold?
A local LLM inference engine written in Zig that serves models on Apple Silicon Metal and NVIDIA CUDA GPUs without Python or MLX. TensorFold is an LLM inference server built as a single Zig binary. It reads model checkpoints, tokenizes requests and runs Metal or CUDA kernels directly, so serving does not require a Python or MLX installation. It targets Apple Silicon Macs and NVIDIA GPUs, including GB10 systems, and exposes OpenAI-compatible chat and completions endpoints plus Responses and Anthropic Messages APIs.
Is TensorFold free to use? What license does it have?
TensorFold is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is TensorFold still maintained?
Actively maintained. The last commit to TensorFold was on Nov 7, 2026.
What can I use TensorFold for?
Running open-weight LLMs locally on a Mac with Apple Silicon. Hosting a self-managed OpenAI-compatible endpoint on an NVIDIA GPU server. Serving large checkpoints across two GB10 or two Mac systems. Evaluating model quality and speed on specific hardware with the built-in benchmark command.
How do I install TensorFold?
Install TensorFold with Homebrew on a Mac with Apple Silicon: brew install ashhart/tensorfold/tensorfold Download the pinned Qwen3.5-2B model weights: tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3 Start the local model server: tensorfold serve mlx-community/Qwen3.5-2B-MLX-4bit --name local-model --context 8192 --no-thinking
Where can I get TensorFold?
The source code and installation instructions are on GitHub: https://github.com/ashhart/TensorFold

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects