Repolode
Submit
magnitudedev

magnitudedev/magnitude

Open source inference engine that tunes its kernels to your hardware to run open models faster on Apple Silicon, NVIDIA, AMD, or CPU.

Open-source alternative to:Ollama

View on GitHubWebsite
Stars
7,186
Forks
494
Open issues
54
Last commit
2 days ago

What is it?

Magnitude is an inference engine for running open-weight models locally. When a model is downloaded, it tries different ways to split up the computation on the actual chip, such as how many GPU threads work together and how far ahead data is loaded, and keeps the fastest configuration that does not change the model's output. The project states it is about 2x faster than Ollama on average across popular local models on Mac and NVIDIA hardware.

It ships as a desktop app for macOS, Windows, and Linux that includes a `magnitude` CLI. On Linux and macOS, `magnitude server setup` runs it as a boot-time service that serves browsers and agents on other machines. Users can connect agents such as Pi, OpenCode, Codex, and Claude Code with one click, and anything else can use the OpenAI-compatible API.

Prompts, files, and models stay on the local machine, and no internet connection is needed once a model is downloaded. The project is licensed under Apache 2.0.

Key features

  • 01Tunes kernels per model on the local chip to find faster execution settings.
  • 02Runs on Apple Silicon, NVIDIA, AMD GPUs, or CPU only.
  • 03Includes a built-in speculative decoding drafter for each model.
  • 04Uses quantized KV cache to reduce memory per agent session.
  • 05Shares prefix caches across concurrent sessions.
  • 06Provides an OpenAI-compatible API and one-click agent connections.

Use cases

  • Running open-weight LLMs locally on a Mac or NVIDIA workstation for coding agents.
  • Hosting a local model on a Linux or macOS server that multiple machines can reach.
  • Keeping prompts and files private by avoiding cloud inference APIs.

Quick start

  1. 1

    Download Magnitude from the releases page and install it

  2. 2

    Open the app and choose a recommended model in Discover, then download it

  3. 3

    Connect your agent in Connections and start using it

Things to know

  • Kernel tuning runs when you download a model, which can take up to a minute.
  • Server setup is only documented for Linux and macOS.

Topics

#inference#llm#local-llm#apple-silicon#cuda#ollama-alternative#openai-compatible#speculative-decoding

FAQ

What is magnitude?
Open source inference engine that tunes its kernels to your hardware to run open models faster on Apple Silicon, NVIDIA, AMD, or CPU. Magnitude is an inference engine for running open-weight models locally. When a model is downloaded, it tries different ways to split up the computation on the actual chip, such as how many GPU threads work together and how far ahead data is loaded, and keeps the fastest configuration that does not change the model's output. The project states it is about 2x faster than Ollama on average across popular local models on Mac and NVIDIA hardware.
Is magnitude free to use? What license does it have?
magnitude is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is magnitude still maintained?
Actively maintained. The last commit to magnitude was on Oct 19, 2026.
What can I use magnitude for?
Running open-weight LLMs locally on a Mac or NVIDIA workstation for coding agents. Hosting a local model on a Linux or macOS server that multiple machines can reach. Keeping prompts and files private by avoiding cloud inference APIs.
How do I install magnitude?
Download Magnitude from the releases page and install it Open the app and choose a recommended model in Discover, then download it Connect your agent in Connections and start using it
Where can I get magnitude?
The source code and installation instructions are on GitHub: https://github.com/magnitudedev/magnitude

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects