Run LLMs locally: top picks for 2026

A pure C inference engine that runs very large mixture-of-experts models on ordinary hardware by streaming routed experts from disk.

Open-source, local-first AI meeting assistant that records, transcribes in real time, and generates summaries on macOS and Windows.

Local proxy that lets OpenAI Codex and Claude Code run on any LLM provider, including Claude, Gemini, Grok, DeepSeek, and Ollama.

An edge-native MoE serving engine that runs large open-weight models like DeepSeek-V4 and Qwen3.6 on consumer GPUs by splitting work between GPU and CPU memory.

Open-source local music generation model that creates full songs from text, with editing tools like cover, repaint and stem separation.

CLI tool to fine-tune and post-train LLMs from a single YAML config, with layer streaming for training 8B models on a 4 GB GPU.

Self-hosted AI research assistant that runs deep, multi-step research with local or cloud LLMs and searches the web, academic databases and your own documents.

Open source inference engine that tunes its kernels to your hardware to run each open model as fast as possible.

Swift and Metal runtime that runs Gemma 4 26B-A4B on Apple Silicon Macs with about 2 GB of RAM by streaming experts from SSD.

Windows OCR tool that captures text from screen regions, images, and PDFs and copies it to the clipboard or an editor, running fully on-device.

A hands-on tutorial series for building AI agents from scratch with local LLMs via node-llama-cpp, without using frameworks.

Desktop app that searches your local files by meaning, across photos, PDFs, documents, audio, video and code, running on-device with EmbeddingGemma 2.

A modular Agentic RAG tutorial and project built on LangGraph, with hierarchical indexing, conversation memory, and query clarification.

Runs Claude Code against open AI models on your Apple Silicon Mac via an MLX-based Anthropic API server, with no cloud model or API key.

A Streamlit-based web scraper that uses OpenAI, Gemini, Ollama or LiteLLM models to extract structured data from pages described in natural language.

Self-hosted API that converts PDFs, images and Office documents into Markdown or structured JSON using OCR and Ollama-hosted LLMs, with optional PII removal.

A curated awesome-list of platforms, inference engines, UIs, models, tools and tutorials for running LLMs on your own hardware.

A Zotero plugin that lets you chat with one PDF, several PDFs, or your whole library, with clickable citations and optional fully local models.

Cross-platform desktop app that uses local or remote LLMs to sort files into categories and suggest clearer filenames, with review before any change.

Open-source local audio and music studio for macOS and Windows that runs TTS, voice cloning and AI songs fully offline.

An educational JavaScript tutorial that builds retrieval-augmented generation step by step with local LLMs and no cloud APIs.

Self-hosted job search tracker with AI resume review, job matching, automated job discovery, and an MCP server for AI agents.

A curated Awesome-style list of platforms, tools, models and tutorials for running large language models locally.

A local MCP server that routes prompts to ChatGPT, Claude, Gemini and Perplexity browser sessions or BYOK/local model endpoints for use in coding editors.

Uses an iPhone connected over USB-C to speed up prefill and extend context for a 27B Qwen model running locally on a Mac.

Windows desktop studio for the open YuE2 song model that generates full songs with vocals, an editable ABC score, covers and exact replay on a local NVIDIA GPU.

Spark-X2.5 is an open-weight language model series (4B and 1.7B) aimed at agentic coding and tool use on local and on-device hardware.

Converts PDFs, Office files, scans, screenshots and diagrams into Markdown by sending pages to your own OpenAI-compatible vision model.

A framework for autonomous Android and computer use, letting an LLM plan and execute UI tasks with local or remote models.

An open-source macOS popup that appears on selected text, offering transform, script, shortcut, math and AI actions.

A local reasoning tool where a small model's step-by-step answers are stored as a knowledge graph, all running on your own machine via Ollama.

Local-first macOS app that lets you search photos and video frames in any folder by describing them in plain language, using on-device AI models.

Electron-based screenshot and annotation app that lets AI coding agents like Claude Code render diagrams and previews for visual review.

A local-first AI office assistant that turns one request into editable PPTX, DOCX, XLSX and HTML files, using your own model keys or local models.

A small educational project for running vision-language model inference on Apple Silicon with MLX, scoring candidate answers about one image.

A Tauri desktop app for exploring and querying data with SQL, built on ClickHouse, with optional LLM integration and a web mode.

A local-first browser automation engine and Agent Skill that runs Qwen3.5 models on-device to pick browser actions from visible page elements.

A local-first desktop writing assistant that proofreads, rewrites, and formats drafts on your own machine without an account.

Chrome extension plus local Python service that generates real-time English/Chinese translated captions for YouTube livestreams using Ollama.