Repolode
Submit
RightNow-AI

RightNow-AI/picolm

A minimal C11 LLM inference engine that runs 1B-class LLaMA-architecture GGUF models on low-cost boards with about 45MB RAM, with no dependencies.

View on GitHubWebsite
Stars
2,212
Forks
293
Open issues
23
Last commit
8 months ago

What is it?

PicoLM is a from-scratch LLM inference engine written in roughly 2,500 lines of C11 with no external dependencies. It loads GGUF files directly, including TinyLlama 1.1B and other LLaMA-architecture models, and targets hardware such as Raspberry Pi boards, RISC-V boards and ordinary x86-64 or ARM machines.

The model file is memory-mapped rather than loaded into RAM, so the OS pages in one layer at a time. The project reports about 45MB of runtime memory for TinyLlama with a 2048-token context, using an FP16 KV cache and quantized weights from Q2_K up to F32.

PicoLM also includes a grammar-constrained `--json` output mode that masks logits so the generated text is valid JSON, along with a KV cache save/load option that skips prompt prefill on repeated runs. It is designed to pair with the PicoClaw assistant as a fully offline local model.

Key features

  • 01Reads GGUF v2/v3 files directly and supports multiple K-quant and Q4/Q8 formats.
  • 02Memory-maps model weights so only a layer at a time needs to be in RAM.
  • 03Grammar-constrained JSON mode forces syntactically valid JSON output.
  • 04KV cache persistence skips repeated prompt processing.
  • 05Uses ARM NEON and x86 SSE2 SIMD, plus multi-threaded matrix multiplication.
  • 06Builds with only libc, libm and libpthread on Linux, Windows (MSVC) and macOS.

Use cases

  • Running a small offline chatbot or assistant on a Raspberry Pi or similar single-board computer.
  • Producing structured JSON output from a local model for tool calling on constrained hardware.
  • Experimenting with LLM inference on RISC-V or other embedded targets.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/rightnow-ai/picolm.git
  2. 2

    Enter the project directory

    cd picolm/picolm
  3. 3

    Build PicoLM with CPU auto-detection

    make native
  4. 4

    Download a model

    make model

Things to know

  • The model is a 638 MB download and generation speed is only a few tokens per second on small boards.
  • It targets 1B-class LLaMA-architecture GGUF models, so it does not run other model families.

Topics

#llm#inference#embedded#quantization#raspberry-pi#gguf#arm#openclaw#picoclaw#risc-v

FAQ

What is picolm?
A minimal C11 LLM inference engine that runs 1B-class LLaMA-architecture GGUF models on low-cost boards with about 45MB RAM, with no dependencies. PicoLM is a from-scratch LLM inference engine written in roughly 2,500 lines of C11 with no external dependencies. It loads GGUF files directly, including TinyLlama 1.1B and other LLaMA-architecture models, and targets hardware such as Raspberry Pi boards, RISC-V boards and ordinary x86-64 or ARM machines.
Is picolm free to use? What license does it have?
picolm is open source under the MIT license. Check the license terms before using it in commercial products.
Is picolm still maintained?
No commits for 6+ months. The last commit to picolm was on Feb 22, 2026.
What can I use picolm for?
Running a small offline chatbot or assistant on a Raspberry Pi or similar single-board computer. Producing structured JSON output from a local model for tool calling on constrained hardware. Experimenting with LLM inference on RISC-V or other embedded targets.
How do I install picolm?
Clone the repository: git clone https://github.com/rightnow-ai/picolm.git Enter the project directory: cd picolm/picolm Build PicoLM with CPU auto-detection: make native Download a model: make model
Where can I get picolm?
The source code and installation instructions are on GitHub: https://github.com/RightNow-AI/picolm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects