Repolode
Submit
withcatai

withcatai/node-llama-cpp

Node.js bindings for llama.cpp that let you run LLMs locally, with GPU support and JSON schema-constrained output.

View on GitHubWebsite
Stars
2,200
Forks
229
Open issues
46
Last commit
2 days ago

What is it?

node-llama-cpp is a TypeScript/Node.js library that runs large language models directly on your own machine. It wraps llama.cpp and ships pre-built binaries for macOS, Linux and Windows, falling back to compiling llama.cpp from source with cmake when no prebuilt binary matches the platform.

The library covers the common needs of LLM applications: chat sessions, function calling, embeddings and reranking. It can force model output into a parseable format such as JSON, or into a specific JSON schema at the generation level. Hardware acceleration is available through Metal, CUDA and Vulkan, and the library selects the available backend automatically.

Besides the API, it provides a CLI, so you can chat with a GGUF model from the terminal with a single npx command without writing code.

Key features

  • 01Runs GGUF models locally with Metal, CUDA and Vulkan GPU support
  • 02Pre-built binaries with a cmake fallback that builds llama.cpp from source
  • 03Constrains generated output to JSON or a specific JSON schema
  • 04Supports function calling so models can invoke your code on demand
  • 05Provides embedding and reranking APIs with full TypeScript types
  • 06Includes a CLI for chatting with a model without writing code

Use cases

  • Adding a local LLM to a Node.js or TypeScript app without a cloud API
  • Getting structured JSON output from a model for downstream parsing
  • Building retrieval pipelines that need local embeddings and reranking
  • Trying out GGUF models from the terminal before integrating them

Quick start

  1. 1

    Install the package with npm

    npm install node-llama-cpp
  2. 2

    Chat with a model in the terminal without installing (single command)

    npx -y node-llama-cpp chat
  3. 3

    Load a GGUF model and run the usage example from the README (see the getting started guide for code)

Things to know

  • You need a local GGUF model file, which you must download separately.
  • Pre-built binaries cover only macOS, Linux and Windows; other platforms fall back to building llama.cpp from source with cmake.

In collections

Run LLMs locally

Topics

#llm#nodejs#llama-cpp#local-inference#gguf#typescript#json-schema#function-calling#ai#bindings

FAQ

What is node-llama-cpp?
Node.js bindings for llama.cpp that let you run LLMs locally, with GPU support and JSON schema-constrained output. node-llama-cpp is a TypeScript/Node.js library that runs large language models directly on your own machine. It wraps llama.cpp and ships pre-built binaries for macOS, Linux and Windows, falling back to compiling llama.cpp from source with cmake when no prebuilt binary matches the platform.
Is node-llama-cpp free to use? What license does it have?
node-llama-cpp is open source under the MIT license. Check the license terms before using it in commercial products.
Is node-llama-cpp still maintained?
Actively maintained. The last commit to node-llama-cpp was on Nov 5, 2026.
What can I use node-llama-cpp for?
Adding a local LLM to a Node.js or TypeScript app without a cloud API Getting structured JSON output from a model for downstream parsing Building retrieval pipelines that need local embeddings and reranking Trying out GGUF models from the terminal before integrating them
How do I install node-llama-cpp?
Install the package with npm: npm install node-llama-cpp Chat with a model in the terminal without installing (single command): npx -y node-llama-cpp chat Load a GGUF model and run the usage example from the README (see the getting started guide for code)
Where can I get node-llama-cpp?
The source code and installation instructions are on GitHub: https://github.com/withcatai/node-llama-cpp

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects