Repolode
Submit
floneum

floneum/kalosm

A Rust ecosystem for running pre-trained local or remote AI models (LLMs, Whisper, embeddings) with controllable structured output.

View on GitHubWebsite
Stars
2,235
Forks
135
Open issues
53
Last commit
14 days ago

What is it?

Kalosm is a set of Rust crates for building applications on top of pre-trained models. It provides a simple interface for language models such as Llama, Mistral and Phi, audio transcription with Whisper, image segmentation with Segment Anything, and text embeddings with Bert. Models can be loaded in quantized form and accelerated on GPU.

Beyond model access, Kalosm includes utilities for preparing context for LLMs: extracting text from txt, html, docx, md and pdf files, chunking it, and searching it with vector database integrations. It also offers web crawling and scraping, and microphone or file transcription.

Structured generation is a core feature. Types that derive Parse and Schema can constrain model output to a given shape, and regex or custom grammars can be supplied for formats such as JSON, HTML or XML. Fusor, a compiler-backed CPU and WebGPU runtime for quantized inference, sits underneath as the local backend; it is marked as early in development and not production-ready.

Key features

  • 01Runs pre-trained language, audio, image and embedding models in Rust.
  • 02Supports quantized models with GPU acceleration.
  • 03Constrains output to Rust types or regex and custom grammars via derive macros.
  • 04Extracts and chunks text from txt, html, docx, md and pdf for LLM context.
  • 05Transcribes audio from a microphone or file using Whisper.

Use cases

  • Building a local chatbot in Rust without calling a hosted API.
  • Producing structured JSON output from a local LLM for downstream code.
  • Adding semantic search over documents using local embeddings.
  • Transcribing meeting or microphone audio on your own machine.

Quick start

  1. 1

    Install Rust via rustup

  2. 2

    Create a new Rust project

    cargo new kalosm-hello-world cd ./kalosm-hello-world
  3. 3

    Add Kalosm and Tokio as dependencies

    cargo add kalosm --features llama cargo add tokio --features full
  4. 4

    Add the chatbot code to main.rs and run it

    cargo run --release

Things to know

  • Fusor, the inference backend, is early in development and not ready for production use.
  • Requires the Rust toolchain and compiling from source, which takes effort.
  • Model downloads are large, ranging from tens of megabytes to tens of gigabytes.

In collections

Run LLMs locallyAI video & audio tools

Topics

#llm#rust#local-llm#structured-output#speech-to-text#embeddings#whisper#quantization#ai#candle

FAQ

What is kalosm?
A Rust ecosystem for running pre-trained local or remote AI models (LLMs, Whisper, embeddings) with controllable structured output. Kalosm is a set of Rust crates for building applications on top of pre-trained models. It provides a simple interface for language models such as Llama, Mistral and Phi, audio transcription with Whisper, image segmentation with Segment Anything, and text embeddings with Bert. Models can be loaded in quantized form and accelerated on GPU.
Is kalosm free to use? What license does it have?
kalosm is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is kalosm still maintained?
Actively maintained. The last commit to kalosm was on Oct 24, 2026.
What can I use kalosm for?
Building a local chatbot in Rust without calling a hosted API. Producing structured JSON output from a local LLM for downstream code. Adding semantic search over documents using local embeddings. Transcribing meeting or microphone audio on your own machine.
How do I install kalosm?
Install Rust via rustup Create a new Rust project: cargo new kalosm-hello-world cd ./kalosm-hello-world Add Kalosm and Tokio as dependencies: cargo add kalosm --features llama cargo add tokio --features full Add the chatbot code to main.rs and run it: cargo run --release
Where can I get kalosm?
The source code and installation instructions are on GitHub: https://github.com/floneum/kalosm

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects