Repolode
Submit
containers

containers/ramalama

A developer tool that runs AI models locally inside OCI containers, pulling GPU-specific images automatically and serving them via REST API or chatbot.

Open-source alternative to:Ollama

View on GitHubWebsite
Stars
3,109
Forks
386
Open issues
114
Last commit
2 days ago

What is it?

RamaLama is an open-source CLI that lets you pull, run, and serve AI models from multiple registries using the same workflow as containers. It detects the GPUs on your host and pulls a matching accelerated container image, so you do not need to configure drivers and runtimes on the host system yourself. Models are handled much like container images, with commands for listing, pulling, inspecting, converting, and benchmarking them.

Models can come from Hugging Face, ModelScope, Ollama, and generic OCI registries such as Quay or Docker Hub. Inference runs through llama.cpp, vLLM, or MLX on macOS, and you can interact with a model through a REST API or a chatbot. By default containers run rootless, mount the model read-only, have no network access during `ramalama run`, and are removed when they exit.

The project supports CPU as well as NVIDIA, AMD, Intel, Apple Silicon, Ascend, and Moore Threads hardware, and it installs through PyPI, DNF on Fedora, a macOS installer, or a shell script. Windows is supported through Docker Desktop or Podman Desktop with WSL2.

Key features

  • 01Automatically selects a GPU-specific container image for the host hardware.
  • 02Pulls models from Hugging Face, ModelScope, Ollama, and OCI registries.
  • 03Serves models through a REST API or an interactive chatbot.
  • 04Runs models in rootless containers with read-only mounts and no network access.
  • 05Converts models to OCI images and benchmarks or inspects them with built-in commands.

Use cases

  • Running open-weight LLMs locally without manually installing CUDA, ROCm, or inference runtimes.
  • Serving a model over a local REST API for application development.
  • Trying out models from several registries under a single consistent command set.
  • Running model inference in isolated containers for stricter host separation.

Quick start

  1. 1

    Install RamaLama from PyPI

    pip install ramalama
  2. 2

    Run a model with the ramalama run command (needs Podman or Docker installed)

Things to know

  • Works best with Podman or Docker installed, since models run inside containers.
  • NVIDIA GPU use needs extra host configuration described in the project docs.
  • Windows support requires Docker Desktop or Podman Desktop with WSL2.

In collections

Run LLMs locallyRun and train AI on GPUs

Topics

#llm#containers#inference-server#llama-cpp#local-ai#podman#gpu#oci#ai#cuda

FAQ

What is ramalama?
A developer tool that runs AI models locally inside OCI containers, pulling GPU-specific images automatically and serving them via REST API or chatbot. RamaLama is an open-source CLI that lets you pull, run, and serve AI models from multiple registries using the same workflow as containers. It detects the GPUs on your host and pulls a matching accelerated container image, so you do not need to configure drivers and runtimes on the host system yourself. Models are handled much like container images, with commands for listing, pulling, inspecting, converting, and benchmarking them.
Is ramalama free to use? What license does it have?
ramalama is open source under the MIT license. Check the license terms before using it in commercial products.
Is ramalama still maintained?
Actively maintained. The last commit to ramalama was on Nov 4, 2026.
What can I use ramalama for?
Running open-weight LLMs locally without manually installing CUDA, ROCm, or inference runtimes. Serving a model over a local REST API for application development. Trying out models from several registries under a single consistent command set. Running model inference in isolated containers for stricter host separation.
How do I install ramalama?
Install RamaLama from PyPI: pip install ramalama Run a model with the ramalama run command (needs Podman or Docker installed)
Where can I get ramalama?
The source code and installation instructions are on GitHub: https://github.com/containers/ramalama

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects