Repolode
Submit
mgonzs13

mgonzs13/llama_ros

A set of ROS 2 packages that run GGUF-based LLMs, vision-language and audio models via llama.cpp inside ROS 2 nodes.

View on GitHubWebsite
Stars
268
Forks
50
Open issues
0
Last commit
6 hours ago

What is it?

llama_ros wraps llama.cpp as ROS 2 nodes, so robotics projects can run GGUF language models, vision-language models (via llava.cpp), audio-language, embedding and reranking models inside a ROS 2 graph. Models are configured through ROS 2 parameters or YAML files and can be downloaded directly from Hugging Face by repository and filename.

The package exposes an OpenAI-compatible chat completions interface with tool calling, reasoning and streaming. It also supports speculative decoding with draft-model and ngram-based methods, GBNF grammars, LoRA adapters that can be changed at runtime, and control vectors. Parameters cover GPU offloading, CPU threading, KV cache types, RoPE and YaRN scaling, and memory loading modes.

The repository includes a `ros2 llama` command-line tool for launching a model from a YAML file and sending prompts to it, plus LangChain integration, Docker images, and predefined launch files for several models. Builds target ROS 2 Humble through Lyrical.

Key features

  • 01Runs GGUF LLMs, llava-based VLMs and audio-language models as ROS 2 nodes.
  • 02Provides an OpenAI-compatible chat completions interface with tool calling and streaming.
  • 03Supports speculative decoding with draft-model and ngram-based methods.
  • 04Downloads models from Hugging Face or loads local GGUF files.
  • 05Offers GPU offloading, CUDA build option, and detailed CPU and memory tuning parameters.
  • 06Includes a `ros2 llama` CLI for launching models from YAML and sending prompts.

Use cases

  • Adding a local language model to a ROS 2 robot for natural-language commands or chat.
  • Giving a robot vision-language understanding by feeding camera images to a VLM node.
  • Building RAG or LangChain pipelines that query a local model from ROS 2 nodes.
  • Running embedding or reranking models inside a ROS 2 system.

Quick start

  1. 1

    Clone the llama_ros repository into your ROS 2 workspace src folder

    git clone https://github.com/mgonzs13/llama_ros.git
  2. 2

    Install the Python dependencies from requirements.txt

    pip3 install --break-system-packages --ignore-installed -r requirements.txt
  3. 3

    Install the ROS 2 dependencies, then build the workspace with colcon

    rosdep install --from-paths src --ignore-src -r -y
  4. 4

    Launch a predefined model from its YAML config

    ros2 llama launch ~/ros2_ws/src/llama_ros/llama_bringup/models/StableLM-Zephyr.yaml

Things to know

  • Requires a ROS 2 installation and building from source with colcon.
  • Models are downloaded from Hugging Face and must be configured in launch files or YAML.

In collections

Run LLMs locallyRobotics & embodied AI projects

Topics

#ros2#llm#llama-cpp#gguf#multimodal#robotics#embeddings#speculative-decoding#audio#cpp

FAQ

What is llama_ros?
A set of ROS 2 packages that run GGUF-based LLMs, vision-language and audio models via llama.cpp inside ROS 2 nodes. llama_ros wraps llama.cpp as ROS 2 nodes, so robotics projects can run GGUF language models, vision-language models (via llava.cpp), audio-language, embedding and reranking models inside a ROS 2 graph. Models are configured through ROS 2 parameters or YAML files and can be downloaded directly from Hugging Face by repository and filename.
Is llama_ros free to use? What license does it have?
llama_ros is open source under the MIT license. Check the license terms before using it in commercial products.
Is llama_ros still maintained?
Actively maintained. The last commit to llama_ros was on Nov 6, 2026.
What can I use llama_ros for?
Adding a local language model to a ROS 2 robot for natural-language commands or chat. Giving a robot vision-language understanding by feeding camera images to a VLM node. Building RAG or LangChain pipelines that query a local model from ROS 2 nodes. Running embedding or reranking models inside a ROS 2 system.
How do I install llama_ros?
Clone the llama_ros repository into your ROS 2 workspace src folder: git clone https://github.com/mgonzs13/llama_ros.git Install the Python dependencies from requirements.txt: pip3 install --break-system-packages --ignore-installed -r requirements.txt Install the ROS 2 dependencies, then build the workspace with colcon: rosdep install --from-paths src --ignore-src -r -y Launch a predefined model from its YAML config: ros2 llama launch ~/ros2_ws/src/llama_ros/llama_bringup/models/StableLM-Zephyr.yaml
Where can I get llama_ros?
The source code and installation instructions are on GitHub: https://github.com/mgonzs13/llama_ros

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects