Repolode
Submit
jasonacox

jasonacox/TinyLLM

Guide and chatbot web app for running a small local LLM with an OpenAI-compatible API on consumer-grade hardware via Ollama, vLLM or llama-cpp-python.

Open-source alternative to:ChatGPT

View on GitHub
Stars
354
Forks
43
Open issues
9
Last commit
13 days ago

What is it?

TinyLLM is a setup guide and web chatbot for running a large language model on a small, local machine. It walks through standing up an inference server with Ollama, vLLM or llama-cpp-python, each exposing an OpenAI-compatible API, and then connecting a ChatGPT-style chatbot to that endpoint.

The chatbot is a Python FastAPI app that supports multiple sessions and keeps conversation history. It can summarize pasted URLs and PDFs, fetch news headlines, look up stock prices and weather, and run retrieval-augmented queries against a vector database. The project also includes notes on which quantized GGUF and AWQ models work well on modest GPUs, and a section on pointing the `llm` command-line tool at the local server.

The repository targets Ubuntu Linux and macOS with Apple Silicon or NVIDIA GPUs, and includes Docker commands for running the services.

Key features

  • 01Runs local LLMs through Ollama, vLLM or llama-cpp-python with an OpenAI-compatible API.
  • 02Provides a ChatGPT-style web chatbot with multiple sessions and conversation history.
  • 03Summarizes web pages and PDFs pasted into the chat.
  • 04Fetches news headlines, stock prices and weather via chat commands.
  • 05Supports retrieval-augmented queries against a vector database.
  • 06Includes Ollama settings to keep models loaded in memory.

Use cases

  • Running a private chatbot on a home GPU or Apple Silicon Mac without sending data to a cloud API.
  • Experimenting with quantized open models on hardware with limited VRAM.
  • Giving a local chat interface retrieval access to your own documents through a vector database.

Quick start

  1. 1

    Clone the project repository

    git clone https://github.com/jasonacox/TinyLLM.git
  2. 2

    Start the Ollama server container using the docker run command in the README

  3. 3

    Download and test run the llama3 model inside the container

    docker exec -it ollama ollama run llama3

Things to know

  • Hardware requirements are Ubuntu Linux or macOS only.
  • The Ollama server handles only one session or prompt at a time.
  • The quick Ollama setup assumes Docker and an NVIDIA GPU, and vLLM needs GPUs with more VRAM.

In collections

Run LLMs locallyAI chatbots & chat clients

Topics

#llm#local-llm#chatbot#openai-api#llama-cpp#vllm#ollama#rag#artificial-intelligence#large-language-models

FAQ

What is TinyLLM?
Guide and chatbot web app for running a small local LLM with an OpenAI-compatible API on consumer-grade hardware via Ollama, vLLM or llama-cpp-python. TinyLLM is a setup guide and web chatbot for running a large language model on a small, local machine. It walks through standing up an inference server with Ollama, vLLM or llama-cpp-python, each exposing an OpenAI-compatible API, and then connecting a ChatGPT-style chatbot to that endpoint.
Is TinyLLM free to use? What license does it have?
TinyLLM is open source under the MIT license. Check the license terms before using it in commercial products.
Is TinyLLM still maintained?
Actively maintained. The last commit to TinyLLM was on Oct 25, 2026.
What can I use TinyLLM for?
Running a private chatbot on a home GPU or Apple Silicon Mac without sending data to a cloud API. Experimenting with quantized open models on hardware with limited VRAM. Giving a local chat interface retrieval access to your own documents through a vector database.
How do I install TinyLLM?
Clone the project repository: git clone https://github.com/jasonacox/TinyLLM.git Start the Ollama server container using the docker run command in the README Download and test run the llama3 model inside the container: docker exec -it ollama ollama run llama3
Where can I get TinyLLM?
The source code and installation instructions are on GitHub: https://github.com/jasonacox/TinyLLM

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects