Repolode
Submit
irthomasthomas

irthomasthomas/llm-consortium

A Python plugin for the llm CLI that runs several language models in parallel, iteratively refines their answers, and reaches a consensus via an arbiter model.

View on GitHub
Stars
409
Forks
42
Open issues
10
Last commit
2 months ago

What is it?

LLM Consortium is a plugin for the `llm` package that sends one prompt to several language models at once. An arbiter model then evaluates and synthesizes the individual responses, and the process repeats until a confidence threshold or iteration limit is reached. The approach is based on the idea that checking candidate answers is often easier than producing one, so pooling several models can help on hard problems.

Consortiums are saved as named models and invoked with the standard `llm -m` syntax. Each member model can run multiple instances via a `model:count` syntax. The project offers several judging methods, including full synthesis, ranking, and chain-of-thought ranking, plus an optional semantic strategy that clusters response embeddings before arbitration. Runs are logged to SQLite, and saved embeddings can be exported as HTML visualizations. Conversation continuation with `-c` and `--cid` is supported.

A Python API via `create_consortium` is also available for use outside the command line. Installation is through `llm install` or pip, with optional extras for embeddings and visualization.

Key features

  • 01Sends one prompt to multiple models in parallel and collects their responses.
  • 02Uses an arbiter model to synthesize, rank, or evaluate candidate answers.
  • 03Iterates until a confidence threshold or maximum iteration count is reached.
  • 04Supports per-model instance counts with the model:count syntax.
  • 05Logs all interactions to SQLite and can export embedding visualizations as HTML.
  • 06Provides a Python API through create_consortium.

Use cases

  • Getting a more reliable answer to a hard question by cross-checking several models.
  • Comparing and ranking outputs from different LLM providers in one workflow.
  • Experimenting with multi-model reasoning strategies from the command line or Python code.

Quick start

  1. 1

    Install the llm CLI with uv

    uv tool install llm
  2. 2

    Install the llm-consortium plugin

    llm install "llm-consortium"
  3. 3

    Check that the consortium command is available

    llm consortium --help

Things to know

  • Semantic and embedding features need extra packages (scikit-learn, hdbscan, openai, sentence-transformers) and API keys such as OPENAI_API_KEY or CHUTES_API_TOKEN.
  • Geometric confidence measures agreement between response shapes, not factual correctness.

In collections

Open-source AI agent frameworks & appsCLI & terminal tools

Topics

#llm#multi-agent#llm-ensemble#consensus#cli#python#prompt-engineering#ai#llms

FAQ

What is llm-consortium?
A Python plugin for the llm CLI that runs several language models in parallel, iteratively refines their answers, and reaches a consensus via an arbiter model. LLM Consortium is a plugin for the `llm` package that sends one prompt to several language models at once. An arbiter model then evaluates and synthesizes the individual responses, and the process repeats until a confidence threshold or iteration limit is reached. The approach is based on the idea that checking candidate answers is often easier than producing one, so pooling several models can help on hard problems.
Is llm-consortium free to use? What license does it have?
llm-consortium is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is llm-consortium still maintained?
Updated within 6 months. The last commit to llm-consortium was on Sep 5, 2026.
What can I use llm-consortium for?
Getting a more reliable answer to a hard question by cross-checking several models. Comparing and ranking outputs from different LLM providers in one workflow. Experimenting with multi-model reasoning strategies from the command line or Python code.
How do I install llm-consortium?
Install the llm CLI with uv: uv tool install llm Install the llm-consortium plugin: llm install "llm-consortium" Check that the consortium command is available: llm consortium --help
Where can I get llm-consortium?
The source code and installation instructions are on GitHub: https://github.com/irthomasthomas/llm-consortium

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects