Repolode
Submit
ianarawjo

ianarawjo/ChainForge

A visual, node-based environment for comparing prompts and models by querying multiple LLMs at once and scoring their responses.

View on GitHubWebsite
Stars
3,041
Forks
261
Open issues
70
Last commit
9 days ago

What is it?

ChainForge is a data flow environment for prompt engineering and LLM response evaluation. Instead of chatting with one model at a time, you build a flow on a canvas, send the same prompt template to several models, and compare the outputs side by side across prompt variations and model settings.

The core mechanism is combinatorial: input values fed into a prompt template are crossed with each other and with the selected models, so hundreds of queries can run at once. You can attach Python evaluation scripts to score responses, then visualize results as box-and-whisker or histogram plots, and export the data to a spreadsheet.

ChainForge runs as a local Python server with a React front end, and can also be used in the browser at chainforge.ai/play. Supported providers include OpenAI, Anthropic, Google Gemini, Ollama-hosted local models, Azure OpenAI and Amazon Bedrock, with custom provider scripts for others.

Key features

  • 01Query multiple LLMs in one run and compare their responses
  • 02Generate prompt permutations from template variables and datasets
  • 03Score responses with Python evaluation nodes
  • 04Visualize numeric and boolean metrics as plots
  • 05Export results to spreadsheet format
  • 06Includes RAG nodes for upload, chunking, retrieval and reranking

Use cases

  • Choosing which model performs best for a specific task before committing to one provider
  • Testing how small wording changes in a prompt affect output quality across many inputs
  • Running ground-truth evaluations, such as checking math answers against expected results
  • Batch-generating parametrized LLM outputs and exporting them for analysis

Quick start

  1. 1

    Install ChainForge with pip (Python 3.10 or higher)

    pip install chainforge
  2. 2

    Start the local server

    chainforge serve
  3. 3

    Open localhost:8000 in Chrome, Firefox, Edge or Brave

Things to know

  • Local installs need Python 3.10 or higher.
  • Some features, such as Python evaluators and local Ollama models, are only available in the local version rather than the web version.
  • Full RAG server-side features require an extra install, and FAISS must be installed separately.

In collections

Testing frameworks & test automation tools

Topics

#prompt-engineering#llm-evaluation#large-language-models#visual-programming#multi-model-comparison#llmops#ai#evaluation#llms

FAQ

What is ChainForge?
A visual, node-based environment for comparing prompts and models by querying multiple LLMs at once and scoring their responses. ChainForge is a data flow environment for prompt engineering and LLM response evaluation. Instead of chatting with one model at a time, you build a flow on a canvas, send the same prompt template to several models, and compare the outputs side by side across prompt variations and model settings.
Is ChainForge free to use? What license does it have?
ChainForge is open source under the MIT license. Check the license terms before using it in commercial products.
Is ChainForge still maintained?
Actively maintained. The last commit to ChainForge was on Oct 29, 2026.
What can I use ChainForge for?
Choosing which model performs best for a specific task before committing to one provider Testing how small wording changes in a prompt affect output quality across many inputs Running ground-truth evaluations, such as checking math answers against expected results Batch-generating parametrized LLM outputs and exporting them for analysis
How do I install ChainForge?
Install ChainForge with pip (Python 3.10 or higher): pip install chainforge Start the local server: chainforge serve Open localhost:8000 in Chrome, Firefox, Edge or Brave
Where can I get ChainForge?
The source code and installation instructions are on GitHub: https://github.com/ianarawjo/ChainForge

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects