stalkermustang/llm-bulls-and-cows-benchmark

What is it?

Key features

• 01Supports many LLM providers via LiteLLM
• 02Configurable 3- or 4-digit games, with or without repeated digits
• 03Runs concurrent games with live progress display
• 04Reports success rate, format failures and average turns with confidence intervals
• 05Saves full game histories and generates HTML and Markdown result reports

Use cases

• Comparing reasoning performance across LLMs on a simple, reproducible puzzle
• Testing whether a model follows strict output formatting instructions
• Studying in-context learning from feedback over multiple turns

Topics

FAQ

What is llm-bulls-and-cows-benchmark?

A Python benchmark that measures how well LLMs solve the Bulls and Cows number-guessing game, using LiteLLM to reach many model providers. LLM Bulls and Cows Benchmark is a small framework for testing how well language models play Bulls and Cows, a code-breaking game where a guesser must identify a secret number from bull and cow feedback on each guess. It is designed to exercise both multi-step reasoning and the ability to learn from earlier answers in the same conversation.

Is llm-bulls-and-cows-benchmark free to use? What license does it have?

llm-bulls-and-cows-benchmark is open source under the MIT license. Check the license terms before using it in commercial products.

Is llm-bulls-and-cows-benchmark still maintained?

No commits for 6+ months. The last commit to llm-bulls-and-cows-benchmark was on Jan 31, 2025.

What can I use llm-bulls-and-cows-benchmark for?

Comparing reasoning performance across LLMs on a simple, reproducible puzzle Testing whether a model follows strict output formatting instructions Studying in-context learning from feedback over multiple turns

Where can I get llm-bulls-and-cows-benchmark?

The source code and installation instructions are on GitHub: https://github.com/stalkermustang/llm-bulls-and-cows-benchmark