Repolode
Submit
davidjurgens

davidjurgens/potato

Free, self-hosted annotation platform for NLP, agent evaluation and qualitative research, configured via YAML.

Open-source alternative to:Label StudioNVivoATLAS.ti

View on GitHubWebsite
Stars
437
Forks
79
Open issues
0
Last commit
yesterday

What is it?

Potato is a self-hosted annotation platform for labeling text, audio, video, images and documents, as well as evaluating LLM agent traces. Tasks are defined in a YAML config file, so no coding is needed to set up a labeling interface. It also includes a qualitative data analysis mode with a living codebook, memos and cases.

Beyond classic labeling, Potato imports traces from many agent frameworks (OpenAI, Anthropic, LangChain, LangGraph, CrewAI, MCP and others) and supports step-level, span-level and side-by-side evaluation. It can pre-label data with LLM backends such as OpenAI, Anthropic, Ollama and vLLM, reorder queues with active learning, and track annotator agreement and quality.

Potato supports several authentication methods, including OAuth/SSO and a database backend, and includes deployment commands for a range of cloud hosts and HuggingFace Spaces. The project is licensed under GPL-3.0 and is installable from PyPI as potato-annotation.

Key features

  • 01Define annotation tasks in YAML without writing code
  • 02Label text, audio, video, images and documents
  • 03Evaluate agent traces at trajectory, step, span and comparison levels
  • 04Pre-annotate with LLM backends and active learning queue ordering
  • 05Track inter-annotator agreement, gold standards and attention checks
  • 06Qualitative analysis mode with codebook, memos and cases

Use cases

  • Building NLP datasets for classification, NER or span labeling
  • Evaluating LLM agent or RAG pipeline outputs step by step
  • Running crowdsourced annotation studies on Prolific or MTurk
  • Conducting qualitative interview or document coding with a self-hosted tool

Quick start

  1. 1

    Install the package from PyPI

    pip install potato-annotation
  2. 2

    Start the server with an example config

    potato start examples/classification/single-choice/config.yaml -p 8000
  3. 3

    Open the app in your browser at localhost port 8000

Things to know

  • Licensed under GPL-3.0, a copyleft license that requires derivative works to be distributed under the same terms.
  • You host it yourself, so you must run and maintain the Python server.

In collections

Open-source AI agent frameworks & appsTesting frameworks & test automation toolsNatural language processing tools & libraries

Topics

#annotation#data-labeling#nlp#llm-evaluation#ai-agents#qualitative-research#active-learning#crowdsourcing#agentic-ai#agentic-workflow

FAQ

What is potato?
Free, self-hosted annotation platform for NLP, agent evaluation and qualitative research, configured via YAML. Potato is a self-hosted annotation platform for labeling text, audio, video, images and documents, as well as evaluating LLM agent traces. Tasks are defined in a YAML config file, so no coding is needed to set up a labeling interface. It also includes a qualitative data analysis mode with a living codebook, memos and cases.
Is potato free to use? What license does it have?
potato is open source under the GPL-3.0 license. Check the license terms before using it in commercial products.
Is potato still maintained?
Actively maintained. The last commit to potato was on Oct 22, 2026.
What can I use potato for?
Building NLP datasets for classification, NER or span labeling Evaluating LLM agent or RAG pipeline outputs step by step Running crowdsourced annotation studies on Prolific or MTurk Conducting qualitative interview or document coding with a self-hosted tool
How do I install potato?
Install the package from PyPI: pip install potato-annotation Start the server with an example config: potato start examples/classification/single-choice/config.yaml -p 8000 Open the app in your browser at localhost port 8000
Where can I get potato?
The source code and installation instructions are on GitHub: https://github.com/davidjurgens/potato

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects