spaCy components and architectures that plug Hugging Face transformer models such as BERT, RoBERTa and XLNet into spaCy pipelines.
1.4K177PythonMITUpdated 4 days ago
KRLabsOrg/LettuceDetect
Open span-level hallucination detector that flags unsupported parts of RAG, code-agent and tool-grounded AI answers against source evidence.
- Stars
- 620
- Forks
- 64
- Open issues
- 28
- Last commit
- yesterday
What is it?
LettuceDetect checks AI-generated answers against the evidence they were supposed to be based on. Given a context, a question and an answer, it marks the exact character spans that are unsupported, contradictory or fabricated, instead of returning only a single clean-or-flagged verdict for the whole answer. It covers RAG prose, multilingual QA, coding-agent responses and developer-tool output.
The project offers two kinds of models. Fast encoder models (ModernBERT for English, EuroBERT for several languages, and mmBERT-based v2 models) run locally as token classifiers. Generative models, such as the 2B Qwen-based v2 detector, emit typed spans with a category and subcategory in a single pass. A taxonomy head can also be added to the encoder to label spans. An LLM-based baseline using the OpenAI API is included for comparison.
The Python package installs from PyPI and loads models from Hugging Face in a few lines. It also ships a FastAPI web server with a Python client, a Streamlit demo, and training and evaluation scripts for the RAGTruth dataset.
Key features
- 01Returns exact character spans for unsupported or fabricated content in an answer.
- 02Provides local encoder models for fast inference and generative models that type each span.
- 03Supports English and multilingual RAG answers, plus coding-agent and tool output.
- 04Offers a pip-installable Python API and a FastAPI server with a Python client.
- 05Includes training and evaluation scripts for the RAGTruth dataset.
Use cases
- Checking whether answers from a RAG chatbot are grounded in retrieved documents.
- Flagging invented APIs or wrong values in coding-agent responses against repository source.
- Auditing tool-grounded outputs before showing them to users.
- Benchmarking hallucination detectors on RAGTruth-style data.
Quick start
- 1
Install the package from PyPI
pip install lettucedetect -U - 2
Run the detector on a context, question and answer (see the Python example in the README)
Things to know
- EuroBERT models need transformers below 5.x, since they are not yet compatible with transformers 5.
- LLM-based baselines require an OpenAI API key.
In collections
Topics
#hallucination-detection#rag#nlp#token-classification#pytorch#bert#llm-evaluation#hallucination-evaluation#information-extraction#python
FAQ
- What is LettuceDetect?
- Open span-level hallucination detector that flags unsupported parts of RAG, code-agent and tool-grounded AI answers against source evidence. LettuceDetect checks AI-generated answers against the evidence they were supposed to be based on. Given a context, a question and an answer, it marks the exact character spans that are unsupported, contradictory or fabricated, instead of returning only a single clean-or-flagged verdict for the whole answer. It covers RAG prose, multilingual QA, coding-agent responses and developer-tool output.
- Is LettuceDetect free to use? What license does it have?
- LettuceDetect is open source under the MIT license. Check the license terms before using it in commercial products.
- Is LettuceDetect still maintained?
- Actively maintained. The last commit to LettuceDetect was on Nov 6, 2026.
- What can I use LettuceDetect for?
- Checking whether answers from a RAG chatbot are grounded in retrieved documents. Flagging invented APIs or wrong values in coding-agent responses against repository source. Auditing tool-grounded outputs before showing them to users. Benchmarking hallucination detectors on RAGTruth-style data.
- How do I install LettuceDetect?
- Install the package from PyPI: pip install lettucedetect -U Run the detector on a context, question and answer (see the Python example in the README)
- Where can I get LettuceDetect?
- The source code and installation instructions are on GitHub: https://github.com/KRLabsOrg/LettuceDetect
Summary generated by AI from the project's README. GitHub stats refresh daily.
Related projects
Python library that scores LLM responses for likely hallucinations using uncertainty quantification, with black-box, white-box, judge and ensemble scorers.
AI & Machine Learning#hallucination-detection#uncertainty-quantification#llm-evaluation#confidence-estimation#llm-safety
1.2K+2 this week140PythonApache-2.0Well maintainedUpdated 25 days ago
A Python library that uses LLMs to pull structured data from unstructured text, with each extraction linked back to its source span.
39.3K+59 this week2.8KPythonApache-2.0Updated yesterday
Turns unstructured text into an interactive knowledge graph by using an LLM to extract subject-predicate-object triples.
3.1K+1 this week420PythonApache-2.0Well maintainedUpdated 2 months ago
Fast, state-of-the-art tokenization library in Rust with Python and Node.js bindings for NLP and language model pipelines.
11.2K+8 this week1.2KRustApache-2.0Updated yesterday
A Python topic modeling library that combines transformer embeddings with c-TF-IDF to produce interpretable topics from text.
7.9K+15 this week931PythonMITUpdated 2 days ago