Repolode
Submit
PacificAI

PacificAI/langtest

Python library that generates and runs robustness, bias, fairness and accuracy tests against NLP models and LLMs with one line of code.

View on GitHubWebsite
Stars
561
Forks
53
Open issues
7
Last commit
2 months ago

What is it?

LangTest is a Python library for evaluating language models before they reach production. A user creates a Harness object for a task such as named entity recognition, points it at a model, then generates test cases, runs them and produces a report. The library covers more than 60 test types spanning robustness, bias, representation, fairness and accuracy.

It works with NLP pipelines built on Spark NLP, Hugging Face and Transformers for NER, translation and text classification. It can also test hosted LLMs from OpenAI, Cohere, AI21, the Hugging Face Inference API and Azure OpenAI, covering question answering, toxicity, factuality, sycophancy, summarization and clinical or legal scenarios. For some models, test results can drive automatic augmentation of training data.

The project is Apache-2.0 licensed and maintained by John Snow Labs and the Pacific AI team, with documentation and benchmark datasets published on langtest.org.

Key features

  • 01Generates and runs more than 60 test types from a single line of code.
  • 02Covers robustness, bias, representation, fairness and accuracy.
  • 03Supports NER, translation and text classification pipelines in Spark NLP, Hugging Face and Transformers.
  • 04Tests hosted LLMs from OpenAI, Cohere, AI21, Hugging Face and Azure OpenAI.
  • 05Includes benchmark datasets for question answering, summarization and other tasks.

Use cases

  • Checking an NER model for robustness to typos and casing changes before deployment.
  • Auditing a chatbot or LLM for bias, toxicity or sycophancy.
  • Evaluating medical or legal LLM outputs for domain-specific accuracy.
  • Comparing several models on the same benchmark datasets.

Quick start

  1. 1

    Install langtest with transformers support

    pip install langtest[transformers]
  2. 2

    Create a Harness and generate, run and report tests on a model

Things to know

  • Some test types need paid API keys for LLM providers such as OpenAI, Cohere or AI21.

In collections

Testing frameworks & test automation toolsNatural language processing tools & libraries

Topics

#ai-testing#llm-evaluation#nlp#responsible-ai#model-robustness#bias-detection#benchmarks#ai-safety#artificial-intelligence#benchmark-framework

FAQ

What is langtest?
Python library that generates and runs robustness, bias, fairness and accuracy tests against NLP models and LLMs with one line of code. LangTest is a Python library for evaluating language models before they reach production. A user creates a Harness object for a task such as named entity recognition, points it at a model, then generates test cases, runs them and produces a report. The library covers more than 60 test types spanning robustness, bias, representation, fairness and accuracy.
Is langtest free to use? What license does it have?
langtest is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is langtest still maintained?
Updated within 6 months. The last commit to langtest was on Aug 28, 2026.
What can I use langtest for?
Checking an NER model for robustness to typos and casing changes before deployment. Auditing a chatbot or LLM for bias, toxicity or sycophancy. Evaluating medical or legal LLM outputs for domain-specific accuracy. Comparing several models on the same benchmark datasets.
How do I install langtest?
Install langtest with transformers support: pip install langtest[transformers] Create a Harness and generate, run and report tests on a model
Where can I get langtest?
The source code and installation instructions are on GitHub: https://github.com/PacificAI/langtest

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects

cvs-health / uqlm

Python library that scores LLM responses for likely hallucinations using uncertainty quantification, with black-box, white-box, judge and ensemble scorers.

AI & Machine Learning#hallucination-detection#uncertainty-quantification#llm-evaluation
1.2K+2 this week140PythonApache-2.0Well maintainedUpdated 25 days ago