Repolode
Submit
Giskard-AI

Giskard-AI/giskard-oss

Open-source Python library for writing evals, red-teaming and vulnerability scans against LLM-based and agentic systems.

View on GitHubWebsite
Stars
5,945
Forks
561
Open issues
77
Last commit
5 days ago

What is it?

Giskard is a Python library for testing and evaluating agentic systems. It wraps any target, whether an LLM, a black-box agent or a multi-step pipeline, as a sync or async callable and runs checks against its outputs. Because LLM outputs are non-deterministic, its evals combine plain assertions such as string matching, regex and semantic similarity with LLM-as-judge checks like Groundedness, Conformity and LLMJudge.

The v3 release is organized as a set of focused packages. giskard-checks provides the scenario API, suites and built-in checks for regression testing, RAG grounding and multi-turn conversations. giskard-scan adds a vulnerability scanner that generates adversarial test suites from a plain-language description of an agent, covering prompt injection, jailbreaks and harmful content, plus a quality_scan for knowledge-base evaluation.

The legacy v2 tabular ML scan and RAGET test generator remain available for v2 users, but v2 is no longer actively maintained. Installation requires Python 3.12 or newer and provider SDKs are added through pip extras.

Key features

  • 01Scenario API for building evals from interactions and checks
  • 02Built-in checks including string matching, regex, semantic similarity and LLM judges
  • 03Groundedness, Conformity and LLMJudge checks for RAG and safety evaluation
  • 04Automatic red-team suite generation from a description of your agent
  • 05Prompt-injection probes and OWASP LLM Top-10 threat categories
  • 06Modular packages installable via pip extras

Use cases

  • Regression testing an LLM application after prompt or model changes
  • Checking whether RAG answers are grounded in retrieved context
  • Red-teaming a customer support chatbot for prompt injection and harmful outputs
  • Evaluating multi-turn conversations of an agent end to end

Quick start

  1. 1

    Install the core checks library

    pip install giskard
  2. 2

    Install the vulnerability scanner extra

    pip install "giskard[scan]"
  3. 3

    Install a provider SDK for LLM judges (e.g. OpenAI) and set its API key

    pip install "giskard[openai]"

Things to know

  • Requires Python 3.12 or newer.
  • LLM judges and scanners need a provider API key, such as OpenAI.
  • The v2 tabular ML scan is legacy-only and v2 is no longer actively maintained.

In collections

Testing frameworks & test automation tools

Topics

#llm-evaluation#red-teaming#ai-testing#llm-security#rag-evaluation#agent-evaluation#python#ai-red-team#ai-security#fairness-ai

FAQ

What is giskard-oss?
Open-source Python library for writing evals, red-teaming and vulnerability scans against LLM-based and agentic systems. Giskard is a Python library for testing and evaluating agentic systems. It wraps any target, whether an LLM, a black-box agent or a multi-step pipeline, as a sync or async callable and runs checks against its outputs. Because LLM outputs are non-deterministic, its evals combine plain assertions such as string matching, regex and semantic similarity with LLM-as-judge checks like Groundedness, Conformity and LLMJudge.
Is giskard-oss free to use? What license does it have?
giskard-oss is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is giskard-oss still maintained?
Actively maintained. The last commit to giskard-oss was on Nov 2, 2026.
What can I use giskard-oss for?
Regression testing an LLM application after prompt or model changes Checking whether RAG answers are grounded in retrieved context Red-teaming a customer support chatbot for prompt injection and harmful outputs Evaluating multi-turn conversations of an agent end to end
How do I install giskard-oss?
Install the core checks library: pip install giskard Install the vulnerability scanner extra: pip install "giskard[scan]" Install a provider SDK for LLM judges (e.g. OpenAI) and set its API key: pip install "giskard[openai]"
Where can I get giskard-oss?
The source code and installation instructions are on GitHub: https://github.com/Giskard-AI/giskard-oss

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects