Repolode
Submit
NVIDIA

NVIDIA/garak

An open-source command-line scanner that probes LLMs for failure modes such as prompt injection, data leakage, jailbreaks, and toxic output.

View on GitHubWebsite
Stars
9,635
Forks
1,390
Open issues
510
Last commit
17 hours ago

What is it?

garak is a vulnerability scanner for large language models. It runs a library of probes against a target model and records where the model produces undesirable behavior, covering areas such as hallucination, data leakage, prompt injection, misinformation, toxicity, and jailbreaks. Its design is loosely modeled on network scanners like nmap, but aimed at LLMs and dialog systems.

The tool is driven from the command line. You pick a target type and model name, optionally narrow the run to a probe family or a single probe with --spec, and garak applies each probe's recommended detectors to the responses. Failing prompts are reported with failure rates, and every attempt is written to a JSONL report, with a separate hit log for successful attacks.

garak supports Hugging Face models (local pipelines and hosted endpoints), OpenAI, AWS Bedrock, Replicate, Cohere, Groq, NIM, LiteLLM, llama.cpp-compatible GGUF models, and arbitrary REST endpoints configured through YAML. It installs from PyPI and requires Python 3.11 or newer.

Key features

  • 01Ships a large set of probes for prompt injection, jailbreaks, encoding attacks, hallucination, and toxicity.
  • 02Works with Hugging Face, OpenAI, AWS Bedrock, Replicate, Cohere, Groq, NIM, and REST endpoints.
  • 03Runs all probes by default, or a chosen module or single probe via --spec.
  • 04Writes detailed JSONL run reports and a separate hit log of successful attacks.
  • 05Uses a plugin structure of probes, detectors, generators, and harnesses that can be extended.

Use cases

  • Red-teaming a chatbot or LLM application before release.
  • Comparing how different models hold up against the same set of attacks.
  • Checking whether a self-hosted or API-served model is vulnerable to known jailbreak prompts.
  • Adding automated LLM security checks to a research or evaluation pipeline.

Quick start

  1. 1

    Install garak from PyPI

    python -m pip install -U garak
  2. 2

    Set your OpenAI API key (replace the example value with a real key)

    export OPENAI_API_KEY="sk-123XXXXXXXXXXXX"
  3. 3

    Run an encoding prompt-injection scan against an OpenAI model

    python3 -m garak --target_type openai --target_name gpt-5-nano --spec probes.encoding

Things to know

  • Hosted model scans need an API key from the provider, such as OpenAI, Replicate or Cohere.
  • The README says it is developed and tested mainly on Linux and macOS.

Topics

#llm-security#red-teaming#prompt-injection#jailbreak#vulnerability-scanner#llm-evaluation#ai-safety#ai#security-scanners#vulnerability-assessment

FAQ

What is garak?
An open-source command-line scanner that probes LLMs for failure modes such as prompt injection, data leakage, jailbreaks, and toxic output. garak is a vulnerability scanner for large language models. It runs a library of probes against a target model and records where the model produces undesirable behavior, covering areas such as hallucination, data leakage, prompt injection, misinformation, toxicity, and jailbreaks. Its design is loosely modeled on network scanners like nmap, but aimed at LLMs and dialog systems.
Is garak free to use? What license does it have?
garak is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is garak still maintained?
Actively maintained. The last commit to garak was on Oct 20, 2026.
What can I use garak for?
Red-teaming a chatbot or LLM application before release. Comparing how different models hold up against the same set of attacks. Checking whether a self-hosted or API-served model is vulnerable to known jailbreak prompts. Adding automated LLM security checks to a research or evaluation pipeline.
How do I install garak?
Install garak from PyPI: python -m pip install -U garak Set your OpenAI API key (replace the example value with a real key): export OPENAI_API_KEY="sk-123XXXXXXXXXXXX" Run an encoding prompt-injection scan against an OpenAI model: python3 -m garak --target_type openai --target_name gpt-5-nano --spec probes.encoding
Where can I get garak?
The source code and installation instructions are on GitHub: https://github.com/NVIDIA/garak

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects