Repolode
Submit
ifixai-ai

ifixai-ai/iFixAi

A Python CLI that audits AI agents against 61 inspections, testing whether they stay within their role, resist manipulation and avoid fabrication.

View on GitHubWebsite
Stars
43,588
Forks
1,785
Open issues
3
Last commit
yesterday

What is it?

iFixAi is an independent auditing tool for AI agents. It builds a simulated environment of a business from a profile and policy file, sends the agent the kinds of requests a normal day brings, including requests that push it beyond its role, and has a separate judge model grade each reply. Each finding records what was asked and how the agent responded.

A default run covers 10 inspections that work over a plain chat endpoint, covering tool permissions, prompt injection, policy violations, cross-user data leaks, uncertainty handling and instruction following. The --full-run option runs all 61 inspections, skipping those that cannot apply to the agent. Results are scored from A to F across five pillars, with three mandatory minimums that cap the grade at D if failed.

The judge is chosen from a different provider than the agent under test. Reports are written as JSON and Markdown, and a --min-score threshold can gate CI pipelines. A built-in demo agent lets users see the workflow without an API key.

Key features

  • 01Runs 10 default inspections quickly over a standard chat endpoint.
  • 02Offers 61 inspections through the --full-run option.
  • 03Uses a judge model from a different provider than the agent under test.
  • 04Grades results A to F with mandatory minimums that cap the score.
  • 05Exports reports as JSON and Markdown and supports CI score gating.

Use cases

  • Checking whether a customer support agent respects refund and account-change permissions.
  • Testing an agent for prompt injection before deploying it to production.
  • Producing audit evidence for AI governance frameworks such as the EU AI Act or NIST AI RMF.
  • Adding a regression gate for agent behavior in a CI pipeline.

Quick start

  1. 1

    Create and activate a virtualenv

    python3 -m venv .venv && source .venv/bin/activate
  2. 2

    Install iFixAi from PyPI

    pip install ifixai
  3. 3

    Generate the reference simulation environment

    ifixai author reference -o reference.yaml
  4. 4

    Run the default inspections against the built-in demo agent (no API key needed)

    ifixai run --provider mock --eval-mode self --fixture reference.yaml

Things to know

  • Running a real audit requires a judge model from a different provider and its API key.
  • Requires Python 3.10 or newer.

In collections

CLI & terminal toolsTesting frameworks & test automation tools

Topics

#agent-evaluation#ai-safety#llm-evaluation#prompt-injection#red-teaming#ai-governance#cli#ai#ai-alignment#ai-evaluation

FAQ

What is iFixAi?
A Python CLI that audits AI agents against 61 inspections, testing whether they stay within their role, resist manipulation and avoid fabrication. iFixAi is an independent auditing tool for AI agents. It builds a simulated environment of a business from a profile and policy file, sends the agent the kinds of requests a normal day brings, including requests that push it beyond its role, and has a separate judge model grade each reply. Each finding records what was asked and how the agent responded.
Is iFixAi free to use? What license does it have?
iFixAi is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is iFixAi still maintained?
Actively maintained. The last commit to iFixAi was on Nov 6, 2026.
What can I use iFixAi for?
Checking whether a customer support agent respects refund and account-change permissions. Testing an agent for prompt injection before deploying it to production. Producing audit evidence for AI governance frameworks such as the EU AI Act or NIST AI RMF. Adding a regression gate for agent behavior in a CI pipeline.
How do I install iFixAi?
Create and activate a virtualenv: python3 -m venv .venv && source .venv/bin/activate Install iFixAi from PyPI: pip install ifixai Generate the reference simulation environment: ifixai author reference -o reference.yaml Run the default inspections against the built-in demo agent (no API key needed): ifixai run --provider mock --eval-mode self --fixture reference.yaml
Where can I get iFixAi?
The source code and installation instructions are on GitHub: https://github.com/ifixai-ai/iFixAi

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects

cvs-health / uqlm

Python library that scores LLM responses for likely hallucinations using uncertainty quantification, with black-box, white-box, judge and ensemble scorers.

AI & Machine Learning#hallucination-detection#uncertainty-quantification#llm-evaluation
1.2K+2 this week140PythonApache-2.0Well maintainedUpdated 25 days ago