Repolode
Submit
openlayer-ai

openlayer-ai/jevals

Python library for running agent evals and guardrails on hosted or local Jev-style decision models, with all checks for a trace sent in one request.

View on GitHubWebsite
Stars
112
Forks
12
Open issues
3
Last commit
5 days ago

What is it?

jevals runs evaluations and guardrails for LLM agents using decision models instead of a frontier LLM as judge. Each eval is a typed yes/no, choice or rubric question, and all evals for a trace are packed into a single request that returns calibrated probabilities. The library covers agent behavior (tool choice, grounding, scope, loops), security (prompt and indirect injection, PII/PHI, secrets) and Ragas-style quality metrics.

Backends can be Jev through the TypeSafe or Vercel AI Gateway APIs, decision models on OpenRouter, locally run Kev or Laya on a Mac, Eikos on a GPU, or a regular chat LLM as a slower fallback. Evals can be written in Python or YAML and reused offline, as production monitors, and as gates that allow, escalate or block tool calls before they run.

The project includes a CLI for replaying JSONL traces and calibrating thresholds against human labels, an MCP server for coding agents, and adapters for the OpenAI Agents SDK, LangGraph and the Claude Agent SDK. It is alpha, and the README notes the direct TypeSafe backend and framework adapters are tested against mocks rather than run live.

Key features

  • 01Runs all evals for one trace in a single request to a decision model.
  • 02Ships 37 evals across agent behavior, security and quality, including Ragas-style metrics.
  • 03Defines evals in Python classes or YAML files with a shared schema.
  • 04Gates can allow, escalate or block tool calls and redact PII/PHI before they reach the model.
  • 05Supports multiple backends: Jev via TypeSafe or Vercel, OpenRouter, Kev, Laya, Eikos, or chat LLMs.
  • 06Includes a CLI to replay traces and calibrate thresholds against labeled data.

Use cases

  • Scoring production agent traces for tool misuse, ungrounded answers and scope violations on every request.
  • Blocking indirect prompt injection in tool results or retrieved documents before the agent acts on them.
  • Gating irreversible tool calls such as refunds or database deletes with human escalation.
  • Running offline regression evals for a RAG or agent pipeline without a costly LLM judge.

Quick start

  1. 1

    Install the package from PyPI

    pip install jevals
  2. 2

    Set an API key for the Jev backend through Vercel AI Gateway

    export AI_GATEWAY_API_KEY=...

Things to know

  • Needs a Jev API key, a TypeSafe account, or a self-hosted decision model such as Kev or Eikos to run the calibrated evals.
  • Local options depend on hardware: Kev and Laya need a Mac, and Eikos needs a GPU.
  • Without a decision model, the emulated chat-LLM backend returns only 0.00 or 1.00 probabilities.

In collections

Open-source AI agent frameworks & appsTesting frameworks & test automation tools

Topics

#agents#llm-evaluation#guardrails#prompt-injection#rag#llm-safety#python#evals#jev#llm

FAQ

What is jevals?
Python library for running agent evals and guardrails on hosted or local Jev-style decision models, with all checks for a trace sent in one request. jevals runs evaluations and guardrails for LLM agents using decision models instead of a frontier LLM as judge. Each eval is a typed yes/no, choice or rubric question, and all evals for a trace are packed into a single request that returns calibrated probabilities. The library covers agent behavior (tool choice, grounding, scope, loops), security (prompt and indirect injection, PII/PHI, secrets) and Ragas-style quality metrics.
Is jevals free to use? What license does it have?
jevals is open source under the MIT license. Check the license terms before using it in commercial products.
Is jevals still maintained?
Actively maintained. The last commit to jevals was on Nov 2, 2026.
What can I use jevals for?
Scoring production agent traces for tool misuse, ungrounded answers and scope violations on every request. Blocking indirect prompt injection in tool results or retrieved documents before the agent acts on them. Gating irreversible tool calls such as refunds or database deletes with human escalation. Running offline regression evals for a RAG or agent pipeline without a costly LLM judge.
How do I install jevals?
Install the package from PyPI: pip install jevals Set an API key for the Jev backend through Vercel AI Gateway: export AI_GATEWAY_API_KEY=...
Where can I get jevals?
The source code and installation instructions are on GitHub: https://github.com/openlayer-ai/jevals

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects