Repolode
Submit
arthur-ai

arthur-ai/arthur-engine

Open-source engine for guardrails, tracing, evaluation and monitoring of LLM and agentic AI applications, plus ML model monitoring.

View on GitHubWebsite
Stars
93
Forks
19
Open issues
27
Last commit
5 hours ago

What is it?

The Arthur Engine is a service for developing, monitoring and governing AI and ML workloads. For LLM applications and generative AI workflows, it enforces configurable guardrails that detect PII or sensitive data leakage, hallucinations, prompt injection attempts and toxic language.

For agentic applications, it collects and analyzes OpenInference OpenTelemetry traces from agent workflows and LLM apps, runs continuous evaluations on live traces, and manages and versions prompts. It also supports experiments that compare prompt variants and end-to-end evaluation of retrieval-augmented generation pipelines.

The project also covers traditional ML monitoring and benchmarking, including drift, accuracy, precision, recall, F1 and AUC, though that part requires the Arthur Platform. Custom metrics and an extensible API let it fit into existing application architectures.

Key features

  • 01Real-time guardrails for PII leakage, hallucination, prompt injection and toxic language
  • 02Collects OpenInference OpenTelemetry traces from agentic workflows and LLM apps
  • 03Continuous evaluations run on live traces to catch quality regressions
  • 04Prompt management with versioning and variant experiments
  • 05End-to-end evaluation of RAG pipelines
  • 06Custom metrics and an extensible API

Use cases

  • Adding guardrails to an LLM application to block sensitive data leaks or prompt injection
  • Tracing and evaluating a multi-step agent to detect quality regressions
  • Comparing prompt variants and measuring their effect on quality metrics
  • Monitoring traditional ML model drift and accuracy in production

Quick start

  1. 1

    Run the engine installer for macOS from the README (official install script, run manually)

  2. 2

    Run the engine installer for Windows from the README (official install script, run manually)

  3. 3

    Instrument your agents by following the examples in the genai-engine/examples folder

Things to know

  • ML model monitoring requires the Arthur Platform.
  • Full platform features such as custom dashboards and alerts need a sign-up to the Arthur Platform.
  • Setup relies on running an installer script rather than a standard package-manager install.

In collections

OpenTelemetry & observability platforms

Topics

#llm#guardrails#observability#evaluation#tracing#agentic#monitoring#opentelemetry#benchmarking#genai

FAQ

What is arthur-engine?
Open-source engine for guardrails, tracing, evaluation and monitoring of LLM and agentic AI applications, plus ML model monitoring. The Arthur Engine is a service for developing, monitoring and governing AI and ML workloads. For LLM applications and generative AI workflows, it enforces configurable guardrails that detect PII or sensitive data leakage, hallucinations, prompt injection attempts and toxic language.
Is arthur-engine free to use? What license does it have?
arthur-engine is open source under the MIT license. Check the license terms before using it in commercial products.
Is arthur-engine still maintained?
Actively maintained. The last commit to arthur-engine was on Nov 7, 2026.
What can I use arthur-engine for?
Adding guardrails to an LLM application to block sensitive data leaks or prompt injection Tracing and evaluating a multi-step agent to detect quality regressions Comparing prompt variants and measuring their effect on quality metrics Monitoring traditional ML model drift and accuracy in production
How do I install arthur-engine?
Run the engine installer for macOS from the README (official install script, run manually) Run the engine installer for Windows from the README (official install script, run manually) Instrument your agents by following the examples in the genai-engine/examples folder
Where can I get arthur-engine?
The source code and installation instructions are on GitHub: https://github.com/arthur-ai/arthur-engine

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects