Repolode
Submit
microsoft

microsoft/Power-CAT-Copilot-Studio-Kit

A toolkit of Microsoft Copilot Studio add-ons for building, governing, testing and monitoring custom agents, distributed as Power Platform solutions.

View on GitHubWebsite
Stars
477
Forks
104
Open issues
107
Last commit
2 days ago

What is it?

The Copilot Agent Kit is a set of capabilities that extends Microsoft Copilot Studio. It helps makers batch test custom agents with several test types, including response match, topic match, multi-turn and generative answers scored by AI Builder, and it supports user-defined rubrics for grading generative answers.

The kit also covers governance and operations. Compliance Hub evaluates agent configurations against configurable risk thresholds and opens SLA-tracked compliance cases. Agent Inventory gives administrators a tenant-wide view of agents, while Agent Insights Hub (Preview) aggregates Application Insights telemetry and conversation transcripts into KPI dashboards. Agent Change Tracker records saved changes, and PowerShield manages connector access through an approval workflow for DLP policies.

Other modules include Agent Debugger for stepping through recorded conversations, Prompt Advisor for scoring and refining prompts, Webchat Playground for styling a webchat, an Adaptive Cards Gallery, a SharePoint content sync into agent knowledge bases, and an Agent Library and Component Library for installing prebuilt templates.

Key features

  • 01Batch test custom agents with response, topic, multi-turn and generative-answer test types.
  • 02Governance checks and SLA-tracked compliance cases through Compliance Hub.
  • 03Aggregated agent KPIs and transcript analysis in Agent Insights Hub (Preview).
  • 04Step-by-step inspection of recorded agent conversations with Agent Debugger.
  • 05Prompt scoring and suggested refinements via Prompt Advisor.
  • 06Scheduled SharePoint content sync into an agent's knowledge base.

Use cases

  • Running regression test suites on Copilot Studio agents before release.
  • Enforcing organization-wide governance policies across many agents in a tenant.
  • Monitoring agent performance, errors and credit consumption over time.
  • Gating agent deployments in Power Platform Pipelines based on test results.

In collections

Open-source AI agent frameworks & appsTesting frameworks & test automation tools

Topics

#copilot-studio#power-platform#ai-agents#testing#governance#llm-evaluation#prompt-engineering

FAQ

What is Power-CAT-Copilot-Studio-Kit?
A toolkit of Microsoft Copilot Studio add-ons for building, governing, testing and monitoring custom agents, distributed as Power Platform solutions. The Copilot Agent Kit is a set of capabilities that extends Microsoft Copilot Studio. It helps makers batch test custom agents with several test types, including response match, topic match, multi-turn and generative answers scored by AI Builder, and it supports user-defined rubrics for grading generative answers.
Is Power-CAT-Copilot-Studio-Kit free to use? What license does it have?
Power-CAT-Copilot-Studio-Kit is open source under the MIT license. Check the license terms before using it in commercial products.
Is Power-CAT-Copilot-Studio-Kit still maintained?
Actively maintained. The last commit to Power-CAT-Copilot-Studio-Kit was on Nov 5, 2026.
What can I use Power-CAT-Copilot-Studio-Kit for?
Running regression test suites on Copilot Studio agents before release. Enforcing organization-wide governance policies across many agents in a tenant. Monitoring agent performance, errors and credit consumption over time. Gating agent deployments in Power Platform Pipelines based on test results.
Where can I get Power-CAT-Copilot-Studio-Kit?
The source code and installation instructions are on GitHub: https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects