Repolode
Submit
Azure-Samples

Azure-Samples/AI-Gateway

A collection of hands-on Jupyter labs for managing AI models, MCP tools, and agents through Azure API Management's AI Gateway.

View on GitHubWebsite
Stars
1,008
Forks
520
Open issues
84
Last commit
2 days ago

What is it?

AI Gateway Labs is a set of hands-on Jupyter notebooks that show how to place Azure API Management in front of AI models, tools, and agents. Each lab comes with step-by-step instructions, Bicep infrastructure templates, and APIM policies that can be deployed to an Azure subscription.

The labs cover three areas. For models, they demonstrate backend pool load balancing, token rate limiting, semantic caching, model routing, and a FinOps budget framework. For tools, they cover MCP server access with OAuth, MCP client authorization, function calling with Azure Functions, and realtime audio combined with MCP. For agents, they include examples with Foundry Agent Service, the OpenAI Agents SDK, Gemini with MCP, and A2A-enabled agents.

The repository also includes developer utilities such as a tracing tool, a streaming tester, a rate limit tester, an OpenAI API mock server, and an OAuth client, along with Copilot agent skills for generating new labs.

Key features

  • 01Hands-on notebooks covering model, tool, and agent gateway scenarios
  • 02Bicep templates and APIM policy XML for deploying to Azure
  • 03Token rate limiting, semantic caching, and backend load balancing examples
  • 04MCP server access with OAuth and client authorization flows
  • 05Developer utilities including an OpenAI API mock server and rate limit tester

Use cases

  • Adding rate limiting and cost controls before exposing LLM APIs to internal teams
  • Connecting MCP tools to agents with centralized OAuth credential management
  • Evaluating semantic caching or multi-backend routing for Azure OpenAI deployments
  • Learning how to govern AI agents and tools with an enterprise API gateway

Quick start

  1. 1

    Install uv (Linux/macOS) using the README's official install script

  2. 2

    Clone the repository

    git clone https://github.com/Azure-Samples/AI-Gateway.git
  3. 3

    Enter the project folder and create the virtual environment with dependencies

    cd AI-Gateway && uv sync
  4. 4

    Open VS Code and start with a lab notebook, selecting the .venv kernel

    code .

Things to know

  • Labs deploy resources to an Azure subscription, so an Azure account with Contributor and RBAC Administrator roles is required.
  • Requires the Azure CLI and VS Code with the Jupyter extension.
  • Labs are Azure-specific and not a standalone, cloud-agnostic gateway.

In collections

Open-source AI agent frameworks & appsMCP serversMicroservice frameworks & service infrastructure

Topics

#api-gateway#azure-api-management#llm#mcp#ai-agents#rate-limiting#semantic-caching#jupyter#a2a#agents

FAQ

What is AI-Gateway?
A collection of hands-on Jupyter labs for managing AI models, MCP tools, and agents through Azure API Management's AI Gateway. AI Gateway Labs is a set of hands-on Jupyter notebooks that show how to place Azure API Management in front of AI models, tools, and agents. Each lab comes with step-by-step instructions, Bicep infrastructure templates, and APIM policies that can be deployed to an Azure subscription.
Is AI-Gateway free to use? What license does it have?
AI-Gateway is open source under the MIT license. Check the license terms before using it in commercial products.
Is AI-Gateway still maintained?
Actively maintained. The last commit to AI-Gateway was on Nov 5, 2026.
What can I use AI-Gateway for?
Adding rate limiting and cost controls before exposing LLM APIs to internal teams Connecting MCP tools to agents with centralized OAuth credential management Evaluating semantic caching or multi-backend routing for Azure OpenAI deployments Learning how to govern AI agents and tools with an enterprise API gateway
How do I install AI-Gateway?
Install uv (Linux/macOS) using the README's official install script Clone the repository: git clone https://github.com/Azure-Samples/AI-Gateway.git Enter the project folder and create the virtual environment with dependencies: cd AI-Gateway && uv sync Open VS Code and start with a lab notebook, selecting the .venv kernel: code .
Where can I get AI-Gateway?
The source code and installation instructions are on GitHub: https://github.com/Azure-Samples/AI-Gateway

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects