Repolode
Submit
oxylabs

oxylabs/web-api-mcp

Self-hostable MCP server that gives AI agents search, page scraping, and dedicated scrapers through the Oxylabs Web API.

View on GitHub
Stars
38
Forks
8
Open issues
2
Last commit
21 hours ago

What is it?

This is the official Model Context Protocol server for the Oxylabs Web API. It lets any MCP-capable agent search the live web, read a page as Markdown or JSON (including JavaScript-heavy and bot-protected pages), and call dedicated scrapers for specific sites, all through one MCP connection.

You can use a hosted Remote MCP endpoint with an API key in a request header, or run the Python server yourself over stdio for a local client or over HTTP as a shared service. Self-hosting is configured through environment variables for the API key, retries, a spend cap, and the inline token budget for large responses.

Large pages are written to a temp file and read in chunks on local runs, and the server flags pages that appear to need JavaScript rendering. Structured output can come from a dedicated parser or an AI parser that follows a prompt or JSON schema.

Key features

  • 01Search the live web and get ranked organic results with title, description, and URL.
  • 02Scrape any page as Markdown, or extract named fields as JSON via a prompt or schema.
  • 03Call dedicated scraper endpoints with their own parameters.
  • 04Run through a hosted remote endpoint or self-hosted over stdio or HTTP.
  • 05Offloads oversized pages to disk and reads them in chunks on local runs.
  • 06Optional per-server rate limit to cap API spend.

Use cases

  • Giving a coding or research agent live web search and page reading.
  • Extracting structured fields from product or listing pages into an agent workflow.
  • Running a shared, team-wide scraping service behind an internal network.
  • Keeping API keys and network access inside your own infrastructure.

Quick start

  1. 1

    Get an Oxylabs Web API key from the dashboard

  2. 2

    Set the API key as an environment variable

    export OXYLABS_WEB_API_KEY="your_api_key_here"
  3. 3

    Connect Claude Code to the hosted Remote MCP server

    claude mcp add --transport http --scope user oxylabs-web-api https://mcp.oxylabs.io/web-api-mcp --header "Authorization: Bearer ${OXYLABS_WEB_API_KEY}"

Things to know

  • Requires an Oxylabs Web API key, and AI parser usage is billed on top of scrapes.
  • Self-hosting needs Python 3.10+ and uv, and you manage updates yourself.
  • The HTTP transport requires setting MCP_ALLOWED_HOSTS or it rejects requests.

In collections

Open-source AI agent frameworks & appsMCP serversSelf-hosted apps

Topics

#mcp#web-scraping#web-search#llm-tools#ai-agents#python#self-hosted#mcp-server#mcp-servers#mcp-tools

FAQ

What is web-api-mcp?
Self-hostable MCP server that gives AI agents search, page scraping, and dedicated scrapers through the Oxylabs Web API. This is the official Model Context Protocol server for the Oxylabs Web API. It lets any MCP-capable agent search the live web, read a page as Markdown or JSON (including JavaScript-heavy and bot-protected pages), and call dedicated scrapers for specific sites, all through one MCP connection.
Is web-api-mcp free to use? What license does it have?
web-api-mcp is open source under the MIT license. Check the license terms before using it in commercial products.
Is web-api-mcp still maintained?
Actively maintained. The last commit to web-api-mcp was on Nov 6, 2026.
What can I use web-api-mcp for?
Giving a coding or research agent live web search and page reading. Extracting structured fields from product or listing pages into an agent workflow. Running a shared, team-wide scraping service behind an internal network. Keeping API keys and network access inside your own infrastructure.
How do I install web-api-mcp?
Get an Oxylabs Web API key from the dashboard Set the API key as an environment variable: export OXYLABS_WEB_API_KEY="your_api_key_here" Connect Claude Code to the hosted Remote MCP server: claude mcp add --transport http --scope user oxylabs-web-api https://mcp.oxylabs.io/web-api-mcp --header "Authorization: Bearer ${OXYLABS_WEB_API_KEY}"
Where can I get web-api-mcp?
The source code and installation instructions are on GitHub: https://github.com/oxylabs/web-api-mcp

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects