Repolode
Submit
howie6879

howie6879/ruia

Async Python web scraping micro-framework built on asyncio and aiohttp, aiming to make crawling URLs as convenient as possible.

View on GitHubWebsite
Stars
1,739
Forks
185
Open issues
9
Last commit
8 days ago

What is it?

Ruia is a web scraping micro-framework for Python 3.6+, written on top of asyncio and aiohttp. It lets you define a spider declaratively: you describe the data items to extract and the parsing rules, and the framework handles concurrent requests on the event loop.

The framework is built around extensibility. Middlewares let you hook into the request and response pipeline, and plugins extend its behavior. Installation on Linux and macOS can include uvloop for a faster event loop, and the project also lists JavaScript support among its capabilities.

The README points to a separate ruia-cache project for caching responses during debugging to reduce request load on target sites, and to ruia-shell for interactively debugging scripts. Distributed crawling is listed as not yet implemented.

Key features

  • 01Declarative spider and data item definitions
  • 02Built on asyncio and aiohttp for concurrent requests
  • 03Middleware and plugin system for extension
  • 04Optional uvloop support on Linux and macOS
  • 05JavaScript support

Use cases

  • Scraping many pages concurrently from a list of URLs
  • Building small, asynchronous data collection scripts in Python
  • Extending a crawler with custom middlewares or plugins

Quick start

  1. 1

    Install on Linux or Mac (with uvloop)

    pip install -U ruia[uvloop]
  2. 2

    Install on Windows

    pip install -U ruia
  3. 3

    Read the tutorials to write your first spider

Things to know

  • Ruia is still under development, so its API may change.
  • Distributed crawling/scraping is listed as a TODO and is not yet available.

In collections

Concurrency & async libraries

Topics

#web-scraping#crawler#asyncio#python#aiohttp#spider#async#asyncio-spider#crawling-framework#middlewares

FAQ

What is ruia?
Async Python web scraping micro-framework built on asyncio and aiohttp, aiming to make crawling URLs as convenient as possible. Ruia is a web scraping micro-framework for Python 3.6+, written on top of asyncio and aiohttp. It lets you define a spider declaratively: you describe the data items to extract and the parsing rules, and the framework handles concurrent requests on the event loop.
Is ruia free to use? What license does it have?
ruia is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is ruia still maintained?
Actively maintained. The last commit to ruia was on Oct 30, 2026.
What can I use ruia for?
Scraping many pages concurrently from a list of URLs Building small, asynchronous data collection scripts in Python Extending a crawler with custom middlewares or plugins
How do I install ruia?
Install on Linux or Mac (with uvloop): pip install -U ruia[uvloop] Install on Windows: pip install -U ruia Read the tutorials to write your first spider
Where can I get ruia?
The source code and installation instructions are on GitHub: https://github.com/howie6879/ruia

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects