Repolode
Submit
gocolly

gocolly/colly

A Go framework for writing web scrapers and crawlers that extract structured data from websites.

View on GitHubWebsite
Stars
25,578
Forks
1,857
Open issues
194
Last commit
22 days ago

What is it?

Colly is a scraping and crawling framework for Go. It gives developers a clean interface for writing crawlers, scrapers and spiders that pull structured data out of websites, for uses such as data mining, data processing or archiving.

A collector handles the core crawl loop: it visits URLs, parses HTML with callbacks for matching elements, and can follow links it discovers. It manages request delays and maximum concurrency per domain, handles cookies and sessions automatically, and supports synchronous, asynchronous and parallel scraping. Robots.txt support, response caching and automatic encoding conversion for non-UTF-8 pages are also built in.

The project is distributed as a Go module (github.com/gocolly/colly/v2) and can be configured through environment variables and extended with plugins.

Key features

  • 01Clean API for defining HTML callbacks on matching elements
  • 02Per-domain request delays and concurrency limits
  • 03Automatic cookie and session handling
  • 04Synchronous, asynchronous and parallel scraping modes
  • 05Response caching and robots.txt support
  • 06Automatic encoding conversion for non-Unicode pages

Use cases

  • Extracting product listings or prices from e-commerce sites into a database
  • Collecting articles or quotes from websites for data analysis
  • Archiving or mirroring pages on a schedule
  • Crawling a site to discover and visit all of its links

Quick start

  1. 1

    Install the Colly v2 package with go get

    go get github.com/gocolly/colly/v2
  2. 2

    Create a main.go file with the example code from the README and run it

In collections

Frameworks for building apps and websites

Topics

#web-scraping#crawler#golang#data-extraction#spider#html-parsing#crawling#framework#go#scraper

FAQ

What is colly?
A Go framework for writing web scrapers and crawlers that extract structured data from websites. Colly is a scraping and crawling framework for Go. It gives developers a clean interface for writing crawlers, scrapers and spiders that pull structured data out of websites, for uses such as data mining, data processing or archiving.
Is colly free to use? What license does it have?
colly is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is colly still maintained?
Actively maintained. The last commit to colly was on Oct 16, 2026.
What can I use colly for?
Extracting product listings or prices from e-commerce sites into a database Collecting articles or quotes from websites for data analysis Archiving or mirroring pages on a schedule Crawling a site to discover and visit all of its links
How do I install colly?
Install the Colly v2 package with go get: go get github.com/gocolly/colly/v2 Create a main.go file with the example code from the README and run it
Where can I get colly?
The source code and installation instructions are on GitHub: https://github.com/gocolly/colly

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects