Repolode
Submit
projectdiscovery

projectdiscovery/katana

A fast, configurable web crawler and spider framework from ProjectDiscovery, with standard and headless browser modes for discovering endpoints.

View on GitHub
Stars
17,807
Forks
1,218
Open issues
21
Last commit
23 hours ago

What is it?

Katana is a command-line crawler written in Go for discovering URLs and endpoints on websites. It runs in two modes: a standard mode that uses Go's HTTP client and only parses raw responses, and a headless mode that drives Chrome so it can also capture endpoints produced by JavaScript and XHR requests.

Scope is controlled by root domain, fully qualified domain, or custom regex rules, and output can be written to stdout, a file, or JSONL with selectable fields. Input can come from a single URL, a list file, or stdin, which makes it easy to chain with tools such as httpx.

The project also includes options for JavaScript endpoint parsing, automatic form filling, known-file crawling (robots.txt and sitemap.xml), similar-URL and similar-page filtering, rate limiting, and a machine-learning knowledge base that classifies page types and detects secrets or API endpoints.

Key features

  • 01Standard HTTP crawling and headless Chrome crawling modes
  • 02JavaScript file parsing to discover endpoints
  • 03Scope control by root domain, FQDN, or regex
  • 04Automatic form filling and recorded-flow login replay in headless mode
  • 05Similar-URL and similar-page filtering to reduce duplicate crawling
  • 06Output to stdout, files, or JSONL with configurable fields

Use cases

  • Mapping the attack surface or endpoints of a web application during security testing
  • Building a URL list for further scanning in an automation pipeline
  • Crawling single-page applications that rely on JavaScript-rendered content
  • Collecting site structure while limiting scope to a target domain

Quick start

  1. 1

    Install katana with Go (requires Go 1.26+)

    CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest
  2. 2

    Crawl a target URL

    katana -u https://tesla.com

Things to know

  • Requires Go 1.26 or newer to install.
  • Headless mode needs a Chrome browser, either installed locally or provided by you.
  • Several features are marked experimental, including automatic form filling and headless hybrid crawling.

In collections

CLI & terminal tools

Topics

#web-crawler#spider#headless-browser#security-testing#endpoint-discovery#cli#go#crawler#gocrawler#hacktoberfest

FAQ

What is katana?
A fast, configurable web crawler and spider framework from ProjectDiscovery, with standard and headless browser modes for discovering endpoints. Katana is a command-line crawler written in Go for discovering URLs and endpoints on websites. It runs in two modes: a standard mode that uses Go's HTTP client and only parses raw responses, and a headless mode that drives Chrome so it can also capture endpoints produced by JavaScript and XHR requests.
Is katana free to use? What license does it have?
katana is open source under the MIT license. Check the license terms before using it in commercial products.
Is katana still maintained?
Actively maintained. The last commit to katana was on Nov 6, 2026.
What can I use katana for?
Mapping the attack surface or endpoints of a web application during security testing Building a URL list for further scanning in an automation pipeline Crawling single-page applications that rely on JavaScript-rendered content Collecting site structure while limiting scope to a target domain
How do I install katana?
Install katana with Go (requires Go 1.26+): CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest Crawl a target URL: katana -u https://tesla.com
Where can I get katana?
The source code and installation instructions are on GitHub: https://github.com/projectdiscovery/katana

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects