Skip to content
@spire-rs

Spire

Crawl the web from Rust: the Spire framework and the libraries it's built on.
README.md

Crates

Crate Version What it does
spire spire The crawler and scraper framework: frontiers, HTTP and browser backends, typed routing and extractors.
uagent uagent Browser profiles with user agents and matching navigation headers for Chrome, Edge, Firefox and Safari.
robotxt robotxt Parses and builds robots.txt files following RFC 9309, with the crawl-delay and sitemap extensions.
sitemapo sitemapo Parses and builds text and XML sitemaps and sitemap indexes, with the image, video, news and hreflang extensions.
countio countio Byte counting for readers and writers, for std::io, tokio and futures.

Getting started

The Spire Book is the guide: handling pages, discovering them, fetching over HTTP or with a browser, running and resuming a crawl, and extending spire. The API reference lives on docs.rs.

Pinned Loading

  1. spire spire Public

    The flexible crawler and scraper framework for Rust: route pages to typed handlers, crawl over HTTP or real browsers, built on tokio.

    Rust 6 1

  2. robotxt robotxt Public

    Parse and build robots.txt files in Rust per RFC 9309, with crawl-delay and sitemap extensions.

    Rust 2

  3. sitemapo sitemapo Public

    Parse and build text and XML sitemaps in Rust, with sitemap indexes and image, video, news and hreflang extensions.

    Rust 2

  4. uagent uagent Public

    Browser profiles for Rust: user agents, matching navigation headers and client hints, with deterministic selection for web crawlers.

    Rust 2

Repositories

Showing 6 of 6 repositories

Top languages

Loading…

Most used topics

Loading…