[syphar]

Why

During crawler-driven traffic peaks, the docs.rs daemon reached roughly 12,000 threads and 19,000 open file descriptors. Requests slowed down, PostgreSQL pool acquisition times increased, and SQLite connection initialization failed with Resource temporarily unavailable. This problem lead to unplanned daemon restarts, and also build-servers stopping to build.

Archive indexes currently use SQLx, which runs a dedicated worker thread for each SQLite connection. We open a connection per index lookup, so concurrent requests can create thousands of SQLite worker threads. Slow directory listings and connection initialization can further increase the number of requests—and connections—alive simultaneously.

The investigation also found contention from builds sharing the host. This change addresses the SQLite thread growth; it does not establish that SQLite was the sole cause of the PostgreSQL pool pressure.

Approach

Replace SQLx archive-index access with synchronous rusqlite operations scheduled on a dedicated, fixed-size Rayon pool shared by the archive-index cache.

This separates SQLite connection lifetime from worker-thread lifetime. Connections can remain open without retaining a dedicated thread, while excess SQLite work queues behind the configured workers.

A dedicated pool also keeps this work separate from Tokio's blocking pool. Connection pooling could reduce initialization costs later, but is not required to bound SQLite worker threads.

Changes

• Use rusqlite for archive-index creation and reads, preserving the existing index format.
• Run connection initialization and reads on cache-owned Rayon workers, configured through DOCSRS_ARCHIVE_INDEX_THREADS (default: 32).
• Keep builder-side index creation on spawn_blocking.
• Preserve streaming directory results using bounded batches.
• Handle cancellation safely: skip cancelled queued jobs, retain exclusive connection access until running jobs finish, and stop streaming producers when their receiver is dropped.
• Convert worker panics into caller errors while keeping the pool available.
• Preserve read-only, immutable index access and correctly encode SQLite URI paths.
• Avoid purging a replacement index when another request repairs the cache during a failed open.
• Add tracing spans separating worker queue time from execution time, alongside connection initialization and lookup/listing spans.
• Consolidate and extend archive-index benchmarks using the existing 80,827-entry fixture, covering reused and fresh connections and concurrent request batches.

This bounds worker threads, not the number of queued requests or open connections.