feat(scrape-api): honor inbound X-Request-Id on POST /scrape - #41
Merged
Conversation
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| def test_honored_request_id_on_200(self): | ||
| from fastapi.testclient import TestClient | ||
|
|
||
| import app.main as main_mod |
| def test_request_id_collision_returns_502(self): | ||
| from fastapi.testclient import TestClient | ||
|
|
||
| import app.main as main_mod |
gildesmarais
enabled auto-merge (squash)
August 22, 2026 14:38
gildesmarais
added a commit
to html2rss/html2rss
that referenced
this pull request
Aug 22, 2026
## What changed - `lib/html2rss/request_service/botasaurus_strategy.rb` — each Botasaurus transport `POST /scrape` sends `User-Agent: html2rss/<VERSION>`, `Accept: application/json`, `Accept-Encoding: identity`, and a fresh `X-Request-Id` (UUID) per execute; headers are covered in `botasaurus_strategy_spec.rb`. - `spec/fixtures/botasaurus/openapi.yaml` — fixture synced so the scrape API documents inbound `X-Request-Id`. - `CONTEXT.md` — notes the gem identifies itself on the Botasaurus hop and correlates requests via `X-Request-Id`. - YARD comments on the new transport header constants (lint-yard gate). ## Why Botasaurus scrape-api logs and support need to distinguish html2rss gem traffic from other clients and tie a gem execute to a single API request. Explicit JSON accept/encoding avoids ambiguous Faraday decoding on the hop. ## Risk - Low for non-Botasaurus strategies — change is isolated to `BotasaurusStrategy`. - End-to-end request correlation depends on botasaurus-scrape-api honoring inbound `X-Request-Id` (see sibling PR html2rss/botasaurus-scrape-api#41). Until that merges, the header is sent but may not appear in API diagnostics. ## Review map Review map: whole PR is small — start at `lib/html2rss/request_service/botasaurus_strategy.rb`, then `spec/lib/html2rss/request_service/botasaurus_strategy_spec.rb`. ## Validation - Rebased onto `origin/master` (includes compose Sentry DSN merge). - `make ready` — exit 0 - RuboCop: 318 files, no offenses - yard-lint: no offenses; YARD doc 100% documented - `make schema` — generated schema clean (`git diff --exit-code`) - Fixture validation — `exe/html2rss validate` on spec fixtures - shellcheck — bin scripts clean - RSpec: 1633 examples, 0 failures; line coverage 98.51%, branch 83.74%
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Honor an inbound
X-Request-IdonPOST /scrapeso clients can correlate transport headers withdiagnostics.request_idin JSON responses.What
resolve_request_idpure function for validating and selecting the effective request id200) and error paths (400,422,504, collision handling)Headeron the scrape endpointWhy
html2rss gem clients can align
diagnostics.request_idwith the outbound/inbound transportX-Request-Idfor end-to-end diagnostics.Risk
Backward compatible when the header is absent or invalid (server generates/falls back as before).
Test plan
make check(ruff, format, hadolint, spectral, unittest — 80 tests OK)