Retry a throttled page turn once after a 30-60 s pause; canary walk budget 15 MB

.github/proxy_traffic.py

@@ -5,12 +5,16 @@
 Reads `<out>.latest_attempt.meta.json` (written on every metered run, rows
 or not), else `<out>.meta.json`, and appends
 
-    **<label>**: proxy traffic X.X MB of 10 MB
+    **<label>**: proxy traffic X.X MB of N MB
 
-to $GITHUB_STEP_SUMMARY. A missing or unreadable sidecar, or a run that did
-not go through a local proxy, is reported as such and is not a failure here
-(the exit-code steps judge the run itself). A run that went over its proxy
-budget prints `::error::stopped: proxy budget 10 MB exceeded` and exits 1.
+to $GITHUB_STEP_SUMMARY. N is the run's OWN budget, read from the sidecar's
+`proxy_budget_bytes` (written by the scraper from --proxy-budget-mb; `null`
+or absent reads as "no budget"), so a canary step that raises its budget is
+reported against that budget rather than a fixed figure. A missing or
+unreadable sidecar, or a run that did not go through a local proxy, is
+reported as such and is not a failure here (the exit-code steps judge the
+run itself). A run that went over its proxy budget prints
+`::error::stopped: proxy budget N MB exceeded` and exits 1.
 """
 
 import json

.github/workflows/canary.yml

@@ -163,20 +163,28 @@ jobs:
         # 2026-10-21 walk used 9.6 MB of the 10 MB cap with two of its three
         # pages only 19 cards deep, so three full pages with images would
         # not fit. image_url is not one of the columns this job checks.
+        # --proxy-budget-mb 15, not the scraper's default 10: with
+        # --block-media the 2026-10-22 walk (run 39858684619) used 8.7 MB
+        # for pages 1 and 2 at 50 of 50 cards each, so a full third page and
+        # one paced retry of a throttled turn would not fit under 10.
+        # A turn that gets no grid back (a /graphql 429) is retried ONCE
+        # after a 30-60 s pause (--throttle-wait-seconds, default 45);
+        # throttle_retries in the sidecar says whether that happened.
         run: |
           set +e
           start=$(date -u -d '+45 days' +%F)
           end=$(date -u -d '+47 days' +%F)
           url="https://www.vrbo.com/search?destination=Orlando%2C%20Florida%2C%20United%20States%20of%20America&regionId=2693&sort=RECOMMENDED&startDate=${start}&endDate=${end}&adults=2"
           xvfb-run -a python3 playwright_scraper.py --url "$url" \
-            --pages 3 --delay 15 --retries 4 --proxy-budget-mb 10 --block-media \
+            --pages 3 --delay 15 --retries 4 --proxy-budget-mb 15 --block-media \
             --out canary_walk --dump-html canary_walk.html
           echo "exit_code=$?" >> "$GITHUB_OUTPUT"
 
       - name: Proxy traffic of the walk
         # The walk goes through VRBO_PROXY, which meters what it sends and
-        # stops at --proxy-budget-mb 10. This puts the figure in the step
-        # summary and fails the job when the cap was hit.
+        # stops at --proxy-budget-mb 15. This puts the figure in the step
+        # summary, with the budget read from the run's own sidecar, and fails
+        # the job when the cap was hit.
         if: ${{ !cancelled() && steps.run.outcome != 'skipped' }}
         run: python3 .github/proxy_traffic.py "pagination" canary_walk
 

CHANGELOG.md

@@ -10,6 +10,14 @@ that is a fix, and the release notes lead with it.
 
 ## [Unreleased]
 
+> **Behaviour change.** A page turn that gets no grid back (the site's
+> `/graphql` answered HTTP 429, with no `Retry-After`) is now retried
+> **once** in all three engines, after a random 30-60 s pause, in the same
+> browser session and exit. A run that used to end `next_page_throttled`
+> (exit 6) at once can now take up to about two minutes longer at that
+> page, and goes on to the next page when the retry succeeds.
+> `--throttle-wait-seconds 0` restores the old behaviour.
+
 > **Behaviour change.** On a 2Captcha residential gateway
 > (`*.proxy.2captcha.com`, `*.proxy.rucaptcha.com`), a proxy login with no
 > `-session-` segment now gets one random `-session-<id>-sessTime-30`
@@ -34,6 +42,18 @@ that is a fix, and the release notes lead with it.
 
 ### Added
 
+- `--throttle-wait-seconds` in all three engines (default 45, `0` = no
+  retry): one paced retry of a page turn that got no grid back. The pause is
+  a random 2/3 to 4/3 of the value. The proxy budget is checked before the
+  pause and again before the second press, so a spent budget never buys a
+  retry. A grid that arrives during the pause counts as the turn and is not
+  pressed again. The listing sidecar gains `throttle_retries` (retries made
+  in the run). Measured on the 2026-10-22 canary walk (run 39858684619):
+  pages 1 and 2 turned in 2 s, then the press for page 3 got one `/graphql`
+  429 with no `Retry-After`, and the grid stayed empty for 60 s. Offline
+  checks drive each engine's real page fetch with a stubbed grid: 429 then
+  success reads the page, 429 twice ends exit 6 after exactly two presses,
+  and a budget spent by the failed turn makes no second press.
 - `--proxy-session-minutes` in all three engines, and
   `VRBO_PROXY_SESSION_MINUTES` in `.env.example`. The session id is drawn
   once per run, before the pre-check; with `--proxy-file`, each gateway line
@@ -55,9 +75,10 @@ that is a fix, and the release notes lead with it.
     `proxy_budget_exceeded`, and a closing log line
     `Proxy traffic: X.X MB of Y MB budget`.
   - `--proxy-budget-mb`, and `VRBO_PROXY_BUDGET_MB` (in `.env.example`).
-- CI only: both canary runs pass `--proxy-budget-mb 10`. Each one writes its
-  proxy traffic to the step summary, and fails with
-  `::error::stopped: proxy budget 10 MB exceeded` when the cap was hit
+- CI only: both canary runs pass a `--proxy-budget-mb` (10 for the first
+  page, 15 for the walk; see Changed). Each one writes its proxy traffic to
+  the step summary, and fails with
+  `::error::stopped: proxy budget N MB exceeded` when the cap was hit
   (`.github/proxy_traffic.py`).
 - Offline checks:
   - The meter against known CDP and Playwright sizes.
@@ -88,6 +109,13 @@ that is a fix, and the release notes lead with it.
 
 ### Changed
 
+- CI only: the canary's three-page walk through `VRBO_PROXY` passes
+  `--proxy-budget-mb 15`; the scraper's default stays 10. With
+  `--block-media` the 2026-10-22 walk used 8.7 MB for pages 1 and 2 at 50 of
+  50 cards each, so a full third page and a paced retry would not fit under
+  10. The step summary already reported each run against the budget in its
+  own sidecar (`proxy_budget_bytes`); its docstring said "of 10 MB" and now
+  says so, and the canary checks cover a 15 MB run and a run with no cap.
 - CI only: the whole canary, free and paid jobs alike, now runs weekly (Mondays 06:20 UTC) and on a manual dispatch; the daily schedule is removed.
 
 ### Fixed

README.md

@@ -127,10 +127,17 @@ while every `/graphql` POST behind a press came back **HTTP 429**
 (`{"error":"Too Many Requests","message":"Provisioned request rate has been
 exceeded"}`) and the site's own client gave up.
 
-When that happens the run reports **`partial`, exit 6** — never "the listing
-ended" — and says how many 429s it counted. `--delay` (default **5.0s**,
-higher than the family default) is the cheap lever; `--proxy-file` is the one
-that scales.
+The 429 carries no `Retry-After` header. So when a press gets no grid back,
+the run waits a random **30-60 s** (`--throttle-wait-seconds`, default **45**,
+`0` = off) in the same browser session and exit and presses next **once**
+more. The next page exists only inside that session, so it is never retried
+from a new browser or a new exit. The proxy budget is checked before the
+retry. The sidecar records `throttle_retries`.
+
+If the retry gets no grid either, the run reports **`partial`, exit 6** —
+never "the listing ended" — and says how many 429s it counted (Playwright
+only). `--delay` (default **5.0s**, higher than the family default) is the
+cheap lever; `--proxy-file` is the one that scales.
 
 ### 4. The results list is an inner scroller, and the page body never scrolls
 
@@ -260,6 +267,10 @@ expedia_property_id, price_note, sku, url, source. The only differences were
 Engine flag differences are asserted in both directions by the test suite, so
 closing one needs a README edit rather than a quiet patch:
 
+* `--throttle-wait-seconds SECONDS` — all three engines, but not part of the
+  family flag contract. The pause before the one retry of a page turn that
+  got no grid back: a random 2/3 to 4/3 of it (30-60 s at the default 45).
+  `0` turns the retry off.
 * `--block-retry-delay SECONDS` — all three engines, but not part of the
   family flag contract. Wait before re-fetching a page that came back
   refused, doubling per retry (30 waits 30 s, then 60 s). Unset keeps the
@@ -315,7 +326,10 @@ A proxy is billed by traffic. With `--proxy` / `--proxy-file` / `VRBO_PROXY`:
   Media is loaded by default here, and images are most of a search page's
   bytes, so a long walk through a proxy reaches 10 MB sooner than on a site
   that blocks them. Raise the budget, or pass `--block-media` if `image_url`
-  is not needed. The sidecars record `proxy_bytes`, `proxy_budget_bytes`
+  is not needed. Measured on the canary's walk through a residential exit
+  (2026-10-22, run 39858684619): with `--block-media`, pages 1 and 2 at 50 of
+  50 cards each used **8.7 MB**, so that walk runs at `--proxy-budget-mb 15`
+  to leave room for its third page and a paced retry. The sidecars record `proxy_bytes`, `proxy_budget_bytes`
   (`null` with no cap) and `proxy_budget_exceeded`: in `<out>.meta.json`,
   and in `<out>.latest_attempt.meta.json`, which every metered run writes.
   The log ends with `Proxy traffic: X.X MB of Y MB budget`.

TROUBLESHOOTING.md

@@ -50,12 +50,20 @@ Look at `stop_reason` in `<out>.meta.json`.
 on the page. The data behind a next-press comes over `POST /graphql`, which
 has its own limiter: the HTML keeps answering 200 while the turn is refused
 with `{"error":"Too Many Requests","message":"Provisioned request rate has
-been exceeded"}`. The log says how many 429s it counted.
+been exceeded"}`, with no `Retry-After` header. The log says how many 429s
+it counted (Playwright only; Selenium and pyppeteer cannot see the response).
+
+Before it stops, the run has already retried that turn once: it waited a
+random 30-60 s (`--throttle-wait-seconds`, default 45) in the same browser
+session and exit, then pressed next again. `throttle_retries` in the sidecar
+counts those retries. `next_page_throttled` means the retry was refused too
+(or the proxy budget was spent first, which reads `proxy_budget_exceeded`).
 
 In order of cost:
 
 1. `--delay 15` (the default is 5)
-2. wait a few minutes — this clears on its own
+2. a longer `--throttle-wait-seconds`, or wait a few minutes and re-run —
+   this clears on its own
 3. `--proxy-file` to spread the load across exits
 
 It is reported as partial rather than complete **on purpose**. Reporting a

page_flow.py

@@ -53,7 +53,8 @@
 from __future__ import annotations
 
 import logging
-from typing import Callable, Dict, List, Optional
+import random
+from typing import Callable, Dict, List, Optional, Tuple
 
 from product_parser import (SELECTORS, NEXT_PAGE_SELECTOR, PAGE_CAP,
                             challenge_vendor, detect_block_marker,
@@ -635,25 +636,122 @@ def advance_to_next_page(press_next: Callable[[], bool],
     way, and on this site the reason is almost always that the `/graphql`
     POST behind the turn was answered 429.
     """
-    before = first_card_href()
-    before_sku = sku_from_url(strip_tracking(before or "")) if before else None
+    before_sku = _first_sku(first_card_href)
     if not press_next():
         return NO_BUTTON
+    return _await_turnover(first_card_href, count, sleep, before_sku,
+                           timeout_ms)
+
+
+def _first_sku(first_card_href: Callable[[], Optional[str]]) -> Optional[str]:
+    href = first_card_href()
+    return sku_from_url(strip_tracking(href or "")) if href else None
+
+
+def _turned_over(first_card_href, count, before_sku) -> bool:
+    if count(READY_SELECTOR_LISTING) <= 0:
+        return False
+    now_sku = _first_sku(first_card_href)
+    return bool(now_sku and now_sku != before_sku)
+
+
+def _await_turnover(first_card_href, count, sleep, before_sku,
+                    timeout_ms: int) -> str:
     waited = 0
     while waited < timeout_ms:
         sleep(NEXT_PAGE_POLL_MS)
         waited += NEXT_PAGE_POLL_MS
-        if count(READY_SELECTOR_LISTING) <= 0:
-            continue
-        now = first_card_href()
-        now_sku = sku_from_url(strip_tracking(now or "")) if now else None
-        if now_sku and now_sku != before_sku:
+        if _turned_over(first_card_href, count, before_sku):
             logger.info("next page arrived after %dms", waited)
             return ADVANCED
     logger.info("next page never turned over within %dms", timeout_ms)
     return NO_TURNOVER
 
 
+# One bounded retry of a page turn that never came back.
+#
+# Measured on the 2026-10-22 canary walk through a sticky residential
+# session: pages 1 and 2 turned over in 2 s, then the press for page 3 got
+# one `/graphql` HTTP 429 ("Provisioned request rate has been exceeded",
+# no Retry-After header) and the grid stayed empty for the whole 60 s. The
+# answer carries no wait to honour, so the wait is ours: a random pause
+# between 2/3 and 4/3 of `--throttle-wait-seconds` (30-60 s at the default
+# 45), in the SAME browser, session and exit — the next page exists only
+# inside this session, so a relaunch or a new exit would restart the listing
+# at page 1. Then the turn is tried ONCE more. A second failure ends the run
+# as before (partial, next_page_throttled); there is no third press.
+#
+# The retry keys on the no-turnover STATE rather than on a counted 429: the
+# Playwright engine can count the /graphql 429s, Selenium has no response
+# listener and cannot. All three engines therefore retry the same state and
+# report the count only where it is known.
+THROTTLE_WAIT_DEFAULT_S = 45.0
+THROTTLE_RETRIES_PER_TURN = 1
+
+
+def throttle_pause_s(wait_seconds: float,
+                     uniform: Callable[[float, float], float] = random.uniform
+                     ) -> float:
+    """The randomised pause before retrying a throttled turn, in seconds."""
+    return uniform(wait_seconds * 2.0 / 3.0, wait_seconds * 4.0 / 3.0)
+
+
+def advance_with_throttle_retry(press_next: Callable[[], bool],
+                                first_card_href: Callable[[], Optional[str]],
+                                count: Callable[[str], int],
+                                sleep: Callable[[int], None],
+                                wait_seconds: float,
+                                before_retry: Callable[[], None],
+                                throttled: Callable[[], int] = lambda: 0,
+                                uniform: Callable[[float, float], float]
+                                = random.uniform,
+                                timeout_ms: int = NEXT_PAGE_TIMEOUT_MS
+                                ) -> Tuple[str, int]:
+    """`advance_to_next_page`, with one paced retry of a turn that failed.
+
+    Returns (outcome, retries made). `wait_seconds` 0 disables the retry.
+    `before_retry` is called before the pause and again before the second
+    press; the engines pass the proxy-budget check, which raises once the
+    run's budget is spent, so a spent budget never buys a retry.
+
+    The second attempt compares against the card that was first BEFORE the
+    original press: a grid that arrived late, during the pause, is a turn
+    and is not pressed again (a second press would skip a page). A next
+    button that is gone after the failed turn is NOT the end of the
+    listing here — it reads as another failed turn, never as NO_BUTTON.
+    """
+    before_sku = _first_sku(first_card_href)
+    if not press_next():
+        return NO_BUTTON, 0
+    turn = _await_turnover(first_card_href, count, sleep, before_sku,
+                           timeout_ms)
+    if turn == ADVANCED or not wait_seconds or wait_seconds <= 0:
+        return turn, 0
+    before_retry()
+    pause = throttle_pause_s(wait_seconds, uniform)
+    seen = throttled()
+    logger.warning(
+        "The page turn did not come back within %.0fs%s. Waiting %.0fs in "
+        "the same browser session and exit, then pressing next once more "
+        "(--throttle-wait-seconds %g; 0 disables this retry).",
+        timeout_ms / 1000,
+        f" ({seen} /graphql response(s) were HTTP 429)" if seen else "",
+        pause, wait_seconds)
+    sleep(int(pause * 1000))
+    before_retry()
+    if _turned_over(first_card_href, count, before_sku):
+        logger.info("The next page arrived during the pause; not pressing "
+                    "again.")
+        return ADVANCED, THROTTLE_RETRIES_PER_TURN
+    if not press_next():
+        logger.info("The next button is not on the page after the failed "
+                    "turn; the turn stays failed.")
+        return NO_TURNOVER, THROTTLE_RETRIES_PER_TURN
+    turn = _await_turnover(first_card_href, count, sleep, before_sku,
+                           timeout_ms)
+    return turn, THROTTLE_RETRIES_PER_TURN
+
+
 def page_cap_reached(page_num: int) -> bool:
     return page_num >= PAGE_CAP