Scraper API on the shared output contract; --url keeps its market; docs and packaging

.dockerignore

@@ -10,6 +10,12 @@ __pycache__/
 .fingerprint_cache/
 .venv/
 venv/
+# One virtualenv per engine is what the README recommends, under any name;
+# and captures/ is test material the image never copies. Neither belongs in
+# the build context: ~240 MB of it was sent to the daemon on every build.
+.venv-*/
+venv-*/
+captures/
 env/
 googleplay_apps.*
 *_debug.html

CHANGELOG.md

@@ -98,9 +98,35 @@ discovered from a bill or a diff.
   while a previous session on the same profile closes. Other refusals —
   401, a closed port, a timeout — fail at once.
 
+- **A market in `--url` is kept.** `--gl` and `--hl` defaulted to `US` and
+  `en` and were then written over the URL, so `--url "…/search?q=vpn&hl=ja&gl=JP"`
+  fetched the US store — USD prices, English titles, rows labelled US — with
+  nothing said. The URL's own `gl`/`hl` now win over the defaults; a typed
+  flag still replaces them, and says so. Found while comparing the Scraper
+  API with Playwright: after the fix the two agreed on all 30 apps of that
+  search with no field different.
+- **The Scraper API path follows the same output contract.** It writes a
+  `.meta.json` (so `diff_runs.py` can use its output), fills `page` and
+  `position`, defaults to `--format json` like the browser engines, and
+  normalises `--out` the same way. It also pins `hl`/`gl` in the request:
+  without them Play chose the market from the API's own exit while the rows
+  were labelled with a default. The "not run against this site yet" notes
+  are replaced by what was measured.
+- `diff_runs.py` ends a refusal with advice for the actual reason — the same
+  `--gl`, the same `--mode`, re-run the incomplete side — instead of "re-run
+  the incomplete side" whatever the reason.
+- `.dockerignore` leaves out `.venv-*/`, `venv-*/` and `captures/`: with one
+  virtualenv per engine, as the README recommends, ~240 MB was sent to the
+  build. The image itself copies files by name and is unchanged.
+
 ### Changed
 
 - CI only: the canary now runs weekly (Mondays 06:17 UTC) instead of daily, and on a manual dispatch.
+- README: why `original_price` and `discount_pct` are null on every row (0
+  of 411 rows from the captures, 0 of 877 from live runs).
+- README: the Scraper API is in the engine table.
+- README: pyppeteer on Apple Silicon — its Chromium download has no arm64
+  entry; use it through `--cdp-endpoint` there, or use Playwright.
 
 ## [0.2.0] — 2026-10-15
 

README.md

@@ -206,7 +206,8 @@ package ids were identical.
 |---|---|---|
 | **Playwright** (primary) | `requirements-playwright.txt` | the one to use |
 | Selenium | `requirements-selenium.txt` | no authenticated remote CDP endpoint (`--cdp-endpoint` is refused, exit 2); `--proxy-server` cannot authenticate at all, so credentials are stripped and a warning printed; exposes no HTTP status, so classification leans on the structural signals (Play's own 404 template is recognised from its body); `--fingerprint` is refused |
-| pyppeteer | `requirements-puppeteer.txt` | effectively unmaintained upstream; here for parity; `--fingerprint` is refused |
+| pyppeteer | `requirements-puppeteer.txt` | effectively unmaintained upstream; here for parity; `--fingerprint` is refused. On Apple Silicon Macs, use it through `--cdp-endpoint` or use Playwright: the Chromium pyppeteer downloads comes from its `Mac` (x86_64) snapshot channel — it has no Apple Silicon entry — and was reported to crash on macOS arm64 |
+| Scraper API | `scraper_api_client.py --url …` | no local browser: the 2Captcha Scraper API fetches one page per request (measured 2026-10-22 at $0.0003 or less each). Same rows, `.meta.json` and exit codes as the browser engines; a grid's further shelves and later review pages are the browser engines' job. Which apps fill a category's shelves depends on the requesting address, so a grid can differ from a local run (82 of 96 apps shared in one comparison) while a search came back identical |
 | Scraping Browser API | `--cdp-endpoint` | a managed browser; never stack `--proxy` or `--fingerprint` on it. Playwright and pyppeteer open their own page on it and leave the browser running when the run ends; a handshake refused with HTTP 500/502/503 is retried twice (3 s, 6 s), anything else fails at once with exit 5 |
 
 **Install exactly one per virtualenv.** Playwright and pyppeteer declare
@@ -229,6 +230,12 @@ every repo in this family; Play's own follow. See
 Exit codes: `0` ok · `1` crash · `2` bad usage · `3` blocked · `4` zero
 apps · `5` **the content was never obtained** · `6` partial.
 
+`original_price` and `discount_pct` are part of the family prefix and are
+null on every row: 0 of 411 rows parsed from the captures and 0 of 877 rows
+from live runs on 2026-10-22. Play does show a struck-through price on some
+discounted paid apps; this repository has no capture of one and does not read
+a was-price, so the columns stay empty rather than guess.
+
 `5` and `4` are the distinction worth knowing: a run that never got its page
 must not report "the catalogue is empty", or a dead proxy and an empty
 category look the same to whatever reads the file. A run that gathered
@@ -265,6 +272,9 @@ many cells differ.
   2026-10-22) gets Play's own HTTP 400 page, which this scraper reports as a
   usage error naming the flag — with no retries and no proxy advice. `ZZ`, by
   contrast, is answered with a full page.
+* **A market in `--url` is kept.** `--url "…/search?q=vpn&hl=ja&gl=JP"`
+  fetches Japan; `--gl`/`--hl` replace it only when typed, and say so. Up to
+  0.2.0 the defaults (`US`, `en`) silently replaced it.
 * **`--out` is a prefix.** `--out data/run.json` writes `data/run.json`,
   `data/run.csv` and `data/run.meta.json`, creating `data/` if needed.
 * **`genre_id` is null on every listing row, and that is the store.** A grid

diff_runs.py

@@ -223,9 +223,9 @@ def _print_summary(result: dict) -> None:
 def _run_status(path: str) -> Tuple[Optional[str], Optional[dict]]:
     """Read the `<out>.meta.json` sidecar beside a run's JSON output.
 
-    Returns (status, meta), or (None, None) when there is no sidecar — which
-    is the normal case for output written before run metadata existed, or by
-    `scraper_api_client.py` (single fetch, no pagination to cut short).
+    Returns (status, meta), or (None, None) when there is no sidecar — the
+    case for output written before run metadata existed. Every engine
+    writes one now, `scraper_api_client.py` included.
     """
     meta_path = re.sub(r"\.json$", "", path) + ".meta.json"
     try:
@@ -247,6 +247,16 @@ def _check_comparable(args) -> bool:
     this is a refusal with a --force escape hatch rather than a hard error.
     """
     problems = []
+    # What would make the two runs comparable, one line per KIND of problem,
+    # in the order found. The closing advice used to be "re-run the
+    # incomplete side" whatever the reason, which is wrong for two complete
+    # runs of different markets and for a reviews run.
+    advice = []
+
+    def advise(line):
+        if line not in advice:
+            advice.append(line)
+
     modes = {}
     for label, path in (("--old", args.old), ("--new", args.new)):
         status, meta = _run_status(path)
@@ -265,16 +275,20 @@ def _check_comparable(args) -> bool:
                 f"{label} ({path}) is a {mode!r} run, which is not one row "
                 f"per sku. This tool diffs one row per sku on price, so there "
                 f"is nothing here it can compare.")
+            advise("compare reviews runs on review_id yourself; this tool "
+                   "diffs app rows")
         if status != "complete":
             problems.append(
                 f"{label} ({path}) was a {status!r} run — stopped after "
                 f"{meta.get('pages_completed')} of {meta.get('pages_requested')} "
                 f"page(s), reason {meta.get('stop_reason')!r}")
+            advise("re-run the incomplete side")
     if len(set(modes.values())) > 1:
         problems.append(
             f"the two runs are different modes ({modes}). A listing row and a "
             f"detail row carry different fields, so `added`/`removed` would "
             f"describe the mode change rather than the catalogue.")
+        advise("diff two runs of the same --mode")
 
     # A CURRENCY MISMATCH. Google Play prices per market: the README's
     # table shows one app quoted in USD, EUR, JPY, INR and BRL from one
@@ -296,13 +310,15 @@ def _check_comparable(args) -> bool:
                 f"{label} ({path}) holds more than one currency ({sorted(seen)}) "
                 f"— that run was redirected mid-way and its own prices are not "
                 f"comparable with each other, let alone with another run's.")
+            advise("re-run that side")
     if len(set(currencies.values())) > 1:
         problems.append(
             f"the two runs quote different currencies ({currencies}). "
             f"Google Play prices per market, so these are most likely runs "
             f"of different --gl values, and their prices and ratings are not "
             f"comparable. Diff two runs of the same --gl. `source` cannot "
             f"catch it: it is 'play.google.com' on both sides.")
+        advise("diff two runs of the same --gl")
 
     # A MARKET mismatch, read from the rows' own `gl`. The currency above
     # cannot tell two markets that share one apart — DE and FR are both EUR —
@@ -323,6 +339,7 @@ def _check_comparable(args) -> bool:
             f"the two runs describe different markets ({markets}). Ratings and "
             f"prices are per --gl, so every difference would be the market "
             f"rather than the store moving. Diff two runs of the same --gl.")
+        advise("diff two runs of the same --gl")
 
     if not problems:
         return True
@@ -334,9 +351,9 @@ def _check_comparable(args) -> bool:
     print("[!] Refusing to diff these two runs:")
     for line in problems:
         print(f"      {line}")
-    print("    Re-run the incomplete side, or pass --force to compare anyway "
-          "(added/removed will include products that were simply never "
-          "fetched).")
+    print("    To compare them: " + "; ".join(advice) + ".")
+    print("    --force compares anyway, and then the differences describe the "
+          "mismatch above rather than the store.")
     return False
 
 

engine_core.py

@@ -48,6 +48,7 @@ def close() -> None
 import re
 import sys
 import time
+import urllib.parse
 from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple
 
 import captcha_solver
@@ -127,11 +128,14 @@ def build_parser() -> argparse.ArgumentParser:
     ap.add_argument("--query", help="a search query (--mode search)")
     ap.add_argument("--app-id", help="an Android package id, e.g. com.whatsapp")
     ap.add_argument("--developer", help="a developer id or name")
-    ap.add_argument("--gl", default="US", type=country_code,
+    ap.add_argument("--gl", default=None, type=country_code,
                     help="the MARKET, as a two-letter country code. Changes "
                          "the rating, the currency and which apps are listed. "
-                         "A code Play does not serve is exit 2. Default US.")
-    ap.add_argument("--hl", default="en", help="the display language. Default en.")
+                         "A code Play does not serve is exit 2. Default: "
+                         "the gl in --url, else US.")
+    ap.add_argument("--hl", default=None,
+                    help="the display language. Default: the hl in --url, "
+                         "else en.")
     ap.add_argument("--pages", type=positive_int, default=1,
                     help="how many fetches to make. On a grid this follows "
                          "the page's own shelves; on reviews it follows the "
@@ -205,12 +209,42 @@ def build_parser() -> argparse.ArgumentParser:
     return ap
 
 
+DEFAULT_HL, DEFAULT_GL = "en", "US"
+
+
+def _settle_market(args: argparse.Namespace) -> None:
+    """The market of a `--url` run: the URL's own, unless a flag was typed.
+
+    `--gl` and `--hl` used to DEFAULT to US and en and were then written over
+    the URL, so `--url ".../search?q=vpn&hl=ja&gl=JP"` was fetched as
+    `hl=en&gl=US` — measured 2026-10-22: USD prices and English titles, rows
+    labelled US — with nothing said. A market in the URL is something the
+    caller typed and outranks a default. A flag the caller typed outranks the
+    URL, and says so.
+    """
+    query = dict(urllib.parse.parse_qsl(urllib.parse.urlsplit(args.url).query))
+    for flag, attr, default in (("--gl", "gl", DEFAULT_GL), ("--hl", "hl", DEFAULT_HL)):
+        in_url = query.get(attr)
+        if attr == "gl" and in_url:
+            in_url = in_url.upper()
+        typed = getattr(args, attr)
+        if typed and in_url and typed != in_url:
+            print(f"[info] {flag} {typed} replaces {attr}={in_url} in --url",
+                  file=sys.stderr)
+        setattr(args, attr, typed or in_url or default)
+
+
 def resolve_target(args: argparse.Namespace) -> Tuple[Optional[str], Optional[str], Optional[str]]:
-    """`(mode, url, error)` — what this invocation is actually asking for."""
+    """`(mode, url, error)` — what this invocation is actually asking for.
+
+    Also settles the MARKET, and writes it back to `args.gl` / `args.hl` so the
+    request, the rows and every message agree on it.
+    """
     if args.url:
         reason = parser.refusal_reason(args.url)
         if reason:
             return None, None, reason
+        _settle_market(args)
         mode = parser.mode_for_url(args.url)
         # An app page is the address of BOTH app-level modes: `--mode app`
         # reads the page, `--mode reviews` pages that app's reviews. Refusing
@@ -226,6 +260,7 @@ def resolve_target(args: argparse.Namespace) -> Tuple[Optional[str], Optional[st
                 f"{mode} URL")
         return mode, parser.with_market(args.url, args.hl, args.gl), None
 
+    args.hl, args.gl = args.hl or DEFAULT_HL, args.gl or DEFAULT_GL
     if args.mode == "listing":
         if not args.category:
             return None, None, "--mode listing needs --category (e.g. GAME) or --url"

product_parser.py

@@ -1513,7 +1513,11 @@ def parse_products(markup: str, url: str = "", category: Optional[str] = None,
     """
     mode = mode_for_url(url) if url else None
     if mode == "app":
-        row = parse_detail(markup, package_from_url(url), hl=hl, gl=gl, page=page)
+        # Position 1, as the browser engines' app run gives it: one page is
+        # one app. Left unset here, an app row from the Scraper API came back
+        # with `position` null beside the same row's `page` 1.
+        row = parse_detail(markup, package_from_url(url), hl=hl, gl=gl, page=page,
+                           position=1 if page is not None else None)
         rows = [row] if row is not None else []
     else:
         rows = parse_grid(markup, hl=hl, gl=gl, page=page)