From 87328ad4d60491efd85f6c91f68916df08ec2bb4 Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Fri, 16 Oct 2026 11:18:45 +0000 Subject: [PATCH] Run the paid canary weekly instead of daily Paid canary paths (proxy, Scraping Browser, captcha) run weekly on Mondays; free paths keep their daily schedule. A manual dispatch runs everything. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8 --- .github/workflows/canary.yml | 2 +- CHANGELOG.md | 6 ++++++ CONTRIBUTING.md | 2 +- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/.github/workflows/canary.yml b/.github/workflows/canary.yml index bcad7db..46968ae 100644 --- a/.github/workflows/canary.yml +++ b/.github/workflows/canary.yml @@ -2,7 +2,7 @@ name: canary on: schedule: - - cron: "41 6 * * *" # once daily; avoid :00 so it doesn't pile up with everyone else's + - cron: "41 6 * * 1" # weekly, Mondays (every run bills a Scraping Browser session); avoid :00 so it doesn't pile up with everyone else's workflow_dispatch: {} # dispatch manually at least once before trusting the badge jobs: diff --git a/CHANGELOG.md b/CHANGELOG.md index bc46de5..6191565 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,12 @@ rather than being a silent violation of that. ## [Unreleased] +### Changed + +- CI only: the canary bills a Scraping Browser session (`PERPLEXITY_CDP_ENDPOINT`), + so it now runs weekly (Mondays 06:41 UTC) instead of daily; a manual dispatch + still runs it. + ## [0.2.0] — 2026-10-16 > **Behaviour changes.** `--retries`, `--retry-delay`, diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index da66d80..9a3e53b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,6 +4,6 @@ 2. Run `python3 smoke_test.py` (or `pytest tests/test_smoke.py`). It is offline and must pass with **no** engine installed; install one engine (`pip install -r requirements-playwright.txt`, or `-selenium` / `-puppeteer`) to exercise its code too. 3. **If you touched `page_parser.py`**: every parser check runs on real captures in `tests/fixtures/` (the `/rest/article/` API for a classic Page and a Discover article, the Discover feed, and a Cloudflare challenge page). If the site changed shape, save a scrubbed capture of what it serves now next to them, update the parser to match, and add a `smoke_test.py` check against the new fixture. Don't change the parser on a guess without a capture behind it. 4. Keep the three engines behaving identically: same exit codes, same `Product` schema, same CLI flags. The fetch logic itself lives once in `page_flow.fetch_article()`; an engine only supplies its driver's own page operations. If one engine has to diverge (see `selenium_scraper.py`'s CDP-credential limitation), say why in a comment. -5. Open a PR. CI runs the offline suite on two Python versions, builds the wheel and the Docker image, and runs one `engine-smoke` job per engine in its own virtualenv. A daily `canary` job scrapes the live site over the Scraping Browser API when the repo has the `PERPLEXITY_CDP_ENDPOINT` secret (see `TESTING.md`). +5. Open a PR. CI runs the offline suite on two Python versions, builds the wheel and the Docker image, and runs one `engine-smoke` job per engine in its own virtualenv. A weekly (Mondays) `canary` job scrapes the live site over the Scraping Browser API when the repo has the `PERPLEXITY_CDP_ENDPOINT` secret (see `TESTING.md`). Bug reports and feature requests: open an issue. Please include the exact command you ran (without credentials) and the `.meta.json` sidecar from the run — or say that none was written, which is itself informative (see `output_writer.finish_run`). If an article came back blocked or empty, a `--dump-html` capture is the most useful thing you can attach.