Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# and eighteen candidate bot-challenge markers counted at zero across seven
# captures.
#
# So these jobs run a real scrape every morning from a runner with no
# So these jobs run a real scrape every Monday from a runner with no
# secrets, and they are expected to be GREEN. That is not a convenience — it
# is what keeps the central claim of this repo's README honest. If Mercor
# ever puts these routes behind a challenge, a key or a rate limit, the badge
Expand All @@ -32,7 +32,7 @@ on:
schedule:
# 05:40 UTC. Not on the hour: every scheduled workflow on GitHub fires at
# :00 and the queue there is long.
- cron: "40 5 * * *"
- cron: "40 5 * * 1"
workflow_dispatch:

permissions:
Expand Down Expand Up @@ -300,7 +300,7 @@ jobs:
# page is one job detail page, and the addresses come from the
# sitemap ∪ index enumeration rather than from a guessed convention.
#
# Four pages, not four hundred. This runs daily against a site that
# Four pages, not four hundred. This runs weekly against a site that
# imposes no rate limit, and that is a reason to be careful rather
# than a licence: the check is that enumeration and multi-page fetch
# still work, which four pages proves as well as four hundred would.
Expand Down Expand Up @@ -360,7 +360,7 @@ jobs:
want("the enumeration still finds a catalogue",
isinstance(enumerated, int) and enumerated >= 100,
f"jobs_enumerated was {enumerated}")
# BOTH sources answered. This is the daily check that makes a lost
# BOTH sources answered. This is the scheduled check that makes a lost
# sitemap loud: without it a run could enumerate from the index
# alone, fetch every address it was given, and be green. Since
# v0.1.1 such a run is `partial`, which the status check above
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ it from a bill or from a diff.
> `scraper_api_client.py`.

### Changed

- CI only: the canary now runs weekly (Mondays 05:40 UTC) instead of daily, and on a manual dispatch.
- **A refused page has the same stop reason in every engine.** A 403 now
ends with `blocked_refused` in Selenium and pyppeteer, which said
`blocked_cloudflare (hard block)`; Playwright already said
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,10 +98,10 @@ Then the rest of the presentation, in the order that matters:
1. `python3 smoke_test.py` green, and the canary dispatched at least once.
Unlike most siblings in this family the canary here is **not** gated on a
secret: every route this scraper reads is served to a bare GitHub runner,
so it runs a real 3-page scrape daily and is expected to be GREEN. That
so it runs a real 3-page scrape weekly (Mondays) and is expected to be GREEN. That
is not a convenience — it is what keeps the README's central claim
honest. If Mercor ever puts these routes behind a challenge or a key, the
badge goes red the next morning and the claim is retested without anyone
badge goes red on the next weekly run and the claim is retested without anyone
having to remember to.
2. The repo description, homepage and topics set (see the family notes on
what those should say).
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -470,9 +470,9 @@ to parse **identically** to its untrimmed original before being committed.
CI runs the suite offline on the oldest and newest supported Python, builds
and runs the Docker image, and installs each engine in its own virtualenv to
check the versions users actually get. The **canary** runs a real scrape
daily from a bare GitHub runner with no secrets — which is what keeps this
weekly (Mondays) from a bare GitHub runner with no secrets — which is what keeps this
README's central claim honest. If Mercor ever puts these routes behind a
challenge or a key, the badge goes red the next morning and the claim is
challenge or a key, the badge goes red on the next weekly run and the claim is
retested without anyone having to remember to.

---
Expand Down
Loading