Skip to content

Run robustness from the #59 feedback: heartbeat, mid-run origin abort, CSP-proof CSS, selector-aware ceiling - #96

Merged
tmashara merged 2 commits into
mainfrom
claude/issue-59-run-robustness
Oct 30, 2026
Merged

tmashara merged 2 commits into
mainfrom
claude/issue-59-run-robustness

Conversation

@tmashara

Copy link
Copy Markdown
Owner

Part of #59. This PR takes the remaining small, root-cause items from that feedback. The rest is either already fixed on main or should be split out (lists at the end).

Changes

  • Heartbeat. When output is piped, stderr gets renderdiff: n/N shots every tenth of the run, plus one at the end. Stdout's compact summary is unchanged. SKILL.md now says to run long runs in the background and watch these lines. --help, README and reference.md are updated.
  • Origin dies mid-run. When a shot errors with ERR_CONNECTION_REFUSED, renderdiff checks both origins again with the preflight's alive(), which moved to config.ts so cli and run share it.
    • If one is down, a shared down flag stops every worker's loop. A queue.push retry can't revive the run, because the loop checks the flag, not the queue.
    • The browser still closes, and renderdiff exits 2 with ✗ right (<url>) stopped responding mid-run.
    • This happens before saveRecording, so no partial --cache-right HAR is saved, and before any report is written.
    • A refused third-party URL can't trigger it.
  • CSP.
    • Root cause: Playwright's addStyleTag runs inside _raceWithCSPError, so it rejects on any "Content Security Policy" console error logged while it injects. A site's own blocked analytics script is enough (docs/case-study.md); that's why these errors moved between routes and widths from run to run. A <style> is also subject to style-src.
    • Fix: renderdiff's CSS is now added as a constructed stylesheet in document.adoptedStyleSheets, through one helper used at both injection sites. CSP doesn't govern constructed sheets, and adopted sheets sort after the document's own.
    • One side effect: an @import in extraCss is ignored. That's noted on the option.
    • The case-study bullet now gives the cause and the fix.
  • Height ceiling. The ceiling now tests the area actually shot (clip ?? page), so --selector or @sel on a very long page compares that element. An element that is itself past the ceiling is still refused (element is …).
  • get.geojs.io is now in the default blocklist. It's a geo-IP lookup that decides whether a consent banner shows.
  • results.json docs (reference.md and docs/output.md) say the per-line results sit inside one JSON document, with trailing commas.

Tests

  • guardrails:
    • New fixture /csp: both sides send style-src 'self' and are styled only via /shared/base.css, and a consent banner is on the left only. The shot must compare with 0 diff pixels, which shows the hide selectors applied.
    • An inline right origin that dies after its first page: the run must exit 2, name the right side, and write no report.
    • /long @h1 at 1440 must compare.
    • The /hang fixture now wedges on the adoptedStyleSheets setter instead of a <style> mutation.
  • controls: the piped run's stderr must end with renderdiff: 25/25 shots (the total), with 2–10 heartbeat lines.
  • With the capture.ts change stashed, the CSP and selector-ceiling tests fail.

Verification: npm run typecheck, npm run lint and npm run format:check are clean. npm test passes 357/357 and npm run test:visual passes 389/389, both on macOS.

Already fixed on main since 0.5.0

Left for separate issues

  • A route's before chaining the global one (changes documented semantics).
  • CLI routes reusing a config route's hooks, or warning.
  • A route name or state field.
  • waitFor and pinned iframe heights.
  • A bounding box for unstable pixels, and folding small unstable counts.
  • report --compact, WebP and gh-pages.
  • --open over HTTP; renderdiff serve.
  • --routes-file.
  • --rerun-failed across config changes.
  • A "lower --concurrency" hint on timeouts.

🤖 Generated with Claude Code

tmashara and others added 2 commits October 30, 2026 10:02
- Piped runs print `renderdiff: n/N shots` to stderr every tenth of the run.
- An origin that stops responding mid-run aborts with exit 2, before any
  report or --cache-right recording is written.
- Inject renderdiff's CSS as an adopted constructed stylesheet: addStyleTag
  rejects on any CSP violation logged during injection, and a <style> is
  subject to style-src.
- The area ceiling tests what is shot, so a selector on a very long page compares.
- Block get.geojs.io by default; say results.json is one JSON document.

Part of #59.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@tmashara
tmashara merged commit 46f3525 into main Oct 30, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant