Feedback from a CSS-notation parity run (0.13.0): rerun reuses unstable shots, one-sided asset retry, reach-first skill path, on-demand baseline #129

From a CSS parity run on renderdiff 0.13.0 on metabase.com. Marketing had 208 routes × 5 widths (1040 shots) and docs had 49 × 5 (245), both with --computed, comparing a base build against a change build. The change was a stylelint --fix + Prettier pass over ~140 style files: property reordering, rgb() notation, merged shorthands, ::before, and width >= media ranges. The result was parity, 0 differing px on both apps. Items are ranked by time lost.

• --rerun-failed reruns zero-diff shots that were only slightly unstable. reusable() (src/run.ts) requires unstablePixels === 0, so anti-aliased corners (16–55 px, 0 diff px) count as failed. Clearing 6 differing shots reran 121, which took 5.5 min instead of ~1. Reuse a shot with diffPixels === 0 whose unstable px stay under a small cap, or reuse every unstable-only shot unless a flag like --rerun-unstable is passed. Related: Feedback from a 1,285-shot parity run (0.5.0): computed origin noise, progress, base-vs-base, sharing reports #59 (folding tiny unstable rows into clean).

• A stylesheet that fails on one side isn't retried within the run. Under load, fonts.googleapis.com timed out on the left side for 3 shots (failed left: stylesheet …, fonts -"Lato", 87–91% with +1045 px height). Each one cost a separate --rerun-failed pass. shoot() already retries transient exceptions once. It would help to also retry a shot once when a stylesheet or font failed on only one side, and to report it only if it fails again.

• The skill contradicts itself for CSS-only changes. The mode table says "shoot the routes the changed rules reach", but Loop steps 3 and 5 say "run all routes" and "Finish with a fresh full run". It also has no case for "same meaning, different notation": a build diff where every changed rule is a reordering or notation change, so parity is proven by the build diff and shots add nothing. Two things would have saved a full run:

  • A note that Astro (and similar frameworks) inline <style> blocks and style="" into every page, so "HTML differs" doesn't mean the markup changed.
  • A by-meaning CSS diff, either as a recipe or as renderdiff reach <base dir> <change dir>. It would pair renamed hashed assets, normalise notation (rgb() commas/slash, hex case and short form, currentColor, 0px, min-width: ↔ width >=), and flag only rules whose declaration set changed or whose overlapping shorthand/longhand pair swapped order. A 70-line postcss script did this here in 4 s over 1,548 pages.

  Related: Skill: for CSS-only changes, diff the shipped CSS first to find the reach #102, Skill: for CSS-only changes, diff the build outputs first #110, Feedback from a CSS-only docs run (0.12.0): rule coverage, PR crops of clean shots, hidden subtrees in dom, results.json fields #121.

• Base vs base on demand, only where the change run differs. The full marketing base-vs-base run took 23.6 min and found nothing that recheck (varies) didn't already paint out in the change run. Run the change first, then reload the right origin twice only for the routes that differed, e.g. --baseline after a run, or a mode where --rerun-failed shoots right vs right. Related: Feedback from a 1,285-shot parity run (0.5.0): computed origin noise, progress, base-vs-base, sharing reports #59, Recheck differing shots for load-to-load variation #47.

• Right-side shots can't be reused across runs. --cache-right replays network (HAR), but both clocks start at the run's own minute, so a time-dependent page renders differently each run. A pinned --clock <iso> would let repeated change runs reuse the reference shots.

• One more example for Feedback from a CSS-only docs run (0.12.0): rule coverage, PR crops of clean shots, hidden subtrees in dom, results.json fields #121's rule-coverage line. The only rule whose declarations changed (a chat bubble's border-radius) exists only after a JS interaction. The 1040-shot run passed without ever rendering it, and a hooked route was needed to check it. A per-stylesheet coverage line would have flagged a changed rule that matched 0 elements.