Skip to content

fix(dev): bound the thermal freeze and poll fast under the guard - #89

Merged
Dreamcoder08 merged 5 commits into
mainfrom
fix/safe-run-liveness
Sep 29, 2026
Merged

Dreamcoder08 merged 5 commits into
mainfrom
fix/safe-run-liveness

Conversation

@Dreamcoder08

Copy link
Copy Markdown
Owner

Closes #61. Hardens scripts/safe-run.sh (the dev laptop's thermal guard), calibrated with real-heat measurements.

Behavior

  • Hard ceiling SAFE_TEMP_HARD (85 °C) freezes at once, even in a grace window; after SAFE_MAX_FREEZE (180 s) a freeze is forced open with a SAFE_GRACE (60 s) window unless the reading fell ≥ 2 °C (then it is load-related and no grace is granted) — a stuck-hot sensor can no longer stall a run indefinitely
  • SAFE_POLL (0.25 s), wall-clock timing independent of the poll rate; rise guard = ≥ 8 °C within ~1 s
  • SAFE_CPU default 100%; thaw failures warn once; thresholds validated (exit 2); unreadable zones skipped; thermal glob documented and paths with spaces handled
  • Helpers in scripts/lib/safe-run-lib.sh (safe-run.sh 130 lines)

Evidence

  • Unit: 170/170 (11 safe-run tests: stuck-hot progress at 1 s and 0.1 s polls, hard ceiling stays frozen, fast-rise early pause, failing thaw warns once, six invalid-setting cases, signals leave no scope)
  • Real heat (two busy loops for 30–45 s under the guard, sampled 1 s and 0.25 s): the only sensor (acpitz, critical trip 103 °C) jumps from ~65 to 92–95 °C within one sample when the CPU boosts and falls back to 70 °C in ~10 s once frozen. Peak 95 °C (old settings) / 92 °C (new); never near 103 °C, 3 pause cycles each run, no orphaned scope afterwards
  • Honest limit: the guard reacts after the spike; the ~10 °C margin to the critical trip is the hardware's (cooling), not something software can widen. Disabling CPU boost (root) would; that is left to the owner

Reviewed follow-ups from #85/#86 are unrelated to this PR.

After SAFE_MAX_FREEZE, a zone whose reading never fell by 2 C gets a SAFE_GRACE window in which only SAFE_TEMP_HARD re-freezes, so a stuck-hot sensor can no longer pause a run indefinitely. At or above the hard ceiling there is no forced thaw. A rise of 8 C within three polls above RESUME pauses early. A failing thaw warns once and never claims to resume. New settings are validated (exit 2), and the thermal read and validation move to scripts/lib/safe-run-lib.sh to keep the script within budget.
…uck hot

Covers the stuck-hot grace, no grace for a load-related zone, the hard ceiling staying frozen, the early pause on a fast rise and a single warning on thaw failure, all against fake zone files.
A real-heat check measured the sensor jumping from 67 C to 95 C in under a second (critical trip 103 C), so a 1 s poll left only ~8 C of margin. The watchdog now polls every SAFE_POLL seconds (default 0.25) and measures the freeze cap, the grace and the 8 C rise window on the wall clock, so they mean the same at any poll rate. SAFE_CPU defaults to 100% (one core-equivalent) and SAFE_TEMP_HARD to 85000. Zones are read with the read builtin to avoid a fork per zone per poll. Tests pin SAFE_POLL=1, validate SAFE_POLL, and rerun the stuck-hot progress case at 0.1 s.
…rd ceiling

A zone that stays at or above SAFE_TEMP_HARD never thaws, which is right for safety but hung the caller forever on a stuck sensor. After SAFE_HARD_ABORT seconds (default 600) frozen at or above the ceiling, the scope is killed without ever being thawed and safe-run exits 75 (EX_TEMPFAIL). Behaviour below the ceiling is unchanged. The rise window, rise delta and fall check are now named readonly constants.
@Dreamcoder08
Dreamcoder08 merged commit 8958dca into main Sep 29, 2026
1 check passed
@Dreamcoder08
Dreamcoder08 deleted the fix/safe-run-liveness branch September 29, 2026 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

safe-run.sh: bound the max-freeze escape hatch and review nits

1 participant