Skip to content

Add optional per-task check_timeout_s (default stays 60 s) - #147

Open
eazene wants to merge 1 commit into
NateBJones-Projects:mainfrom
eazene:per-task-check-timeout
Open

eazene wants to merge 1 commit into
NateBJones-Projects:mainfrom
eazene:per-task-check-timeout

Conversation

@eazene

@eazene eazene commented Oct 5, 2026

Copy link
Copy Markdown

Checks that do real work were failing on the fixed 60-second check timeout, not on the work.

We run each executed check inside a container/VM launcher, so the 60 s also covers starting the sandbox and waiting
for a free slot. Multi-suite checks were taking about 54 s. One of them went over 60 s after a correct worker run:
the task failed, we re-checked it by hand, and the run was voided on our scoreboard. Workers already have a per-task
timeout_s; checks had no equivalent.

What this adds

An optional task field, check_timeout_s:

  • Value. A JSON integer from 1 to 3600. When absent, CHECK_TIMEOUT_S (60) applies exactly as today, so existing
    manifests are unchanged.
  • Validation. Anything else fails manifest loading with the task key: 0, negatives, 3601, 90.0, 2.5,
    "90", true or null. So lint, run, --baseline and --dry-run all refuse the manifest. A bool is
    rejected even though it is an int subclass.
  • Use. Verifier._run_check uses the task's effective value; the process-group termination path is unchanged.
    The timeout message names the value (check timed out after <n>s).
  • Visibility. The effective value is recorded in the run-state task record next to timeout_s and shown in the
    --dry-run listing.
  • Dataclass. TaskSpec.check_timeout_s defaults to None, and the effective_check_timeout_s property reads the
    module constant at call time, so tests (and anyone patching the default) see the current value.

The README task-field table gains one row.

Tests

tests/test_check_timeout.py (9 tests, about 4 s), stdlib unittest, with RINGER_NO_SELF_UPDATE=1 for the CLI
cases:

  • absent field → effective 60
  • 1 and 3600 accepted
  • every rejected value above, with the task key in the error
  • with the default patched to 1 s, a task with 5 s passes a 2 s check, and a task without the field times out (the
    default is read at run time)
  • check_timeout_s: 1 stops a 5 s check, and the message says 1s
  • Manifest.from_obj refuses a bad task
  • --dry-run shows check_timeout_s: 45 and check_timeout_s: 60
  • lint exits non-zero on "90"

Full suite: 278 tests OK locally on macOS with Python 3.12.

Notes

  • The cap. 3600 is there to catch typos such as an extra zero. Happy to change it, or drop it to match
    timeout_s, if you prefer.
  • Real-world use. We've carried this as a local patch on a vendored copy and used it on real runs, with checks at
    120, 180 and 240 s. If it lands, we drop the patch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F74VKWWyJNxArUTPPaiBpb

A task may set check_timeout_s, a whole number of seconds from 1 to 3600, to
replace the fixed 60 s CHECK_TIMEOUT_S for its executed check. Without the
field nothing changes. Invalid values (0, negative, above 3600, decimals,
strings such as "90", bools, null) fail manifest loading with the task key,
so lint, run, --baseline and --dry-run all refuse the manifest. The effective
value is used by Verifier._run_check, named in the timeout message, recorded
in the run-state task record next to timeout_s and shown in --dry-run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01F74VKWWyJNxArUTPPaiBpb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant