Skip to content

Report harness timeouts as TIMED_OUT, reply from daemon outer handler, add stall/timeout regression tests - #7

Open
davidoj wants to merge 1 commit into
mainfrom
fix/timed-out-status-and-daemon-tests
Open

Report harness timeouts as TIMED_OUT, reply from daemon outer handler, add stall/timeout regression tests#7
davidoj wants to merge 1 commit into
mainfrom
fix/timed-out-status-and-daemon-tests

Conversation

@davidoj

@davidoj davidoj commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Replaces #3 and #4, whose main fixes landed in #5. This carries only the pieces #5 did not cover, rebased on current main.

What #5 already covered

What this PR adds

1. A blown budget is TIMED_OUT, not FAILED / CRASHED

On main a parent-side "Daemon timed out" comes back as FAILED on the secure path (timeout text buried in per-test feedback) and CRASHED on the insecure path — indistinguishable from a wrong answer. #5's description says training should drop rather than zero a timed-out grade, but a caller could only do that by string-matching feedback. Both paths now return VerificationStatus.TIMED_OUT (already defined; already what the insecure verifier's own alarm reports).

Only subprocess_error is classified. That key is written by the daemon loops and the parent, never by submission code, so a submission cannot spoof a timeout via its own exception text. Per-test alarm timeouts inside a batch still score that test as failed, unchanged.

calc_reward is unaffected: it returns 1.0 iff PASSED, so TIMED_OUT scores exactly as FAILED/CRASHED did. verify_problem_consistency compares against PASSED only.

2. Daemon outer exception handler replies instead of only logging

The one piece of #4 not in main. The except Exception at the end of each daemon loop logged and sent nothing, so any unexpected exception (fork failure, pipe setup, …) cost the parent its whole budget. It now sends a best-effort Daemon error: … reply; the send is itself guarded, so a gone parent still yields only a log line.

3. Regression tests (#5 merged without any)

Insecure-path tests skip on platforms where the daemon child cannot set RLIMITs (macOS: Failed to set memory limits). Fixtures set insecure_test_cases so the test_case_leak insecure verifier actually runs the probe instead of passing on zero leaked cases.

Deliberately not carried over from #3

  • The 5s post-warmup clamp. It re-caps every steady-state grade at 5s regardless of the configured budget — the original 1s bug at a different number — and the wedged-daemon case it guarded against is mostly gone now that the stall is fixed.
  • The max(10, …) floor. Moot with a 6s per-test cap.

Testing

Platform Result
Linux, python:3.12-slim, unprivileged container 23 passed (stall + timeout + marker)
macOS, Python 3.12 21 passed, 2 skipped (insecure daemon cannot set RLIMITs)

On main, test_exceeded_budget_reports_timed_out_not_failed_secure fails with FAILED / "Daemon timed out" and the insecure analogue fails with CRASHED.

🤖 Generated with Claude Code

https://claude.ai/code/session_014rSFHpdtkVYpGAEafamz5R

…, add regression tests

A parent-side daemon timeout ("Daemon timed out") was reported as FAILED on
the secure path and CRASHED on the insecure path, indistinguishable from a
wrong answer. It now maps to VerificationStatus.TIMED_OUT on both paths so a
caller can drop (not zero) timed-out grades. Only `subprocess_error` is
classified, which submission code can never write. calc_reward is unaffected
(1.0 iff PASSED).

Both daemon loops now send a best-effort "Daemon error" reply from the outer
exception handler instead of only logging, so an unexpected exception costs
one request rather than the parent's whole budget.

Tests: test_daemon_stall.py (dead child via os._exit / SIGKILL on both paths,
runaway code still bounded) guards the #5 stall fix; test_timeout_budget.py
guards warmup-stable verdicts and the TIMED_OUT mapping. Insecure-path tests
skip on platforms where the daemon child cannot set RLIMITs (macOS).

Supersedes #3 and #4.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_014rSFHpdtkVYpGAEafamz5R
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant