Skip to content

Say when the simulator restarted under a failed iOS test step (#508) - #514

Merged
ramesh130 merged 1 commit into
mainfrom
issue-508-ios-stale-live-sigterm
Oct 13, 2026
Merged

ramesh130 merged 1 commit into
mainfrom
issue-508-ios-stale-live-sigterm

Conversation

@ramesh130

Copy link
Copy Markdown
Owner

Closes #508

The cause: the simulator restarted under the runner, not a hang

The SIGTERM came from outside the test. The simulator restarted about five seconds into an eight-second test. What restarted it is not known.

The original result bundle is gone. ios/check deletes .build/ios-check on every run, and #430's worktree has been removed. The evidence below is from the host's logs for that run (2026-10-12, 17:15–17:27 local, simulator sp-430). Those logs are still on disk.

  • The simulator booted again mid-run. Its own system.log (~/Library/Logs/CoreSimulator/<udid>/system.log) shows a new syslogd at 17:24:17. The run's only other boots were xcodebuild's own: one at the start, and later runs.
  • Everything in the simulator ended together:
    • 17:24:16.43: coreaudiod released all of the simulator's audio contexts, including those made at its boot.
    • 17:24:16.51: the Resilience target's xctest (pid 79037) ended.
    • 17:24:16.66: the simulator's SimRenderServer and SimMetalHost died. Apart from the final shutdown, these are the only such deaths in the run.
  • The test was not hanging. The Resilience process started at 17:23:05. In a passing run, this test starts about 66 s into the target and ends at about 74 s. So the process died about 5 s into an 8 s test.
  • No Claude session ran simctl at that time. I searched every session transcript on the host for simctl, xcodebuild and ios/check between 05:30 and 06:30 UTC.
  • iOS i5: the fault sweep, the rung register and stale-live-playlist detection #421's earlier ProbeTests run has the same fingerprint. It crashed "with signal term" at 00:41:00 on 2026-10-12, and its simulator booted again at 00:40:56.

Why it cannot be a hang

  • Nothing kills a hung test here. Neither ios/check nor a scheme or test plan sets -test-timeouts-enabled or a time allowance, and xcodebuild leaves them off by default. A probe test that slept 418 s in one-second steps passed.
  • The test's own waits are bounded. Both are harness.until(timeout: 90 s). A wait that never ends would therefore fail as a ConditionTimeout, never as a SIGTERM.

Reproduction

There is no race in this test, so there is no racing side to make lose. A held response or an extra settle makes the test slower, not killed, for the reason above. What I reproduced instead is the external cause, injected deterministically:

  • Tearing the simulator down mid-test. xcrun simctl spawn <udid> launchctl reboot userspace, 3 s after the test starts, reports exactly Test crashed with signal term. on this test. It did so in both runs: the test alone, and its class.
    • One difference from 17:24. This teardown left the device shut down, with the class's next test failing to launch. In the original, the device booted again within a second and the run went on.
  • Killing the runner. kill -TERM on the xctest process gives the same text. That shows how unspecific the text is: any SIGTERM to the runner reads as this crash.

What changed

No test code changed. The test was not defective, and no change to it can survive the device restarting under it. Nothing is retried or masked, and no assertion was touched.

A diagnostic in ios/check, for whoever investigates the next one.

  • What it does. After a failed test step (iOS, tvOS, ios/MoQ), it counts the device's boots, as the distinct syslogd pids in that system.log. If the device booted more than xcodebuild's own one boot, it says the simulator restarted during the step.
  • What it does not do. It never changes pass/fail: a restart still fails the step.
  • How it was checked.
    • Driven by hand on sp-508: it was silent after the step's own boot and printed after a second boot.
    • It survives a device with no log yet under set -euo pipefail.
    • This detector would have named the 17:24 restart.
  • Its one stated gap. A restarted syslogd that is given a pid an earlier boot had would be missed.

ios/CLAUDE.md gains a bullet: "crashed with signal term" can be the simulator restarting, and is never a hung test.

Acceptance criteria

  • Cause identified, with evidence: met. It is external: a simulator restart whose initiator is unknown.
  • Reproduced deterministically by making the racing side lose: answered by argument rather than literally. There is no racing side, so the external cause was injected instead.
  • Holds whatever the host speed, weakens nothing:
    • "Weakens nothing" is met.
    • "Host speed" does not apply: the failure does not depend on host speed.
  • Removed from the known-flake list: that list lives in the batch prompts, so the coordinator drops it. What replaces it is general: any test can be ended by a simulator restart, and ios/check now says when one happened.
  • 20 iterations and ios/check: met.
    • xcodebuild test -test-iterations 20 -only-testing:SuperPlayerResilienceTests/StaleLivePlaylistPlaybackTests/testAPlaylistHeldByACacheEndsOnTheTypedErrorNamingTheIntermediary passed 20/20 on sp-508.
    • The whole class passed.
    • SP_SIMULATOR=sp-508 ios/check passed before the review fixes and on the final commit: 708 tests, 12 on tvOS, 38 in ios/MoQ.
    • Two runs between those failed on tests this change cannot reach, with the host's load average near 56 from other batch workers: Flake (iOS): FaultScriptPlaybackTests testALatencyHolds… fails under load #506's testALatencyHolds…, and NetworkShapingTests/testACapUnderATraceGovernsWhenItIsTheSlower (3.73 s against a 3.5 s bound). The detector correctly printed nothing for either.

No device run: nothing here needs one.

A possible follow-up: find what restarts a simulator mid-run. It has now happened twice (#421's probe and #508).

Review summary

Standards:

  • Hard findings, fixed:
    • The ios/CLAUDE.md bullet stated one observation as a rule. It now says "can be".
    • It claimed coverage the tvOS step lacked. The tvOS step is now wired too.
  • Soft findings, applied:
  • Kept: the existing per-step failure blocks and the repeated shutdown check, which predate this change. Folding them is a refactor beyond the issue.

Spec:

  • Asked that each criterion be answered honestly, met or argued, rather than ticked. The section above does that.
  • Asked that ios/check be shown passing with the diff. It was, before and after the fixes.
  • Asked whether the reproduction matched the original fingerprint. It matches in the SIGTERM, and differs in that the device stayed down, which is said above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MJAhQmFgdRRAKUaMHibQLe

…that is what #508's "signal term" was

The stale-live-playlist test was not hanging: no time allowance is set, so nothing kills a hung test,
and its waits are bounded. The host's logs show the simulator booting again five seconds into the
test, with every process in it ending together. ios/check now counts the device's boots after a
failed step and names a restart; a restart still fails the step.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MJAhQmFgdRRAKUaMHibQLe
@ramesh130
ramesh130 merged commit 1832ef0 into main Oct 13, 2026
3 checks passed
@ramesh130
ramesh130 deleted the issue-508-ios-stale-live-sigterm branch October 13, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flake (iOS): StaleLivePlaylistPlaybackTests testAPlaylistHeldByACache… crashes with signal term

1 participant