[YUZHEthefool]

Problem and outcome

Finite QEMU checks currently wait for a mandatory 900-second observation window even after their guest has finished. This PR collects those checks after a verified PID 1 cleanup witness and a two-second diagnostic tail. The existing timeout becomes a maximum deadline; all final workload assertions and PASS / QUALIFIED / FAIL distinctions remain. An independent 32-vCPU SMP lane now checks the larger guest topology in regular CI and Extended.

Extended previously serialized its lanes inside one 180-minute job per OS. This PR also splits those lanes into independent jobs and scans multi-GiB stress diagnostics in bounded chunks, avoiding the reproduced whole-file reader memory failure.

Related issue or audit finding

Related: #63 (EXT-CI). Plan item: EXT-CI.

Design and scope

• qemu_run.py owns each functional QEMU process. It requires an ordered runtime summary, fixture result where applicable, consistent PID 1 exit evidence, and exactly one CI-EXIT ... stage=teardown-complete witness. After two seconds it negotiates QMP, sends quit, reaps QEMU, and records the exit status, timing, stop method and final serial-text hash.
• Boot/runtime, SMP, musl, Ring-3 and IOMMU activation use the shared supervisor. Final parsers still inspect completed logs; a completion trigger alone cannot pass a gate. Missing/duplicate records, deadline, spontaneous exit, cancellation, QMP failure and forced kill fail closed. Failure cleanup only signals the owned child process group.
• The kernel adds one PID 1 logging witness after synchronous teardown, parent notification and local guard/Arc drops. It does not change lifecycle state, locking or scheduling and does not assert global quiescence or completion of deferred callbacks.
• The 32-vCPU lane has a 3600-second guest deadline (90-minute regular CI job; 120-minute Extended job) and requires all 32 CPUs online and exactly one 31/31 AP acknowledgement. It runs once without automatic retry, shares the regular normal-build artifact, and retains separate artifacts. make test-smp-32core exposes the same gate locally. Its guest CPUs share the host runner's CPUs; this is functional coverage, not physical 32-core throughput. The original 8/16-vCPU suite retains its qualification rules.
• Regular CI defaults to completion. Manual dispatch accepts qemu_gate_mode=window for the former full observation scope, including musl. QEMU_GATE_MODE=window preserves GNU timeout status 124, one TERM and no KILL.
• Extended explicitly stays in window mode. Its 11 lanes × two Ubuntu images include the new independent 32-vCPU lane and preserve the existing lanes' CPU counts, guest durations, profiles and verdict rules; stress keeps the complete 900-second soak after READY. The 8/16-vCPU suite and stress lanes have no automatic retry. The combined local extended entrypoint remains available and applies the same single-attempt SMP policy.
• Stress diagnostics are scanned through EOF in bounded chunks, preserving split-marker, Unicode/replacement, separate-file and error-precedence behavior. A bounded-memory regression fails with the original reader and passes with the new reader.

The scope and receipt format are documented in scripts/ci/README.md.

Kernel safety and invariants

The kernel change is logging only. Completion results cover the workload through synchronous PID 1 cleanup plus the bounded tail; they do not claim the former 900-second post-completion observation coverage. Extended and explicit window mode retain that coverage. Old images without the new witness require window mode.

QUALIFIED remains distinct from PASS. No stress threshold is relaxed. The existing 4-vCPU thread lane does not qualify the 16-vCPU THREAD-HB requirement in #62.

Validation evidence

Current head: 28159ee5097fd5903eb4c9443c8af24c53f10829.

• PR85-1 audit repair: the 8/16-vCPU suite now runs once in both Extended entrypoints. Its two full 900-second windows leave 30 minutes of the 60-minute job for build, final log analysis and upload; an automatic whole-suite retry no longer consumes that headroom.
• The new failure/incomplete regression covers both entrypoints with CI_GUEST_RETRIES=1: all four subcases fail before the repair and pass after it. The focused local suite passed 15 tests, with shell syntax and whitespace checks passing.
• The user requested a 3600-second 32-vCPU timeout and merge without waiting for the new CI result. The SMP caller, shared supervisor and receipt validation now support that bound together; 8/16-vCPU defaults and limits remain 900 seconds. The regular and Extended 32-vCPU jobs allow 90 and 120 minutes for setup/build, guest execution, final analysis and upload.
• Current-head targeted local checks: 22 tests passed; one additional prerequisite test could not complete because the exported snapshot lacks Git metadata. The passing checks cover topology-dependent defaults and invalid limits, workflow headroom, existing failure propagation, and a real synthetic QMP process completing with a 3600-second receipt. The Linux-gated synthetic supervisor test was enabled only in memory on macOS; no real guest run is claimed. Shell syntax and whitespace checks passed.
• Current-head full CI has not been awaited or qualified. This merge instruction does not turn the prior failure or pending validation into a pass.
• Previous-head CI run 42361466791, head 35dcb3e: 13 jobs successful; 32-vCPU and aggregate failed. The quality job passed 244 tests, zero failures/errors/skips. The 32-vCPU receipt records a 900-second deadline, 32/32 CPUs online, no completed runtime summary or full AP acknowledgement, QMP quit and no forced kill. Its interrupt trace is 22,338,066,065 bytes uncompressed. Increasing the deadline is a requested configuration change; it has not yet demonstrated completion of that workload.

Previous-head regular CI, head 1667afc before the retry-budget repair:

• Previous-head CI run 42358985265: 15/15 jobs successful, including the aggregate; 13 artifacts uploaded. The quality job reports 243 tests, zero failures/errors/skips. Lint, shell syntax, fixture validation, actionlint, rustfmt, Clippy and REUSE passed.
• New 32-vCPU job: QUALIFIED (3), one attempt. Its downloaded artifact confirms 32/32 CPUs online, exactly one 31/31 AP acknowledgement, three R175 cross-CPU checks, and PASS for the scheduler, IPI and TLB checks. Runtime summary: 37 passed / 37 deferred / 0 failed, with one warning; this remains distinct from strict PASS.
• The 32-vCPU receipt binds to the final serial hash and records completion at 550.2 s, QMP exit at 555.6 s, status 0 and no forced kill. The gate including final log validation took 791.1 s. Its full interrupt trace is 19,863,460,256 bytes uncompressed; full-trace execution and collection remain expensive at this topology.
• No kernel scheduling, locking or lifecycle code was changed by the 32-vCPU addition. The new 32-vCPU Extended full-window lanes have not been run; regular completion coverage does not qualify that stability scope.

Historical functional validation, head f90b448 before the 32-vCPU lane:

• Local harness regression: 238 passed, 363 subtests passed. Tests exercise the actual supervisor with an owned QMP process, final-log faults during shutdown, cancellation/reaping, rejected quit, forced kill, receipt binding, and shell-gate integration. Extended mode/duration are pinned.
• make lint, make fmt-check, make clippy, normal kernel build, and whitespace checks passed.
• Real QEMU completion checks: boot, runtime and 4-vCPU SMP retain QUALIFIED; musl 1/4-vCPU checks PASS. A separate 1-vCPU Ring-3 check PASSed with 51 assertions and zero failures in 109 seconds. Every receipt records QMP quit, exit 0 and the two-second tail.
• Regular CI run 42357546969 completed successfully on f90b448: 14/14 jobs, including the aggregate; 12 artifacts uploaded. The quality job reports 238 tests, zero failures/errors/skips, and actionlint passed.
• Boot/runtime/4-vCPU SMP: QUALIFIED at 13.4 / 12.9 / 14.6 seconds (40.9 seconds of gate execution versus the former three mandatory 900-second windows; build/setup excluded). The uploaded runtime receipts were independently rechecked: valid final serial hashes, two-second tail, QMP quit, QEMU status 0 and no forced kill.
• musl 1/4-vCPU: PASS at 20.2 / 15.6 seconds. IOMMU activation: QUALIFIED at 9.3 seconds; its failure and device probes PASS.

Historical Extended acceptance, head 429d613 before the completion change:

• Regular CI: 14/14 jobs successful.
• Extended run 42348591710: 20/20 final lane verdicts and 20 artifacts, 19 QUALIFIED / 1 FAIL, no cancellation or exhausted job budget; longest job 39:32. Ubuntu 24.04 block exhausted the existing 12 writer/crash attempts without an active journal tail and did not enter recovery soak. The aggregate correctly failed.
• Full historical table and earlier failures. This record does not qualify the new head's full Extended matrix. STRESS-7 remains open.

Compatibility, performance, and rollout

Finite gates now finish when their recorded completion protocol finishes, bounded by the configured deadline (3600 seconds for 32 vCPUs; otherwise unchanged). Extended retains its full stability windows and per-lane job budgets. Actions' existing always-upload steps retain results after success or failure while the runner remains available.

Use QEMU_GATE_MODE=window or the regular workflow's manual window input to restore the former observation scope without changing the final workload assertions.

Checklist

• The change is focused and free of unrelated refactoring.
• Meaningful supervisor, parser, caller and workflow regressions are included.
• Current-head full CI has not been awaited, as explicitly requested by the user; the prior 32-vCPU timeout and historical/full-window boundaries remain explicit.
• Changed observation coverage and the historical acceptance boundary are explicit.

[YUZHEthefool]

Current-head CI and 32-vCPU SMP — 1667afc

Run 42358985265 completed successfully on 1667afcc82e57628cc50f4c27c9b4a056c6561e1: 15/15 jobs, including the aggregate, with 13 artifacts. The quality job reports 243 tests, zero failures/errors/skips; lint, shell syntax, fixture validation, actionlint, rustfmt, Clippy and license checks passed.

The 32-vCPU job returned QUALIFIED (3) in one attempt. Its downloaded ci-smp-32 artifact confirms:

• 32/32 CPUs online, and exactly one 31/31 AP acknowledgement.
• Three R175 cross-CPU checks, plus scheduler, IPI ping-pong and TLB shootdown checks PASS.
• Runtime summary: 37 passed, 37 deferred, 0 failed; one warning remains. This is not a strict PASS.
• Valid final serial hash and completion receipt: completion at 550.2 s, QMP shutdown by 555.6 s, QEMU status 0, no forced kill. Total gate time including final log checks: 791.1 s.
• Full interrupt trace retained: 19,863,460,256 bytes uncompressed (the complete compressed artifact is 426,217,452 bytes).

This tests 32 guest vCPUs on a standard runner reporting four host CPUs. It is functional topology coverage, not physical 32-core throughput. The gate runs once without automatic retry and fails a partial 32-CPU topology or missing/duplicate AP acknowledgement. The committed regression fixtures cover those cases and completion-mode collection.

The new 32-vCPU Extended lanes retain full-window mode but have not been run on this head. Historical functional and Extended results below keep their original commit scope; STRESS-7 remains open.

Historical functional CI — f90b448

Regular CI run 42357546969 completed successfully on f90b448bbfd95009fd7f9b7c4141328a9331e10b: 14/14 jobs, including the aggregate, with 12 artifacts uploaded. The quality job reports 238 tests, zero failures/errors/skips; lint, shell syntax, fixture validation, actionlint, rustfmt, Clippy and license checks passed.

Functional gate	Final verdict	Gate seconds
boot	QUALIFIED (3)	13.4
runtime	QUALIFIED (3)	12.9
SMP 4-vCPU	QUALIFIED (3)	14.6
musl 1-vCPU	PASS (0)	20.2
musl 4-vCPU	PASS (0)	15.6
IOMMU activation	QUALIFIED (3)	9.3

Boot/runtime/SMP now take 40.9 seconds of gate execution, versus the former three mandatory 900-second windows. This excludes build/setup and reflects the explicitly changed observation scope, not a kernel execution speedup. These gates and musl/IOMMU activation required no retry. IOMMU's failure and device probes also PASS.

The uploaded ci-runtime artifact was independently verified: all three receipts bind to the final serial hashes, retain the two-second tail, and record QMP quit, QEMU status 0, and no forced kill. QUALIFIED remains distinct from PASS. Committed regressions cover late faults during shutdown, incomplete/duplicate completion evidence, deadline, spontaneous exit, cancellation/reaping, rejected QMP quit, forced kill and receipt mismatch.

Functional completion covers the workload through synchronous PID 1 cleanup plus a two-second tail. It does not claim the former 900-second post-completion stability observation. Extended explicitly retains window mode and the stress profiles retain their full 900-second soak after READY. The regular workflow's manual qemu_gate_mode=window input restores the former observation scope, including musl.

The full Extended acceptance below is historical evidence for 429d613, not an Extended matrix qualification of the new head. Its retained block failure and STRESS-7 remain open.

Historical Extended acceptance — 429d613

Run 42348591710 completed on 429d6134530fc73b851792c3e91e5b6f100737e9: 20/20 lanes returned a final verdict and uploaded 20 artifacts: 19 QUALIFIED (3), 1 FAIL (1). No lane was cancelled or exhausted its job budget. The longest job took 39:32 against its 60-minute budget. The workflow and its aggregate correctly remain failure because of the block-lane failure below.

Lane	Ubuntu 22.04: verdict; job elapsed	Ubuntu 24.04: verdict; job elapsed
smp-8-16	QUALIFIED (3); 37:00	QUALIFIED (3); 39:32
ext3	QUALIFIED (3); 17:31	QUALIFIED (3); 17:11
iommu	QUALIFIED (3); 20:06	QUALIFIED (3); 21:00
stress-memory	QUALIFIED (3); 16:55	QUALIFIED (3); 17:25
stress-cpu	QUALIFIED (3); 19:37	QUALIFIED (3); 18:41
stress-smp	QUALIFIED (3); 20:16	QUALIFIED (3); 19:16
stress-thread	QUALIFIED (3); 19:14	QUALIFIED (3); 18:44
stress-process	QUALIFIED (3); 17:50	QUALIFIED (3); 17:20
stress-block	QUALIFIED (3); 18:20	FAIL (1); 4:18
stress-combined	QUALIFIED (3); 18:51	QUALIFIED (3); 18:36

Elapsed times cover job start through completion, including setup and artifact upload, but exclude queue time. IOMMU activation is QUALIFIED; both device and failure probes are PASS on both images. A successful Actions job that accepts exit 3 is not a strict guest PASS.

Retained failure: Ubuntu 24.04 stress-block exhausted its existing 12 bounded writer/crash attempts without capturing an active Zero-Intent JBD2 tail. It returned FAIL: no bounded crash attempt captured an active journal tail after 137.9 seconds in the gate. The recovery soak was not entered because its precondition failed. Its artifact is retained; the failed lane was not rerun and the criterion was not relaxed.

The formerly interrupted Ubuntu 22.04 stress-smp lane now completed with QUALIFIED and an artifact. This outcome does not prove what caused the previous runner shutdowns.

Parser repair: 429d613 replaces whole-file diagnostic reads/copies with bounded scanning through EOF. A 128-MiB trace with a late fault reproduces MemoryError in the old reader under a 96-MiB additional virtual-address-space budget; the new reader detects it. All 223 script tests passed locally and in the cloud quality job. Lint also passed. The regular PR CI is now completed successfully: 14/14 jobs, including the aggregate. Boot, runtime and SMP each returned QUALIFIED (3) after their unchanged observation windows; musl on 1/4 vCPUs returned PASS. The boot/runtime/SMP group took 45:53 before artifact upload and did not rerun a guest gate.

EXT-CI runtime acceptance is complete: each split lane returns evidence inside its budget, including a genuine failure. PR #85 remains unmerged for review. No guest window, CPU count, workload, heartbeat threshold or verdict rule was weakened. STRESS-7 remains open, and the 4-vCPU thread lanes do not satisfy #62's 16-vCPU requirement.

Preserved historical acceptance and runner interruptions at 524e624

Tested head: 524e624d0ad7ec4837a4146a4928c5d500f69779 on ci/ext-ci. Run 42341190538, attempt 1.

The first attempt produced 17 QUALIFIED guest verdicts, 2 FAIL verdicts, and 1 infrastructure-interrupted lane. A green job here accepts exit 3; it is not a strict guest PASS. Build steps passed. The workflow and its aggregate correctly remain failed.

Lane	Ubuntu 22.04: verdict; job elapsed	Ubuntu 24.04: verdict; job elapsed
smp-8-16	QUALIFIED (3); 39:32	QUALIFIED (3); 40:37
ext3	QUALIFIED (3); 17:32	QUALIFIED (3); 16:42
iommu	QUALIFIED (3); 19:24	QUALIFIED (3); 19:52
stress-memory	FAIL (1); 17:30	QUALIFIED (3); 17:45
stress-cpu	QUALIFIED (3); 19:51	QUALIFIED (3); 19:49
stress-smp	INCOMPLETE (runner shutdown); 17:46	QUALIFIED (3); 19:31
stress-thread	QUALIFIED (3); 19:19	QUALIFIED (3); 18:47
stress-process	QUALIFIED (3); 18:12	QUALIFIED (3); 18:17
stress-block	FAIL (1); 18:43	QUALIFIED (3); 18:59
stress-combined	QUALIFIED (3); 18:58	QUALIFIED (3); 18:50

IOMMU's activation gate is QUALIFIED; its device/failure probes are PASS. Times above include job setup and artifact upload. The longest completed lane took 40:37 against its 60-minute budget.

The Ubuntu 22.04 stress-smp job received “The runner has received a shutdown signal” after 17:46, before its 70-minute budget. Its gate step was cancelled and artifact upload skipped, even though the job-level conclusion is failure. It has no final stress verdict and cannot count as a completed lane. Only this infrastructure-interrupted job was manually rerun on the same SHA; the two actual failing soaks were retained, without retry.

Both failing gates report stress_protocol: last profile round has no HEARTBEAT:

• memory: final profile record seq 11996 at 899.581 s after READY; last complete heartbeat seq 11995.
• block: writer attempt 3 captured an active JBD2 transaction; recovery ran to 900.149 s after READY, ending with profile record seq 7902 after heartbeat seq 7901.

These are retained FAIL outcomes. No duration, CPU count, profile, heartbeat threshold, or verdict parser was changed. This does not close STRESS-7. The thread results are the existing 4-vCPU scope and do not qualify the 16-vCPU requirement in #62.

The same Ubuntu 22.04 stress-smp lane was interrupted again in attempt 2 after 17:34, and in attempt 3 after 18:22. Both received runner shutdown signals, cancelled the gate step and skipped artifact upload. Neither produced a guest verdict. The public job logs record runner shutdown and cancelled gate steps; they do not identify the underlying cause.