From 1b9a85d3fe4b7c8c320ba7dffa5bc28fd08c4882 Mon Sep 17 00:00:00 2001 From: Florian DAVID <150798857+fdaviddpt@users.noreply.github.com> Date: Sun, 11 Oct 2026 04:03:19 +0200 Subject: [PATCH 1/6] PROOF (temporary) for #1014: lower faulthandler_timeout to prove the dump fires Temporary commit to capture CI evidence that pytest's faulthandler_timeout names a hung test and dumps its traceback. Will be removed before the final PR: this lowers faulthandler_timeout to 5s and adds a throwaway test that sleeps 8s so the dump fires on a real CI run (windows + linux). Co-Authored-By: Max --- pyproject.toml | 4 ++++ tests/test_proof_1014_faulthandler.py | 13 +++++++++++++ 2 files changed, 17 insertions(+) create mode 100644 tests/test_proof_1014_faulthandler.py diff --git a/pyproject.toml b/pyproject.toml index 0fb94782..e8a301a4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,3 +7,7 @@ testpaths = ["tests"] # human. Rendered as a bare `s`, a green run with silent skips is visually # identical to a green run that checked everything (#306). addopts = "-rs --durations=25 --cov=pipeline --cov-report=term-missing --cov-fail-under=80" + +# TEMPORARY PROOF VALUE for #1014 -- lowered to make the dump fire fast on CI. +# Will be raised to the real margin (900s) before the PR is finalized. +faulthandler_timeout = 5 diff --git a/tests/test_proof_1014_faulthandler.py b/tests/test_proof_1014_faulthandler.py new file mode 100644 index 00000000..148a6aab --- /dev/null +++ b/tests/test_proof_1014_faulthandler.py @@ -0,0 +1,13 @@ +"""Throwaway proof for #1014 -- removed before the PR is finalized. + +Sleeps past the temporarily-lowered `faulthandler_timeout` in +pyproject.toml so a CI run names this test and dumps its traceback. This +file and the lowered pyproject.toml value are both reverted to the real +settings once the proof is captured from a job log. +""" + +import time + + +def test_proof_sleep_past_faulthandler_timeout(): + time.sleep(8) From faf224f617052753d0baadb29ed1f719db36bed9 Mon Sep 17 00:00:00 2001 From: Florian DAVID <150798857+fdaviddpt@users.noreply.github.com> Date: Sun, 11 Oct 2026 09:37:23 +0200 Subject: [PATCH 2/6] Fix #1014: faulthandler_timeout + step timeout so a hung pytest leg names its test A Windows pytest leg was cancelled twice by the job-level timeout-minutes: 30 (PR #1013 job 114288023016, PR #1020 job 114358206393), and a job cancelled by its own timeout writes NO log at all, so we could not tell which test hung or whether it was the runner or our code (#1002's real wscript.exe tests have already hung 40+ minutes once). - pyproject.toml: faulthandler_timeout = 900 under [tool.pytest.ini_options] (built into pytest, no new dependency). Dumps every thread's traceback, naming the hung test, once any single test runs longer than 900s. Sized with ~30x margin over the slowest single test observed on a recent green windows-latest run (30.14s, run 38091996750 job 114330142530), and below the new step timeout so the dump lands in a kept log. - .github/workflows/tests.yml: a step-level timeout-minutes: 25 on both "Run tests" and "Run the full suite against the compiled hooks" -- below the job's 30-minute one, so a hang fails the STEP and the job still completes with its log (including the faulthandler dump) rather than being cancelled with no log at all. - A trailing, Windows-only, if: always() step on both jobs that lists any surviving wscript.exe/bash.exe/conhost.exe/powershell.exe processes with PID/PPID/command line, so a leaked child is visible. Not hypothetical: the same green run above still had to force-terminate a leaked bash and a leaked wscript process at job teardown. PROVEN on PR #1022 (same branch, run 38103924282, before this commit): a temporary faulthandler_timeout=5 plus a throwaway test sleeping 8s produced a log on BOTH a Windows leg (pytest (windows-latest, 3.10), job 114365322297, lines 703-708: "Timeout (0:00:05)!" naming tests/test_proof_1014_faulthandler.py::test_proof_sleep_past_faulthandler_timeout with its full traceback) and a Linux leg (pytest (ubuntu-latest, 3.10), job 114365322191, lines 1072-1079: same dump, same test). The throwaway test and the lowered value are removed/restored in this commit; the real values above are what ships. No user-visible behavior changes -- CI-only -- so no changelog fragment; PR labeled no-changelog instead. faulthandler_timeout is independent of --cov (coverage collection is unaffected) and of pytest-xdist (confirmed via repo-wide grep: this repo has no xdist configured, and the setting is per-worker/per-process either way if that changes). Co-Authored-By: Max --- .github/workflows/tests.yml | 18 ++++++++++++++++++ pyproject.toml | 17 ++++++++++++++--- tests/test_proof_1014_faulthandler.py | 13 ------------- 3 files changed, 32 insertions(+), 16 deletions(-) delete mode 100644 tests/test_proof_1014_faulthandler.py diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index e5ba8ba2..c8dc2edd 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -87,7 +87,16 @@ jobs: Add-MpPreference -ExclusionPath "${{ github.workspace }}" Add-MpPreference -ExclusionPath "$env:RUNNER_TEMP" - name: Run tests + timeout-minutes: 25 run: pytest + - name: List surviving wscript/bash/conhost/powershell processes (Windows) + if: always() && runner.os == 'Windows' + shell: pwsh + run: | + Get-CimInstance Win32_Process | + Where-Object { $_.Name -in @('wscript.exe', 'bash.exe', 'conhost.exe', 'powershell.exe') } | + Select-Object ProcessId, ParentProcessId, Name, CommandLine | + Format-List # #900: each hooks.json-registered hook script is inlined and # comment-stripped for the release tree (compile_hooks.py) so the @@ -139,4 +148,13 @@ jobs: # above already uses on this same matrix. run: python .github/scripts/compile_hooks.py --repo . --apply - name: Run the full suite against the compiled hooks + timeout-minutes: 25 run: pytest + - name: List surviving wscript/bash/conhost/powershell processes (Windows) + if: always() && runner.os == 'Windows' + shell: pwsh + run: | + Get-CimInstance Win32_Process | + Where-Object { $_.Name -in @('wscript.exe', 'bash.exe', 'conhost.exe', 'powershell.exe') } | + Select-Object ProcessId, ParentProcessId, Name, CommandLine | + Format-List diff --git a/pyproject.toml b/pyproject.toml index e8a301a4..b2c85080 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,6 +8,17 @@ testpaths = ["tests"] # identical to a green run that checked everything (#306). addopts = "-rs --durations=25 --cov=pipeline --cov-report=term-missing --cov-fail-under=80" -# TEMPORARY PROOF VALUE for #1014 -- lowered to make the dump fire fast on CI. -# Will be raised to the real margin (900s) before the PR is finalized. -faulthandler_timeout = 5 +# #1014: a job cancelled by its own job-level timeout-minutes writes NO log at +# all, so a hung test used to be unidentifiable. faulthandler_timeout (builtin +# to pytest, no new dependency) dumps every thread's traceback -- naming the +# hung test -- once any single test runs longer than this. Sized from a green +# windows-latest run's own --durations=25 output (run 38091996750, job +# 114330142530): the slowest single test there was 30.14s, so 900s leaves +# ~30x margin over normal variance while still firing well inside the step's +# own 1500s (25min) timeout below, which is itself below the job's 30min +# timeout -- the dump lands in the kept log before the step is killed. +# Compatible with --cov (faulthandler is independent of coverage collection) +# and with pytest-xdist if it is ever added (this repo currently has none, +# confirmed by a repo-wide grep); faulthandler_timeout is a per-worker/ +# per-process setting either way. +faulthandler_timeout = 900 diff --git a/tests/test_proof_1014_faulthandler.py b/tests/test_proof_1014_faulthandler.py deleted file mode 100644 index 148a6aab..00000000 --- a/tests/test_proof_1014_faulthandler.py +++ /dev/null @@ -1,13 +0,0 @@ -"""Throwaway proof for #1014 -- removed before the PR is finalized. - -Sleeps past the temporarily-lowered `faulthandler_timeout` in -pyproject.toml so a CI run names this test and dumps its traceback. This -file and the lowered pyproject.toml value are both reverted to the real -settings once the proof is captured from a job log. -""" - -import time - - -def test_proof_sleep_past_faulthandler_timeout(): - time.sleep(8) From 6a6a3d2e174fb26b379bfe618711fc1c7ec04ec9 Mon Sep 17 00:00:00 2001 From: Florian DAVID <150798857+fdaviddpt@users.noreply.github.com> Date: Sun, 11 Oct 2026 12:12:43 +0200 Subject: [PATCH 3/6] PROOF (temporary) for #1014 round 2: faulthandler_exit_on_timeout kills a genuine hang This PR's own CI (run 38122057767, job 114418645594, pytest windows-latest/3.9) ran the "Run tests" step ~33.6 min past its own 25min timeout and still ended `cancelled` with no log at all -- the exact failure #1014 set out to fix, on this PR's own branch. Neither the step's nor the job's timeout-minutes visibly stopped it. Temporarily lowers faulthandler_timeout to 20s and adds a test that sleeps forever (never returns on its own) to prove faulthandler_exit_on_timeout kills the pytest PROCESS itself well inside that bound, converting a hang into a failed step with a traceback in a kept log -- rather than depending on a runner-level kill already observed not to fire. Removed in the finalizing commit on this branch; the real values ship there. Co-Authored-By: Claude Sonnet 5 --- pyproject.toml | 28 +++++++++++++++---- .../test_proof_1014_round2_exit_on_timeout.py | 22 +++++++++++++++ 2 files changed, 44 insertions(+), 6 deletions(-) create mode 100644 tests/test_proof_1014_round2_exit_on_timeout.py diff --git a/pyproject.toml b/pyproject.toml index b2c85080..f4f1ed8b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -14,11 +14,27 @@ addopts = "-rs --durations=25 --cov=pipeline --cov-report=term-missing --cov-fai # hung test -- once any single test runs longer than this. Sized from a green # windows-latest run's own --durations=25 output (run 38091996750, job # 114330142530): the slowest single test there was 30.14s, so 900s leaves -# ~30x margin over normal variance while still firing well inside the step's -# own 1500s (25min) timeout below, which is itself below the job's 30min -# timeout -- the dump lands in the kept log before the step is killed. +# ~30x margin over normal variance. +# +# #1014 round 2: this PR's OWN CI hit the exact failure it set out to fix +# (run 38122057767, job 114418645594, pytest windows-latest/3.9): "Run +# tests" ran ~33.6 min -- past the step's own 25min timeout below -- yet +# the job ended `cancelled`, not a failed step, and wrote NO log at all. +# Neither the step's nor the job's own timeout-minutes visibly stopped the +# hang; the leading theory is a Windows runner cannot always kill a +# process tree held by a hung grandchild (a green sibling leg of the same +# run, job 114418645663, had to force-terminate an orphaned bash.exe/ +# wscript.exe at teardown -- see +# tests/test_windows_real_wscript_v0_42_3_1002.py's own cleanup fixture). +# faulthandler_exit_on_timeout makes the dump kill the pytest PROCESS +# itself (_exit, bypassing whatever held it) well inside this same 900s +# bound, so a hang becomes a failed step with its traceback in a log that +# survives, rather than depending on a runner-level kill that was already +# observed not to fire. +# # Compatible with --cov (faulthandler is independent of coverage collection) # and with pytest-xdist if it is ever added (this repo currently has none, -# confirmed by a repo-wide grep); faulthandler_timeout is a per-worker/ -# per-process setting either way. -faulthandler_timeout = 900 +# confirmed by a repo-wide grep); both ini options are per-worker/per-process +# settings either way. +faulthandler_timeout = 20 +faulthandler_exit_on_timeout = true diff --git a/tests/test_proof_1014_round2_exit_on_timeout.py b/tests/test_proof_1014_round2_exit_on_timeout.py new file mode 100644 index 00000000..31596662 --- /dev/null +++ b/tests/test_proof_1014_round2_exit_on_timeout.py @@ -0,0 +1,22 @@ +"""PROOF (temporary) for #1014 round 2 -- removed in the finalizing commit +on this branch. + +Shows that `faulthandler_exit_on_timeout` actually kills a genuinely hung +test PROCESS, converting a hang into a FAILED step with a traceback in a +kept log, rather than depending on the runner's own step/job timeout -- +which this PR's own earlier CI run showed does not reliably fire on +Windows (run 38122057767, job 114418645594: cancelled, no log, after the +step ran ~33.6 min past its own 25min timeout). + +This sleeps forever (never returns on its own), unlike the earlier, +already-landed #1014 proof's 8s sleep -- that one only proved the dump +fires; this one proves the PROCESS actually exits. pyproject.toml's +`faulthandler_timeout` is lowered to a small value for this one proof +commit so the CI run itself stays short. +""" + +import time + + +def test_proof_genuine_hang_is_killed_by_faulthandler_exit_on_timeout(): + time.sleep(10**9) From da1b0b528ef062c5b859a68957aba9a9ff12b77f Mon Sep 17 00:00:00 2001 From: Florian DAVID <150798857+fdaviddpt@users.noreply.github.com> Date: Sun, 11 Oct 2026 12:13:01 +0200 Subject: [PATCH 4/6] Tee pytest output + upload as artifact; clean up v0.42.3 wscript test orphans (#1014) Belt-and-braces alongside faulthandler_exit_on_timeout: the "Run tests" and "Run the full suite against the compiled hooks" steps now tee pytest's output to a file and upload it with actions/upload-artifact under `if: always()`, so a failed or timed-out step still leaves the log behind even if the step's own captured output is somehow lost. Does not cover a job-level `cancelled` -- GitHub Actions skips steps entirely in that case -- which is exactly why the process-level kill is the primary fix. Also fixes the orphan processes a green leg of this PR's own CI already surfaced (run 38122057767, job 114418645663): the OLD v0.42.3 shape test_windows_real_wscript_v0_42_3_1002.py drives backgrounds wscript.exe with a trailing `&` and disowns it by design (reproducing the shipped trap-B silent skip), so nothing in that shape or this test module ever waited on or killed what it launched -- the job's own teardown had to force-terminate a leftover bash.exe/wscript.exe pair at the end of the whole run. Adds an autouse fixture that force-terminates, after every test in the module, any wscript.exe/bash.exe/conhost.exe whose command line mentions that test's own tmp_path. Co-Authored-By: Claude Sonnet 5 --- .github/workflows/tests.yml | 44 +++++++++++++++++- .../test_windows_real_wscript_v0_42_3_1002.py | 46 +++++++++++++++++++ 2 files changed, 88 insertions(+), 2 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index c8dc2edd..45aac171 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -87,8 +87,35 @@ jobs: Add-MpPreference -ExclusionPath "${{ github.workspace }}" Add-MpPreference -ExclusionPath "$env:RUNNER_TEMP" - name: Run tests + # #1014 round 2: this PR's own CI already hit a Windows leg that + # ran past this timeout and still ended `cancelled` with no log at + # all (job 114418645594) -- the step-level kill is not trusted to + # fire on its own, so pyproject.toml's faulthandler_exit_on_timeout + # makes a hung test kill the pytest PROCESS itself well inside + # this bound. `tee` is belt-and-braces on top of that: even if a + # hang somehow outlives the process-level exit, whatever reached + # the log file before that point is uploaded as an artifact below + # under `if: always()`, rather than depending only on the step's + # own captured output surviving. timeout-minutes: 25 - run: pytest + shell: bash + run: | + set -o pipefail + pytest 2>&1 | tee pytest-output.log + - name: Upload pytest output log + # `if: always()` so a failed or timed-out step still uploads + # whatever reached the file -- the one case this cannot cover is + # a job-level `cancelled` (#1014's original failure mode), where + # GitHub Actions stops running steps entirely rather than + # executing the ones still pending; faulthandler_exit_on_timeout + # above is what keeps a hang from reaching that case at all. + if: always() + uses: actions/upload-artifact@v4 + with: + name: pytest-output-${{ matrix.os }}-${{ matrix.python-version }} + path: pytest-output.log + if-no-files-found: ignore + retention-days: 7 - name: List surviving wscript/bash/conhost/powershell processes (Windows) if: always() && runner.os == 'Windows' shell: pwsh @@ -148,8 +175,21 @@ jobs: # above already uses on this same matrix. run: python .github/scripts/compile_hooks.py --repo . --apply - name: Run the full suite against the compiled hooks + # Same #1014-round-2 reasoning as the plain pytest job's own "Run + # tests" step above. timeout-minutes: 25 - run: pytest + shell: bash + run: | + set -o pipefail + pytest 2>&1 | tee pytest-output.log + - name: Upload pytest output log + if: always() + uses: actions/upload-artifact@v4 + with: + name: pytest-output-hooks-${{ matrix.os }} + path: pytest-output.log + if-no-files-found: ignore + retention-days: 7 - name: List surviving wscript/bash/conhost/powershell processes (Windows) if: always() && runner.os == 'Windows' shell: pwsh diff --git a/tests/test_windows_real_wscript_v0_42_3_1002.py b/tests/test_windows_real_wscript_v0_42_3_1002.py index 803a3523..d07a59cc 100644 --- a/tests/test_windows_real_wscript_v0_42_3_1002.py +++ b/tests/test_windows_real_wscript_v0_42_3_1002.py @@ -53,6 +53,52 @@ _WSH_KEY = r"HKCU\Software\Microsoft\Windows Script Host\Settings" +def _kill_processes_matching(substr: str) -> list[str]: + """Windows only: force-terminate any wscript.exe/bash.exe/conhost.exe + whose command line mentions SUBSTR (this test's own tmp_path). + + #1014 CI finding (run 38122057767, job 114418645663 -- a GREEN leg of + this same PR's own CI): the hidden-launch route this module drives + left an orphan bash.exe and wscript.exe alive all the way to the job's + own teardown step, which had to force-terminate them itself (log lines + 2256-2258, "Cleaning up orphan processes"). The OLD v0.42.3 shape under + test here backgrounds wscript.exe with a trailing `&` and disowns it -- + by design, since trap B (the silent skip this module exists to + observe) depends on the caller never waiting on it -- so nothing in + the shape itself, or in this test module before this fixture, ever + killed what it launched. + + Returns what it killed, for diagnostics; killing nothing is not an + error -- traps A and B can both mean the hidden route never actually + reached a running wscript.exe at all. + """ + if sys.platform != "win32": + return [] + ps = ( + "Get-CimInstance Win32_Process | " + "Where-Object { $_.Name -in @('wscript.exe','bash.exe','conhost.exe') -and " + f"$_.CommandLine -like '*{substr}*' }} | " + "Select-Object ProcessId, Name | ForEach-Object { " + "Write-Output \"$($_.ProcessId) $($_.Name)\"; " + "Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue }" + ) + r = subprocess.run(["powershell", "-NoProfile", "-Command", ps], + capture_output=True, text=True, timeout=15, check=False) + return [line for line in r.stdout.splitlines() if line.strip()] + + +@pytest.fixture(autouse=True) +def _cleanup_hidden_launch_orphans(tmp_path): + """Kill, after every test in this module, any wscript.exe/bash.exe this + test's own hidden-launch call spawned and never waited on (see + `_kill_processes_matching`'s docstring for the CI evidence). Matched on + this test's own tmp_path so cleanup never reaches a process some other, + unrelated test launched. + """ + yield + _kill_processes_matching(str(tmp_path)) + + def _run(script: str, env: dict | None = None, timeout: float = 20, cwd: Path | None = None) -> subprocess.CompletedProcess: """CI finding (#1002 round 3, job 114204833202 and siblings): without From d960981fc7463a79010dcb50366a171ac2110d62 Mon Sep 17 00:00:00 2001 From: Florian DAVID <150798857+fdaviddpt@users.noreply.github.com> Date: Sun, 11 Oct 2026 23:01:10 +0200 Subject: [PATCH 5/6] Finalize #1014: real values + changelog, after round-2 proof narrowed the Linux-only claim The round-2 proof (run 38122057767, job 114445674357) showed the faulthandler traceback naming the stuck test IS kept in the log on Linux, followed by the step's own 25-minute timeout error -- i.e. the step timeout is what keeps the log, not faulthandler_exit_on_timeout: pytest was not observed to exit on its own before the step timeout fired. Windows was not proven. - Removes tests/test_proof_1014_round2_exit_on_timeout.py (the deliberately hanging proof test) and restores faulthandler_timeout to its real value (900s). - Drops faulthandler_exit_on_timeout: unproven by the round-2 evidence, so this repo does not rely on it. - Updates the workflow comments that had claimed faulthandler_exit_on_timeout does the killing, to instead say what is actually proven: the step timeout keeps the log, proven on Linux only. - Adds the changelog.d/1014.fixed.md fragment this PR was missing. Co-Authored-By: Claude Sonnet 5 --- .github/workflows/tests.yml | 25 +++++++------ pyproject.toml | 35 +++++++++---------- .../test_proof_1014_round2_exit_on_timeout.py | 22 ------------ 3 files changed, 31 insertions(+), 51 deletions(-) delete mode 100644 tests/test_proof_1014_round2_exit_on_timeout.py diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 45aac171..6f75ea62 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -90,13 +90,15 @@ jobs: # #1014 round 2: this PR's own CI already hit a Windows leg that # ran past this timeout and still ended `cancelled` with no log at # all (job 114418645594) -- the step-level kill is not trusted to - # fire on its own, so pyproject.toml's faulthandler_exit_on_timeout - # makes a hung test kill the pytest PROCESS itself well inside - # this bound. `tee` is belt-and-braces on top of that: even if a - # hang somehow outlives the process-level exit, whatever reached - # the log file before that point is uploaded as an artifact below - # under `if: always()`, rather than depending only on the step's - # own captured output surviving. + # fire on its own on every OS. pyproject.toml's faulthandler_timeout + # dumps the stuck test's traceback well inside this bound, and the + # step's own timeout-minutes is what actually ends the step and + # keeps the log (proven on Linux: run 38122057767, job + # 114445674357 -- the faulthandler traceback IS in the kept log, + # followed by the step-timeout error). `tee` is belt-and-braces on + # top of that: whatever reached the log file before the step ends + # is uploaded as an artifact below under `if: always()`, rather + # than depending only on the step's own captured output surviving. timeout-minutes: 25 shell: bash run: | @@ -105,10 +107,11 @@ jobs: - name: Upload pytest output log # `if: always()` so a failed or timed-out step still uploads # whatever reached the file -- the one case this cannot cover is - # a job-level `cancelled` (#1014's original failure mode), where - # GitHub Actions stops running steps entirely rather than - # executing the ones still pending; faulthandler_exit_on_timeout - # above is what keeps a hang from reaching that case at all. + # a job-level `cancelled` (#1014's original failure mode, observed + # on Windows: job 114418645594), where GitHub Actions stops + # running steps entirely rather than executing the ones still + # pending. That path is not proven fixed here -- see the issue + # filed for the unproven Windows cancellation case. if: always() uses: actions/upload-artifact@v4 with: diff --git a/pyproject.toml b/pyproject.toml index f4f1ed8b..028bfe3c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -14,27 +14,26 @@ addopts = "-rs --durations=25 --cov=pipeline --cov-report=term-missing --cov-fai # hung test -- once any single test runs longer than this. Sized from a green # windows-latest run's own --durations=25 output (run 38091996750, job # 114330142530): the slowest single test there was 30.14s, so 900s leaves -# ~30x margin over normal variance. +# ~30x margin over normal variance, and fires well inside the step's own +# 25min (1500s) timeout below. # -# #1014 round 2: this PR's OWN CI hit the exact failure it set out to fix -# (run 38122057767, job 114418645594, pytest windows-latest/3.9): "Run -# tests" ran ~33.6 min -- past the step's own 25min timeout below -- yet -# the job ended `cancelled`, not a failed step, and wrote NO log at all. -# Neither the step's nor the job's own timeout-minutes visibly stopped the -# hang; the leading theory is a Windows runner cannot always kill a -# process tree held by a hung grandchild (a green sibling leg of the same -# run, job 114418645663, had to force-terminate an orphaned bash.exe/ -# wscript.exe at teardown -- see -# tests/test_windows_real_wscript_v0_42_3_1002.py's own cleanup fixture). -# faulthandler_exit_on_timeout makes the dump kill the pytest PROCESS -# itself (_exit, bypassing whatever held it) well inside this same 900s -# bound, so a hang becomes a failed step with its traceback in a log that -# survives, rather than depending on a runner-level kill that was already -# observed not to fire. +# #1014 round 2 PROOF (run 38122057767, job 114445674357, pytest +# linux/3.x -- a deliberately hanging test, with faulthandler_timeout +# temporarily lowered and the step timeout left in place): the +# faulthandler traceback naming the stuck test IS present in the kept +# log, followed by "##[error]The action 'Run tests' has timed out after +# 25 minutes." -- i.e. the STEP timeout is what kept the log, not +# faulthandler_exit_on_timeout: pytest was not observed to exit on its +# own before the step timeout fired. faulthandler_exit_on_timeout is +# therefore not enabled here; it was not shown to do anything in this +# proof, and this repo does not rely on it. Only Linux was proven this +# way -- the earlier round-2 Windows failure (run 38122057767, job +# 114418645594: cancelled, no log, after ~33.6 min past the step's own +# 25min timeout) is NOT shown fixed by this setting; see issue filed +# separately for that gap. # # Compatible with --cov (faulthandler is independent of coverage collection) # and with pytest-xdist if it is ever added (this repo currently has none, # confirmed by a repo-wide grep); both ini options are per-worker/per-process # settings either way. -faulthandler_timeout = 20 -faulthandler_exit_on_timeout = true +faulthandler_timeout = 900 diff --git a/tests/test_proof_1014_round2_exit_on_timeout.py b/tests/test_proof_1014_round2_exit_on_timeout.py deleted file mode 100644 index 31596662..00000000 --- a/tests/test_proof_1014_round2_exit_on_timeout.py +++ /dev/null @@ -1,22 +0,0 @@ -"""PROOF (temporary) for #1014 round 2 -- removed in the finalizing commit -on this branch. - -Shows that `faulthandler_exit_on_timeout` actually kills a genuinely hung -test PROCESS, converting a hang into a FAILED step with a traceback in a -kept log, rather than depending on the runner's own step/job timeout -- -which this PR's own earlier CI run showed does not reliably fire on -Windows (run 38122057767, job 114418645594: cancelled, no log, after the -step ran ~33.6 min past its own 25min timeout). - -This sleeps forever (never returns on its own), unlike the earlier, -already-landed #1014 proof's 8s sleep -- that one only proved the dump -fires; this one proves the PROCESS actually exits. pyproject.toml's -`faulthandler_timeout` is lowered to a small value for this one proof -commit so the CI run itself stays short. -""" - -import time - - -def test_proof_genuine_hang_is_killed_by_faulthandler_exit_on_timeout(): - time.sleep(10**9) From 3b00b1fcb7353bd5646f2f1acf4f5e581bd15297 Mon Sep 17 00:00:00 2001 From: Florian DAVID <150798857+fdaviddpt@users.noreply.github.com> Date: Sun, 11 Oct 2026 23:01:22 +0200 Subject: [PATCH 6/6] Add changelog fragment for #1014 Co-Authored-By: Claude Sonnet 5 --- changelog.d/1014.fixed.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 changelog.d/1014.fixed.md diff --git a/changelog.d/1014.fixed.md b/changelog.d/1014.fixed.md new file mode 100644 index 00000000..a8d25bd1 --- /dev/null +++ b/changelog.d/1014.fixed.md @@ -0,0 +1,13 @@ +- Fixed #1014: a hung pytest leg on CI used to be cancelled by its own + job-level `timeout-minutes` with no log at all, leaving no way to tell + which test hung. `pyproject.toml` now sets `faulthandler_timeout = 900` + so a stuck test dumps every thread's traceback well before CI's own + 25-minute step timeout fires, the "Run tests" step now runs under its + own `timeout-minutes: 25` and tees output to a log file that is uploaded + as an artifact under `if: always()`, and a Windows leg additionally + lists any surviving `wscript.exe`/`bash.exe`/`conhost.exe`/ + `powershell.exe` processes at teardown. Proven on Linux (run 38122057767, + job 114445674357): the faulthandler traceback naming the stuck test is + present in the kept log. The equivalent Windows failure mode (a job + ending `cancelled` with no log at all, run 38122057767, job + 114418645594) is not shown fixed by this change; tracked separately.