Skip to content

ci: compiler probe tests fail normally; scan ci.yml for swallowed failures - #10120

Merged
TrapsterDK merged 1 commit into
mainfrom
ci-no-swallow-10060
Nov 9, 2026
Merged

TrapsterDK merged 1 commit into
mainfrom
ci-no-swallow-10060

Conversation

@TrapsterDK

Copy link
Copy Markdown
Collaborator

Closes #10060

  • hermetic and starter-consumer: new "Run the compiler probe tests" step fails normally (bazel exit 4, all targets skipped as incompatible, is accepted); "Record compiler evidence" stays if: always() and only reads results.
  • Removed every continue-on-error and the cp ... || true in ci.yml; the ci job's evidence merge now fails the gate if it fails.
  • ci_workflow::swallows plus ci_swallows_no_failure_outside_the_allow_list fail on any un-allow-listed || true or continue-on-error.
  • Job names, needs, required and triggers are unchanged.

Probe failures hidden before will now fail hermetic and starter shard 0; that is intended.

Tested: bazel test //tools/ci:aggregate_gates_test passes. ci.yml was not dispatched; nested-Bazel and consumer tests were not run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hwESun1HL2eSVd1d5B

@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

Review: approve

Checked: job/step logic, if: conditions, gate and job names, permissions, triggers, secret handling, and that evidence is still uploaded when tests fail.

  • Logic is right. "Run the compiler probe tests" (.github/workflows/ci.yml:187, :648) is !cancelled(). It fails normally and only accepts bazel exit 4. "Record compiler evidence" stays always() and only reads results. The scan and upload steps stay always() and are gated on the scan. Evidence is still uploaded when the probes or the qualify step fail.
  • If the facts/producers build fails, no compiler-targets file is written, Record exits 0 with a notice, and assert-no-secret skips the missing path. The job still fails through the probe step, so nothing is hidden.
  • Job names, needs, the ci aggregate, required=(...), permissions and triggers are unchanged. No new secrets reach a step, and nothing is echoed.
  • Note: .github/workflows/ci.yml:1062-1088. The ci gate job now fails when the artifact download, compiler_table.sh or the compiler-evidence-all upload fails. A transient artifact-service error will block the merge queue even when every needed job passed. This is intended per ci: evidence steps must not swallow test failures with || true #10060; just calling it out.
  • Note: tools/ci/ci_workflow/src/lib.rs swallows() only matches the literal || true in ci.yml run blocks. It misses ||true, || :, || exit 0 and set +e. It also misses anything inside .github/actions/*/action.yml.
  • Conflict: ci: one hermetic-evidence composite action for the hermetic jobs #10129 rewrites the same steps into composite actions. Those actions keep || true on the probe bazel build/bazel test and on the cp, which would undo this PR in a place the scanner can't see. Land in sequence.
  • Note: tools/ci/aggregate_gates_test.rs:545. The scanner self-test depends on reusable-consumer.yml still containing || true, so an unrelated cleanup there would break this test.
  • The failing "analysis" check is an unrelated ocaml fixture compile_check error ("params files are not supported"). It also fails on evidence: post evidence/<platform>/<task> commit statuses #10122 and ci: one hermetic-evidence composite action for the hermetic jobs #10129. ci.yml itself has not run on this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hwESun1HL2eSVd1d5B

@TrapsterDK
TrapsterDK marked this pull request as ready for review November 9, 2026 16:25
@TrapsterDK
TrapsterDK merged commit d45f5a8 into main Nov 9, 2026
1 check failed
@TrapsterDK
TrapsterDK deleted the ci-no-swallow-10060 branch November 9, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: evidence steps must not swallow test failures with || true

1 participant