Skip to content

evidence: post evidence/<platform>/<task> commit statuses - #10122

Open
TrapsterDK wants to merge 3 commits into
mainfrom
evidence-status
Open

TrapsterDK wants to merge 3 commits into
mainfrom
evidence-status

Conversation

@TrapsterDK

Copy link
Copy Markdown
Collaborator

Closes #10047

evidence.yml gets a pending job (statuses: write, posts pending at dispatch) and a finish job (actions: read, statuses: write, if: always(), posts success, failure or error per matrix leg). The evidence job keeps contents: read. No new triggers. Docs and landing command updated; workflow_permissions_test gets a statuses marker; new evidence_status_test.

Not run: the Bazel tests. Local builds failed with missing llvm execroot files (remote cache error) even with the cache disabled. Only YAML parse and rustfmt were checked. No nested-Bazel or consumer tests were run. After merge, run /evidence on a test PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hwESun1HL2eSVd1d5B

@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

Review: changes requested

Good: the only trigger is still workflow_dispatch. statuses: write is scoped to pending and finish, and finish adds only actions: read. The evidence job keeps contents: read. Inputs reach run through env, not ${{ }}, and the token is not echoed.

  • .github/workflows/evidence.yml:586: || true on the final status post. A transient API error leaves evidence/<platform>/<task> stuck at pending, and the job still goes green. Combined with the landing-command change below, gh pr checks <n> --watch then waits forever. Let finish fail when a post fails. :93 has the same problem in a milder form: no status is posted and nothing reports it.
  • .github/workflows/evidence.yml:573: the default shell on ubuntu-slim is bash -e without pipefail. If the gh api .../jobs call fails, jq -s still gets empty input and returns []. Every platform then gets error ("run ended without a result") instead of the step failing. Add set -o pipefail or shell: bash.
  • .github/workflows/evidence.yml:91, :584: the context evidence/<platform>/<task> has no scope. Two runs on one commit with different scope overwrite each other, so a later passing //go/... run hides an earlier failing //haskell/... run. CONTRIBUTING.md documents the replacement, but it makes the status unreliable as a signal.
  • .github/workflows/README.md:27: dropping --required from the landing command changes the documented merge gate. Every check is now waited on and fail-fast, including statuses from exploratory /evidence runs that are expected to fail. Raise this separately from the status feature, or keep --required and make the evidence context required where it is wanted.
  • Low: the workflow-level env: BUILDBUDDY_API_KEY (:66) is now also exported to pending and finish, which only call gh. Consider job-level env for those two jobs.
  • No Bazel tests were run, and the "analysis" check fails on an unrelated ocaml fixture error.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hwESun1HL2eSVd1d5B

@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

Fixed: removed || true from both status posts (pending and final); set -o pipefail in both steps so a failed jobs API call fails the step; context is now evidence/<platform>/<task>/<scope> (scope sanitized, 60 chars) with docs and test updated; restored --required in .github/workflows/README.md. Rebased on origin/main. Tests pass: //tools/ci:evidence_status_test, //tools/ci:config_consistency_test. Left as is: job-level env for the Low item.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hwESun1HL2eSVd1d5B

@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

Review: changes requested

Fixed since the last review: no || true on either post, set -o pipefail in both steps (evidence.yml pending and finish run blocks), scope sanitized into the context, --required back in .github/workflows/README.md:27. Trigger is still only workflow_dispatch, permissions are minimal (statuses: write, plus actions: read on finish), no secrets are echoed.

  • .github/workflows/evidence.yml:66: BUILDBUDDY_API_KEY is still a workflow-level env:, so pending and finish receive the secret although they only call gh. The diff does not change it. Least privilege requires moving it to job-level env on evidence (or onto the steps that use it).

Not run: Bazel tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hwESun1HL2eSVd1d5B

@TrapsterDK

Copy link
Copy Markdown
Collaborator Author

Fixed: moved BUILDBUDDY_API_KEY from workflow-level env to job-level env on the evidence job only (the only job running Bazel), so pending and finish get no secrets. Added evidence_status_test assertion that no workflow, pending, or finish env (job or step) references secrets. Adjusted secret_flags_test upload-scan check to also honor job-level key. Rebased on origin/main. Tests pass: //tools/ci:evidence_status_test, //tools/ci:secret_flags_test, //tools/ci:config_consistency_test.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xut2hwESun1HL2eSVd1d5B

@TrapsterDK
TrapsterDK marked this pull request as ready for review November 9, 2026 16:34

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

evidence: post a commit status so auto-merge waits for requested evidence

1 participant