feat: packaged CI gate — JUnit XML output + composite GitHub Action#37
Open
uipreliga wants to merge 8 commits into
Open
feat: packaged CI gate — JUnit XML output + composite GitHub Action#37uipreliga wants to merge 8 commits into
uipreliga wants to merge 8 commits into
Conversation
New src/coder_eval/reports_junit.py: generate_junit_xml(run_dir) reads the run.json spine (RunSummary), optional */*/suite.json gates (SuiteRollup), and per-failed-row task.json (best-effort plain-dict) to build a JUnit XML string; write_junit_xml is the thin persist wrapper. Counts always equal emitted children; status classification goes through FinalStatus.category; all agent-derived text is scrubbed of illegal XML 1.0 chars. Production code never parses XML. Adds defusedxml (dev-only) for test-side round-trip parsing and a shared write_run_json conftest fixture reused by later phases. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add --junit-xml to `coder-eval run` (writes the JUnit report after the run summary is persisted and before the failure exit-code gate, so a red run still produces a report; write errors propagate). Add 'junit' to `coder-eval report --format` (regenerates from any run dir; defaults to <run-dir>/junit.xml, -o overrides; missing run.json → clean red error). Output-only — does not enter the config merge. Docs: CI-pipeline tutorial JUnit section + CLAUDE.md tree entry. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add action.yml at the repo root: a composite action that installs a pinned coder-eval (or the local checkout via `version: local`), runs the suite, always emits run-dir/junit-path outputs and a run.md job summary, then exits with coder-eval's real exit code. Inputs cross into bash via env: only. release.yml now bumps the action's `version:` default inside the amended release commit (anchored on a `# <-- kept in sync` comment, with a loud grep guard) and force-moves the `v<major>` tag after pushing the release tag. pr-checks.yml gains a fork-gated `action-dogfood` job that exercises the action via `uses: ./` on one Haiku task and verifies the JUnit + run.json output — the sensor for this surface, since it isn't covered by pytest. The action is deliberately agent-agnostic: it installs no coding-agent runtime, so callers provide the Claude CLI (as the dogfood job does). Documented in the README and the CI tutorial along with the pull_request_target security caveat. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Companion to the pyproject dev-group addition in 1/3. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Harden reports_junit.py against schema-skewed / crafted run.json rows, which are untyped dicts (RunSummary.task_results is list[dict[str, Any]]) and may be blob-pulled from elsewhere: - Path containment: a crafted variant_id/task_id could steer the task.json lookup outside the run dir (absolute value discards run_dir, ".." walks up). Reject unsafe path components and verify containment after resolve(). - UnicodeDecodeError is a ValueError but NOT a json.JSONDecodeError, so a task.json with undecodable bytes aborted the whole report instead of falling back. Catch ValueError. - Ambiguous replicate: with replicate_index absent and several replicate dirs, the writer attributed replicate 00's failure detail to the row. Degrade to the status-only body instead of misattributing. - bool is an int subclass, so a bool replicate_index rendered as "[01]" and a bool duration as a time. Exclude bool explicitly. - Guard non-finite/negative durations (NaN would emit an invalid time="nan"). - Coerce non-str variant_id/task_path rather than crashing on dict keys/Path(). - Skipped testcases now use the suffix-stripped path, not the bare stem, so two skipped tasks sharing a basename keep distinct JUnit identities. - A row with no status reads as "<missing>" rather than "None". Found by the final multi-model review (gpt-5.6, gemini-3.1-pro, gpt-5.3-codex); each fix has a regression test that fails without it. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
uipreliga
requested review from
akshaylive,
bai-uipath and
tmatup
as code owners
July 22, 2026 01:23
|
Claude Code is working… I'll analyze this and get back to you. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # Every time attribute must be a finite, non-negative float (JUnit requirement). | ||
| for case in cases: | ||
| t = float(case.get("time")) | ||
| assert t == t and t >= 0.0 # not NaN, not negative |
_skipped_suite built the testcase name with str(Path(path).with_suffix("")),
which yields "tasks\opt" on Windows and "tasks/opt" on Linux. That made the
JUnit testcase identity depend on the OS that generated the report, so the same
logical run would split into two identities in CI history/flake tracking (and it
broke the Windows smoke job).
Normalize separators and parse with PurePosixPath so the emitted name is always
"/"-separated. Adds a regression test feeding a Windows-style path.
Also switch the action's model in the README/tutorial examples from Haiku to
Sonnet (docs only; the dogfood job stays on Haiku for per-PR cost).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
uipreliga
force-pushed
the
feat/ci-gate-junit-action
branch
from
July 22, 2026 01:39
186fa90 to
f6c02fe
Compare
Co-Authored-By: Claude <[email protected]>
…tion # Conflicts: # .github/workflows/release.yml # uv.lock Co-Authored-By: Claude <[email protected]>
tmatup
approved these changes
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes coder-eval consumable as a CI gate: a JUnit XML report generated from any run directory, plus a composite GitHub Action published from the repo root.
What's in it
1.
src/coder_eval/reports_junit.py(new) —generate_junit_xml(run_dir)reads a finalized run directory and returns JUnit XML;write_junit_xmlis the thin persist wrapper (mirrors thebuild_run_summary/write_run_summaryseam).run.jsonspine via theRunSummarymodel (SSOT, not hand-parsed), optional*/*/suite.jsongates viaSuiteRollup, and per-failed-rowtask.jsonas a plain dict (best-effort, so schema skew in blob-pulled/older runs degrades to a status-only body instead of aborting).<testsuite>per variant, one<testcase>per task row ([NN]suffix for replicates);skippedandsuite-gatessynthetic suites. Count attributes are derived from the emitted children, so they can't drift.FinalStatus(value).category— an explicit allowlist with an explicit unknown-value fallback, never a denylist (CE018).defusedxmlis added dev-only for test-side round-trips.2. CLI wiring — output-only flags; neither enters the 5-layer config merge.
coder-eval run --junit-xml PATH— written after the run summary is persisted and before the failure exit-code gate, so a red run still emits a report. A write error propagates (a silently missing file would make a downstream reporter fail confusingly).coder-eval report <run-dir> -f junit [-o PATH]— regenerates from any existing run dir (defaults to<run-dir>/junit.xml).3.
action.yml+ release automation + dogfoodversion: localescape hatch, always emitsrun-dir/junit-pathoutputs and appendsrun.mdto$GITHUB_STEP_SUMMARY, then exits with coder-eval's real exit code. Every input crosses into bash viaenv:— zero${{ }}interpolation into script bodies.release.ymlbumps the action's pinnedversion:default inside the amended release commit (anchored on a# <-- kept in synccomment, with agrepguard that fails the release loudly rather than shipping a stale pin) and force-moves thev<major>tag.pr-checks.ymlgains a fork-gatedaction-dogfoodjob — the sensor for this surface, since it isn't reachable from pytest.Design note (deviation from the original plan)
The action is deliberately agent-agnostic: it installs
coder-evalbut no coding-agent runtime.claude-agent-sdkdoes not bundle theclaudebinary, so callers provide it — the dogfood job installs Node +@anthropic-ai/claude-codeitself, and the README/tutorial document this. Keeps the action usable for codex/antigravity consumers instead of forcing a Claude CLI install on everyone.Review fixes
A multi-model review (gpt-5.6, gemini-3.1-pro, gpt-5.3-codex) found several robustness gaps around the untyped
task_resultsrow dicts, all fixed with regression tests that fail without the fix:variant_id/task_idcould steer thetask.jsonlookup outside the run dir (an absolute value discardsrun_dir;..walks up). A test confirmed a real leak before the fix.UnicodeDecodeErroris aValueErrorbut not ajson.JSONDecodeError, so undecodable bytes aborted the whole report despite the documented graceful degrade.replicate_indexabsent and several replicate dirs, replicate 00's failure detail was misattributed to the row; now degrades instead.boolis anintsubclass — a boolreplicate_indexrendered as[01]; NaN/inf durations emitted an invalidtime="nan".variant_id/task_pathcrashed the dict key /Path(); same-basename skipped tasks collapsed into one JUnit identity.Testing
make verifygreen: 3454 passed, 2 skipped (both pre-existing environment skips), 90.98% coverage (threshold 80%);reports_junit.py~98%.tmp_path, no agents, no API.sed(proven to rewrite only the version line and preserve the trailing comment), an empirical simulation of the action's argv building underset -uo pipefail, andactionlint(clean on all additions).What can't be verified until this PR runs
action-dogfood— this PR both adds the job and is its first subject. Composite-action quirks (setup-uvPATH propagation,$GITHUB_ACTION_PATHinstall, output plumbing) are only provable in CI.🤖 Generated with Claude Code