fix(ci): show example test failure reasons

[xiaofeilovecoding]

Why

Refs #1010.

When the examples suite fails, the gate reports the test name but drops the failure message from the JUnit report. This makes the CI failure hard to diagnose.

What changed

• Keep failure details on stderr so the gate can still read the same classification from stdout.
• Show up to five non-budget failures, capped at 1,000 characters each, with a marker when truncated.
• Add 14 checks to the existing budget-gate tests for diagnostics, output limits, and unchanged exit codes.

Test plan

• bash scripts/test-examples-budget-gate.sh: 58 passed. The same tests with the original classifier: 49 passed, 9 failed.
• bash -n scripts/test-examples-budget-gate.sh and Python compile() syntax check: passed.
• git diff --check: passed.

Run on Windows with Python 3.14.6 and Git Bash 5.3.9. Full SDK suites and live examples were not run locally.

How I can prove I was successful

No playable artifact — see Test plan. The regression tests run the real gate with JUnit fixtures and check that failure details appear in its log.

Anything surprising?

This fixes the missing diagnostics in #1010. The underlying evaluator failure still needs investigation; budget handling is unchanged.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 163f1054-9660-490c-bdec-12edc64fa9d5

📥 Commits

Reviewing files that changed from the base of the PR and between ad220e6 and bc2d47c.

📒 Files selected for processing: 2

• scripts/ci/classify-examples-failures.py
• scripts/test-examples-budget-gate.sh

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

Walkthrough

The example failure classifier now emits normalized, bounded diagnostics for up to five non-budget failures. The test harness checks diagnostic content and confirms that classification output and suite exit status remain unchanged.

Changes

Failure diagnostics

Layer / File(s)	Summary
Emit and verify failure diagnostics scripts/ci/classify-examples-failures.py, scripts/test-examples-budget-gate.sh	The classifier emits normalized diagnostics for up to five non-budget failures and truncates long text at 1,000 characters. Harness checks cover report content, budget-only output, classifier stdout, and suite exit status.

Suggested reviewers: langwatch-agent

Priority: ⬇️ Low

Change: Bug fix

Merge Risk: ⚪ Minimal · up to bc2d4

The diagnostic change appears ready to merge after normal checks; no merge-blocking issue was identified.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly and concisely describes the main change: showing example test failure reasons in CI.
Description check		The description directly explains the missing failure messages, the diagnostic limits, the test updates, and the validation results.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit reads each test report,
And trims long errors to their sort.
Five failures leave a trace behind,
Budget-only ones stay confined.
The checks keep stdout as before,
Then hop away across the floor.}

_{Comment @coderabbitai help to get the list of available commands.}