feat: add auditable evaluation and prompt optimization loop

[mingri31164]

Closes #91

Summary

• add a reproducible baseline -> failure attribution -> prompt optimization -> validation regression -> gate -> audit pipeline
• run every baseline and candidate through AgentEvaluator, with fake-model trace mode and call_agent mode supported
• keep AgentOptimizer as an optional search backend with update_source=False; only the external gate may approve source write-back
• classify final-response, tool-call, parameter, rubric, knowledge-recall, format, and execution failures with evidence and key traces
• persist JSON/Markdown reports, every candidate prompt, raw evaluator results, costs, timing, seed, and input hashes

Acceptance behavior

Candidate	Train score	Validation score	Decision
baseline	0.0000	0.6667	reference
balanced_candidate	0.6667	1.0000	accept
overfit_candidate	0.6667	0.3333	reject

The overfit candidate improves training while adding failures in val_critical_safety and val_stable_math. It is rejected by validation delta, hard-fail, critical-case, new-failure, and train/validation gain-gap checks.

Verification

• .venv/bin/pytest tests/evaluation/test_eval_optimize_loop_example.py -q -ra: 18 passed
• .venv/bin/pytest -q tests/evaluation: all ~901 evaluation tests passed
• fake model + deterministic judge + trace mode: 18 local calls, ~0.03 seconds, no API key
• Flake8, JSON parsing, and git diff --check: passed
• broader repository suite (excluding existing Claude/CLI Python 3.11 collection incompatibility) reached 100%; only the existing Python 3.11 enum compatibility assertion failed

E2E evidence

The screenshot is rendered directly from the generated optimization_report.json and shows baseline/candidate scores, both round decisions, overfit rejection checks, per-case delta, cost, and source-write audit.

Deliverables

• examples/optimization/eval_optimize_loop/ entry script, prompt, config, 3 train + 3 validation cases, README, and 459-character design note
• checked-in optimization_report.json and optimization_report.md sample outputs
• regression tests for gate accuracy, attribution, trace/call-agent modes, cost budget, overfitting, write-back, and rollback on optimizer failure

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[helloopenworld]

AI Code Review

没有命令注入的风险。让我来撰写最终的审查结论。

发现的问题

🚨 Critical

（未发现 Critical 问题。）

⚠️ Warning

• examples/optimization/eval_optimize_loop/pipeline.py:403-408：except AssertionError: pass 过于宽泛，会吞掉无关断言错误

  • AgentEvaluator 内部用 _EvaluationCasesFailed(AssertionError) 表示 case 失败（见 _agent_evaluator.py:78-93），其文档明确指出内部包装器应捕获该具体子类以避免“unrelated AssertionError 被静默吞掉”。当前裸 except AssertionError 会把评估器或依赖库中任何 AssertionError（如 assert allclose）一并吞掉，随后 get_result() 可能返回部分/陈旧结果且无任何告警。建议改为 except _EvaluationCasesFailed 或 AgentEvaluator 暴露的具体异常类型，至少在 result is None 之外对非预期 AssertionError 重新抛出。

• examples/optimization/eval_optimize_loop/pipeline.py:862：默认 run_id 用本地时间 datetime.now()，与审计字段 started_at（UTC）时区不一致且可能碰撞

  • run_id = run_id or datetime.now().strftime("%Y%m%dT%H%M%S") 使用本地时间且精度到秒，而 started_at/finished_at 用 UTC；同一秒内多次运行还会产生相同 run_id，破坏审计可追溯性。建议改用 UTC（datetime.now(timezone.utc)）并叠加更高精度或唯一后缀，与 _utc_now() 保持一致。

• examples/optimization/eval_optimize_loop/pipeline.py:600：max_candidate_cost_usd 缺省为 inf，与“默认 fail closed”设计不一致

  • float(gate_config.get("max_candidate_cost_usd", float("inf"))) 在 gate 配置漏配该项时退化为不设成本上限，使成本检查永远通过。DESIGN/README 强调“所有检查按 AND 语义执行并默认 fail closed”。建议缺省为一个有限保守值或在配置缺失时显式报错，避免成本 gate 被静默跳过。

💡 Suggestion

• examples/optimization/eval_optimize_loop/pipeline.py:33：from fake_model import FakePromptModel 为顶层裸导入，使 pipeline.py 必须依赖 sys.path 注入（run_pipeline.py/测试都手动 sys.path.insert）才能导入，限制了作为包模块复用。若希望示例可被 examples.optimization... 路径导入，可改为相对导入或 try/except 兜底；当前作为示例可接受。

• examples/optimization/eval_optimize_loop/pipeline.py:62-67：_write_json 用 path.write_text 直接覆盖，而报告文件（optimization_report.json，约 1450 行）写入非原子；并发运行或写入中断可能产生半截文件。可复用已有的 _atomic_write_text（tmp+replace）以保持与 prompt 回写一致的原子性。

总结

整体风险较低，无 Critical 阻塞问题；新增示例与测试逻辑自洽，trace/call_agent 双路径、gate AND 语义、失败归因和源 prompt 原子回写均与设计一致。主要建议修复 except AssertionError 过宽、run_id 时区/碰撞、以及成本 gate 缺省值三处 Warning，以更好地兑现“可复现、可追责、默认 fail closed”的设计目标。

测试建议

• 建议补充一个“评估器内部抛出非 _EvaluationCasesFailed 的 AssertionError”的注入测试，验证 pipeline 不会静默吞掉并继续产出报告（覆盖 pipeline.py:403-408 的修复）。
• 建议补充同一秒内两次 run_pipeline 默认 run_id 的隔离性测试，或显式断言 run_id 唯一性。

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@6a2f7f9). Learn more about missing BASE report.

Additional details and impacted files

@@            Coverage Diff             @@
##             main        #235   +/-   ##
==========================================
  Coverage        ?   88.22774%           
==========================================
  Files           ?         482           
  Lines           ?       45157           
  Branches        ?           0           
==========================================
  Hits            ?       39841           
  Misses          ?        5316           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mingri31164]

Addressed the automated review findings in bbda5b0:

• catch only _EvaluationCasesFailed; unrelated evaluator assertions now propagate, with an injection regression test
• generate default run IDs from UTC microseconds plus a unique suffix, with uniqueness/format coverage
• fail closed when the candidate cost budget is missing, negative, or non-finite
• support package-relative and direct-script imports for FakePromptModel
• write JSON/Markdown artifacts atomically through unique temporary files and replace

Re-verified with 16 focused tests, the complete tests/evaluation suite, Flake8, YAPF diff, JSON parsing, git diff --check, and the fake/trace E2E run.

[helloopenworld]

AI Code Review

我已经掌握了完整的情况。正在撰写代码审查报告。

发现的问题

⚠️ Warning

• examples/optimization/eval_optimize_loop/pipeline.py:2478-2512（_validate_inputs）：agent_optimizer 后端缺少必要前置校验。

  • _validate_inputs 只校验了 evaluate 段，但 agent_optimizer 后端在 pipeline.py:2553-2563 处会用 config["evaluate"] 与 config["optimize"] 构造 optimizer 配置。若用户切到该后端却漏配 optimize 段，不会在启动校验阶段失败，而会在 _agent_optimizer_candidate 内抛出 KeyError: 'optimize'（或 SDK 深层 ValueError），此时 audit/、resolved_optimizer_config.json 已落盘且 baseline 评估已完成，错误定位成本高。建议在该函数对 backend == "agent_optimizer" 时补一条 if "optimize" not in config: raise ValueError(...)，与 optimizer_backend、gate 的 fail-fast 风格一致。

• examples/optimization/eval_optimize_loop/pipeline.py:2731-2732（execution_mode 未做白名单校验）：

  • execution_mode 取值仅依赖 run_evaluation 在第一次 baseline 评估时抛 ValueError，而此时 audit/evaluation_config.json、resolved_optimizer_config.json 已经写入磁盘。建议在 _validate_inputs 中把 execution_mode in {"trace","call_agent"} 一并纳入启动校验，使非法值在写任何产物前 fail-fast，与现有 optimizer_backend 校验保持对称。

总结

整体逻辑稳健：失败归因、AND gate、原子写回与 baseline 恢复、trace/call_agent 双路径均有对应测试覆盖，未发现安全或核心功能阻塞问题。仅 _validate_inputs 对 agent_optimizer 后端与 execution_mode 的前置校验不完整，建议补齐以保持 fail-fast 风格。

测试建议

• 建议补一条用例：optimizer_backend="agent_optimizer" 但配置缺 optimize 段时，run_pipeline 在启动阶段（未写 audit 产物前）即抛出明确 ValueError。
• 建议补一条用例：非法 execution_mode（如 "foo"）在 _validate_inputs 阶段被拒绝，而非延后到首次评估。

[mingri31164]

Follow-up review findings are addressed in 098149d:

• _validate_inputs now rejects unsupported execution modes before evaluation or audit writes
• the agent_optimizer backend now requires a non-empty optimize object up front
• the existing evaluate validation was tightened to require a non-empty object
• parameterized regression tests assert both failures occur before the audit/ directory is created

Re-verification: 18 focused tests, complete tests/evaluation, Flake8, YAPF diff, git diff --check, and fake/trace E2E all pass.