feat(examples): add eval-optimize-loop closed-loop optimization pipeline#104
feat(examples): add eval-optimize-loop closed-loop optimization pipeline#104woshidage77 wants to merge 53 commits into
Conversation
Implement 6-phase automatic optimization closed loop for tRPC-Agent: evaluation -> failure attribution -> prompt optimization -> regression verification -> acceptance gate -> audit trail. - Phase 1: Baseline evaluation with fake/real dual-mode support - Phase 2: 6-category failure attribution with 4-layer rule chain - Phase 3: attribution-driven prompt optimization (FakeOptimizer) - Phase 4: Candidate validation with delta matrix - Phase 5: Configurable acceptance gate (5 rules + overfit detection) - Phase 6: Audit trail with JSON + Markdown dual-format reports Includes 99 unit tests covering all phases and full pipeline integration. Fake mode enables complete pipeline execution without API keys. Related: Issue trpc-group#6 (犀牛鸟 2026)
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #104 +/- ##
==========================================
Coverage ? 87.84906%
==========================================
Files ? 482
Lines ? 45157
Branches ? 0
==========================================
Hits ? 39670
Misses ? 5487
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
I have read the CLA Document and I hereby sign the CLA |
…ator path - Replace NotImplementedError in optimizer._run_real with AgentOptimizer.optimize() call - Add validator._run_real using AgentEvaluator.evaluate_eval_set() with fallback - Add call_agent.py: echo_call_agent + create_plate_call_agent adapter - Add SDK-format evalsets (train.sdk / val.sdk) and optimizer.sdk.json - Add DESIGN.md (400-char design note) - Harden auditor: threading.Lock() + atomic writes (_atomic_write_text) + .tmp cleanup - Add --mode real-agent CLI option to run_pipeline.py - Keep fake mode intact as fast smoke-test fallback (99 tests pass)
AI Code Review已确认。现在让我写下审查结论。 发现的问题🚨 Critical
|
…, encoding, hashlib Critical: - Wrap Phase 1-6 in try/finally to prevent lock directory leak on exception Warnings: - Plumb --mode arg to BaselineRunner/OptimizationRunner/ValidationRunner - Fix overfit detection: derive candidate train scores separately - Replace builtin hash() with hashlib.sha256 for deterministic cross-process id - Replace image_id positional lookup with path-based case mapping - Fix garbled Chinese in BASE_PROMPTS, CATEGORY_OPTIMIZATION_HINTS, and prompt files Suggestion: - Read critical case ids dynamically from val.evalset.json critical field
AI Code Review发现的问题🚨 Critical
|
Critical: - Rewrite BASE_PROMPTS with readable English content (was all ? chars) - Rewrite CATEGORY_OPTIMIZATION_HINTS with readable English content - Fix SDK config JSONs (train.sdk, val.sdk, optimizer.sdk) with clean fields - Fix test assertions to check real section headers (Workflow, Preprocessing Guide, etc) Warning: - Remove unimplemented --mode trace from CLI choices 99 tests pass
AI Code Review我已经获取了所需的行号。注意 发现的问题🚨 Critical
|
…implify overfit detection CLI gate prevents cryptic NotImplementedError when --mode real or --mode real-agent is used. Fake mode remains the only active path with all 99 tests passing. Changes (AI review round 3): - Add explicit CLI rejection for real/real-agent modes - Remove dead real-agent OptimizationRunner branch - Simplify BaselineRunner/ValidationRunner to always use fake - Unify overfit detection to simulated approach (real branch was broken: reused BaselineRunner without optimized prompt)
AI Code Review基于对 发现的问题🚨 Critical
|
…arning items Critical: - Replace mkdir lock with PID-file lock (survives SIGKILL) - Expand fake-mode gate comments (clarify simulated values) - Add C3 comment on baseline _run_real_split placeholder Warnings: - Plumb run_mode through BaselineRunner/OptimizationRunner/ValidationRunner - Differentiate CANDIDATE_PREDICTIONS per failure category - Derive dominant_condition from real case.conditions (not hardcoded map) - Add FutureWarning when constructing real-mode runners - Add logging.exception in call_agent fallback path - Clarify gate majority strategy (strict majority, ties=reject) - Warn on _read_critical_case_ids fallback - Split long lines in auditor build_trail - Comment auditor latency_ms as fake-mode placeholder
AI Code Review现在让我确认一下测试中关于 对于 现在让我检查 现在让我检查一个真正的问题： 让我验证最具体的 bug。让我检查 现在让我检查 让我检查一个具体的关键 bug： 让我检查 JSON 报告写入器中 在 现在让我检查关键问题： 现在让我检查更令人担忧的安全项： 让我检查 现在让我看看我发现的最具体的正确性 bug。让我检查 让我检查 过拟合检测的真实 bug：在 让我检查 候选者预测中包含 现在让我检查最具体的 bug： 现在让我确认实际的问题。让我重新审视 让我检查 现在让我评估 让我验证一个真正的问题： 让我检查 现在让我检查最重要的一项 —— 现在让我检查 让我检查我标记的一个具体正确性 bug： 让我检查 现在让我确认一个真正的 bug。重新阅读 让我最后检查一下 让我检查 让我检查最后一个真正的问题 —— 让我验证 在 fake_model.py 中，case_id 提取存在 bug： 现在让我检查真正可操作的安全/正确性问题。让我重新审视 让我完成审查。让我检查一个我可能遗漏的更重要的正确性 bug —— 让我验证 让我验证 让我检查 现在让我确认真正的发现。让我更仔细地检查 现在让我看看是否有任何关键问题。让我检查 基于我的分析，让我编写审查报告。关键问题很少；主要是一些警告和建议。让我确定： 警告发现：
让我验证 #6 是否真的有影响 —— 审计通过 让我检查 让我验证 让我确定并撰写审查报告。让我再次检查 #5 —— 回退到 让我验证一个更具体的 bug： 现在让我撰写审查报告。 发现的问题🚨 Critical
|
…arning + 1 suggestion Critical: - _read_critical_case_ids: return [] on error instead of guessing [val_001] Warnings: - PID lock: only remove lock if our PID owns it (TOCTOU mitigation) - _make_candidate_id: remove time.time(), use deterministic hash+iteration - sys.path: pop before re-raising ImportError in baseline _run_real_split - Gate overfit: note fake-mode simulation in check description - candidate_train_scores: documented as fake placeholder Suggestion: - BaselineCaseResult.to_dict: add trajectory field
AI Code Review发现的问题🚨 Critical
|
…arning + 1 suggestion Critical: - _pid_alive: fix Linux dead-PID false-positive (ctypes.windll AttributeError was caught by blind except Exception, returning True for all dead processes. Now: ProcessLookupError -> dead, PermissionError -> alive, platform-gated Windows ctypes branch, Unix returns os.kill result) Warnings: - --max-iter: plumb args.max_iter to OptimizationRunner config - Lock file: use output_dir instead of hardcoded BASE_DIR/output - test_knowledge_recall_from_trajectory: assert exact category - test_param_error_from_trajectory: assert priority-based classification - baseline image_id hash: retain C3 placeholder comment (no code change) Suggestion: - auditor: rename total_latency_ms -> avg_latency_ms (semantic fix)
AI Code Review现在我已经有足够的信息来撰写审查报告了。让我整合一下我的发现。 发现的问题🚨 Critical
|
…arning + 1 suggestion
Critical:
- PID lock: atomic write via temp file + fsync + os.replace (was non-atomic
open('w') without flush, partial writes on crash)
- real mode PlateEvaluator: ACK via reply, docstring already marks PLACEHOLDER
Warnings:
- --max-iter: change default from 3 to None, use is not None check
- gate all_must_pass: reject when checks list is empty (was all([])=True)
- test_knowledge_recall: assert exact category instead of loose in() + pass
- REGRESSION_PREDICTIONS: differentiate val_002/val_003 from baseline
- baseline image_id: retain existing C3 comment (no code change)
Suggestion:
- Remove unused imports: time from optimizer, BaselineResult/AttributionReport
from auditor, unused AttributionCluster/CATEGORY_META from optimizer
…ne (R27 Suggestion 2)
…th save() JSON path
- _generate_md now uses baseline.get(name) or {} default, consistent with
save() which writes {} for None baseline values (R27 Warning 2)
AI Code Review我已审查 发现的问题
|
- Remove 'or {}' from baseline.get(name) which made 'if r is None' dead code
and would crash with AttributeError on r.summary if key truly missing.
- Fix comment indentation to match loop body scope.
- Warning 1 (real mode pass semantics) noted but deferred: real mode blocked
at CLI; API-reachable path is documented limitation.
AI Code Review我将停止重试 compile 命令（它一直被阻止），并继续进行我已经通过直接读取文件所收集到的分析。基于我对 diff 和相关上下文的全面审查，我现在已有足够的信息来出具审查结论。 发现的问题🚨 Critical
|
…ncoding - [Critical] test_gate.py: pass lock_path to release_pipeline_lock (POSIX now deletes lock file; no stale residue in tmp_path) - [Critical] baseline.py: unify test_plates -> test_images (match call_agent._resolve_image_path directory assumption) - [Warning] evalset JSON: fix garbled description fields (???? -> English)
AI Code Review验证已完成。测试 发现的问题🚨 Critical
|
- attribution.py L207: Rule 4 now only fires when candidates is empty (true fallback), restoring multi-category classification. Previously final_answer_mismatch (priority 1) always won over trajectory/judge signals. - test_attribution.py: test_unattributed_case now constructs correct=True with no signals, asserts category==unattributed (was always-true assertion). test_param_error_from_trajectory updated for new Rule 4 semantics. - test_gate.py L227: subprocess uses sys.executable instead of 'python'. - test_validator.py L144: split 'or' assertion into two distinct assertions for improved/score_delta to prevent near-tautology.
AI Code Review我已经完全了解情况了。让我来写审查意见。 发现的问题🚨 Critical
|
…k makedirs guard - Delete docs/agent-memory.md: contained internal PR competition data, reviewer names, local absolute paths — not suitable for open source repo. - lock.py L24: guard _os.makedirs() against empty dirname when lock_path is a bare filename (edge case; current callers always pass absolute paths).
AI Code Review基于对 发现的问题
|
…own + Win32 retry - lock.py _release_flock: stop removing lock file after close. Removing after close creates a window where another process can flock the same inode, then the remove deletes it, and a third process creates an independent flock at the same path — two concurrent holders break mutual exclusion. - lock.py _acquire_pid_lock_win32: retry after corrupt lock cleanup instead of returning None (exit 75). Single corrupt file no longer blocks CI; pipeline runs normally after recovery. - gate.py _check_critical_cases: detect 'unknown' critical ids (absent from both baseline and candidate, e.g. config drift) and fail-closed. Extracted _fmt_critical_detail to module-level helper for readable three-category detail strings. - test_gate.py: corrupt-lock assertions updated rc==75 -> rc==0 (retry now succeeds after cleanup).
AI Code Review基于对 发现的问题
|
… config copy - lock.py _acquire_flock: remove O_TRUNC from open flags. A contended process would truncate the holder's diagnostic PID/timestamp before flock failed. Now uses ftruncate+lseek only after flock succeeds. write/fsync wrapped in try/except to close fd on exception. - lock.py _acquire_pid_lock_win32: replace unbounded recursion with _retry param (max 2 retries after cleanup) to prevent stack overflow on persistent lock corruption. - run_pipeline.py L109: copy pipeline config dict before mutating random_seed to avoid polluting load_config()'s cached object.
…timizer.py Line 222 had 16-space indent (method-body + 2 levels), placing an orphaned comment between return and priority_queue. Moved to 8-space method-body level with correct description.
AI Code Review现在让我更仔细地检查 现在检查一个更严重的问题： 让我检查一下门控（gate）的 现在我已经有足够的信息了。让我再确认一下： 让我再看看 让我写下审查意见。 发现的问题🚨 Critical
|
Critical: baseline.py _run_real_split hard key access -> .get() with KeyError, replace bare assert with RuntimeError for ground_truth assign. Warning: gate.py no_new_hard_fail skips cases not in baseline (None sentinel instead of 1.0 default) to avoid false-positive new-hard-fail. Warning: run_pipeline.py baseline_cost now uses total_cost_baseline from validator instead of avg_cost*total for consistent comparison. Warning: lock.py document Windows stale-lock takeover TOCTOU race between _pid_alive and os.replace; clarify post-facto detection limitation. Warning: validator.py char_correct calculation clarified with comment re consistency with fake_judge._char_match_score denominator. Suggestion: optimizer.py OptimizationRunner.__init__ validates max_iterations >= 1 to prevent empty priority_queue[:0] downstream. Tests: updated test_skips_new_case_not_in_baseline to match new gate semantics. 106 tests pass, pipeline 6-phase E2E OK.
…Check All branches return GateCheck (never None), so the Optional wrapper and the 'if cost_check is not None' guard in decide() were dead code. Also removes now-unused typing.Optional import.
AI Code Review我已经完整审查了 发现的问题
|
…l-read test Warning: gate.py _check_critical_cases description '\u95ee\u95ee case \u95ee\u95ee\u95ee\u95ee\u95ee\u95ee' -> '\u5173\u952e case \u4e0d\u9000\u6b65', fixes garbled text in audit JSON/markdown reports consumed by downstream tools. Suggestion: auditor.py avg_latency_ms comment clarifies per-case avg semantics instead of misleading 'per-entry average (renamed from...)'. Warning: add test_critical_read_failure_rejects to test_gate.py verifying the run_pipeline.py fail-closed override path when _read_critical_case_ids returns None (evalset unreadable).
AI Code Review发现的问题🚨 Critical未发现明确的 Critical 阻塞问题。
|
…mments Warning: extract AcceptanceGate.fail_closed_for_unreadable_evalset() so test_critical_read_failure_rejects exercises the same code as run_pipeline.py production path instead of a logic copy. Warning: _read_critical_case_ids now returns (ids, error_msg) tuple; exception type/message flows into gate detail via the helper instead of being consumed by bare except+print. Warning: add TestPipelineFakeE2E with 4 subprocess smoke tests: exits_zero, produces_report_json, produces_audit_dir, rejects_broken_evalset. All run on both POSIX and Windows. Suggestion: restore 38 garbled \u2018????' comments across optimizer.py, test_optimizer.py, test_validator.py with readable English equivalents. 111 tests pass, pipeline 6-phase E2E OK.
AI Code Review我已经审查了完整的 发现的问题
|
Warning: attribution Rule 2 substring matching 'failed' in search_text would match 'deblur_failed' in preprocess steps, causing false-positive tool_call_error attribution. Fix: check individual raw_steps and exclude preprocess-stage failures from tool-error matching. Suggestion: mark FakeLLM/FakeLLMResponse export in fake/__init__.py as legacy (not used by current pipeline, baseline reads FAKE_PREDICTIONS directly). Retained for external reuse.
AI Code Review发现的问题💡 Suggestion
总结未发现明显阻塞问题。该 PR 为 eval-optimize 流水线示例（fake 模式），核心逻辑（归因、优化、验证、gate、锁、审计）经过多轮评审、边界处理与 fail-closed 设计较为完备，测试与配置内部自洽。仅存两条维护性层面的注释/语义不一致，不影响当前正确性。 测试建议暂无额外测试建议；现有测试已覆盖 fake 全链路、gate 各规则、锁生命周期与 fail-closed 路径。如后续启用 real 模式，需补充 cost/overfit 真实数据路径的覆盖。 |
3 participants
Implement 6-phase automatic optimization closed loop for tRPC-Agent: evaluation -> failure attribution -> prompt optimization -> regression verification -> acceptance gate -> audit trail.
Includes 99 unit tests covering all phases and full pipeline integration. Fake mode enables complete pipeline execution without API keys.
Related: Issue #6 (犀牛鸟 2026)