Repository navigation
feat(evaluation): Improve iteration loop efficiency - #1101
Conversation
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
changed the title
…eration-loop # Conflicts: # backend/app/services/evaluations/fast.py # docs/wiki/modules/evaluations.md
OpenAPI changes ⚪ No API surface changesNote This PR does not modify the API contract.
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
| {"finalize_node": "finalize_node", "start_improve_node": "start_improve_node"}, | ||
| ) | ||
| graph.add_edge("start_improve_node", "wait_improve_node") | ||
| graph.add_edge("wait_improve_node", "start_eval_node") |
There was a problem hiding this comment.
wait_improve_node currently has an unconditional edge, so round_failed never reaches finalize_node and the webhook doesn't fire. Reuse route_after_eval here, mapping the continue branch to start_eval_node, so failures finalize correctly and both wait nodes follow the same pattern.
graph.add_conditional_edges(
"wait_improve_node",
route_after_eval,
{"finalize_node": "finalize_node", "start_improve_node": "start_eval_node"},
)
| ) | ||
|
|
||
|
|
||
| def _mark_iteration_run_failed( |
There was a problem hiding this comment.
here mark the row FAILED but never hit the callback_url, caller just waits forever. row already has the url, can we POST a failure report from here too?
| so a caller passing an oversized value degrades to the safety cap instead of | ||
| failing outright. | ||
| """ | ||
| resolved_max_rounds = min( |
There was a problem hiding this comment.
Silently capping max_rounds at 25 is surprising when the caller asks for 100. Since each round costs money, we should reject invalid values instead. Add le=EVAL_ITERATION_MAX_ROUNDS_HARD_CAP to EvaluationIterationCreateRequest so FastAPI returns a 422 and the limit is reflected in OpenAPI.
just nitpick
|
🎉 This PR is included in version 1.5.0-main.11 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
2 participants
Issue
Closes #1159
Summary
improve-prompt, then re-running the eval — repeated by hand each round.POST /api/v2/evaluations/iterationskicks off a self-driving eval → improve-prompt → eval loop via a LangGraph state machine (start_eval_node→wait_eval_node→start_improve_node→wait_improve_node, looping back or finalizing), returns202immediately, and delivers the round-by-round report tocallback_urlonce it stops.Adherence to Ground Truth+Adherence to Prompt;Adherence to Knowledge Baseis recorded per round for visibility only and never gates stopping. Stops on 3 consecutive low-delta rounds (ceiling_reached), amax_roundscap (max_rounds_reached), or a hard round failure (round_failed).evaluation_iteration_runtable is thin bookkeeping only (status/stop_reason/dataset/config/callback) — round-by-round state lives in a Postgres LangGraph checkpoint keyed bythread_id = str(iteration_run.id)./cron/evaluationstick (dispatch_pending_evaluation_iteration_resumes) rather than a new scheduler; no orchestrator polls a provider directly.082_add_evaluation_iteration_run.pyadds the new table;docs/wiki/modules/evaluations.mdanddomain-map.mdupdated per the maintenance rule.Checklist
Before submitting a pull request, please ensure that you mark these task.
fastapi run --reload app/main.pyordocker compose upin the repository root and test.Notes
Design detail:
docs/wiki/modules/evaluations.md(Async section) anddocs/architecture/kaapi-evaluations-ARCHITECTURE.md.