Repository navigation
Evaluation: Automate prompt improvement #1159
Description
Is your feature request related to a problem?
Improving a config's prompt requires a manual evaluation loop, which is time-consuming and inefficient. Currently, there's no automated way to handle this cycle and receive a report upon convergence.
Describe the solution you'd like
- Implement
POST /api/v2/evaluations/iterationsto automate the evaluation and prompt improvement process via a LangGraph state machine. - Define stopping criteria based on mean scores of
Adherence to Ground TruthandAdherence to Prompt, with visibility tracking forAdherence to Knowledge Base. - Establish conditions for stopping after 3 consecutive rounds under the ceiling-delta threshold, reaching a maximum round cap, or encountering a hard failure.
- Store round-by-round state in a Postgres LangGraph checkpoint, with minimal bookkeeping for the
evaluation_iteration_runtable. - Utilize the existing
/cron/evaluationstick for resumption instead of a new scheduler. - Deliver a final report and best round results to the caller's
callback_urlfollowing the same convention as prompt-improvement callbacks.
Original issue
Problem
Improving a config's prompt today is a manual loop: run an eval, read scores, call improve-prompt, wait, kick off another eval, repeat. There's no way to hand the whole cycle to Kaapi and get back a report once it converges.
Proposal
Add POST /api/v2/evaluations/iterations to run a self-driving loop that chains fast-eval and v2 prompt improvement via a LangGraph state machine:
start_eval → wait_eval → (conditional) start_improve → wait_improve → loop back to start_eval, or → finalize.
- Stop score = mean of
Adherence to Ground Truth+Adherence to Prompt;Adherence to Knowledge Baseis tracked per round for visibility only and never gates stopping. - Stops on 3 consecutive rounds under the ceiling-delta threshold (
ceiling_reached), amax_roundscap (max_rounds_reached), or a hard round failure (round_failed). - Round-by-round state lives in a Postgres LangGraph checkpoint (
thread_id = str(iteration_run.id)); theevaluation_iteration_runtable is thin bookkeeping only (status/stop_reason/dataset/config/callback). - Resumption piggybacks on the existing
/cron/evaluationstick rather than a new scheduler — no orchestrator polls a provider directly. - Final round-by-round report + best round delivered to the caller's
callback_url, same best-effort convention as prompt-improvement callbacks.
Notes
Full design detail: docs/wiki/modules/evaluations.md (Async section) and docs/architecture/kaapi-evaluations-ARCHITECTURE.md.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsClosed
Activity
[-]Automated eval → improve-prompt → eval iteration loop[/-][+]Evaluation: Automate prompt improvement[/+]github-actions commentedon Aug 26, 2026
🎉 This issue has been resolved in version 1.5.0-main.11 🎉
The release is available on GitHub release
Your semantic-release bot 📦🚀