Skip to content

Evals: Migrate prompt iteration to async job polling #248

Description

@Ayush8923

Is your feature request related to a problem?
The recent change to the improve-prompt endpoint has disrupted the synchronous handling of prompt iterations. Clients now receive a job_id instead of the expected configuration details, leading to failures in the iteration process.

Describe the solution you'd like
Make the following enhancements:

  1. Create a new BFF proxy route at app/api/evaluations/[id]/improve-prompt/[jobId]/route.ts (GET) to poll the backend.
  2. Update the existing POST proxy at app/api/evaluations/[id]/improve-prompt/route.ts to pass through the 202 response with job_id.
  3. Revise handleIteratePrompt in app/(main)/evaluations/[id]/page.tsx to:
    • POST and read job_id, then initiate polling on a 2s interval until the status is no longer PENDING/PROCESSING.
    • On SUCCESS, retrieve config_id and version, clear cache, display success toast, and redirect appropriately.
    • On FAILED, show error_message in a toast.
    • On timeout, show error toast and clear loading state.
Original issue

Describe the current behavior

Context

Backend converted improve-prompt from a synchronous call to an async Celery job.

The endpoint no longer returns the new config directly. It now returns a job_id that the client must poll until the job completes.

Current (broken) behavior

handleIteratePrompt in app/(main)/evaluations/[id]/page.tsx POSTs to /api/evaluations/{id}/improve-prompt and expects a synchronous body { success, data: { config_id, id, version } }, then redirects to the prompt editor.

With the new backend this response is 202 { job_id, status, message, ... } — no config, no success flag — so prompt iteration no longer works.

Describe the enhancement you'd like

  1. New BFF proxy route app/api/evaluations/[id]/improve-prompt/[jobId]/route.ts
    (GET) forwarding to the backend poll endpoint via apiClient.
  2. Update POST proxy app/api/evaluations/[id]/improve-prompt/route.ts — pass
    through the 202 + job_id body unchanged (drop the sync success/data shape
    assumption).
  3. Rewrite handleIteratePrompt (app/(main)/evaluations/[id]/page.tsx):
    • POST → read job_id.
    • Poll GET on an interval (e.g. 2s, with max attempts / timeout) while status is PENDING/PROCESSING. Keep isImprovingPrompt loading state active during polling.
    • On SUCCESS: pull config_id + version from config_version (confirm ConfigVersionPublic field names), invalidateConfigCache(), success toast, redirect to /configurations/prompt-editor config=...&version=...&from=evaluations.
    • On FAILED: show error_message in an error toast.
    • On timeout: error toast, clear loading state.

Activity

  1. self-assigned this
    on Jul 15, 2026
  2. changed the title [-]Evals: Migrate prompt iteration to async job polling[/-] [+]Improve Prompt: Synchronous handling fix[/+] on Jul 15, 2026
  3. moved this to To Do in Kaapi-devon Jul 15, 2026
  4. changed the title [-]Improve Prompt: Synchronous handling fix[/-] [+][-]Evals: Migrate prompt iteration to async job polling[/+] on Jul 15, 2026
  5. changed the title [-][-]Evals: Migrate prompt iteration to async job polling[/-] [+]Evals: Migrate prompt iteration to async job polling[/+] on Jul 15, 2026
  6. moved this from To Do to In Progress in Kaapi-devon Jul 18, 2026
  7. moved this from In Progress to Closed in Kaapi-devon Jul 21, 2026
  8. github-actions commented on Jul 21, 2026

    @github-actions

    🎉 This issue has been resolved in version 0.4.0-main.2 🎉

    The release is available on GitHub release

    Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

  • Status
    Closed

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions