Repository navigation
Prompt Iteration: Asynchronous Anthropic call #1018
Description
Is your feature request related to a problem?
The prompt-iteration endpoint (POST /{evaluation_id}/improve-prompt) runs a blocking, non-streaming Anthropic call synchronously inside the request. Because the handler is a sync def, FastAPI runs it via run_in_threadpool (Starlette/anyio, default 40 threads), so the full LLM round-trip occupies one of those shared threads for its entire duration. It does not freeze the async event loop, but enough concurrent calls exhaust the threadpool and queue other sync requests — degrading responsiveness under load.
Describe the solution you'd like
- Ensure the endpoint returns quickly while the improvement process completes asynchronously.
- Consider reusing existing infrastructure like the
Jobmodel and Celery task pattern.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
[-]Move blocking Anthropic call in improve-prompt off the request path[/-][+]Endpoint Improvement: Asynchronous Anthropic call[/+][-]Endpoint Improvement: Asynchronous Anthropic call[/-][+]Prompt Iteration: Asynchronous Anthropic call[/+]github-actions commentedon Jul 17, 2026
🎉 This issue has been resolved in version 1.4.0-main.5 🎉
The release is available on GitHub release
Your semantic-release bot 📦🚀
github-actions commentedon Jul 24, 2026
🎉 This issue has been resolved in version 1.4.0-main.1 🎉
The release is available on GitHub release
Your semantic-release bot 📦🚀