Skip to content

Evaluation: Speed up smaller runs #741

Closed

Description

@AkhileshNegi

Is your feature request related to a problem?
The current evaluation process using batchAPI takes up to 24 hours to complete, which slows down our ability to iterate on prompts. Although this method is cost-effective and requires less infrastructure, faster evaluations are needed to enhance workflow.

Describe the solution you'd like

  • Implement evaluations for smaller sizes (like 50 or 100) using the response API for quicker results.
  • Continue using batchAPI for larger evaluations to maintain cost efficiency.

Activity

  1. changed the title [-]Evaluation: faster runs for smaller evaluation[/-] [+]Evaluation: Faster response API evaluations[/+] on Apr 9, 2026
  2. changed the title [-]Evaluation: Faster response API evaluations[/-] [+]Evaluation: Speed up smaller runs[/+] on Apr 9, 2026
  3. self-assigned this
    on Apr 9, 2026
  4. moved this to In Progress in Kaapi-devon May 8, 2026
  5. linked a pull request that will close this issueEvaluation: Fast Evals #870on Jun 3, 2026
  6. moved this from In Progress to Closed in Kaapi-devon Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

  • Status
    Closed

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions