Repository navigation
Evaluation: Optimize Langfuse API calls #1022
Description
Is your feature request related to a problem?
Fetching evaluation results requires one Langfuse API call per trace, leading to ~80 calls for an 80-item run. This rapidly exhausts the rate limit on the Hobby plan, resulting in 429 errors and incomplete data.
Describe the solution you'd like
- Tag each trace with its run identifier upon creation.
- Use Langfuse's batch/list APIs to fetch results filtered by tag, reducing API calls from ~80 to ~2.
- Maintain current per-trace fetch as a fallback for older runs without tags.
Original issue
Is your feature request related to a problem?
When fetching evaluation results, we make one Langfuse API call per trace in a dataset run. A run with 80 items means ~80 API calls, and all evaluations in a project share the same rate-limit budget. On the Hobby plan this quickly hits the rate limit (429 errors), causing result syncs to fail or return partial data.
Describe the solution you'd like
Tag each trace with its run identifier when the trace is created, then fetch results using Langfuse's batch/list APIs filtered by that tag instead of fetching traces one by one. This reduces an 80-item run from ~80 API calls to ~2, staying well within rate limits even with multiple concurrent evaluations.
Notes:
- Older runs won't have tags, so keep the current per-trace fetch as a fallback for them.
Activity
[-]Batch-fetch evaluation traces from Langfuse using tags to avoid rate limits[/-][+]Evaluation: Optimize Langfuse API calls[/+]AkhileshNegi commentedon Aug 7, 2026
We are moving away from langfuse as required and having llm as judge in house so this is not needed