Repository navigation
Evaluation: Add Gemini model support #879
Description
Is your feature request related to a problem?
Text evaluations currently only support OpenAI models, leaving Gemini models unsupported for this purpose. This limitation restricts organizations from assessing the accuracy of Gemini models against their datasets.
Describe the solution you'd like
- Enable text evaluations for Google Gemini models
- Ensure compatibility with the existing dataset format, scoring, and workflow used for OpenAI evaluations.
Original issue
Describe the current behavior
A clear description of how it currently works and what the limitations are.
Text evaluations only support OpenAI. Gemini models cannot be used for text evaluation.
Describe the enhancement you'd like
A clear and concise description of the improvement you want to see.
Allow text evaluations to run against Google Gemini models, using the same dataset format, scoring, and workflow as OpenAI evaluations.
Why is this enhancement needed?
Explain the benefits (e.g., performance, usability, maintainability, scalability).
Organizations need to evaluate Gemini model accuracy against their golden datasets — whether to compare against OpenAI or to validate Gemini outputs independently.
Additional context
Add any other context, metrics, screenshots, or examples about the enhancement here.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
[-]Evaluation: Gemini Eval Integration (TEXT)[/-][+]Evaluation: Add Gemini model support[/+]