Repository navigation
Evaluation: Category-wise analytics for Evals #844
Description
Is your feature request related to a problem?
We currently lack the ability to provide category-wise analytics for Evals. This limits users' insights into evaluation performance and makes it harder to track metrics by category.
Describe the solution you'd like
- Allow users to add categories to golden QnA in their CSV file uploads.
- Show Evals metrics grouped by category, including:
- Total eval count per category
- Average cosine score
- Average correctness score
- Include the category in the raw data, staying consistent in CSV exports.
- Implement for text-based evals only with flexible free text categories.
- Ensure that missing categories do not disrupt the process, using a default or leaving them blank.
Initiative 1: Category wise analytics
User side action (i/p): User will be able to add category to the golden QnA in the input csv file they upload
System side behaviour
System will show Evals metrics grouped as per category
Metrics to show
Total eval count per category
Avg cosine score
Avg correctness score
System will also show category in the raw data, even in CSV file
Scope:
Only text based evals for now
Category will be free text as of now
If no category added, then it should not break the flow, you can keep it as blank or create default other category
Open Questions:
Should we limit no of categories that can be input?
Should we also limit category text length ?
E.g
Input
| id | category | question | expected_answer |
|---|---|---|---|
| 1 | Health | What are symptoms of anemia? | Fatigue, weakness and low hemoglobin are common symptoms. |
| 2 | Education | How to apply for scholarship? | Submit application form with required documents online. |
| 3 | Livelihood | What is a self help group? | A community-based financial support group, usually for women. |
| 4 | Health | Can pregnant women take iron tablets daily? | Yes, as prescribed by a doctor or health worker. |
| 5 | Livelihood | What documents are needed for job card registration? | Aadhaar card, address proof and identity documents. |
Activity
[-]Evals insights layer - Category wise insights[/-][+]Evaluation: Category-wise analytics for Evals[/+]