Skip to content

Evaluation: Insights layer #843

Closed
1 of 1 issue completed
Closed

Description

@PritamSGB

Objective:
The overall objective is to help non profit users to get more actionable, data driven insights from Eval runs that can help them to further optimise the prompt. Few initiatives listed below

Initiative 1: Enable users to analyze eval performance across thematic categories (e.g PII handling, Hinglish handling, corner case, hallucination checks, Facts handling etc) to identify weak areas and improve prompts/models faster accordingly. This was specific request from Sneha, VG etc.
Github ticket

Initiative 2 (long term): Use AI model to recommend prompt changes. One of the input for this can be category wise scoring as part of initiative 1

PRD Document: https://docs.google.com/document/d/1Z5mRQIS4rli-vWGT31-O93uFn1o_CFrWOA4oMO6uw9M/edit?tab=t.h9hid11svst#heading=h.joj9q8ysj80v

Activity

  1. moved this to To Do in Kaapi-devon May 15, 2026
  2. changed the title [-]Evals Insights layer[/-] [+]Evaluation: Thematic category analysis[/+] on May 15, 2026
  3. changed the title [-]Evaluation: Thematic category analysis[/-] [+]Evaluation: Insights layer[/+] on May 15, 2026
  4. moved this from To Do to In Progress in Kaapi-devon Jul 13, 2026
  5. linked a pull request that will close this issueEvaluation: Prompt Generation #966on Jul 13, 2026
  6. moved this from In Progress to Closed in Kaapi-devon Jul 13, 2026
  7. closed this as completedby moving to Closed in Kaapi-devon Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

sub-parentChild of parent label for roadmap view

Projects

  • Status
    Closed

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions