[nekomeowww]

Problem

A run page showed hybrid = (exact + judge) / 2: a number without a name that averaged two different questions. Other pages showed exact. For example, airi evals/tts-speech-formula recorded "the listener wrote the formula back" as exact and "the reading says the meaning" as judge, and the viewer showed their average.

There were only two score slots:

• A third quantity, such as latency, could not become a criterion.
• Benchmarks with several named rewards (Harbor reward.json, EvalScope metrics) could not be imported.

Result

Scores have names and definitions:

• One score per task is primary. Every page, the CLI, and the leaderboard show it with its label, next to the completion rate.
• A combined score is declared explicitly. Nothing is averaged unless the eval asks for it.
• A value can carry its rationale and its grader.

Comparison: afc74d7 (main, v0.9.0) → adcb7dd (head revision of nekomeowww/feat/named-scores).

The branch has five commits:

• 1f7c221 adds the create-pr agent skill, adapted from moeru-ai/auv.
• 6bb1d99 is the named-scores change.
• abf892f fixes the run summary. It counted a case that errored in its first auto attempt and passed in a later one as errored. That bug is on main too.
• 870a82f moves the per-case scoring rule into one function, countCase in core/scoring. scoreCases, cases.jsonl records, and the viewer all call it.
• adcb7dd removes vieval/core/processors/results, which nothing called.

API changes

Surface / change	Before	After	Caller action
recordScore — changed	context.recordScore(0.8), context.recordScore(0.6, 'judge')	context.recordScore(correct, 0.8, { rationale, grader }) with correct = defineScore('formula.correct', { label }) or a name string	Required: give every score a name
defineScore — added	Not available	defineScore(name, { label, description, primary, combine, errors, grader })	Optional
Derived scores — added	Not available (implicit hybrid)	defineScore('pass', { combine: { all: [correct, { score: natural, atLeast: 0.5 }] }, primary: true }), listed in describeTask(..., { scores: [pass] })	Optional
recordUnscored — added	Not available	context.recordUnscored(natural, 'grader_failed')	Optional
describeTask options — added	Not available	{ scores: ScoreDefinition[] } announces scores before the first case	Required for derived scores
Scoring state rule — changed	A passed case without recordScore scored exact: 1. An empty selection scored exact: 1.	No implicit score. A failed case counts 0, and errored or timed-out cases count by errors: 'exclude' | 'zero' (default exclude). An empty selection has no scores. The completion rate counts each case once, by its final outcome.	Read passed or a declared score instead of the implicit exact
TaskRunOutput — changed	{ coverage?: CaseCoverage, scores: RunScore[] }	{ completion?, passed?, primary?, scores: ScoreSummary[] }	Required for custom defineTask executors; build summaries with scoreCases from vieval/core/scoring
vieval/core/scoring — changed	scoreCaseOutcomes, hybridAverage, strictExactOf, CaseCoverage	scoreCases, deriveScores, poolScoreSummaries, resolveHeadline, emptyScoreSummary, ScoreSummary, HeadlineResult	Required if imported
vieval/core/runner aggregation — changed	exactAverage, judgeAverage, hybridAverage, strictExactAverage, coverage	scores: ScoreSummary[], completion, headline	Required if read
vieval/core/assertions — removed	AssertionOutcome.scoreKind, expectCustom({ scoreKind }), toRunScores(outcomes)	Outcomes keep score; record it with context.recordScore(definition, outcome.score)	Required if used
vieval/core/processors/results — removed	processRunResults(results, { threshold, maxFailedRuns })	None. No config field, CLI command, or code called it.	Gate on run-summary.json headline yourself if you used it

import { defineScore, describeTask } from 'vieval'

const correct = defineScore('formula.correct', { label: { 'en': 'Written back correctly', 'zh-Hans': '听写正确' }, grader: { kind: 'rule', id: 'grader' } })
const natural = defineScore('formula.natural', { label: { 'en': 'Natural reading', 'zh-Hans': '读法自然' }, grader: { kind: 'model', id: 'judge' } })
const pass = defineScore('formula.pass', { combine: { all: [correct, natural] }, primary: true })

describeTask('read-aloud', ({ caseOf }) => {
  caseOf('case', async (context) => {
    context.recordScore(correct, 1)
    context.recordScore(natural, 1, { rationale: 'says "f of x"', grader: { kind: 'model', id: 'judge', model: 'gpt-5.5', votes: 3, reducer: 'median' } })
  })
}, { scores: [pass] })

CLI changes

Command	Before	After
vieval run, project summary line	… <n> runs, cases …, hybrid <mean>, coverage <rate>	Pass 52% · completed 100% on its own line (with (n errored, n timed out) when some cases have no result); passed X% when no score is recorded; one line per task when the primary scores differ
vieval run, case counts, failure list, and resume hint	each case counted by its first case.ended: cases 304 passed | 24 failed | 8 errored, and the 8 cases offered for resume although a later auto attempt passed	each case counted by its last case.ended: cases 312 passed | 24 failed | 0 errored
vieval run (comparison leaderboard)	ranks by hybrid, prints hybrid … exact … coverage …	ranks by the primary score, with the completion rate in its own column; prints Primary scores differ (a, b), so the rows rank by passed cases. when needed
vieval report case-compare	--score-kind <kind> (default exact)	--score <name> (default: the primary score of the reports, or their only score; otherwise an error that lists the names)
vieval report attempts	COVERAGE column	COMPLETED column
Every report reader	read event schema 1–2	rejects older files with This report was written by an older vieval. Run it again.

vieval run
vieval report case-compare <left-report> <right-report> --score formula.pass

Report file and viewer API changes

File / route	Before	After
events.jsonl	schemaVersion: 2; score.recorded { kind: 'exact' | 'judge', score }	schemaVersion: 3; new score.defined { name, label?, description?, primary?, combine?, errors?, grader? }; score.recorded { name, value } or { name, unscored: { reason } }, with optional rationale, grader, metadata, derived
run-summary.json	no version; result.overall.{exactAverage, judgeAverage, hybridAverage, strictExactAverage, coverage}	schemaVersion: 2; top-level and per-project headline (HeadlineResult) and scoreDefinitions; result holds ScoreSummary lists
cases.jsonl	schemaVersion: 1; scores keyed by exact / judge	schemaVersion: 2; scores keyed by score name (the counted value), unscored, scoreDetails
metrics-summary.json	score buckets by kind	a ScoreSummary per score name
OTLP	CLI gauge vieval.score.exact, viewer gauge vieval.case.score.exact	vieval.score.<name> in both; task spans carry vieval.task.primary_score
Viewer contract (@vieval/ui/contract)	exact, judge, hybrid and *Average fields in about ten schemas	ScoreSummary, CaseScoring, HeadlineResult, CaseScores; scoreDefinitions, scoreRecords on case detail, scores on metrics, scoreGraders on tasks and comparison columns
Viewer runs of older versions	legacy source, rebuilt from cases.jsonl	unsupported source and state: listed, every route but GET /api/runs/:runKey answers 422

The viewer:

• shows the primary score with its label and the completion rate on every page
• has a Scores table with each score's formula, grader, and unscored reasons
• has one score badge per score on the case page, with rationale, grader, metadata, and a link to the grader's model calls
• puts scores in the metric panels
• warns when compared columns were graded by different kinds, models, or versions

Migration

• Name every recordScore call. Use defineScore for labels, the primary score, and derived scores.
• List derived scores in describeTask(..., { scores }).
• Custom defineTask executors return ScoreSummary[]. scoreCases([...outcomes], definitions) builds them.
• Run old evals again: older reports are not read.
• The docs (en and zh-hans), both READMEs, packages/ui/API.md, the examples, and the test projects are updated.

Validation

At adcb7dd:

• pnpm typecheck: passes.
• pnpm lint: 0 errors. Two warnings were there before this change.
• New regression test: packages/vieval/src/cli/run.test.ts, "counts a case that errored in one auto attempt and passed in a later one as passed". It fails at 6bb1d99 and passes at abf892f.
• pnpm test:run: all 661 tests pass in the last run.
  • packages/vieval/src/core/runner/collect.test.ts sometimes times out at 5 s under full-suite load. This PR does not touch it, and it passes on its own.
• pnpm build: passes.
• Real runs: packages/ui/test/projects/showcase was run twice with the built CLI.
  • The summary line reads Pass 52% · completed 100%.
  • The case line reads cases 312 passed | 24 failed | 0 errored.
  • The viewer pages were checked in a browser: home, project, run, Scores tab, and case page.

Sources

At adcb7dd:

• Event contract: packages/vieval/src/events/index.ts
• Scoring rule (countCase, scoreCases), derived scores, pooling, and headline: packages/vieval/src/core/scoring/index.ts and its tests
• DSL: packages/vieval/src/dsl/scores.ts and packages/vieval/src/dsl/task.ts
• Report files: packages/vieval/src/cli/report-records.ts and packages/vieval/src/cli/report-artifacts.ts
• Viewer contract: packages/ui/src/shared/contract.ts and packages/ui/API.md

🤖 Generated with Claude Code

[github-actions]

Approval required for deploying to Cloudflare Workers (Preview) for docs.

Name	Link
Waiting for approval	For maintainers, approve here