diff --git a/.agents/skills/create-pr/SKILL.md b/.agents/skills/create-pr/SKILL.md
new file mode 100644
index 0000000..fb98a08
--- /dev/null
+++ b/.agents/skills/create-pr/SKILL.md
@@ -0,0 +1,146 @@
+---
+name: create-pr
+description: Prepare or update vieval pull requests with source-backed disclosures of CLI, eval DSL / package export, and report / viewer API changes, before-and-after usage, and migration notes. Also use for retrospective API audits of specified PRs or revisions.
+---
+
+# Create PR
+
+<!-- Adapted from moeru-ai/auv `.agents/skills/create-pr`: the AUV surfaces (Rust APIs, Protobuf / Runner) are replaced
+     by the surfaces vieval exposes (CLI, eval DSL and package exports, report files and the viewer HTTP API). -->
+
+Make the PR useful to both reviewers and downstream callers: eval authors, report readers, and tools that read
+vieval's files. Explain the final behavior, show exact calling changes, and preserve enough source context to audit
+those claims later. Follow the repository's AGENTS.md.
+
+## Establish the comparison
+
+- Identify the repository, PR or branch, base, and head. Record resolved commit
+  SHAs in the disclosure. Use the actual PR diff, not every change since `main`.
+- For an open PR, compare its merge base with its head. For a merged or historical
+  change, identify the reviewed or landed revision range and state which one you
+  used. Do not infer historical behavior from today's working tree. A commit in
+  Git is not proof of merge or release; use “head revision” unless that status
+  was verified.
+- For stacked PRs, distinguish changes introduced here from changes inherited
+  from the base. State dependencies and merge order. Recheck the base after an
+  upstream merge; do not claim independence merely because a PR has its own URL.
+- Start with changed files, then inspect definitions, exports, and real callers
+  at both revisions. Use `git show`, `git diff`, and targeted `rg` searches.
+  PR prose and commit messages are context, not proof of an API contract.
+
+## Inventory caller-visible changes
+
+Trace each changed contract across the layers that actually expose it: the eval DSL
+(`describeTask`, `caseOf`, `recordScore`, `defineMetric`, ...), package exports in
+`packages/vieval/package.json` (`vieval`, `vieval/config`, `vieval/core/*`, `vieval/events`, ...), config types,
+the CLI, the report files, and the viewer (`@vieval/ui`). Do not scan unrelated modules or invent obligations for
+untouched layers.
+
+Always resolve these two surfaces explicitly for a caller-visible PR, even when
+one is unchanged:
+
+- **CLI:** inspect command definitions and dispatch (`packages/vieval/src/cli`), then show the exact command
+  line before and after. Include subcommands, flags, defaults, terminal output that people read (summary lines,
+  leaderboards), exit codes, and stdout/stderr behavior when relevant.
+  A prose feature name such as “adds report support” is not a CLI disclosure.
+- **Report files and viewer API:** inspect `events.jsonl` (`RUN_EVENT_SCHEMA_VERSION`, event names and payload
+  schemas in `packages/vieval/src/events`), `run-summary.json`, `cases.jsonl`, `metrics-summary.json`, `otlp/*`,
+  and the viewer HTTP contract (`packages/ui/src/shared/contract.ts`, `packages/ui/API.md`). Show exact event,
+  field, schema version, and route changes, and whether older files are still read. If the diff does not change
+  this surface, say **No report file or viewer API changes**; do not omit the surface or infer a file change from
+  CLI behavior.
+
+Classify findings as added, removed, changed, or internal-only. Include:
+
+- Exported functions, types, fields, and re-exports.
+- Parameter order and types, required arguments, defaults, options,
+  return values, error messages, and serialization shapes.
+- Accepted values, aggregation and scoring rules, and file layouts when existing evals or reports behave
+  differently.
+
+Check these distinctions before describing a migration:
+
+- A new path or re-export is not a removal if the old path still exists.
+- A rename of an unexported helper is not a public API break.
+- A new public type does not mean the functions that run evals accept that type. Read their
+  parameter types; show actual accepted inputs, not an intended future API.
+- Source compatibility, file compatibility, and runtime behavior are different.
+  An optional payload field keeps old logs readable; a renamed or retyped one needs a schema version bump.
+  A reader that ignores a new field does not implement the behavior it asks for.
+- The ability to parse or compile an input does not prove it is scored or shown.
+- An unchanged signature can require migration when defaults, scoring semantics,
+  file versions, or what the viewer reads change.
+
+Report removals explicitly, including when none were found in the inspected
+public surfaces. Mention unchanged APIs only when readers could reasonably
+mistake them for removed or changed APIs. Do not list every untouched function.
+
+## Write the disclosure
+
+Lead with the concrete problem and resulting behavior. Use an English
+Conventional Commit title with the relevant scope (`feat(vieval): ...`, `feat(ui): ...`). Use `chore(docs): ...` for
+primarily documentation changes. Keep committed PR text in clear, simple English unless the user requests another
+language.
+
+For API changes, include a Markdown table with side-by-side inline code:
+
+| Surface / change | Before | After | Caller action |
+|---|---|---|---|
+| DSL method — changed | Exact old call | Exact new call | Required migration |
+| Public type — added | Not available | Exact construction or use | Optional use or new requirement |
+| Export — removed | Exact old import | Supported replacement, or none | Required migration |
+
+Replace the illustrative rows with verified findings; omit categories with no
+rows. Put short code in backticks inside cells. Escape pipe characters as needed
+for Markdown tables. Put longer compilable examples in fenced blocks below the
+table and refer to them from the relevant row. Do not put fenced blocks inside
+table cells. Define setup when a snippet would otherwise be ambiguous. Label partial snippets and placeholders; do
+not imply they were run.
+
+For caller-visible PRs, give CLI and report files / viewer API their own headings in the
+published body. Under **CLI changes**, use a before/after table plus copyable
+command examples (and the changed output lines) for every added or changed command family. Use `Not available`
+for a genuinely new command rather than hiding the before state. Under
+**Report file and viewer API changes**, use the same before/after form for file or
+route changes, or the explicit unchanged statement required above. Other DSL or export changes may follow in the
+general API table. Do not substitute source links or a list of command names for concrete caller syntax.
+
+For viewer UI changes, add screenshots of the changed pages when the user asks for them or they were taken while
+verifying the change.
+
+Then add only the migration notes that matter:
+
+- List added APIs with usable examples, and removed APIs with replacements.
+  The comparison table can serve as this inventory; do not duplicate it.
+- State required edits versus optional adoption. Preserve meaningful before/after behavior differences.
+- Name package version requirements when supported by the code. Do not
+  invent released versions or compatibility negotiation.
+- Link each group of claims to commit-pinned source definitions, callers, tests,
+  or durable contract notes. Include both revisions when explaining a removal or
+  signature change. Local revision/path references are acceptable for offline
+  drafts; replace them with permalinks when publishing to GitHub.
+- Report validation commands and outcomes at the tested revision (`pnpm typecheck`, `pnpm lint`,
+  `pnpm test:run`, builds, real runs, viewer checks). Distinguish compilation, automated tests, real runs, and
+  pending CI. Do not claim examples were executed when only their signatures were inspected.
+
+For changes with no caller-visible impact, use a short explicit statement and
+relevant validation instead of an empty table. For a retrospective audit, produce
+the same disclosure with the historical comparison range; do not create a new PR
+or rewrite old PRs unless requested.
+
+## Deliver within the requested scope
+
+Before publishing, compare the final diff with the table and migration notes.
+Update obsolete rows when scope changes; do not append contradictory follow-ups.
+Confirm that the final body contains the **CLI changes** and **Report file and viewer API changes** headings and
+that their claims match the inspected definitions.
+Use a structured body argument or `gh ... --body-file` to preserve code and
+newlines. Read the published body back to verify it reflects the intended text.
+
+Creating or updating a PR must follow the user's existing authorization. A draft
+or audit request alone does not authorize publishing, merging, or changing other
+PRs. When publication is already requested, complete the preparation and publish
+without an extra confirmation. Keep unrelated working-tree changes out of the PR.
+The skill does not require raw logs, screenshots, or temporary probe scripts to
+be committed; concise validation results and source links are sufficient unless
+the task specifically requires those artifacts.
diff --git a/.agents/skills/create-pr/agents/openai.yaml b/.agents/skills/create-pr/agents/openai.yaml
new file mode 100644
index 0000000..6c6c942
--- /dev/null
+++ b/.agents/skills/create-pr/agents/openai.yaml
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Create PR"
+  short_description: "Disclose CLI, DSL, and report changes and migration steps in PRs"
+  default_prompt: "Use $create-pr to prepare a PR with API changes, before-and-after examples, and migration notes."
diff --git a/README.md b/README.md
index 18e4653..b7cbba8 100644
--- a/README.md
+++ b/README.md
@@ -140,6 +140,33 @@ describeTask('arithmetic-quality', ({ casesFromInputs }) => {
 })
 ```
 
+Declare the scores a task measures with `defineScore`. Each score is a named value in `0..1` where higher is better; one score per task is `primary` and becomes the headline, shown next to the completion rate. A derived score (`combine: { all | mean | weighted }`) is computed by the runtime when the case try settles:
+
+```ts
+import { caseOf, defineScore, describeTask } from 'vieval'
+
+const correct = defineScore('formula.correct', { label: { 'en': 'Correct', 'zh-Hans': '正确' } })
+const natural = defineScore('formula.natural', { grader: { id: 'judge', kind: 'model' } })
+const pass = defineScore('formula.pass', {
+  combine: { all: [correct, { atLeast: 0.5, score: natural }] },
+  label: 'Pass',
+  primary: true,
+})
+
+describeTask('read-formula', () => {
+  caseOf('fraction', async (context) => {
+    const { listenerScore, verdict } = await readAloud('1/2')
+    context.recordScore(correct, listenerScore)
+    if (verdict == null)
+      context.recordUnscored(natural, 'invalid_response')
+    else
+      context.recordScore(natural, verdict.score, { rationale: verdict.reason })
+  })
+}, { scores: [correct, natural, pass] })
+```
+
+A failed case counts `0` in every score. An errored or timed-out case is left out (`errors: 'exclude'`, the default) or counts `0` (`errors: 'zero'`), and lowers the completion rate either way. An unscored value never counts. Latency and cost are metrics (`defineMetric` with `better: 'lower'`), not scores.
+
 `describeEval` remains exported as an alias of `describeTask`, but new examples should prefer `describeTask` because task/case semantics are the primary runtime model.
 
 Every `describeTask` call in an eval file is its own task:
@@ -182,7 +209,8 @@ Each scheduled task receives stable matrix metadata:
 - `project`: the complete unit of work: discovery rules (`include`, `exclude`, `root`), `runMatrix` / `evalMatrix`, `models`, `inferenceExecutors`, optional `executor`, `plugins`, `reporters`, and `scheduler`.
 - `experiment`: run metadata derived from explicit `--experiment` or, when omitted, stable matrix row metadata. It does not create an extra scheduler layer.
 - `task`: one eval definition discovered from files and expanded across inference executors plus run/eval matrix rows.
-- `case`: the scoring and evidence source inside a task. `context.recordScore(...)` contributes normalized score evidence; `context.recordMetric(...)` emits benchmark metadata for reports.
+- `case`: the scoring and evidence source inside a task. `context.recordScore(score, value)` records a named score declared with `defineScore` (a value in `0..1`, higher is better; one score per task is the primary score); `context.recordUnscored(score, reason)` records that the grader could not produce one; `context.recordMetric(...)` emits benchmark data such as latency or cost for reports.
+- `completion rate`: the share of cases that produced a result (`passed` or `failed`, not `errored` or `timeout`). It is shown next to the primary score everywhere.
 - `attempt`: a full task rerun used to estimate reliability. With `autoAttempt`, each full attempt contributes evidence, so a fail-then-pass pair scores as a success rate rather than replacing the earlier failure.
 - `retry`: an in-case retry. With `autoRetry`, a case can recover inside one attempt; a retry pass still counts as that attempt passing.
 
@@ -260,6 +288,7 @@ If a project provides no `executor`, `vieval run` still discovers eval files, sc
 
 ```ts
 import { defineConfig } from 'vieval'
+import { scoreCases } from 'vieval/core/scoring'
 
 export default defineConfig({
   projects: [
@@ -282,7 +311,8 @@ export default defineConfig({
           id: task.id,
           inferenceExecutorId: task.inferenceExecutor.id,
           matrix: task.matrix,
-          scores: [{ kind: 'exact', score: success ? 1 : 0 }],
+          // One measurement as one passed outcome; `scoreCases` builds `scores`, `primary`, `completion`, and `passed`.
+          ...scoreCases([{ scores: { 'motion.success': success ? 1 : 0 }, state: 'passed', unscored: {} }]),
         }
       },
       include: ['evals/*.eval.ts'],
@@ -363,7 +393,7 @@ export default defineConfig({
 
 - One live terminal window shows `Workspaces`, `Projects`, `Tasks`, `Cases`, `Concurrency`, and `Time` rows for all workspaces.
 - A failing workspace does not stop the others.
-- At the end, a leaderboard prints one row per workspace and project, sorted by hybrid score, then the failed workspaces with the last lines of their stderr.
+- At the end, a leaderboard prints one row per workspace and project, ranked by the primary score with the completion rate in its own column, then the failed workspaces with the last lines of their stderr. If the workspaces have different primary scores, the leaderboard says so and ranks by passed cases.
 - The exit code is `1` when any workspace failed or had failing cases.
 - `--json` prints the comparison output.
 - Reports go to `.vieval/reports` next to the comparison config. The experiment defaults to the comparison id, and all workspaces share one attempt.
@@ -393,7 +423,7 @@ vieval viewer [reportRoot...] [--port <port>] [--host <host>] [--open]
 vieval report analyze <report-directory>
 vieval report index <report-directory> [--output <path>] [--format table|json|jsonl]
 vieval report cases <report-directory> [--where <key=value>] [--group-by <key>] [--format table|json|jsonl]
-vieval report compare <left-report-directory> <right-report-directory> [--case-key <key>] [--score-kind <kind>] [--format table|json]
+vieval report compare <left-report-directory> <right-report-directory> [--case-key <key>] [--score <name>] [--format table|json]
 ```
 
 `vieval run --config <comparison config>` runs a comparison; there is no separate `vieval compare` command. `vieval report compare` diffs two existing reports.
@@ -427,11 +457,12 @@ A concurrency value is shared by the work where it is declared: top-level `sched
 
 ## Public Entrypoints
 
-- `vieval`: `defineConfig`, `loadEnv`, `requiredEnvFrom`, `describeTask`, `describeEval`, `caseOf`, `casesFromInputs`, and `expect`.
+- `vieval`: `defineConfig`, `loadEnv`, `requiredEnvFrom`, `describeTask`, `describeEval`, `caseOf`, `casesFromInputs`, `defineScore`, `defineMetric`, and `expect`.
 - `vieval/config`: lower-level `defineEval`, `defineTask`, matrix types, task context types, model definitions, and plugin contracts.
 - `vieval/plugins/chat-models`: `ChatModels`, `ChatProviders`, `chatModelFrom`, `chatProviderFrom`, `chatModelMatrix`, runtime config helpers, and chat telemetry helpers.
 - `vieval/core/runner`: collection, scheduling, task context, cache runtime, scheduler, execution, and aggregation utilities.
-- `vieval/core/assertions`: assertion primitives and pipeline helpers.
+- `vieval/core/assertions`: assertion primitives and pipeline helpers. Outcomes carry a `score`; record it with `context.recordScore(definition, outcome.score)`.
+- `vieval/core/scoring`: the one scoring rule (`scoreCases`), derived scores (`deriveScores`), pooling (`poolScoreSummaries`), and headline resolution (`resolveHeadline`). Side-effect free.
 - `vieval/core/inference-executors`: env helpers and remote provider executors.
 - `vieval/testing/expect-extensions`: Vitest expect extensions for testing eval behavior.
 
diff --git a/docs/.vitepress/theme/home/blocks/EvalCode.vue b/docs/.vitepress/theme/home/blocks/EvalCode.vue
index 2cb943f..6a0002e 100644
--- a/docs/.vitepress/theme/home/blocks/EvalCode.vue
+++ b/docs/.vitepress/theme/home/blocks/EvalCode.vue
@@ -8,22 +8,23 @@ type Tone = 'comment' | 'function' | 'keyword' | 'plain' | 'string'
 
 /** Tokens per line of the sample eval file; the shape follows `docs/content/en/guide/learn/tasks-cases-and-inputs.md`. */
 const lines: [string, Tone][][] = [
-  [['import', 'keyword'], [' { describeTask, expect } ', 'plain'], ['from', 'keyword'], [' \'vieval\'', 'string']],
+  [['import', 'keyword'], [' { defineScore, describeTask, expect } ', 'plain'], ['from', 'keyword'], [' \'vieval\'', 'string']],
   [],
   [['const', 'keyword'], [' inputs = ', 'plain'], ['await', 'keyword'], [' loadEpisodes', 'function'], ['(', 'plain'], ['\'s3://evals\'', 'string'], [')', 'plain']],
+  [['const', 'keyword'], [' solved = ', 'plain'], ['defineScore', 'function'], ['(', 'plain'], ['\'desktop.solved\'', 'string'], [', { primary: ', 'plain'], ['true', 'keyword'], [' })', 'plain']],
   [],
   [['describeTask', 'function'], ['(', 'plain'], ['\'desktop\'', 'string'], [', ({ casesFromInputs }) => {', 'plain']],
   [['  casesFromInputs', 'function'], ['(', 'plain'], ['\'episode\'', 'string'], [', inputs, ', 'plain'], ['async', 'keyword'], [' (ctx) => {', 'plain']],
   [['    const', 'keyword'], [' { task, goal } = ctx.matrix.inputs', 'plain']],
   [['    const', 'keyword'], [' run = ', 'plain'], ['await', 'keyword'], [' agent.', 'plain'], ['play', 'function'], ['(task)', 'plain']],
   [['    expect', 'function'], ['(run.state).', 'plain'], ['toEqual', 'function'], ['(goal)', 'plain']],
-  [['    ctx.', 'plain'], ['recordScore', 'function'], ['(', 'plain'], ['await', 'keyword'], [' judge', 'function'], ['(run))', 'plain']],
+  [['    ctx.', 'plain'], ['recordScore', 'function'], ['(solved, ', 'plain'], ['await', 'keyword'], [' judge', 'function'], ['(run))', 'plain']],
   [['  })', 'plain']],
   [['})', 'plain']],
 ]
 
 /** Lines each case passes through, in order: read inputs, run the agent, assert, score. */
-const CASE_LINES = [6, 7, 8, 9]
+const CASE_LINES = [7, 8, 9, 10]
 const LINE_HEIGHT_EM = 1.8
 const BEAT = 420
 
diff --git a/docs/content/en/api/index.md b/docs/content/en/api/index.md
index d0dc5df..00bd1a1 100644
--- a/docs/content/en/api/index.md
+++ b/docs/content/en/api/index.md
@@ -6,11 +6,11 @@ Vieval publishes focused entrypoints so application code can import the narrowes
 
 | Import | Import when |
 | --- | --- |
-| `vieval` | Author eval files or config with the task DSL, config helpers, and the preconfigured assertion API; representative imports are `describeTask`, `defineConfig`, and `expect`. |
-| `vieval/core/assertions` | Build and evaluate structured assertion pipelines and convert their outcomes into runner scores with exports such as `evaluateAssertions`, `expectRubric`, and `toRunScores`. |
+| `vieval` | Author eval files or config with the task DSL, config helpers, and the preconfigured assertion API; representative imports are `describeTask`, `defineScore`, `defineMetric`, `defineConfig`, and `expect`. |
+| `vieval/core/assertions` | Build and evaluate structured assertion pipelines with exports such as `evaluateAssertions`, `expectRubric`, and `expectCustom`. Each outcome carries a `score` in `0..1`; the case records it with `context.recordScore(definition, outcome.score)`. |
 | `vieval/expect` | Import the standalone `expect` with Vieval matchers pre-installed without importing the rest of the main authoring surface. |
 
-Start with [Tasks, Cases, and Inputs](/en/guide/learn/tasks-cases-and-inputs) and [Assertions, Scores, and Metrics](/en/guide/learn/assertions-scores-and-metrics). The task DSL and the structured core assertion pipeline are both public, but outcomes from `vieval/core/assertions` are not connected to task cases automatically.
+Start with [Tasks, Cases, and Inputs](/en/guide/learn/tasks-cases-and-inputs) and [Assertions, Scores, and Metrics](/en/guide/learn/assertions-scores-and-metrics). The task DSL and the structured core assertion pipeline are both public, but outcomes from `vieval/core/assertions` are not connected to task cases automatically: record each outcome's `score` into the score it measures.
 
 ## Config and integration
 
@@ -30,8 +30,7 @@ These are public integration surfaces for hosts that need to assemble or drive e
 | `vieval/cli` | Embed the import-safe top-level CLI parser and command dispatcher through `parseTopLevelCliArguments` and `runTopLevelCli`. |
 | `vieval/core/runner` | Assemble collection, scheduling, and execution directly with exports such as `collectEvalEntries`, `createRunnerSchedule`, and `runScheduledTasks`. |
 | `vieval/core/scheduler` | Admit projects, tasks, and cases through concurrency pools and middleware without the runner's collection APIs, using `createScheduler` and `Scheduler.child`. |
-| `vieval/core/scoring` | Score case outcomes with the same rule as tasks and the viewer, using `scoreCaseOutcomes`, `strictExactOf`, and `isMeasuredState`. It has no side effects, so browser code can import it. |
-| `vieval/core/processors/results` | Apply built-in result policies and inspect the gate through `processRunResults` and the `ResultGateDecision` type. |
+| `vieval/core/scoring` | Score case outcomes with the same rule as tasks, reports, and the viewer. `scoreCases(outcomes, definitions)` returns the score summaries, the completion rate, the share of passed cases, and the primary score name; a custom `defineTask` can return its result as `TaskRunOutput`. `deriveScores` computes the derived (`combine`) scores of one case try, `poolScoreSummaries` pools summaries of the same name across task runs, `resolveHeadline` resolves the headline of a project or run, and `isMeasuredState` tells whether a state produced a result. It has no side effects, so browser code can import it. |
 | `vieval/core/inference-executors` | Build custom inference integrations with exports such as `createProviderAdapter`, `createRetryPolicy`, and `createOpenAIFromEnv`. |
 
 Read [Reliable Execution](/en/guide/learn/reliable-execution) before composing runner, scheduler, retry, or cache behavior. For aggregation, gating, and persisted evidence, continue with [Reports and Comparisons](/en/guide/learn/reports-and-comparisons).
diff --git a/docs/content/en/guide/core-concepts.md b/docs/content/en/guide/core-concepts.md
index c3dd652..3d53257 100644
--- a/docs/content/en/guide/core-concepts.md
+++ b/docs/content/en/guide/core-concepts.md
@@ -25,7 +25,7 @@ run --json -> machine-readable stdout
 run -> report artifacts (.vieval/reports by default) -> analyze / compare
 ```
 
-A **workspace** is a config that lists `projects`, inline or collected from other configs such as `packages/*`. A **comparison** is a config that lists `workspaces` and runs each one in its own process, side by side. A **project** defines a discovery boundary and project-level configuration. A discovered eval definition may carry a **task**, while DSL definitions such as `describeTask` register tasks directly. Matrix rows expand the registered tasks into concrete scheduled work. A task contains one or more **cases**, where assertions, normalized scores, and custom metrics provide execution evidence.
+A **workspace** is a config that lists `projects`, inline or collected from other configs such as `packages/*`. A **comparison** is a config that lists `workspaces` and runs each one in its own process, side by side. A **project** defines a discovery boundary and project-level configuration. A discovered eval definition may carry a **task**, while DSL definitions such as `describeTask` register tasks directly. Matrix rows expand the registered tasks into concrete scheduled work. A task contains one or more **cases**, where assertions, named scores, and custom metrics provide execution evidence. One score per task is its **primary** score; reports show it as the headline, next to the **completion rate**, the share of cases that produced a result.
 
 Execution reliability has two scopes. An **attempt** reruns a full task and retains the evidence from every attempt. A **retry** lets a failed case run again within an attempt. Output is explicit: `--json` prints a machine-readable result to standard output, and every run writes the report artifacts consumed by analysis and comparison commands, by default to `.vieval/reports` next to the config. `--report-out <directory>` changes that root and `--no-report` turns reports off.
 
diff --git a/docs/content/en/guide/learn/assertions-scores-and-metrics.md b/docs/content/en/guide/learn/assertions-scores-and-metrics.md
index 2e363f1..aff07ac 100644
--- a/docs/content/en/guide/learn/assertions-scores-and-metrics.md
+++ b/docs/content/en/guide/learn/assertions-scores-and-metrics.md
@@ -10,15 +10,15 @@ next:
 
 # Assertions, Scores, and Metrics
 
-A case becomes useful evidence when it makes success observable. Start with deterministic assertions, then add normalized scores or report metadata only when the evaluation needs them.
+A case becomes useful evidence when it makes success observable. Start with deterministic assertions, then declare named scores for the qualities you measure, and record metrics for everything else that reports should keep.
 
 ## Start with a deterministic assertion
 
-Vieval exports a Vitest-compatible `expect`. A failed matcher throws an `AssertionError` inside the case, so the case ends `failed`: it produced a wrong result and contributes an `exact` score of `0`. A case that completes without a custom score contributes `1`.
+Vieval exports a Vitest-compatible `expect`. A failed matcher throws an `AssertionError` inside the case, so the case ends `failed`: it produced a wrong result.
 
-Any other error, such as a provider rate limit, a dropped connection, or a bug, ends the case `errored`: it produced no result. A timeout ends it `timeout`, which also has no result. These cases stay out of the `exact` score. Instead they lower the coverage, the share of settled cases that produced a result. The run summary keeps the older every-case score as `strictExactAverage`, where cases without a result count as `0`. Read the score together with its coverage: a high score over a low coverage measured only part of the dataset.
+Any other error, such as a provider rate limit, a dropped connection, or a bug, ends the case `errored`: it produced no result. A timeout ends it `timeout`, which also has no result. These cases lower the **completion rate**, the share of cases that produced a result. Vieval shows the completion rate next to the headline score everywhere, so read the two together: a high score over a low completion rate measured only part of the dataset.
 
-To mark a wrong result, use `expect` or `recordScore(0)`. Do not throw a plain `Error`, which reads as "no result". If your test object can crash on hard inputs and a crash should count as a wrong result, set `classifyError` on the task or case:
+To mark a wrong result, use `expect`. Do not throw a plain `Error`, which reads as "no result". If your test object can crash on hard inputs and a crash should count as a wrong result, set `classifyError` on the task or case:
 
 ```ts
 import { isRetriableProviderError } from 'vieval/core/inference-executors'
@@ -40,51 +40,192 @@ describeTask('normalized answer', ({ caseOf }) => {
 })
 ```
 
-This is the clearest starting point for exact values, schemas, required fields, and other rules that code can decide reliably.
+This is the clearest starting point for exact values, schemas, required fields, and other rules that code can decide reliably. A task that records no score has no score: its headline is the share of passed cases, labeled "Passed cases".
 
 ## Separate the evidence types
 
 | Term | Meaning in Vieval | Effect on a task case |
 | --- | --- | --- |
 | Assertion | A check that produces a pass/fail decision. `expect` controls the case by throwing on failure; `vieval/core/assertions` also exposes assertion functions that return structured outcomes. | A thrown matcher fails the case. Core assertion outcomes are not connected to the task DSL automatically. |
-| Score | A normalized number in `0..1`, recorded with `context.recordScore(value, kind)`. Public score kinds are `exact` and `judge`; `exact` is the default. | Contributes to task-result aggregation when the case passes. Its emitted event may also be persisted in a report case record. |
-| Metric | Named benchmark metadata recorded with `context.recordMetric(name, value)`. Values may be strings, numbers, booleans, `null`, or arrays of those telemetry values. | Appears in report events and case records, but does not change pass/fail state or score aggregation. |
-| Rubric | An assertion whose `judge` callback returns a reason and normalized score, then compares that score with `minScore` (default `0.7`). | Produces an `AssertionOutcome` with score kind `judge`; the author decides how to bridge that outcome into a task case. |
+| Score | A named value in `0..1` where higher is better, declared with `defineScore` and recorded with `context.recordScore(score, value)`. One score per task is the primary score. | Aggregated per name over the task's cases. A failed case counts as `0` in every score. |
+| Completion rate | The built-in share of cases that produced a result: `(passed + failed) / (passed + failed + errored + timeout)`. Skipped cases are left out. | Always reported next to the primary score. An eval cannot turn it off. |
+| Metric | Named benchmark data recorded with `context.recordMetric(name, value)`: latency, cost, token counts, categories. Values may be strings, numbers, booleans, `null`, or arrays of those. | Appears in report events and case records, but does not change the case state or any score. |
+| Rubric | An assertion whose `judge` callback returns a reason and normalized score, then compares that score with `minScore` (default `0.7`). | Produces an `AssertionOutcome`; the case records its `score` into a score of its choice. |
 
-The `recordScore` and `recordMetric` methods are available on the case callback context received by `caseOf` and `casesFromInputs`.
+The `recordScore`, `recordUnscored`, and `recordMetric` methods are available on the case callback context received by `caseOf` and `casesFromInputs`.
 
-## Record scores and metrics from a case
+## Declare a score
 
-Use `recordScore` for a value that should be averaged. Use `recordMetric` for dimensions or observations that reports should retain without treating them as the result itself.
+`defineScore(name, metadata)` declares a score once, as `defineMetric` does for metrics. The name is the stable identifier written to reports. `label` and `description` take one string, or one string per BCP 47 locale; the viewer picks the reader's language and falls back to `en`, then to the first entry. Hovering the score in the viewer shows the description.
 
 ```ts [evals/retrieval.eval.ts]
-import { describeTask, expect } from 'vieval'
+import { defineScore, describeTask, expect } from 'vieval'
+
+const recall = defineScore('retrieval.recall', {
+  description: {
+    'en': 'Share of the expected documents that the retriever returned.',
+    'zh-Hans': '检索结果中包含的期望文档所占的比例。',
+  },
+  label: { 'en': 'Recall', 'zh-Hans': '召回率' },
+})
 
 describeTask('retrieval', ({ caseOf }) => {
   caseOf('finds the expected documents', ({ recordMetric, recordScore }) => {
     const expected = new Set(['doc-a', 'doc-b'])
     const retrieved = ['doc-a', 'doc-c']
     const matches = retrieved.filter(id => expected.has(id)).length
-    const recall = matches / expected.size
 
-    recordScore(recall, 'exact')
+    recordScore(recall, matches / expected.size)
     recordMetric('benchmark.case.id', 'retrieval-basic')
     recordMetric('retrieved.count', retrieved.length)
 
-    expect(recall).toBeGreaterThan(0)
+    expect(matches).toBeGreaterThan(0)
   })
+}, { scores: [recall] })
+```
+
+- The value must be finite and inside `0..1`. Within one case try, the last `recordScore` call per name wins.
+- `recordScore` also accepts a bare name, for example `recordScore('retrieval.recall', 0.5)`. The score then has no label or description.
+- A definition is announced to the report the first time a case records it. List the task's scores in the `scores` option of `describeTask` to announce them when the task starts. A listed score is known before the first case, and it counts as missing for a passed case that never records it.
+
+## Choose the primary score
+
+Pages, leaderboards, comparisons, and the CLI summary line show one score per task first, with its label next to its value. That is the **primary** score.
+
+- Mark it with `primary: true`. At most one score of a task is primary; a second one throws when it is announced.
+- A task with exactly one score uses that score as primary without the flag.
+- A task with several scores and none marked primary has no primary score. Its headline falls back to the share of passed cases.
+- When the tasks of a run have different primary scores, the run has no run-level headline score. Each task's primary score is listed with its task name, and the completion rate stays as the run headline.
+
+## Combine scores
+
+A derived score is computed from other scores of the same case try. Use it when the headline answers a question that combines several measurements, for example "correct and natural":
+
+```ts [evals/read-aloud.eval.ts]
+import { defineScore, describeTask } from 'vieval'
+
+const correct = defineScore('tts.correct', {
+  label: { 'en': 'Written back correctly', 'zh-Hans': '听写正确' },
+})
+const natural = defineScore('tts.natural', {
+  description: 'The judge median: natural 1, mixed 0.5, spelled 0.',
+  label: { 'en': 'Natural reading', 'zh-Hans': '读法自然' },
 })
+const pass = defineScore('tts.pass', {
+  // A bare input means `atLeast: 1`, so a mixed reading (0.5) does not pass.
+  combine: { all: [correct, natural] },
+  label: { 'en': 'Pass', 'zh-Hans': '通过' },
+  primary: true,
+})
+
+describeTask('read aloud', ({ caseOf }) => {
+  caseOf('reads a fraction', async ({ recordScore }) => {
+    const { listenerScore, readingScore } = await readAloud('1/2')
+    recordScore(correct, listenerScore)
+    recordScore(natural, readingScore)
+    // `pass` is computed when the try settles; recording it throws.
+  })
+}, { scores: [correct, natural, pass] })
+```
+
+`combine` takes one of three forms, so the viewer can show the formula, for example "all of: correct, natural":
+
+| Form | Value |
+| --- | --- |
+| `{ all: [a, { score: b, atLeast: 0.5 }] }` | `1` when every input reaches its threshold, else `0`. A bare input passes at `1`. `all` is always binary. |
+| `{ mean: [a, b] }` | The mean of the inputs. |
+| `{ weighted: [{ score: a, weight: 2 }, { score: b, weight: 1 }] }` | The mean weighted by `weight`. Weights are normalized. |
+
+- A derived score must be listed in the `scores` option of `describeTask`, because no case records it.
+- When a try settles, the runtime computes every derived score whose inputs were all recorded in that try. If an input is missing, the derived score is missing for that case.
+- A case cannot record a derived score; `recordScore` throws.
+- When an eval needs another rule, compute the value in the case and record a plain score whose `description` explains the rule.
+
+## How a case state counts
+
+One rule decides how each case counts in each score. The task result, `run-summary.json`, `cases.jsonl`, and the viewer all use it:
+
+| Case state | Every score of the task |
+| --- | --- |
+| `passed` | The recorded value, or missing when the case did not record it. |
+| `failed` | `0`. An assertion failed, so the case measured a wrong result. The recorded values are kept as details, but they are not counted. |
+| `errored`, `timeout` | Left out by default (`errors: 'exclude'`). With `errors: 'zero'` the case counts as `0`. |
+| `skipped` | Left out. |
+
+Set `errors` per score. `exclude` follows the view that a provider outage is not a result of the test object; the completion rate then shows how many cases were left out. `zero` matches benchmarks that count every error as a wrong answer, for example Harbor and Terminal-Bench:
+
+```ts
+const solved = defineScore('bench.solved', { errors: 'zero', primary: true })
 ```
 
-Scores must be finite and inside `0..1`. Within one case, another `recordScore` call with the same kind replaces the earlier value; use one final value per score family.
+Each score summary reports `mean` (over the counted cases, or `null`), `n` (cases that counted), and `total` (settled cases that were not skipped). An empty case selection, for example a `--case` filter that matches nothing, returns no scores.
 
-Task-result aggregation and persisted report records have different timing. If a case later fails, task-result aggregation ignores its custom score contributions and adds an exact `0`. If it errors or times out, task-result aggregation leaves it out of the exact score. Each `recordScore` call also emits an event immediately, however, so a score emitted before the failure may remain in the persisted case record. For a `failed` case, the case-end event supplies an exact `0` only when that record does not already have an exact score. An `errored` or `timeout` record gets no default exact score, so `vieval report` means leave it out. When diagnosing a report, read the case `state` together with its recorded scores.
+## Record a score the grader could not produce
 
-Metrics are evidence for filtering, grouping, and diagnosing reports. A metric alone never makes a case pass, fail, or receive a higher aggregate score.
+Sometimes a case runs to the end, but its grader cannot produce a value: the judge's answer cannot be parsed, the grader fails, or the output is empty. That is neither a wrong result nor an execution error. Record it with `recordUnscored`:
+
+```ts
+caseOf('reads a fraction', async ({ recordScore, recordUnscored }) => {
+  const verdict = await judgeReading(audio)
+  if (verdict == null) {
+    recordUnscored(natural, 'invalid_response')
+    return
+  }
+  recordScore(natural, verdict.score)
+})
+```
+
+- `reason` is a string. The viewer shows its own labels for `grader_failed`, `invalid_response`, `no_response`, and `refusal`. Any other string is shown as written.
+- An unscored value never counts, whatever `errors` says. It is not `0`, because it says nothing about the test object.
+- An unscored record and a value of the same name replace each other in one try. The last call wins.
+- A derived score with an unscored input is unscored with the reason `input_unscored`.
+- The case state does not change. A passed case with an unscored value stays passed, and the completion rate does not change.
+- Each score summary counts unscored values per reason in `unscored`, for example `{ grader_failed: 2 }`. They are part of `total`, not of `n`.
+
+## Explain a value: rationale and grader
+
+A recorded value can say why it has its value, and what produced it. The viewer shows both on the case page, next to the score badge:
+
+```ts
+recordScore(natural, median(verdicts.map(verdict => verdict.score)), {
+  grader: { id: 'judge', kind: 'model', model: judgeModel.model, reducer: 'median', votes: verdicts.length },
+  metadata: { verdicts },
+  rationale: verdicts.map(verdict => verdict.reason).join('\n'),
+})
+```
+
+- `rationale` is a judge's reasoning or the rule that matched.
+- `grader` says what produced the value. `kind` is `rule` (deterministic code), `model` (an LLM or another model), `human` (a reviewer), or `external` (another harness's verifier). `id`, `model`, `version`, `votes`, and `reducer` are optional.
+- `metadata` holds other details, shown as an object tree.
+- Declare the usual grader once on the definition, `defineScore(name, { grader: { kind: 'model', id: 'judge' } })`. A record overrides it, for example when the judge model is a matrix axis.
+- A derived score has no grader of its own. The viewer shows the graders of its inputs.
+- `grader.id` matches the `role` of the judge's `inference.*` events, so the viewer can link a score to its judge calls.
+
+The grader is part of what a score means. Two runs whose graders differ in `kind`, `model`, or `version` do not measure the same thing, so a comparison of such runs shows a warning on that score.
+
+## Scores versus metrics
+
+A score answers "how good was this case?" with a value in `0..1` where higher is better. Everything else is a metric: ranges, directions, units, counts, and categories belong to metrics.
+
+Latency is a metric, not a score. Give it a direction so that reports know lower is better:
+
+```ts
+import { defineMetric } from 'vieval'
+
+const latency = defineMetric('read_aloud.latency', {
+  better: 'lower',
+  label: { 'en': 'Read-aloud latency', 'zh-Hans': '朗读耗时' },
+  unit: 'ms',
+})
+
+recordMetric(latency, elapsedMs)
+```
+
+Metrics are evidence for filtering, grouping, and diagnosing reports. A metric alone never makes a case pass, fail, or change a score.
 
 ## Give metrics a readable name
 
-The viewer shows metric names such as `osworld.model.usage.output_tokens` as they are. Declare the metric once with `defineMetric` to give it a label, a description, a unit, and a direction. Label and description take one string, or one string per BCP 47 locale; the viewer picks the reader's language and falls back to `en`, then to the first entry.
+The viewer shows metric names such as `osworld.model.usage.output_tokens` as they are. Declare the metric once with `defineMetric` to give it a label, a description, a unit, and a direction. Label and description use the same localized shape as scores.
 
 ```ts [evals/agent.eval.ts]
 import { defineMetric, describeTask } from 'vieval'
@@ -109,7 +250,7 @@ describeTask('agent', ({ caseOf }) => {
 })
 ```
 
-The first value that carries metadata emits a `metric.defined` event for the task; the same metadata is not emitted again, and changed metadata is emitted again. Keep one set of metadata per metric name: the viewer shows the latest one. In the viewer, hovering a metric shows its label, name, description, unit, direction, and the distribution of its values across the task's cases.
+The first value that carries metadata emits a `metric.defined` event for the task; the same metadata is not emitted again, and changed metadata is emitted again. Keep one set of metadata per metric name: the viewer shows the latest one. In the viewer, hovering a metric shows its label, name, description, unit, direction, and the distribution of its values across the task's cases. Scores appear in the same metric panels, before the metrics, always as percentages where higher is better.
 
 `recordArtifact` takes `label` and `description` in the same shape, for example `recordArtifact('final.webp', { contentType: 'image/webp', path, label: { 'en': 'Final screen', 'zh-Hans': '最终画面' } })`.
 
@@ -136,12 +277,21 @@ const [outcome] = await evaluateAssertions([
 
 This example is deterministic and makes no model call. `expectRubric` does not select a model or invoke a provider by itself. A `judge` callback may use local logic, a human result, or model inference.
 
-The core assertion pipeline and the task DSL are currently separate public surfaces. `evaluateAssertions` returns outcomes; it does not automatically call the case callback context's `recordScore`, `recordMetric`, or `expect`. If you bridge an outcome manually, decide explicitly whether you are recording its numeric score, failing the case from `outcome.pass`, retaining its reason as a metric, or some combination. Throwing an `AssertionError` (for example through `expect`) makes the task result treat the case as an exact failure; throwing another error makes it `errored` unless `classifyError` says otherwise. Score events emitted before the throw may still appear in the persisted case record.
+The core assertion pipeline and the task DSL are separate public surfaces. `evaluateAssertions` returns outcomes; it does not call the case context's `recordScore`, `recordMetric`, or `expect`. Bridge an outcome explicitly into the score it measures:
+
+```ts
+const concise = defineScore('answer.concise', { grader: { id: 'concise-answer', kind: 'rule' } })
+
+recordScore(concise, outcome.score, { rationale: outcome.reason })
+expect(outcome.pass).toBe(true)
+```
+
+Decide deliberately whether the outcome also fails the case. Throwing an `AssertionError` (for example through `expect`) makes the case `failed`, so every score of the task counts it as `0`; throwing another error makes it `errored` unless `classifyError` says otherwise.
 
 ::: warning Model-backed judging has runtime consequences
 If your `judge` callback invokes a model, it needs the relevant provider configuration and credentials and adds latency and cost. Case concurrency can fan out judge requests, so apply limits that match provider capacity. Review the provider's data policy and choose deliberately which case inputs, outputs, judge prompts, reasons, metrics, or events you retain in report artifacts.
 :::
 
-A common boundary error is treating a metric as a score or assuming a rubric outcome is wired into the case automatically. Keep the decision, numeric evidence, and diagnostic metadata separate in the code.
+A common boundary error is treating a metric as a score or assuming a rubric outcome is wired into the case automatically. Keep the decision, the scores, and the diagnostic metrics separate in the code.
 
 Next, learn how model registrations and runtime adapters fit this flow in [Models and Inference Executors](/en/guide/learn/models-and-inference-executors). The assertion primitives are listed under the public [`vieval/core/assertions` entrypoint](/en/api/).
diff --git a/docs/content/en/guide/learn/matrices-and-datasets.md b/docs/content/en/guide/learn/matrices-and-datasets.md
index 0aab051..dd96c6f 100644
--- a/docs/content/en/guide/learn/matrices-and-datasets.md
+++ b/docs/content/en/guide/learn/matrices-and-datasets.md
@@ -100,6 +100,7 @@ Within every layer, Vieval applies controls in this order:
 
 ```ts [evals/layered.eval.ts]
 import { defineEval, defineTask } from 'vieval/config'
+import { scoreCases } from 'vieval/core/scoring'
 
 export default defineEval({
   description: 'Shows matrix layering.',
@@ -125,12 +126,12 @@ export default defineEval({
       },
     },
     run(context) {
-      return {
-        scores: [{
-          kind: 'exact',
-          score: context.task.matrix.eval.rubric === 'strict' ? 1 : 0,
-        }],
-      }
+      // A custom task returns score summaries; `scoreCases` builds them from case outcomes.
+      return scoreCases([{
+        scores: { 'layered.strict': context.task.matrix.eval.rubric === 'strict' ? 1 : 0 },
+        state: 'passed',
+        unscored: {},
+      }])
     },
   }),
 })
diff --git a/docs/content/en/guide/learn/reliable-execution.md b/docs/content/en/guide/learn/reliable-execution.md
index e674d25..7f70914 100644
--- a/docs/content/en/guide/learn/reliable-execution.md
+++ b/docs/content/en/guide/learn/reliable-execution.md
@@ -138,7 +138,7 @@ const providerRetry = {
 
 A case retry replays the whole case. To retry only the failed provider call, wrap that call in `runWithRetry` from `vieval/core/inference-executors`. Its default delay waits for the provider's `Retry-After` or an exponential backoff with jitter, whichever is longer, and passing `{ signal }` stops the wait when the run is interrupted.
 
-`autoAttempt: 1` permits one additional full task attempt. It starts only when a case still fails after its retries and that case has remaining attempts. Because the full registered case set runs again, attempt evidence includes both the earlier and later outcomes. For example, one failed attempt followed by one passing attempt contributes `0.5`, not a replacement pass of `1`.
+`autoAttempt: 1` permits one additional full task attempt. It starts only when a case still fails after its retries and that case has remaining attempts. Because the full registered case set runs again, attempt evidence includes both the earlier and later outcomes. For example, one failed attempt followed by one passing attempt that records `1` gives that case a mean of `0.5` in the score, not a replacement pass of `1`.
 
 ::: tip Use repetition for the question it answers
 Attempts measure nondeterministic reliability because every completed attempt contributes evidence. Retries recover transient case failures inside one attempt; a retry that eventually passes makes that attempt pass. Retries therefore hide the earlier transient failure from task scoring, although lifecycle events still record the tries. Use retries only when recovery, rather than failure frequency, is the intended measurement.
@@ -166,7 +166,7 @@ its slot back while it waits, and each later run gets its own `autoRetry` retrie
 be next, and without a delay a requeue would be an immediate retry.
 
 Until its last run ends, the case has not ended. It emits `case.requeued` instead of `case.ended`, the viewer shows it
-as waiting to rerun (not done yet), and it does not count in scores or coverage. Only the final run is scored. A case
+as waiting to rerun (not done yet), and it does not count in scores or in the completion rate. Only the final run is scored. A case
 whose retries and requeues all ran out ends `errored`; resume it later with `--resume`.
 
 ### Resume an attempt
@@ -234,13 +234,13 @@ export default defineConfig({
 
 Task aggregation and report artifacts answer related but different questions:
 
-- A failed or timed-out case adds exact score evidence of `0`; a passing case without a custom score adds `1`.
-- A passing case contributes its custom `exact` and `judge` scores. A failed case's custom score is not used by task aggregation.
+- A failed case counts as `0` in every score of its task. An errored or timed-out case is left out of a score with `errors: 'exclude'` (the default) and counts as `0` with `errors: 'zero'`; either way it lowers the completion rate.
+- A passing case contributes the values it recorded. A passing case that did not record a score is missing from that score, not counted as `1`.
 - Every completed auto attempt contributes another set of case outcomes to the task's aggregated score.
 - Reporter lifecycle, score, and metric events are emitted as execution happens. Unless the run uses `--no-report`, `events.jsonl` retains those events.
-- `cases.jsonl` is a final normalized projection keyed by task and case, so repeated retries or automatic attempts do not become independent case rows. For a task with automatic attempts, the record lists every attempt in `attempts`. Read raw events when the exact sequence matters (`case.started` and `case.ended` carry `attemptIndex`), and read the task aggregate in `run-summary.json` when attempt-weighted scoring matters.
+- `cases.jsonl` is a final normalized projection keyed by task and case, so repeated retries or automatic attempts do not become independent case rows. Its `scores` hold the counted value of each score under the same rule as the task result, so a failed case shows `0`. For a task with automatic attempts, the record lists every attempt in `attempts`. Read raw events when the exact sequence matters (`case.started` and `case.ended` carry `attemptIndex`), and read the task aggregate in `run-summary.json` when attempt-weighted scoring matters.
 
-As described in [Assertions, Scores, and Metrics](/en/guide/learn/assertions-scores-and-metrics), a score event emitted before a later failure can remain in a persisted case record even though task aggregation counts that outcome as a failure. Diagnose state and score together.
+As described in [Assertions, Scores, and Metrics](/en/guide/learn/assertions-scores-and-metrics), a value recorded before a later failure stays in `events.jsonl` and in the case record's `scoreDetails`, but it is not counted: the case's `scores` show `0`. Diagnose state and score together.
 
 A common failure is using `autoRetry` to estimate reliability: the recovered case appears as one passing attempt. Use `autoAttempt` when earlier failures should remain in the aggregate, and remember that it reruns cases that already passed. Also avoid treating `--attempt` as a request for more automatic attempts: it only labels the run.
 
diff --git a/docs/content/en/guide/learn/reports-and-comparisons.md b/docs/content/en/guide/learn/reports-and-comparisons.md
index 9d46615..15ae4c6 100644
--- a/docs/content/en/guide/learn/reports-and-comparisons.md
+++ b/docs/content/en/guide/learn/reports-and-comparisons.md
@@ -29,11 +29,11 @@ Every project is its own run. Vieval places each run under the report root as `<
 
 | Artifact | Confirmed contents |
 | --- | --- |
-| `run-summary.json` | The run output of exactly one project, with `schemaVersion: 1`, `runId`, `reportDirectory`, identities, project status, case summaries and failures, matrix summaries, and aggregated run scores. `vieval report` commands reject other versions. |
-| `events.jsonl` | Ordered run, task, case, score, metric, and custom event envelopes captured during execution. |
-| `cases.jsonl` | One normalized final record per observed task/case, with identities, state, timing, a derived retry count, scores, metrics, optional input/output, and, for tasks with automatic attempts, `attemptIndex`, `autoAttempt`, and every attempt in `attempts`. |
-| `metrics-summary.json` | Overall score counts, sums, and averages derived from normalized case records. |
-| `otlp/traces.json`, `otlp/logs.json`, `otlp/metrics.json` | Local OTLP-shaped projections derived from the case records. Each resource and the `vieval.run` span carry `vieval.workspace.id`, `vieval.experiment.id`, `vieval.attempt.id`, and `vieval.run.id`. |
+| `run-summary.json` | The run output of exactly one project, with `schemaVersion: 2`, `runId`, `reportDirectory`, identities, project status, case summaries and failures, matrix summaries, the score summaries of every task run, the `scoreDefinitions` the tasks announced, and the resolved headline (see below). `vieval report` commands reject other versions. |
+| `events.jsonl` | Ordered run, task, case, score (`score.defined`, `score.recorded`), metric, and custom event envelopes captured during execution. |
+| `cases.jsonl` | One normalized final record per observed task/case (`schemaVersion: 2`), with identities, state, timing, a derived retry count, scores, metrics, optional input/output, and, for tasks with automatic attempts, `attemptIndex`, `autoAttempt`, and every attempt in `attempts`. `scores` maps score names to the counted value, `unscored` maps score names to the reason the grader gave none, and `scoreDetails` keeps the `rationale`, `grader`, and `metadata` of each record. |
+| `metrics-summary.json` | Overall score counts, sums, and averages per score name, derived from normalized case records. |
+| `otlp/traces.json`, `otlp/logs.json`, `otlp/metrics.json` | Local OTLP-shaped projections derived from the case records. Score gauges are named `vieval.score.<name>`. Each resource and the `vieval.run` span carry `vieval.workspace.id`, `vieval.experiment.id`, `vieval.attempt.id`, and `vieval.run.id`. |
 
 `vieval report index` creates another artifact, `index/runs.jsonl` by default. It is not written by `vieval run`.
 
@@ -43,12 +43,30 @@ Automatic attempts require one reading caveat. `run-summary.json` contains the t
 
 The `retryCount` field counts retries within the final attempt. For tasks with automatic attempts it uses `attemptIndex`, so additional attempts are not counted as retries. Reports written before `attemptIndex` existed have no such field; for them, repeated starts from additional attempts may still increase `retryCount`.
 
-Event lines keep `schemaVersion: 2` and case records keep `schemaVersion: 1`: new optional fields and new event names do not change the version, and readers ignore fields they do not know. A renamed, removed, or retyped field would increment the version.
+Event lines have `schemaVersion: 3` and case records have `schemaVersion: 2`. Version 3 of the events replaced the fixed `exact` and `judge` score kinds with named scores. New optional fields and new event names do not change the version, and readers ignore fields they do not know. A renamed, removed, or retyped field increments the version. Readers reject reports written with an older version ("This report was written by an older vieval. Run it again."), and the viewer lists such a run as unsupported; there is no migration command.
 
 ::: warning Treat report artifacts as potentially sensitive
 Artifacts may contain case inputs and outputs, error messages, custom metrics, model or benchmark identifiers, and other event payloads. Store them under an appropriate retention policy, restrict access, and remove or redact fields that should not leave the evaluation environment.
 :::
 
+## Read the headline
+
+The headline of each project and of the run is resolved once, when the run ends, and written into `run-summary.json` as `{ headline, completion }`. The CLI, the viewer, and `vieval report` read it instead of deriving it again:
+
+| `headline.kind` | When | Shown as |
+| --- | --- | --- |
+| `score` | Every task that has scores shares one primary score name. | The pooled primary score with its label, for example `Pass 87.5%`. |
+| `per-task` | The tasks have different primary scores. | No run-level score. Each task's primary score is listed with its task name. |
+| `passed-cases` | No task has a primary score. | The share of passed cases, labeled "Passed cases". |
+
+`completion` is the completion rate, `(passed + failed) / (passed + failed + errored + timeout)`, and it is always shown next to the headline. The run summary line prints both, for example:
+
+```text
+Pass 87.5% · completed 96.8% (4 errored)
+```
+
+Every score summary has `name`, `mean` (or `null` when no case counted), `n` (cases that counted), `total` (settled cases that were not skipped), `sum`, and `unscored` (counts per reason). Summaries of a project or a run pool the cases of their tasks; they are not a mean of task means.
+
 ## Index, inspect, and analyze reports
 
 Every report command requires a report path. It can usually be a single run directory or a higher report root that contains multiple runs.
@@ -75,7 +93,7 @@ vieval report analyze <report-directory>
 vieval report attempts --config vieval.config.ts
 ```
 
-`attempts` lists the attempts in a report root, newest first: per experiment and attempt, the workspaces, the passed, failed, and no-result case counts, and the coverage. "←" marks the attempt that `vieval run --resume` continues without an id; notes show which attempt an attempt resumed, interrupts, and runs killed before they wrote a summary. Pass a report root, or `--config` to list the reports next to it; filter with `--experiment` and `--workspace`, and choose `--format table|json`.
+`attempts` lists the attempts in a report root, newest first: per experiment and attempt, the workspaces, the passed, failed, and no-result case counts, and the completion rate. "←" marks the attempt that `vieval run --resume` continues without an id; notes show which attempt an attempt resumed, interrupts, and runs killed before they wrote a summary. Pass a report root, or `--config` to list the reports next to it; filter with `--experiment` and `--workspace`, and choose `--format table|json`.
 
 ## Compare two existing report sets
 
@@ -85,7 +103,7 @@ The report comparison command requires a left baseline and a right candidate:
 vieval report compare <left-report-directory> <right-report-directory>
 ```
 
-It reads normalized cases, aligns them by a case key, and reports matched deltas plus added and removed cases. Per-case and grouped deltas use matched records and are `right - left`; grouping uses the right-side record's selector value. A missing selected score on a matched case is treated as `0`. The overall delta instead subtracts the two full record-set means, ignoring records that lack the selected score, so additions and removals affect it and it is not the mean matched delta. `--score-kind <kind>` defaults to `exact`, and output can be `table` or `json`.
+It reads normalized cases, aligns them by a case key, and reports matched deltas plus added and removed cases. Per-case and grouped deltas use matched records and are `right - left`; grouping uses the right-side record's selector value. A missing selected score on a matched case is treated as `0`. The overall delta instead subtracts the two full record-set means, ignoring records that lack the selected score, so additions and removals affect it and it is not the mean matched delta. `--score <name>` selects the score and defaults to the primary score in `run-summary.json`. Output can be `table` or `json`.
 
 The default alignment key is the `benchmark.case.id` metric when present, then `vieval.case.id`, then `<taskId>::<caseName>`. Use `--case-key <key>` when the benchmark has another stable identifier. An explicit key must exist on every record. Duplicate resolved keys on either side are an error rather than an arbitrary match.
 
@@ -117,7 +135,7 @@ vieval run --config evals/agent-memory/vieval.config.ts
 - The comparison `env` (and an entry's `env`) wins over each workspace config's `env`, and is applied before the workspace's plugins run.
 - `cache: { namespace }` makes all workspaces share `.vieval/cache/<comparison id>/<namespace>/` next to the comparison config.
 - One live terminal window shows `Workspaces`, `Projects`, `Tasks`, `Cases`, `Concurrency`, and `Time` rows for all workspaces. A failing workspace does not stop the others.
-- At the end, a leaderboard prints one row per workspace and project, sorted by hybrid score, then the failed workspaces with the last lines of their stderr.
+- At the end, a leaderboard prints one row per workspace and project, ranked by the primary score, with the completion rate in its own column, then the failed workspaces with the last lines of their stderr. If the workspaces have different primary scores, the leaderboard says so and ranks by passed cases.
 - The exit code is `1` when any workspace failed or had failing cases. `--json` prints the comparison output.
 - Reports go to `.vieval/reports` next to the comparison config. The experiment defaults to the comparison id, and all workspaces share one attempt, so every run of the comparison sits under `<workspace>/<project>/<comparison id>/<attempt>/`.
 - With `--with-viewer`, one viewer serves the comparison's report root.
@@ -168,7 +186,7 @@ vieval report compare \
   .vieval/reports/baseline \
   .vieval/reports/candidate \
   --case-key benchmark.case.id \
-  --score-kind exact \
+  --score retrieval.recall \
   --format table
 ```
 
diff --git a/docs/content/en/guide/learn/tasks-cases-and-inputs.md b/docs/content/en/guide/learn/tasks-cases-and-inputs.md
index 72d3025..94929d1 100644
--- a/docs/content/en/guide/learn/tasks-cases-and-inputs.md
+++ b/docs/content/en/guide/learn/tasks-cases-and-inputs.md
@@ -79,7 +79,7 @@ An explicit `caseOf` name is under your control. Keep it stable when the scenari
 
 For `casesFromInputs`, generated names use the prefix plus a one-based position: `<prefix> #1`, `<prefix> #2`, and so on. Inserting or reordering rows changes those generated names and IDs. If reports must compare the same logical sample across changing dataset order, carry a unique identifier in the input and emit it as `benchmark.case.id` as shown above. That value must be unique across every case in each compared report, including cases from other tasks or projects. Duplicate comparison keys are rejected rather than matched ambiguously.
 
-Report selectors can address direct case fields such as `caseName` and `caseId`, scores, or emitted metric names. Default case comparison looks for `benchmark.case.id`, then `vieval.case.id`, and finally the generated `caseId`.
+Report selectors can address direct case fields such as `caseName` and `caseId`, scores by name (`scores.<name>`), or emitted metric names. Default case comparison looks for `benchmark.case.id`, then `vieval.case.id`, and finally the generated `caseId`.
 
 ::: warning Keep registration inside the task
 `caseOf` and `casesFromInputs` require an active `describeTask` scope. Calling either function outside that scope throws during evaluation-module loading.
diff --git a/docs/content/en/index.md b/docs/content/en/index.md
index d3059a9..a38cdf4 100644
--- a/docs/content/en/index.md
+++ b/docs/content/en/index.md
@@ -20,9 +20,9 @@ home:
       ✓ |desktop-agent| (412 cases)
         cases 407 passed | 5 failed | 0 timeout
       $ pnpm vieval report cases .vieval/reports/candidate --where state=failed
-        ✗ chrome/open-settings  exact 0  wrong tab
+        ✗ chrome/open-settings  pass 0  wrong tab
         ✗ gimp/resize-image  timeout  60m
-        ✗ vscode/rename-symbol  exact 0  old name kept
+        ✗ vscode/rename-symbol  pass 0  old name kept
         ✗ calc/sum-column  rubric 0.42  partial
         ✗ chat/refund-policy  rubric 0.31  hallucinated
   blocks:
diff --git a/docs/content/zh-hans/api/index.md b/docs/content/zh-hans/api/index.md
index f652ab4..7e37d30 100644
--- a/docs/content/zh-hans/api/index.md
+++ b/docs/content/zh-hans/api/index.md
@@ -6,11 +6,11 @@ Vieval 提供以下公开导入路径。编写普通评测时，通常只需从
 
 | 导入路径 | 提供的功能 |
 | --- | --- |
-| `vieval` | 用于编写评测文件和 CLI 配置的常用 API，包括 `describeTask`、`describeEval`、`caseOf`、`casesFromInputs`、`defineConfig`、`loadEnv`、`requiredEnvFrom`，以及已安装 Vieval 匹配器（matcher）的 `expect`。 |
+| `vieval` | 用于编写评测文件和 CLI 配置的常用 API，包括 `describeTask`、`describeEval`、`caseOf`、`casesFromInputs`、`defineScore`、`defineMetric`、`defineConfig`、`loadEnv`、`requiredEnvFrom`，以及已安装 Vieval 匹配器（matcher）的 `expect`。 |
 | `vieval/expect` | 仅导出已安装 Vieval 匹配器的 `expect`。不需要任务 DSL 时，可以使用此入口。 |
-| `vieval/core/assertions` | 创建和执行结构化断言，并将断言结果转换为供运行器使用的分数。常用导出包括 `evaluateAssertions`、`expectRubric` 和 `toRunScores`。 |
+| `vieval/core/assertions` | 创建和执行结构化断言。常用导出包括 `evaluateAssertions`、`expectRubric` 和 `expectCustom`。每个断言结果都带有 `0..1` 之间的 `score`，由用例通过 `context.recordScore(definition, outcome.score)` 记录。 |
 
-从 `vieval/core/assertions` 获得的断言结果不会自动写入任务的用例结果。调用方需要自行执行断言，并将结果传给 `toRunScores` 等后续处理函数。相关用法见[任务、用例与输入](/zh-hans/guide/learn/tasks-cases-and-inputs)和[断言、分数与指标](/zh-hans/guide/learn/assertions-scores-and-metrics)。
+从 `vieval/core/assertions` 获得的断言结果不会自动写入任务的用例结果。调用方需要自行执行断言，并把结果的 `score` 记到它所测量的分数上。相关用法见[任务、用例与输入](/zh-hans/guide/learn/tasks-cases-and-inputs)和[断言、分数与指标](/zh-hans/guide/learn/assertions-scores-and-metrics)。
 
 ## 配置模型和执行器
 
@@ -31,8 +31,7 @@ Vieval 提供以下公开导入路径。编写普通评测时，通常只需从
 | `vieval/cli` | 解析顶层 CLI 参数并分派命令，导出 `parseTopLevelCliArguments` 和 `runTopLevelCli`。 |
 | `vieval/core/runner` | 收集评测、生成调度任务、执行任务并汇总结果。常用导出包括 `collectEvalEntries`、`createRunnerSchedule`、`runScheduledTasks` 和 `aggregateRunResults`。 |
 | `vieval/core/scheduler` | 通过槽位池和中间件放行项目、任务和用例，导出 `createScheduler`；用 `Scheduler.child` 创建只属于某个项目、任务或用例组的池。 |
-| `vieval/core/scoring` | 用与任务和查看器相同的规则为用例结果计分，导出 `scoreCaseOutcomes`、`strictExactOf` 和 `isMeasuredState`。它没有副作用，浏览器代码也可以导入。 |
-| `vieval/core/processors/results` | 处理运行结果并执行内置的结果判定策略，导出 `processRunResults` 和 `ResultGateDecision`。 |
+| `vieval/core/scoring` | 用与任务、报告和查看器相同的规则为用例结果计分。`scoreCases(outcomes, definitions)` 返回各分数的汇总、执行成功率、通过用例比例和主分数名，自定义的 `defineTask` 可以直接把它作为 `TaskRunOutput` 返回。`deriveScores` 计算一次用例尝试的派生分数（`combine`），`poolScoreSummaries` 合并多次任务运行中同名的汇总，`resolveHeadline` 解析项目或运行的标题数值，`isMeasuredState` 判断某个状态是否得出了结果。它没有副作用，浏览器代码也可以导入。 |
 
 如果需要自行组织运行器、调度器、重试和缓存逻辑，请先阅读[可靠执行](/zh-hans/guide/learn/reliable-execution)。结果汇总、判定和报告文件的关系见[报告与比较](/zh-hans/guide/learn/reports-and-comparisons)。
 
diff --git a/docs/content/zh-hans/guide/core-concepts.md b/docs/content/zh-hans/guide/core-concepts.md
index 8658005..1bb5c47 100644
--- a/docs/content/zh-hans/guide/core-concepts.md
+++ b/docs/content/zh-hans/guide/core-concepts.md
@@ -36,7 +36,7 @@ next:
 
 **工作区**是列出 `projects` 的配置，项目可以内联书写，也可以从 `packages/*` 等其他配置中收集。**比较**是列出 `workspaces` 的配置，它为每个工作区启动一个独立进程，并排运行。
 
-**项目**指定从哪个目录发现评测文件，并保存推理执行器、矩阵等项目级配置。通过 `defineEval` 声明的评测项可以包含一个**任务**，`describeTask` 等 DSL 则会直接注册任务。运行器会组合每个评测项、每个推理执行器、每行运行矩阵和每行评测矩阵，为每组组合创建一个待调度任务。任务包含一个或多个**用例**；用例执行后会产生通过或失败的断言结果，也可以记录归一化分数和自定义指标。
+**项目**指定从哪个目录发现评测文件，并保存推理执行器、矩阵等项目级配置。通过 `defineEval` 声明的评测项可以包含一个**任务**，`describeTask` 等 DSL 则会直接注册任务。运行器会组合每个评测项、每个推理执行器、每行运行矩阵和每行评测矩阵，为每组组合创建一个待调度任务。任务包含一个或多个**用例**；用例执行后会产生通过或失败的断言结果，也可以记录具名分数和自定义指标。每个任务有一个**主分数**，报告把它作为标题数值显示，并在旁边显示**执行成功率**，即得出结果的用例所占的比例。
 
 Vieval 提供两种重复执行机制：尝试和重试。每次**尝试**都会执行任务中的全部用例；配置 `autoAttempt` 后，如果仍有符合条件的用例失败或超时，运行器会开始下一次尝试，并把各次结果都计入任务汇总。配置 `autoRetry` 后，**重试**只会在当前尝试中再次执行失败或超时的用例。
 
diff --git a/docs/content/zh-hans/guide/learn/assertions-scores-and-metrics.md b/docs/content/zh-hans/guide/learn/assertions-scores-and-metrics.md
index 3ef627a..fd55157 100644
--- a/docs/content/zh-hans/guide/learn/assertions-scores-and-metrics.md
+++ b/docs/content/zh-hans/guide/learn/assertions-scores-and-metrics.md
@@ -10,15 +10,15 @@ next:
 
 # 断言、分数与指标
 
-评测用例需要明确记录通过或失败，并在必要时补充分数和指标。本节先介绍可由代码直接判断的断言，再说明如何记录取值在 `0..1` 之间的分数和报告指标。
+评测用例需要明确记录通过或失败，并在必要时补充分数和指标。本节先介绍可由代码直接判断的断言，再说明如何声明具名分数、组合分数，以及如何记录报告指标。
 
 ## 从确定性断言开始
 
-Vieval 导出了与 Vitest 兼容的 `expect`。如果匹配失败，`expect` 会抛出 `AssertionError`，当前用例以 `failed` 结束：它得出了错误的结果，计入 `0` 分的 `exact` 分数。没有记录自定义分数的用例通过时，会得到 `1` 分。
+Vieval 导出了与 Vitest 兼容的 `expect`。如果匹配失败，`expect` 会抛出 `AssertionError`，当前用例以 `failed` 结束，表示它得出了错误的结果。
 
-其他错误，例如服务限流、连接中断或代码缺陷，会让用例以 `errored` 结束，表示没有得出结果。超时的用例以 `timeout` 结束，同样没有结果。这些用例不计入 `exact` 分数，而是降低覆盖率，即已结束的用例中得出结果的比例。运行汇总中的 `strictExactAverage` 保留了原来的口径：没有结果的用例按 `0` 分计算。阅读分数时请同时查看覆盖率：覆盖率低时，分数只反映了数据集的一部分。
+其他错误，例如服务限流、连接中断或代码缺陷，会让用例以 `errored` 结束，表示没有得出结果。超时的用例以 `timeout` 结束，同样没有结果。这些用例会降低**执行成功率**（completion rate），即得出结果的用例所占的比例。Vieval 在所有显示主分数的地方都会同时显示执行成功率，阅读时请把两者放在一起看：执行成功率低时，分数只反映了数据集的一部分。
 
-要表示结果错误，请使用 `expect` 或 `recordScore(0)`，不要直接抛出普通的 `Error`，否则会被视为“没有结果”。如果被测对象在困难输入上可能崩溃，而你希望把崩溃算作错误结果，可以在任务或用例上设置 `classifyError`：
+要表示结果错误，请使用 `expect`，不要直接抛出普通的 `Error`，否则会被视为“没有结果”。如果被测对象在困难输入上可能崩溃，而你希望把崩溃算作错误结果，可以在任务或用例上设置 `classifyError`：
 
 ```ts
 import { isRetriableProviderError } from 'vieval/core/inference-executors'
@@ -42,55 +42,196 @@ describeTask('normalized answer', ({ caseOf }) => {
 ```
 :::
 
-精确值、数据结构、必需字段，以及其他可以由代码可靠判断的规则，都适合使用 `expect`。
+精确值、数据结构、必需字段，以及其他可以由代码可靠判断的规则，都适合使用 `expect`。没有记录任何分数的任务就没有分数，它的标题数值是通过用例的比例，并标注为“通过用例”。
 
 ## 选择断言、分数或指标
 
 | 术语 | 在 Vieval 中的含义 | 对任务用例的影响 |
 | --- | --- | --- |
 | 断言（Assertion） | 检查结果是否满足要求。`expect` 失败时会抛出错误；`vieval/core/assertions` 还提供返回结构化结果的断言函数。 | `expect` 抛出的错误会使当前用例失败。核心断言函数的返回值不会自动写入任务结果。 |
-| 分数（Score） | 通过 `context.recordScore(value, kind)` 记录的 `0..1` 数值。分数类型可以是 `exact` 或 `judge`，默认为 `exact`。 | 用例通过时，分数会参与任务结果的汇总；相应事件也会写入报告中的用例记录。 |
-| 指标（Metric） | 通过 `context.recordMetric(name, value)` 记录的具名数据。值可以是字符串、数值、布尔值、`null`，或这些值组成的数组。 | 指标会写入报告事件和用例记录，但不会改变用例状态，也不参与分数汇总。 |
-| 评分标准（rubric） | 一类核心断言。`judge` 回调返回评分理由和分数，Vieval 再将分数与 `minScore` 比较；默认阈值为 `0.7`。 | 返回一个分数类型为 `judge` 的 `AssertionOutcome`。你需要自行决定如何将它写入当前用例。 |
+| 分数（Score） | 用 `defineScore` 声明、用 `context.recordScore(score, value)` 记录的具名数值，取值在 `0..1` 之间，越高越好。每个任务有一个主分数。 | 按分数名在任务的所有用例上汇总。失败的用例在每个分数中都按 `0` 计算。 |
+| 执行成功率（Completion rate） | 内置的数值，表示得出结果的用例比例：`(passed + failed) / (passed + failed + errored + timeout)`，不计跳过的用例。 | 总是显示在主分数旁边，评测无法关闭它。 |
+| 指标（Metric） | 通过 `context.recordMetric(name, value)` 记录的具名数据，例如耗时、费用、token 数、类别。值可以是字符串、数值、布尔值、`null`，或这些值组成的数组。 | 指标会写入报告事件和用例记录，但不会改变用例状态，也不影响任何分数。 |
+| 评分标准（rubric） | 一类核心断言。`judge` 回调返回评分理由和分数，Vieval 再将分数与 `minScore` 比较；默认阈值为 `0.7`。 | 返回一个 `AssertionOutcome`。由用例自己决定把它的 `score` 记到哪个分数上。 |
 
-`caseOf` 和 `casesFromInputs` 收到的用例回调上下文提供 `recordScore` 与 `recordMetric` 方法。
+`caseOf` 和 `casesFromInputs` 收到的用例回调上下文提供 `recordScore`、`recordUnscored` 与 `recordMetric` 方法。
 
-## 在用例中记录分数与指标
+## 声明分数
 
-需要计入任务平均分的数据应使用 `recordScore`。只用于筛选、分组或排查问题的数据应使用 `recordMetric`。
+`defineScore(name, metadata)` 用来声明一个分数，用法和 `defineMetric` 一样。`name` 是写入报告的稳定标识。名称（label）和说明（description）可以是一个字符串，也可以按 BCP 47 语言标签各写一份；viewer 会按读者的语言选择，找不到时依次回退到 `en` 和第一项。在 viewer 中把鼠标悬停在分数上，就能看到它的说明。
 
 ::: code-group
 ```ts [evals/retrieval.eval.ts]
-import { describeTask, expect } from 'vieval'
+import { defineScore, describeTask, expect } from 'vieval'
+
+const recall = defineScore('retrieval.recall', {
+  description: {
+    'en': 'Share of the expected documents that the retriever returned.',
+    'zh-Hans': '检索结果中包含的期望文档所占的比例。',
+  },
+  label: { 'en': 'Recall', 'zh-Hans': '召回率' },
+})
 
 describeTask('retrieval', ({ caseOf }) => {
   caseOf('finds the expected documents', ({ recordMetric, recordScore }) => {
     const expected = new Set(['doc-a', 'doc-b'])
     const retrieved = ['doc-a', 'doc-c']
     const matches = retrieved.filter(id => expected.has(id)).length
-    const recall = matches / expected.size
 
-    recordScore(recall, 'exact')
+    recordScore(recall, matches / expected.size)
     recordMetric('benchmark.case.id', 'retrieval-basic')
     recordMetric('retrieved.count', retrieved.length)
 
-    expect(recall).toBeGreaterThan(0)
+    expect(matches).toBeGreaterThan(0)
   })
+}, { scores: [recall] })
+```
+:::
+
+- 分数必须是 `0..1` 之间的有限数值。在同一次用例尝试中，同名分数以最后一次 `recordScore` 为准。
+- `recordScore` 也可以直接传分数名，例如 `recordScore('retrieval.recall', 0.5)`，这时分数没有名称和说明。
+- 用例第一次记录某个定义时，Vieval 会把它写入报告。也可以在 `describeTask` 的 `scores` 选项中列出任务的分数，让它们在任务开始时就写入报告。列出的分数在第一个用例之前就是已知的；通过的用例如果没有记录它，就算作缺失。
+
+## 选择主分数
+
+页面、排行榜、对比和 CLI 汇总行会优先显示每个任务的一个分数，并在数值旁边显示它的名称。这个分数就是**主分数**（primary）。
+
+- 用 `primary: true` 标记主分数。一个任务最多只有一个主分数，声明第二个时会抛出错误。
+- 只有一个分数的任务不需要标记，这个分数就是主分数。
+- 有多个分数但都没有标记的任务没有主分数，标题数值会回退为通过用例的比例。
+- 如果一次运行中各任务的主分数不同，运行级别就不显示标题分数，而是按任务名分别列出各自的主分数，运行的标题数值只保留执行成功率。
+
+## 组合分数
+
+派生分数由同一次用例尝试中的其他分数计算得出。当标题数值要回答的是几项测量组合起来的问题时，例如“听写正确并且读法自然”，就可以使用派生分数：
+
+::: code-group
+```ts [evals/read-aloud.eval.ts]
+import { defineScore, describeTask } from 'vieval'
+
+const correct = defineScore('tts.correct', {
+  label: { 'en': 'Written back correctly', 'zh-Hans': '听写正确' },
+})
+const natural = defineScore('tts.natural', {
+  description: { 'en': 'The judge median: natural 1, mixed 0.5, spelled 0.', 'zh-Hans': '评委中位数：natural 1，mixed 0.5，spelled 0。' },
+  label: { 'en': 'Natural reading', 'zh-Hans': '读法自然' },
+})
+const pass = defineScore('tts.pass', {
+  // 不带阈值的输入相当于 `atLeast: 1`，所以 mixed（0.5）不算通过。
+  combine: { all: [correct, natural] },
+  label: { 'en': 'Pass', 'zh-Hans': '通过' },
+  primary: true,
 })
+
+describeTask('read aloud', ({ caseOf }) => {
+  caseOf('reads a fraction', async ({ recordScore }) => {
+    const { listenerScore, readingScore } = await readAloud('1/2')
+    recordScore(correct, listenerScore)
+    recordScore(natural, readingScore)
+    // `pass` 会在这次尝试结束时自动计算；直接记录它会抛出错误。
+  })
+}, { scores: [correct, natural, pass] })
 ```
 :::
 
-分数必须是 `0..1` 之间的有限数值。在同一个用例中，如果再次记录相同类型的分数，新值会替换旧值。因此，每种分数类型最好只记录一次最终结果。
+`combine` 只支持三种形式，这样 viewer 才能显示公式，例如“全部满足：听写正确、读法自然”：
+
+| 形式 | 取值 |
+| --- | --- |
+| `{ all: [a, { score: b, atLeast: 0.5 }] }` | 每个输入都达到阈值时为 `1`，否则为 `0`。不带阈值的输入要达到 `1` 才算满足。`all` 的结果总是 0 或 1。 |
+| `{ mean: [a, b] }` | 各输入的平均值。 |
+| `{ weighted: [{ score: a, weight: 2 }, { score: b, weight: 1 }] }` | 按 `weight` 加权的平均值，权重会自动归一化。 |
+
+- 派生分数必须列在 `describeTask` 的 `scores` 选项中，因为没有用例会记录它。
+- 每次用例尝试结束时，只要派生分数的所有输入都已记录，运行时就会计算它。缺少任何一个输入时，这个用例的派生分数也是缺失的。
+- 用例不能直接记录派生分数，否则 `recordScore` 会抛出错误。
+- 如果需要其他计算规则，请在用例里自己算出数值，记录为普通分数，并在 `description` 中写明规则。
+
+## 用例状态如何计入分数
+
+每个用例在每个分数中如何计算，只由一条规则决定。任务结果、`run-summary.json`、`cases.jsonl` 和 viewer 都使用这条规则：
+
+| 用例状态 | 任务的每个分数 |
+| --- | --- |
+| `passed` | 用例记录的值；没有记录时为缺失。 |
+| `failed` | `0`。断言失败，说明用例得出了错误的结果。用例记录过的值会作为详情保留，但不参与计算。 |
+| `errored`、`timeout` | 默认不计入（`errors: 'exclude'`）。设置 `errors: 'zero'` 时按 `0` 计算。 |
+| `skipped` | 不计入。 |
+
+`errors` 按分数分别设置。`exclude` 的出发点是：服务故障不是被测对象的结果，被排除的用例数量由执行成功率体现。`zero` 适合把每个错误都算作答错的基准测试，例如 Harbor 和 Terminal-Bench：
+
+```ts
+const solved = defineScore('bench.solved', { errors: 'zero', primary: true })
+```
+
+每个分数的汇总包含 `mean`（计入用例的平均值，没有用例计入时为 `null`）、`n`（计入的用例数）和 `total`（已结束且未跳过的用例数）。如果没有选中任何用例，例如 `--case` 过滤后为空，任务不会返回分数。
+
+## 记录无法评分的情况
+
+有时用例运行完了，评分方却给不出分数：评委的回答无法解析、评分程序出错，或者输出为空。这既不是错误的结果，也不是执行错误。请用 `recordUnscored` 记录：
+
+```ts
+caseOf('reads a fraction', async ({ recordScore, recordUnscored }) => {
+  const verdict = await judgeReading(audio)
+  if (verdict == null) {
+    recordUnscored(natural, 'invalid_response')
+    return
+  }
+  recordScore(natural, verdict.score)
+})
+```
+
+- `reason` 是一个字符串。viewer 会为 `grader_failed`、`invalid_response`、`no_response` 和 `refusal` 显示专门的名称，其他字符串按原样显示。
+- 无论 `errors` 如何设置，无法评分的值都不参与计算。它不等于 `0`，因为它没有反映被测对象的任何情况。
+- 同一次尝试中，同名的无法评分记录和分数值会互相替换，以最后一次调用为准。
+- 派生分数的输入中只要有一个无法评分，派生分数也无法评分，原因为 `input_unscored`。
+- 用例状态不变。通过的用例即使有无法评分的值，仍然是通过；执行成功率也不变。
+- 每个分数的汇总会在 `unscored` 中按原因统计次数，例如 `{ grader_failed: 2 }`。这些用例计入 `total`，但不计入 `n`。
+
+## 说明分数的来由：理由与评分方
+
+记录分数时，可以说明为什么是这个值，以及由谁给出。viewer 会在用例页面的分数标记旁显示这些信息：
+
+```ts
+recordScore(natural, median(verdicts.map(verdict => verdict.score)), {
+  grader: { id: 'judge', kind: 'model', model: judgeModel.model, reducer: 'median', votes: verdicts.length },
+  metadata: { verdicts },
+  rationale: verdicts.map(verdict => verdict.reason).join('\n'),
+})
+```
+
+- `rationale` 是评委的推理过程，或命中的规则。
+- `grader` 说明分数由谁给出。`kind` 可以是 `rule`（确定性代码）、`model`（LLM 或其他模型）、`human`（人工评审）或 `external`（其他评测框架的验证程序）。`id`、`model`、`version`、`votes` 和 `reducer` 都是可选的。
+- `metadata` 存放其他详情，以对象树的形式显示。
+- 常用的评分方可以在定义里声明一次：`defineScore(name, { grader: { kind: 'model', id: 'judge' } })`。单次记录可以覆盖它，例如评委模型本身是矩阵的一个维度时。
+- 派生分数没有自己的评分方，viewer 会显示其输入的评分方。
+- `grader.id` 与评委 `inference.*` 事件中的 `role` 一致，viewer 据此把分数和对应的评委调用关联起来。
+
+评分方是分数含义的一部分。如果两次运行的评分方在 `kind`、`model` 或 `version` 上不同，它们测量的就不是同一件事，对比这两次运行时，该分数旁边会显示警告。
+
+## 分数与指标的区别
+
+分数回答的是“这个用例做得有多好”，取值在 `0..1` 之间，越高越好。其他数据都应该记为指标：取值范围、优劣方向、单位、计数和类别都属于指标。
+
+耗时是指标，而不是分数。为它设置方向，报告就知道数值越低越好：
+
+```ts
+import { defineMetric } from 'vieval'
 
-任务汇总分数和报告中的用例记录并非同时写入。每次调用 `recordScore` 都会立即发出事件，所以用例失败前记录的分数仍可能出现在报告中。
+const latency = defineMetric('read_aloud.latency', {
+  better: 'lower',
+  label: { 'en': 'Read-aloud latency', 'zh-Hans': '朗读耗时' },
+  unit: 'ms',
+})
 
-如果用例最终失败，任务汇总会忽略这些自定义分数，并为该用例计入一个 `exact: 0`；如果用例出错或超时，任务汇总不会把它计入 `exact` 分数。对于失败的用例，只有报告中的用例记录还没有 `exact` 分数时，Vieval 才会补写 `exact: 0`。`errored` 和 `timeout` 的记录不会补写分数，因此 `vieval report` 计算平均值时会跳过它们。因此，排查结果时应同时检查用例的 `state` 和分数记录。
+recordMetric(latency, elapsedMs)
+```
 
-指标用于筛选、分组和诊断报告。单独记录指标不会让用例通过或失败，也不会提高任务汇总分数。
+指标用于筛选、分组和诊断报告。单独记录指标不会让用例通过或失败，也不会改变任何分数。
 
 ## 给指标起一个易读的名字
 
-viewer 会按原样显示 `osworld.model.usage.output_tokens` 这类指标名。用 `defineMetric` 定义一次指标，就可以给它加上名称（label）、说明（description）、单位（unit）和方向（better）。名称和说明可以是一个字符串，也可以按 BCP 47 语言标签各写一份；viewer 会按读者的语言选择，找不到时依次回退到 `en` 和第一项。
+viewer 会按原样显示 `osworld.model.usage.output_tokens` 这类指标名。用 `defineMetric` 定义一次指标，就可以给它加上名称（label）、说明（description）、单位（unit）和方向（better）。名称和说明的格式与分数相同。
 
 ::: code-group
 ```ts [evals/agent.eval.ts]
@@ -117,7 +258,7 @@ describeTask('agent', ({ caseOf }) => {
 ```
 :::
 
-第一个带有元数据的值会为当前任务发出一个 `metric.defined` 事件。相同的元数据不会重复发出，元数据变化时会再发一次。每个指标名最好只保留一套元数据，因为 viewer 只显示最新的一套。在 viewer 里，鼠标悬停在指标上，可以看到它的名称、原始指标名、说明、单位、方向，以及它在这个任务所有用例中的取值分布。
+第一个带有元数据的值会为当前任务发出一个 `metric.defined` 事件。相同的元数据不会重复发出，元数据变化时会再发一次。每个指标名最好只保留一套元数据，因为 viewer 只显示最新的一套。在 viewer 里，鼠标悬停在指标上，可以看到它的名称、原始指标名、说明、单位、方向，以及它在这个任务所有用例中的取值分布。分数也会出现在同一个指标面板中，排在指标前面，并始终以越高越好的百分比显示。
 
 `recordArtifact` 也接受同样格式的 `label` 和 `description`，例如 `recordArtifact('final.webp', { contentType: 'image/webp', path, label: { 'en': 'Final screen', 'zh-Hans': '最终画面' } })`。
 
@@ -146,16 +287,21 @@ const [outcome] = await evaluateAssertions([
 
 这个示例只执行本地代码，不会调用模型。`expectRubric` 本身既不选择模型，也不请求模型服务；是否调用模型完全由 `judge` 回调决定。这个回调也可以使用本地规则或人工评分结果。
 
-核心断言 API 与任务 DSL 是两套独立的公开接口。`evaluateAssertions` 只返回断言结果，不会自动调用用例上下文中的 `recordScore`、`recordMetric` 或 `expect`。`toRunScores(outcomes)` 可以把这些结果转换成 `RunScore[]`，但同样不会将分数写入当前用例。
+核心断言 API 与任务 DSL 是两套独立的公开接口。`evaluateAssertions` 只返回断言结果，不会调用用例上下文中的 `recordScore`、`recordMetric` 或 `expect`。请显式地把结果记到它所测量的分数上：
 
-得到 `outcome` 后，你可以调用 `recordScore(outcome.score, outcome.scoreKind)` 记录分数，在 `outcome.pass` 为 `false` 时抛出错误，也可以将评分理由记为指标。选择哪种方式取决于你希望它如何影响用例状态和任务汇总。
+```ts
+const concise = defineScore('answer.concise', { grader: { id: 'concise-answer', kind: 'rule' } })
+
+recordScore(concise, outcome.score, { rationale: outcome.reason })
+expect(outcome.pass).toBe(true)
+```
 
-如果你通过抛出 `AssertionError`（例如借助 `expect`）使当前用例失败，任务汇总会为该用例计入 `exact: 0`；抛出其他错误时，除非 `classifyError` 另有规定，用例会以 `errored` 结束。抛错前已经发出的分数事件仍可能保留在报告的用例记录中。
+是否同时让用例失败，需要你自己决定。抛出 `AssertionError`（例如借助 `expect`）会让用例以 `failed` 结束，任务的每个分数都会把它按 `0` 计算；抛出其他错误时，除非 `classifyError` 另有规定，用例会以 `errored` 结束。
 
 ::: warning 模型评分会带来运行成本
 如果 `judge` 回调调用模型，就需要配置对应的模型服务和凭据，并承担请求延迟和费用。并发运行用例时，可能同时发出多次评分请求，因此应根据服务方的速率限制和配额设置并发上限。还应检查服务方如何处理数据，并决定哪些用例输入、模型输出、评分提示词、评分理由、指标和事件可以写入报告文件。
 :::
 
-常见错误包括把指标当成分数，或误以为评分标准断言的结果会自动写入当前用例。编写评测时，应分别处理用例状态、参与汇总的分数，以及用于排查问题的指标。
+常见错误包括把指标当成分数，或误以为评分标准断言的结果会自动写入当前用例。编写评测时，应分别处理用例状态、分数，以及用于排查问题的指标。
 
 下一步在[模型与推理执行器](/zh-hans/guide/learn/models-and-inference-executors)中学习如何注册并选择模型。所有核心断言 API 都可以从 [`vieval/core/assertions`](/zh-hans/api/) 导入。
diff --git a/docs/content/zh-hans/guide/learn/matrices-and-datasets.md b/docs/content/zh-hans/guide/learn/matrices-and-datasets.md
index 2f341b1..117b782 100644
--- a/docs/content/zh-hans/guide/learn/matrices-and-datasets.md
+++ b/docs/content/zh-hans/guide/learn/matrices-and-datasets.md
@@ -114,6 +114,7 @@ Vieval 按以下顺序合并矩阵配置：
 
 ```ts [evals/layered.eval.ts]
 import { defineEval, defineTask } from 'vieval/config'
+import { scoreCases } from 'vieval/core/scoring'
 
 export default defineEval({
   description: 'Shows matrix layering.',
@@ -139,12 +140,12 @@ export default defineEval({
       },
     },
     run(context) {
-      return {
-        scores: [{
-          kind: 'exact',
-          score: context.task.matrix.eval.rubric === 'strict' ? 1 : 0,
-        }],
-      }
+      // 自定义任务需要返回分数汇总，`scoreCases` 可以根据用例结果生成它。
+      return scoreCases([{
+        scores: { 'layered.strict': context.task.matrix.eval.rubric === 'strict' ? 1 : 0 },
+        state: 'passed',
+        unscored: {},
+      }])
     },
   }),
 })
diff --git a/docs/content/zh-hans/guide/learn/reliable-execution.md b/docs/content/zh-hans/guide/learn/reliable-execution.md
index d3c5957..9fc7cbe 100644
--- a/docs/content/zh-hans/guide/learn/reliable-execution.md
+++ b/docs/content/zh-hans/guide/learn/reliable-execution.md
@@ -150,7 +150,7 @@ const providerRetry = {
 
 用例重试会重新执行整个用例。如果只想重试失败的那一次服务调用，可以用 `vieval/core/inference-executors` 的 `runWithRetry` 包住这次调用。它默认在服务的 `Retry-After` 和带随机抖动的指数退避之间取较长的一个作为等待时间；传入 `{ signal }` 后，运行被中断时会立即停止等待。
 
-`autoAttempt: 1` 表示首次尝试结束后，最多再运行一次完整用例集。只有某个用例在用完重试次数后仍然失败或超时，并且尚未达到自己的尝试上限，Vieval 才会开始下一次尝试。每次完成的尝试都会参与任务分数计算。例如，同一个用例第一次失败、第二次通过时，最终的 `exact` 平均分为 `0.5`，后一次通过不会覆盖前一次失败。
+`autoAttempt: 1` 表示首次尝试结束后，最多再运行一次完整用例集。只有某个用例在用完重试次数后仍然失败或超时，并且尚未达到自己的尝试上限，Vieval 才会开始下一次尝试。每次完成的尝试都会参与任务分数计算。例如，同一个用例第一次失败、第二次通过并记录 `1` 分时，它在该分数上的平均值为 `0.5`，后一次通过不会覆盖前一次失败。
 
 ::: tip 根据要回答的问题选择重复方式
 如果要观察多次执行的通过率，请使用 `autoAttempt`，因为每次尝试的结果都会参与任务分数计算。`autoRetry` 更适合应对网络抖动等临时故障：只要某次重试通过，这次尝试就按通过计算，不会把此前的失败计入任务分数。各次运行仍会写入生命周期事件，便于排查问题。
@@ -172,7 +172,7 @@ describeTask('read aloud', build, {
 
 只有没有得出结果的运行（`errored`、`timeout`）会被重新排队；断言失败已经是一个结果。用例在等待期间会让出槽位；之后的每次运行都会重新获得 `autoRetry` 次重试，所以一个用例最多运行 `(1 + autoRetry) × (1 + requeue)` 次。剩下的用例不多时，队列末尾可能马上就轮到它，这时没有 `requeueDelay` 的重新排队就等于立刻重试。
 
-在最后一次运行结束之前，用例都还没有结束：它发出的是 `case.requeued` 而不是 `case.ended`，查看器把它显示为等待重跑（尚未完成），它也不计入分数和覆盖率。只有最终那次运行会计分。重试和重新排队次数都用完后仍没有结果的用例以 `errored` 结束，之后可以用 `--resume` 继续。
+在最后一次运行结束之前，用例都还没有结束：它发出的是 `case.requeued` 而不是 `case.ended`，查看器把它显示为等待重跑（尚未完成），它也不计入分数和执行成功率。只有最终那次运行会计分。重试和重新排队次数都用完后仍没有结果的用例以 `errored` 结束，之后可以用 `--resume` 继续。
 
 ### 继续之前的尝试
 
@@ -241,13 +241,13 @@ export default defineConfig({
 
 任务分数和报告文件保留的信息不同：
 
-- 失败或超时的用例会为任务增加一个值为 `0` 的 `exact` 分数；没有自定义分数的通过用例会增加 `1`。
-- 通过用例发出的 `exact` 和 `judge` 分数会参与任务计算；失败用例在运行中发出的自定义分数不会参与任务计算。
+- 失败的用例在任务的每个分数中都按 `0` 计算。出错或超时的用例在 `errors: 'exclude'`（默认）的分数中不计入，在 `errors: 'zero'` 的分数中按 `0` 计算；无论哪种情况，它都会降低执行成功率。
+- 通过的用例按它记录的值参与计算。通过但没有记录某个分数的用例，在该分数中算作缺失，而不是按 `1` 计算。
 - 每次完成的自动尝试都会再向任务分数加入一组用例结果。
 - 执行过程会产生生命周期、分数和指标事件。除非使用 `--no-report`，这些事件都会按顺序写入 `events.jsonl`。
-- `cases.jsonl` 最终为每个「任务 ID + 用例 ID」组合写一条记录，多次重试或自动尝试不会分别占一行。对于启用自动尝试的任务，该记录的 `attempts` 会列出每次尝试的结果。要查看准确的执行顺序，请读取 `events.jsonl`（`case.started` 和 `case.ended` 带有 `attemptIndex`）；要查看包含所有尝试结果的任务分数，请读取 `run-summary.json`。
+- `cases.jsonl` 最终为每个「任务 ID + 用例 ID」组合写一条记录，多次重试或自动尝试不会分别占一行。记录中的 `scores` 按与任务结果相同的规则保存每个分数的计入值，因此失败用例显示为 `0`。对于启用自动尝试的任务，该记录的 `attempts` 会列出每次尝试的结果。要查看准确的执行顺序，请读取 `events.jsonl`（`case.started` 和 `case.ended` 带有 `attemptIndex`）；要查看包含所有尝试结果的任务分数，请读取 `run-summary.json`。
 
-正如[断言、分数与指标](/zh-hans/guide/learn/assertions-scores-and-metrics)所述，用例在失败前发出的分数事件可能仍会出现在报告记录中，但任务分数会把该用例计为失败。排查问题时，请同时检查用例状态和分数。
+正如[断言、分数与指标](/zh-hans/guide/learn/assertions-scores-and-metrics)所述，用例在失败前记录的分数仍会保留在 `events.jsonl` 和用例记录的 `scoreDetails` 中，但不参与计算，用例记录的 `scores` 显示为 `0`。排查问题时，请同时检查用例状态和分数。
 
 不要使用 `autoRetry` 统计多次执行的通过率：重试后恢复的用例在任务分数中只算一次通过。如果希望早期失败也参与最终分数，请使用 `autoAttempt`，并注意它会重新运行已经通过的用例。`--attempt` 不会增加自动尝试次数，它只为运行设置标签。
 
diff --git a/docs/content/zh-hans/guide/learn/reports-and-comparisons.md b/docs/content/zh-hans/guide/learn/reports-and-comparisons.md
index c98ad42..d5b81d9 100644
--- a/docs/content/zh-hans/guide/learn/reports-and-comparisons.md
+++ b/docs/content/zh-hans/guide/learn/reports-and-comparisons.md
@@ -35,11 +35,11 @@ vieval run \
 
 | 文件 | 内容 |
 | --- | --- |
-| `run-summary.json` | 恰好一个项目的运行结果，包括 `schemaVersion: 1`、`runId`、`reportDirectory`、各类 ID、项目状态、用例计数与失败信息、矩阵摘要和汇总分数。`vieval report` 命令遇到其他版本时会报错。 |
-| `events.jsonl` | 按发生顺序记录运行、任务、用例、分数、指标和自定义事件。 |
-| `cases.jsonl` | 每个「任务 ID + 用例 ID」组合的最终记录，包括状态、耗时、根据事件推算的重试次数、分数、指标以及可选的输入和输出。 |
-| `metrics-summary.json` | 从 `cases.jsonl` 计算出的分数数量、总和与平均值。 |
-| `otlp/traces.json`、`otlp/logs.json`、`otlp/metrics.json` | 从用例记录转换得到的本地 OTLP 数据。 |
+| `run-summary.json` | 恰好一个项目的运行结果，包括 `schemaVersion: 2`、`runId`、`reportDirectory`、各类 ID、项目状态、用例计数与失败信息、矩阵摘要、每次任务运行的分数汇总、任务声明的 `scoreDefinitions`，以及解析好的标题数值（见下文）。`vieval report` 命令遇到其他版本时会报错。 |
+| `events.jsonl` | 按发生顺序记录运行、任务、用例、分数（`score.defined`、`score.recorded`）、指标和自定义事件。 |
+| `cases.jsonl` | 每个「任务 ID + 用例 ID」组合的最终记录（`schemaVersion: 2`），包括状态、耗时、根据事件推算的重试次数、分数、指标以及可选的输入和输出。`scores` 按分数名保存计入的值，`unscored` 按分数名保存无法评分的原因，`scoreDetails` 保存每条记录附带的 `rationale`、`grader` 和 `metadata`。 |
+| `metrics-summary.json` | 从 `cases.jsonl` 按分数名计算出的数量、总和与平均值。 |
+| `otlp/traces.json`、`otlp/logs.json`、`otlp/metrics.json` | 从用例记录转换得到的本地 OTLP 数据。分数指标名为 `vieval.score.<name>`。 |
 
 `vieval report index` 默认会创建 `index/runs.jsonl`，也可以通过 `--output` 指定其他位置；`vieval run` 本身不会写入这个索引文件。
 
@@ -49,12 +49,30 @@ vieval run \
 
 `cases.jsonl` 中的 `retryCount` 只统计最后一次尝试内的重试。对于启用自动尝试的任务，它根据 `attemptIndex` 计算，后续尝试不会被算作重试。在 `attemptIndex` 出现之前写入的报告没有这个字段，其中后续尝试的重复开始事件仍可能让 `retryCount` 增加。
 
-事件行仍为 `schemaVersion: 2`，用例记录仍为 `schemaVersion: 1`：新增可选字段或新事件名不改变版本号，读取方会忽略不认识的字段；重命名、删除字段或改变字段类型时才会增加版本号。
+事件行为 `schemaVersion: 3`，用例记录为 `schemaVersion: 2`。事件的第 3 版用具名分数取代了固定的 `exact` 和 `judge` 两类分数。新增可选字段或新事件名不改变版本号，读取方会忽略不认识的字段；重命名、删除字段或改变字段类型时才会增加版本号。读取方会拒绝旧版本写入的报告（“This report was written by an older vieval. Run it again.”），viewer 会把这类运行标记为不支持；没有迁移命令，请重新运行。
 
 ::: warning 报告文件可能包含敏感信息
 报告可能包含用例输入和输出、错误信息、自定义指标、模型或评测基准标识符，以及其他事件数据。请设置合适的保留期限和访问权限，并删除或脱敏不应离开评测环境的字段。
 :::
 
+## 读取标题数值
+
+每个项目和整个运行的标题数值只在运行结束时解析一次，以 `{ headline, completion }` 的形式写入 `run-summary.json`。CLI、viewer 和 `vieval report` 都直接读取它，不会各自重新计算：
+
+| `headline.kind` | 条件 | 显示方式 |
+| --- | --- | --- |
+| `score` | 所有带分数的任务使用同一个主分数名。 | 合并后的主分数及其名称，例如 `通过 87.5%`。 |
+| `per-task` | 各任务的主分数不同。 | 不显示运行级别的分数，按任务名分别列出各自的主分数。 |
+| `passed-cases` | 没有任何任务有主分数。 | 通过用例的比例，标注为“通过用例”。 |
+
+`completion` 是执行成功率，即 `(passed + failed) / (passed + failed + errored + timeout)`，总是显示在标题数值旁边。运行结束时的汇总行会同时输出两者，例如：
+
+```text
+Pass 87.5% · completed 96.8% (4 errored)
+```
+
+每个分数汇总都包含 `name`、`mean`（没有用例计入时为 `null`）、`n`（计入的用例数）、`total`（已结束且未跳过的用例数）、`sum` 和 `unscored`（按原因统计的次数）。项目和运行的汇总会把各任务的用例合并后再计算，而不是对各任务的平均值再求平均。
+
 ## 索引、检查与分析报告
 
 每个报告命令都需要一个报告路径。这个路径通常既可以是单次运行目录，也可以是包含多次运行的上层目录。
@@ -97,7 +115,7 @@ vieval report attempts --config vieval.config.ts
 
 :::
 
-`attempts` 按时间从新到旧列出报告目录中的尝试：每个实验和尝试一行，显示涉及的工作区、通过、失败和没有结果的用例数，以及覆盖率。"←" 标记 `vieval run --resume` 不指定标识时会继续的那次尝试；备注列显示该尝试继续自哪次尝试、是否被中断，以及没来得及写入摘要就被终止的运行。可以传入报告目录，或用 `--config` 列出配置文件旁边的报告；用 `--experiment` 和 `--workspace` 筛选，用 `--format table|json` 选择输出格式。
+`attempts` 按时间从新到旧列出报告目录中的尝试：每个实验和尝试一行，显示涉及的工作区、通过、失败和没有结果的用例数，以及执行成功率。"←" 标记 `vieval run --resume` 不指定标识时会继续的那次尝试；备注列显示该尝试继续自哪次尝试、是否被中断，以及没来得及写入摘要就被终止的运行。可以传入报告目录，或用 `--config` 列出配置文件旁边的报告；用 `--experiment` 和 `--workspace` 筛选，用 `--format table|json` 选择输出格式。
 
 ## 比较两组已有报告
 
@@ -115,7 +133,7 @@ vieval report compare <基线报告目录> <候选方案报告目录>
 
 1. **先对齐用例。** 命令从基线和候选方案中读取 `cases.jsonl`，根据用例键找到同一个用例。候选方案独有的记录记为新增，基线独有的记录记为移除。
 2. **再计算匹配项。** 单个用例和分组的差值都按「候选方案减基线」计算，并且只使用已经对齐的记录。使用 `--group-by` 时，分组值取自候选方案。
-3. **处理缺失分数。** 已对齐的用例如果在某份报告中缺少 `--score-kind` 指定的分数，对应分数按 `0` 计算。`--score-kind` 默认为 `exact`。
+3. **处理缺失分数。** 已对齐的用例如果在某份报告中缺少 `--score <name>` 指定的分数，对应分数按 `0` 计算。`--score` 默认取 `run-summary.json` 中的主分数。
 4. **单独计算总体差值。** 命令分别对基线和候选方案中所有包含该分数的记录求平均值，再用候选平均值减去基线平均值。新增和移除的用例会影响总体差值；缺少所选分数的记录不会进入这一步。因此，总体差值不一定等于匹配用例差值的平均值。
 
 输出格式支持 `table` 和 `json`。
@@ -158,7 +176,7 @@ vieval run --config evals/agent-memory/vieval.config.ts
 - 比较配置的 `env`（以及单个条目的 `env`）优先于各工作区配置的 `env`，并在工作区插件运行之前生效。
 - `cache: { namespace }` 让所有工作区共用比较配置旁边的 `.vieval/cache/<比较 ID>/<namespace>/`。
 - 终端中只有一个实时窗口，显示所有工作区的 `Workspaces`、`Projects`、`Tasks`、`Cases`、`Concurrency` 和 `Time` 行。某个工作区失败不会中止其他工作区。
-- 运行结束后会输出排行榜：每个工作区的每个项目占一行，按混合分数（hybrid）排序；随后列出失败的工作区及其 stderr 的最后几行。
+- 运行结束后会输出排行榜：每个工作区的每个项目占一行，按主分数排序，执行成功率单独占一列；随后列出失败的工作区及其 stderr 的最后几行。如果各工作区的主分数不同，排行榜会提示这一点，并改为按通过用例排序。
 - 只要有工作区失败或存在失败的用例，退出码就是 `1`。`--json` 会输出比较结果。
 - 报告写入比较配置旁边的 `.vieval/reports`。实验 ID 默认为比较 ID，所有工作区共用同一个尝试，因此比较中的每次运行都位于 `<工作区>/<项目>/<比较 ID>/<尝试>/` 下。
 - 使用 `--with-viewer` 时，只启动一个查看器，展示比较的报告根目录。
@@ -237,7 +255,7 @@ vieval report compare \
   .vieval/reports/baseline \
   .vieval/reports/candidate \
   --case-key benchmark.case.id \
-  --score-kind exact \
+  --score retrieval.recall \
   --format table
 ```
 
diff --git a/docs/content/zh-hans/guide/learn/tasks-cases-and-inputs.md b/docs/content/zh-hans/guide/learn/tasks-cases-and-inputs.md
index a31157f..f3e895c 100644
--- a/docs/content/zh-hans/guide/learn/tasks-cases-and-inputs.md
+++ b/docs/content/zh-hans/guide/learn/tasks-cases-and-inputs.md
@@ -85,7 +85,7 @@ caseOf(..., { input }) 或 casesFromInputs(...)
 
 在参与比较的每一份报告中，`benchmark.case.id` 都不能重复，即使这些用例来自不同任务或项目。如果出现重复值，比较命令会报错，而不会自行猜测应该如何配对。
 
-报告选择器可以读取 `caseName`、`caseId` 等用例字段，也可以读取分数和已记录的指标。比较用例时，Vieval 默认依次尝试 `benchmark.case.id`、`vieval.case.id` 和自动生成的 `caseId`。
+报告选择器可以读取 `caseName`、`caseId` 等用例字段，也可以按名称读取分数（`scores.<name>`）和已记录的指标。比较用例时，Vieval 默认依次尝试 `benchmark.case.id`、`vieval.case.id` 和自动生成的 `caseId`。
 
 ::: warning 在任务或评测中注册用例
 `caseOf` 和 `casesFromInputs` 必须在 `describeTask` 或 `describeEval` 的回调中调用。在其他位置调用时，Vieval 会在加载评测文件时抛出错误。
diff --git a/docs/content/zh-hans/index.md b/docs/content/zh-hans/index.md
index bd605cd..68f5548 100644
--- a/docs/content/zh-hans/index.md
+++ b/docs/content/zh-hans/index.md
@@ -20,9 +20,9 @@ home:
       ✓ |desktop-agent| (412 cases)
         cases 407 passed | 5 failed | 0 timeout
       $ pnpm vieval report cases .vieval/reports/candidate --where state=failed
-        ✗ chrome/open-settings  exact 0  wrong tab
+        ✗ chrome/open-settings  pass 0  wrong tab
         ✗ gimp/resize-image  timeout  60m
-        ✗ vscode/rename-symbol  exact 0  old name kept
+        ✗ vscode/rename-symbol  pass 0  old name kept
         ✗ calc/sum-column  rubric 0.42  partial
         ✗ chat/refund-policy  rubric 0.31  hallucinated
   blocks:
diff --git a/examples/run-matrix-multi-models/evals/extract.eval.ts b/examples/run-matrix-multi-models/evals/extract.eval.ts
index fe7fc3f..ab08020 100644
--- a/examples/run-matrix-multi-models/evals/extract.eval.ts
+++ b/examples/run-matrix-multi-models/evals/extract.eval.ts
@@ -1,7 +1,19 @@
-import { caseOf, describeTask } from 'vieval'
+import { caseOf, defineScore, describeTask } from 'vieval'
 
 import { extract } from './agent'
 
+/**
+ * The one score of this task, so it is also the task's primary score.
+ */
+const extractScore = defineScore('sleep_matrix.extract', {
+  description: {
+    'en': 'Mock extraction quality: a random value in 0..1 returned by the agent.',
+    'zh-Hans': '模拟的抽取质量：agent 返回的 0..1 之间的随机值。',
+  },
+  grader: { id: 'mock', kind: 'rule' },
+  label: { 'en': 'Extract', 'zh-Hans': '抽取' },
+})
+
 async function scoreExtract(input: number) {
   return input
 }
@@ -9,6 +21,6 @@ async function scoreExtract(input: number) {
 describeTask('sleep-matrix', () => {
   caseOf('sleep mock', async (context) => {
     const res = await extract()
-    context.recordScore(await scoreExtract(res))
+    context.recordScore(extractScore, await scoreExtract(res))
   })
-})
+}, { scores: [extractScore] })
diff --git a/packages/ui/API.md b/packages/ui/API.md
index 2ab40cd..13e9aa9 100644
--- a/packages/ui/API.md
+++ b/packages/ui/API.md
@@ -33,13 +33,13 @@ schemas. This document explains the routes and the rules behind the numbers.
 | `live-bus` | Run of this process (`vieval run --with-viewer`) | `onRunReady` → `bus.onAny`; every event from `seq` 1. |
 | `live-tail` | Run directory without an end event and without `run-summary.json` (another process is writing it) | Byte-offset tail of `events.jsonl`: only complete lines, a half-written line waits for the next read, multi-byte characters stay intact, a truncated or replaced file restarts from byte 0. `fs.watch` + 1 s polling while a stream is open; on demand otherwise. |
 | `offline` | Run directory with an end event or `run-summary.json` | Same reader, read to the end once; official numbers from `run-summary.json`. |
-| `legacy` | `events.jsonl` with event schema version 1 | Rebuilt from `cases.jsonl` + `run-summary.json` (no rounds, retries, logs). |
+| `unsupported` | `events.jsonl` with event schema version below 3 | Listed, not opened: every route but `GET /api/runs/:runKey` answers 422 "This report was written by an older vieval. Run it again." |
 
 All sources feed one reducer (`src/shared/reduce.ts`). Envelopes with a `seq` at or below the last applied one are
 dropped (duplicates); missing numbers are counted in `seqGaps`; invalid lines in `parseErrors`.
 
 `state`: `running`, `stale` (no end event and the last event is older than `--stale-after`, default 60 min),
-`ended`, `interrupted` (`run.interrupted` or `interruptSignal` in the summary), `legacy`.
+`ended`, `interrupted` (`run.interrupted` or `interruptSignal` in the summary), `unsupported`.
 
 ## Routes
 
@@ -49,27 +49,45 @@ dropped (duplicates); missing numbers are counted in `seqGaps`; invalid lines in
 |---|---|---|
 | `GET /api/health` | `{ ok, apiVersion, mode, reportRoots, evidenceRoots, logDirectories }` | |
 | `GET /api/tree` | `TreeResponse` | `workspaces[].projects[].experiments[].attempts[].runs[]` (`RunListItem`). Project = report path segment. Current vieval writes one run per project; older reports may use `multi-project` for runs of several projects (see `projectIds`). |
-| `GET /api/runs?workspace=&project=&experiment=&attempt=&state=&source=` | `RunListItem[]` | Newest first. `counts`, `exactAverage`/`judgeAverage`/`hybridAverage` (from `run-summary.json` once written, else from events). `reportRoot` is the absolute report root the run was found in (tells archived roots apart), `reportPath` the directory below it. `matrix`: run-matrix axes of the run's tasks with their distinct values in natural order (`{ model: ['gpt-5.5'], repeat: ['1', '2', '10'] }`), `{}` without a run matrix. |
+| `GET /api/runs?workspace=&project=&experiment=&attempt=&state=&source=` | `RunListItem[]` | Newest first. `counts`, `headline` (a `HeadlineResult`, see "Scores": from `run-summary.json` once written, else from events; `null` before any case settled) and `headlineDefinitions` (the latest `score.defined` of each score in `headline`, for its label and description). `reportRoot` is the absolute report root the run was found in (tells archived roots apart), `reportPath` the directory below it. `matrix`: run-matrix axes of the run's tasks with their distinct values in natural order (`{ model: ['gpt-5.5'], repeat: ['1', '2', '10'] }`), `{}` without a run matrix. |
 | `GET /api/runs/:runKey` | `RunListItem` | |
 
 ### Run
 
 | Route | Response | Notes |
 |---|---|---|
-| `GET /api/runs/:runKey/snapshot` | `RunSnapshot` | Identity, state, `seq`, projects, tasks → cases → rounds → tries (the execution tree), `progress`, `current`, `summary`, `eventCounts`. |
+| `GET /api/runs/:runKey/snapshot` | `RunSnapshot` | Identity, state, `seq`, projects, tasks → cases → rounds → tries (the execution tree), `progress`, `current`, `summary` (`caseSummary` and `headline` of `run-summary.json`), `headline` (from events), `scoreDefinitions`, `eventCounts`. Each task has `scoring` (a `CaseScoring`) and `scoreGraders` (every distinct grader kind, model, and version its score records carried, by score name); each case and round has `scores` (`{ values, unscored }`). |
 | `GET /api/runs/:runKey/progress` | `ProgressResponse` | `progress` + `current` + `state` + `seq`, for polling. |
 | `GET /api/runs/:runKey/events?after=<seq>&limit=<1..5000>` | `{ events, nextAfter, seq }` | Raw envelopes after `after` (default 0, limit default 1000). |
 | `GET /api/runs/:runKey/stream?after=<seq>` | `text/event-stream` | See "Live updates". `Last-Event-ID` is honored. |
 | `GET /api/runs/:runKey/task?taskId=` | `TaskNode` | |
-| `GET /api/runs/:runKey/case?taskId=&caseId=` | `CaseDetail` | Node + `input`, `output`, `metrics`, `metricDefinitions` (latest `metric.defined` of each metric), `artifacts` (with `label` / `description` when recorded), `evidence`, `inference` (inference and tool-call events), `taskDisplayName`. |
+| `GET /api/runs/:runKey/case?taskId=&caseId=` | `CaseDetail` | Node + `input`, `output`, `metrics`, `metricDefinitions` (latest `metric.defined` of each metric), `scoreDefinitions` (latest `score.defined` of each score of the task), `scoreRecords` (every `score.recorded` of the case in event order, with `roundIndex`, `seq`, and the record's `rationale`, `grader`, `metadata`, and `derived`), `artifacts` (with `label` / `description` when recorded), `evidence`, `inference` (inference and tool-call events), `taskDisplayName`. |
 | `GET /api/runs/:runKey/timeline?taskId=&caseId=` | `TimelineResponse` | Every try with round, retry index, times, state; per task a summary per round (passed/failed/running, start/end). |
 | `GET /api/runs/:runKey/logs?taskId=&caseId=&stream=stdout\|stderr&round=&offset=&limit=<1..5000>` | `LogsResponse` | Console output of cases (`case.output`), split into lines. Output outside cases is not recorded by vieval. |
-| `GET /api/runs/:runKey/metrics?taskId=&caseId=` | `MetricsResponse` | Numeric metrics as series with `min/max/avg/sum/count`; other values listed; `definitions`: the latest `metric.defined` (label, description, unit, better) of each listed metric; inference totals. |
-| `GET /api/runs/:runKey/scores` | `ScoresResponse` | `overall`, `byTask`, `byRound`, `cases`, `summary` (from `run-summary.json`). |
-
-**Score rule** (same as `describeTask`): every settled round of every case contributes; a case that did not pass
-scores exact 0; a passed case without `recordScore` scores exact 1; otherwise its recorded `exact` (default 1) and
-`judge`. `hybrid = (exact + judge) / 2` when both exist. Task scores are averaged into `overall`.
+| `GET /api/runs/:runKey/metrics?taskId=&caseId=` | `MetricsResponse` | Numeric metrics as series with `min/max/avg/sum/count`; other values listed; `definitions`: the latest `metric.defined` (label, description, unit, better) of each listed metric; inference totals. `scores`: each score beside the metrics, with one point per case (the counted value of its last round) and the `ScoreSummary` of the selection; `scoreDefinitions` for their labels. |
+| `GET /api/runs/:runKey/scores` | `ScoresResponse` | `overall` (`headline` and each score pooled over the tasks), `byTask` and `byRound` (a `CaseScoring` each), `cases` (`scores` of each case), `definitions` (latest `score.defined` of each score), `summary` (the `HeadlineResult` of `run-summary.json`, once written). |
+
+**Scores**: the viewer computes scores with `scoreCases` of `vieval/core/scoring`, the rule of `describeTask`, so
+live runs, `run-summary.json`, and `cases.jsonl` agree. Every settled round of every case contributes. Per score:
+
+- `passed`: the recorded (or derived) value; a passed case that did not record the score is missing, not 1.
+- `failed`: 0. Recorded values stay in `scoreRecords` but are not counted.
+- `errored`, `timeout`: left out, or 0 when the score's `score.defined` has `errors: 'zero'`.
+- `skipped`: left out. An unscored value (`recordUnscored`) never counts; it is counted per reason in `unscored`.
+
+A `ScoreSummary` is `{ name, mean, n, total, sum, unscored }`: `mean = sum / n` (`null` when `n` is 0), `n` the cases
+that counted, `total` the settled cases that were not skipped. A `CaseScoring` (one task, one round) is
+`{ scores, primary?, completion, passed }`: `primary` is the score declared `primary`, or the only score of the task;
+`completion` is the completion rate (`(passed + failed) / settled`, with `errored` / `timeout` counts in `unscored`);
+`passed` is the share of passed cases. Summaries of several tasks pool their cases (`poolScoreSummaries`); they are
+not a mean of task means.
+
+A `HeadlineResult` is `{ headline, completion }`, resolved with `resolveHeadline`: `headline.kind` is `score` (every
+task with scores shares one primary name; its pooled summary), `per-task` (primary names differ; one summary per task
+in `tasks`), or `passed-cases` (no primary score; the share of passed cases in `score`). Show `completion` next to the
+headline wherever it appears. Labels and descriptions come from the `scoreDefinitions` / `headlineDefinitions` of the
+response (`score.defined`: `label`, `description`, `primary`, `combine`, `errors`, `grader`); a name without a
+definition is shown as written.
 
 **Progress and ETA**: `T` = `run.started.totalTasks`; a task's case total is its first `case.started.total` (unknown
 until the task starts; unknown totals are extrapolated from the mean of known ones: `basis:
@@ -132,9 +150,9 @@ evidence link.
 
 | Route | Response |
 |---|---|
-| `GET /api/runs/:runKey/otel/traces` | Span tree `run → project → task → (round) → case → (try)` rebuilt from events, with ids, times, status. |
+| `GET /api/runs/:runKey/otel/traces` | Span tree `run → project → task → (round) → case → (try)` rebuilt from events, with ids, times, status. Task spans carry `vieval.score.<name>` (the task mean) and `vieval.task.primary_score`; case spans carry `vieval.score.<name>` and `vieval.score.<name>.unscored`. |
 | `GET /api/runs/:runKey/otel/logs` | Console lines (INFO/WARN), case failures and inference failures (ERROR), interruption (WARN). |
-| `GET /api/runs/:runKey/otel/metrics` | Gauges: numeric metrics, `vieval.case.duration_ms`, `vieval.case.score.{exact,judge}`. |
+| `GET /api/runs/:runKey/otel/metrics` | Gauges: numeric metrics, `vieval.case.duration_ms`, and one `vieval.score.<name>` per score. |
 | `GET /api/runs/:runKey/otel/raw/:signal` | `otlp/<traces\|logs\|metrics>.json` written by vieval when the run ended (`404` before). |
 
 Limits: no OTLP receiver, no user-defined spans, no cross-process trace correlation, no sampling, querying, or
@@ -143,7 +161,9 @@ storage. Spans of the live OpenTelemetry runtime are not collected.
 ### Comparisons
 
 All return `ComparisonResponse`: `columns` (each with the runs it covers), `rows` (one per case name; `cells[column]`
-= `{ state, exact?, judge?, roundCount }`; `changed` when state or exact score differs between columns), `summary`.
+= `{ state, scores, roundCount }`, `scores` being `{ values, unscored }`; `changed` when the state or the value of
+`primary` differs between columns), `primary` (the primary score the rows compare, when the columns share one; without
+it, `changed` uses the state only), `scoreDefinitions`, `summary`. Each column's runs carry `headline`.
 
 | Route | Columns |
 |---|---|
@@ -151,26 +171,28 @@ All return `ComparisonResponse`: `columns` (each with the runs it covers), `rows
 | `GET /api/compare/attempts?workspace=&project=&experiment=` | One per run of the experiment (attempt label + run id), oldest first. |
 | `GET /api/compare/runs?runs=<runKey>,<runKey>[,…]` | The given runs, in order (≥ 2). |
 
+Each column carries `scoreGraders`, the graders that produced each score in its runs. The viewer warns on the primary score when the graders of the columns differ in kind, model, or version, because the values then do not measure the same thing.
+
 ### Matrix
 
 | Route | Response | Notes |
 |---|---|---|
 | `GET /api/matrix?workspace=&project=&experiment=` | `MatrixResponse` | Every run in scope (filters as in `/api/runs`; none = all runs; an unknown value is an empty matrix). |
 
-A cell is one eval entry × run-matrix row × eval-matrix row (from the task id). Every run of a cell has a `coverage`:
+A cell is one eval entry × run-matrix row × eval-matrix row (from the task id). Every run of a cell has a `coverage` (the case coverage of the cell, not the completion rate):
 `full` (every expected case settled and at least one has a result; errored cases count as settled, over the same cases
 as the headline run), `other` (every expected case settled, but over other cases than the headline run, told apart by
 `identity`: a smoke or limited run, or another case set under the same entry), `partial` (some expected cases never ran or never settled), or `none` (no case has a result). `latest` is the task
 of the newest run (by start time) that covered the cell fully or is still running it, else of the newest run: run,
-experiment, attempt, task state, coverage, expected case total (`caseTotal`), case counts, the exact score of every case
-(`caseScores`), exact and judge score. The headline run is the newest one that settled every case it planned over cases no other run covers a strict
+experiment, attempt, task state, coverage, expected case total (`caseTotal`), case counts, the primary score of every
+case (`caseScores`), and the task's `scoring` (a `CaseScoring`). The headline run is the newest one that settled every case it planned over cases no other run covers a strict
 superset of; `caseCount` is its number of cases. A partial re-run or a run over other cases therefore never replaces a full run as the
 cell's score. `superseded` is `true` when a newer run ran the same entry and run row with other eval values (the eval
 axes changed since). `history` lists every run with that cell, oldest first
-(one point per run: attempt, coverage, case counts, exact, state, start time). Two tasks of one
+(one point per run: attempt, coverage, case counts, `score` (mean of the cell's primary score), `completion`, state, start time). Two tasks of one
 run in the same cell (several inference executors) keep the last one in queue order. `axes` lists the run-matrix axes,
 then the eval-matrix axes, each with its values in natural order. Cells are ordered by entry, then run row and eval
-row in natural order. Tasks whose id has no matrix are left out. The route builds every run's task nodes once.
+row in natural order. Tasks whose id has no matrix are left out. `scoreDefinitions` holds the latest `score.defined` of each score in the cells. The route builds every run's task nodes once.
 
 ## Live updates (SSE)
 
@@ -196,7 +218,7 @@ data: {"seq":251,"state":"ended"}                   ← terminal state reached;
 ```
 
 Client pattern: `GET /snapshot`, then open the stream with `after=snapshot.seq` and apply each `run-event` with the
-same reducer (`@vieval/ui/reduce`). Finished and legacy runs send their backlog, one `progress`, and `end`.
+same reducer (`@vieval/ui/reduce`). Finished runs send their backlog, one `progress`, and `end`.
 
 ## Security
 
diff --git a/packages/ui/README.md b/packages/ui/README.md
index fc744f1..27e9651 100644
--- a/packages/ui/README.md
+++ b/packages/ui/README.md
@@ -178,7 +178,7 @@ See [API.md § Security](./API.md#security) for the status codes.
 | A new run does not appear in the sidebar | The sidebar re-reads report roots every 10 s. Check that the run writes below one of the roots in `GET /api/health`. |
 | Evidence says "Path is outside the allowed evidence roots." | The `*.evidence` metric points outside the `.vieval` directory. Add its directory with `--evidence-root`. |
 | *Command logs* is empty | The viewer looks for `.vieval/logs/*.jsonl` next to the report root. Logs without the case id, pod, or a time in the case's window are not matched. |
-| No rounds, tries, or console logs for an old run | The run used event schema version 1. These runs only have `cases.jsonl` (shown as `legacy`). |
+| An old run is listed but does not open | It was written by a vieval before named scores (event schema version below 3). Run it again. It is shown as `unsupported`. |
 | Component tests fail with "browserType.launch" | Run `pnpm exec playwright install chromium` once. |
 | `vieval run` without `--with-viewer` changed behavior | It should not: the viewer package is not imported then. Report it with the command and the `stdout` of both versions. |
 
diff --git a/packages/ui/client/src/caseMetrics.test.ts b/packages/ui/client/src/caseMetrics.test.ts
index f5e99e8..0476d33 100644
--- a/packages/ui/client/src/caseMetrics.test.ts
+++ b/packages/ui/client/src/caseMetrics.test.ts
@@ -1,8 +1,8 @@
-import type { CaseNode, MetricsResponse } from '../../src/shared/contract'
+import type { CaseNode, MetricsResponse, ScoreDefinition } from '../../src/shared/contract'
 
 import { describe, expect, it } from 'vitest'
 
-import { buildCaseTable, derivedMetricName, matchesFilters, metricLabel, separationOf } from './caseMetrics'
+import { buildCaseTable, columnDefinitions, derivedMetricName, formatColumnValue, matchesFilters, metricDefinition, metricLabel, separationOf } from './caseMetrics'
 
 function caseNode(index: number, name: string, state: CaseNode['state']): CaseNode {
   const caseId = `${index}:${name}`
@@ -19,14 +19,22 @@ function caseNode(index: number, name: string, state: CaseNode['state']): CaseNo
     metricCount: 0,
     name,
     rounds: [],
-    scores: {},
+    scores: { unscored: {}, values: {} },
     startedAt: '2026-10-13T00:00:00.000Z',
     state,
     taskId: 'task',
   }
 }
 
-function metrics(series: Record<string, [caseId: string, value: unknown, roundIndex?: number][]>): MetricsResponse {
+/**
+ * A metrics response of the task `task`: `series` as recorded metrics, `scores` as each case's counted score value,
+ * with the scores' `score.defined`.
+ */
+function metrics(
+  series: Record<string, [caseId: string, value: unknown, roundIndex?: number][]>,
+  scores: Record<string, [caseId: string, value: number][]> = {},
+  scoreDefinitions: ScoreDefinition[] = [],
+): MetricsResponse {
   const point = (name: string, [caseId, value, roundIndex]: [string, unknown, number?], seq: number) => ({ caseId, name, roundIndex, seq, taskId: 'task', time: '2026-10-13T00:00:00.000Z', value })
   const entries = Object.entries(series)
   return {
@@ -38,6 +46,12 @@ function metrics(series: Record<string, [caseId: string, value: unknown, roundIn
       points: points.map((item, seq) => point(name, item, seq)),
       summary: { avg: 0, count: points.length, max: 0, min: 0, sum: 0 },
     })),
+    scoreDefinitions,
+    scores: Object.entries(scores).map(([name, points]) => ({
+      name,
+      points: points.map(([caseId, value]) => ({ caseId, taskId: 'task', value })),
+      summary: { mean: null, n: points.length, name, sum: 0, total: points.length, unscored: {} },
+    })),
   }
 }
 
@@ -158,16 +172,18 @@ describe('buildCaseTable across tasks', () => {
 describe('buildCaseTable categories', () => {
   /**
    * @example
-   * expect(table.duplicates).toEqual([{ name: 'exact', of: 'state' }, { name: 'domain', of: 'group' }])
+   * expect(table.duplicates).toEqual([{ name: 'domain', of: 'group' }, { name: 'pass', of: 'state' }])
    */
   it('hides categories that split the cases exactly like an earlier one', () => {
-    const scored = cases.map((node, index) => ({ ...node, scores: index === 3 ? {} : { exact: node.state === 'passed' ? 1 : 0 } }))
-    const table = buildCaseTable(scored, metrics({
+    const table = buildCaseTable(cases, metrics({
       domain: cases.map(node => [node.caseId, node.name.split('/')[0]]),
+    }, {
+      // The running case has no counted value yet.
+      pass: cases.slice(0, 3).map(node => [node.caseId, node.state === 'passed' ? 1 : 0]),
     }))
 
     expect(table.categories).toEqual(['state', 'group'])
-    expect(table.duplicates).toEqual([{ name: 'domain', of: 'group' }, { name: 'exact', of: 'state' }])
+    expect(table.duplicates).toEqual([{ name: 'domain', of: 'group' }, { name: 'pass', of: 'state' }])
     expect(table.groups.map(group => group.representative)).toEqual(['duration_s'])
   })
 })
@@ -197,18 +213,21 @@ describe('separationOf', () => {
    * expect(separation.get('steps')).toEqual({ auc: 1, higherIn: 'failed' })
    */
   it('folds the AUC to 0.5..1, keeps the direction, and leaves scores out', () => {
-    const scored = cases.map((node, index) => ({ ...node, scores: { exact: index === 1 ? 0 : 1 } }))
-    const table = buildCaseTable(scored, metrics({
+    const table = buildCaseTable(cases, metrics({
       captures: cases.map((node, index) => [node.caseId, [5, 1, 6, 7][index]!]),
       flat: cases.map((node, index) => [node.caseId, index % 2]),
       steps: cases.map((node, index) => [node.caseId, [2, 50, 4, 9][index]!]),
+    }, {
+      // Three distinct values, so the score is a numeric axis like the metrics.
+      natural: cases.map((node, index) => [node.caseId, [1, 0, 0.5, 1][index]!]),
     }))
     const separation = separationOf(table)
 
     // The one failed case has the most steps, and the fewest captures.
     expect(separation.get('steps')).toEqual({ auc: 1, higherIn: 'failed' })
     expect(separation.get('captures')).toEqual({ auc: 1, higherIn: 'passed' })
-    expect(separation.has('exact')).toBe(false)
+    expect(table.groups.map(group => group.representative)).toContain('natural')
+    expect(separation.has('natural')).toBe(false)
   })
 
   // ROOT CAUSE:
@@ -216,17 +235,97 @@ describe('separationOf', () => {
   // In a scored eval every case passes (a score of 0 is a result), so there was no failed case to compare against and
   // the ranking was empty; the chart opened on metrics in recording order instead of the ones that lower the score.
   //
-  // We fixed this by comparing cases below the mean `exact` score with the rest when no case failed.
+  // We fixed this by comparing cases below the mean primary score with the rest when no case failed.
   /**
    * @example
    * expect(separation.get('steps')).toEqual({ auc: 1, higherIn: 'lower-scored' })
    */
-  it('ranks against the exact score when every case passed', () => {
-    const passed = cases.slice(0, 3).map((node, index) => ({ ...node, scores: { exact: [1, 0, 1][index]! }, state: 'passed' as const }))
+  it('ranks against the primary score when every case passed', () => {
+    const passed = cases.slice(0, 3).map(node => ({ ...node, state: 'passed' as const }))
     const table = buildCaseTable(passed, metrics({
       steps: passed.map((node, index) => [node.caseId, [2, 50, 4][index]!]),
-    }))
+    }, {
+      correct: passed.map(node => [node.caseId, 1]),
+      pass: passed.map((node, index) => [node.caseId, [1, 0, 1][index]!]),
+    }, [{ name: 'correct' }, { name: 'pass', primary: true }]))
 
+    expect(table.primary).toBe('pass')
     expect(separationOf(table).get('steps')).toEqual({ auc: 1, higherIn: 'lower-scored' })
   })
+
+  /**
+   * @example
+   * expect(separationOf(table).size).toBe(0)
+   */
+  it('ranks nothing when every case passed and no score is primary', () => {
+    const passed = cases.slice(0, 3).map(node => ({ ...node, state: 'passed' as const }))
+    const table = buildCaseTable(passed, metrics({
+      steps: passed.map((node, index) => [node.caseId, [2, 50, 4][index]!]),
+    }, {
+      correct: passed.map((node, index) => [node.caseId, [1, 0, 1][index]!]),
+      natural: passed.map((node, index) => [node.caseId, [0, 1, 1][index]!]),
+    }))
+
+    expect(table.primary).toBeUndefined()
+    expect(separationOf(table).size).toBe(0)
+  })
+})
+
+describe('buildCaseTable scores', () => {
+  /**
+   * @example
+   * expect(table.scores).toEqual(['pass', 'correct'])
+   */
+  it('joins the counted score values, the primary score first and before the metrics', () => {
+    const table = buildCaseTable(cases, metrics({
+      steps: cases.map((node, index) => [node.caseId, [2, 8, 30, 64][index]!]),
+    }, {
+      correct: cases.map((node, index) => [node.caseId, [1, 0.5, 0, 1][index]!]),
+      // The failed case counts 0 whatever it recorded; the running one is not counted yet.
+      pass: cases.slice(0, 3).map((node, index) => [node.caseId, [1, 0, 0.5][index]!]),
+    }, [{ name: 'correct' }, { name: 'pass', primary: true }]))
+
+    expect(table.primary).toBe('pass')
+    expect(table.scores).toEqual(['pass', 'correct'])
+    expect(table.rows.map(row => row.numbers.pass)).toEqual([1, 0, 0.5, undefined])
+    // Scores lead the axes; the metrics follow in recording order.
+    expect(table.groups.map(group => group.representative).slice(0, 2)).toEqual(['pass', 'correct'])
+  })
+
+  /**
+   * @example
+   * expect(table.rows.map(row => row.numbers.pass)).toEqual([1, 0, undefined, undefined])
+   */
+  it('reads the recorded values of the cases while the metrics are not loaded', () => {
+    const recorded = cases.map((node, index): CaseNode => ({
+      ...node,
+      scores: { unscored: index === 2 ? { pass: 'grader_failed' } : {}, values: index < 2 ? { pass: [1, 0][index]! } : {} },
+    }))
+    const table = buildCaseTable(recorded, undefined)
+
+    expect(table.rows.map(row => row.numbers.pass)).toEqual([1, 0, undefined, undefined])
+    // The only score of the cases is their primary one.
+    expect(table.primary).toBe('pass')
+  })
+})
+
+describe('columnDefinitions', () => {
+  /**
+   * @example
+   * expect(metricLabel('pass', definitions)).toBe('Pass')
+   */
+  it('reads scores as higher-is-better columns with their formula, shown as percentages', () => {
+    const scoreDefinitions: ScoreDefinition[] = [
+      { label: 'Written back correctly', name: 'correct' },
+      { combine: { inputs: [{ name: 'correct' }, { atLeast: 0.5, name: 'natural' }], kind: 'all' }, label: 'Pass', name: 'pass', primary: true },
+    ]
+    const definitions = columnDefinitions([{ better: 'lower', name: 'latency', unit: 'ms' }], scoreDefinitions)
+
+    expect(metricLabel('pass', definitions)).toBe('Pass')
+    expect(metricDefinition('pass', definitions)?.better).toBe('higher')
+    expect(metricDefinition('pass', definitions)?.formula).toBe('all of: Written back correctly, natural ≥ 50%')
+    expect(formatColumnValue(0.875, metricDefinition('pass', definitions))).toBe('87.5%')
+    expect(formatColumnValue(1234.5, metricDefinition('latency', definitions))).toBe('1.2K ms')
+    expect(formatColumnValue(1.23456, undefined)).toBe('1.23')
+  })
 })
diff --git a/packages/ui/client/src/caseMetrics.ts b/packages/ui/client/src/caseMetrics.ts
index b43a4a3..664f94b 100644
--- a/packages/ui/client/src/caseMetrics.ts
+++ b/packages/ui/client/src/caseMetrics.ts
@@ -1,11 +1,14 @@
 import type { ScaleContinuousNumeric } from 'd3'
 
-import type { CaseNode, CaseState, MetricDefinition, MetricsResponse } from '../../src/shared/contract'
+import type { CaseNode, CaseState, MetricDefinition, MetricsResponse, ScoreDefinition } from '../../src/shared/contract'
+import type { ColumnDefinition } from './scores'
 
 import { scaleLinear, scaleSymlog } from 'd3'
 
 import { parseTaskId } from '../../src/shared/reduce'
+import { formatScore } from './format'
 import { localize } from './localize'
+import { primaryScoreName, scoreColumn } from './scores'
 
 /** One case of a task, flattened for the multi-metric charts (parallel coordinates, scatterplot matrix). */
 export interface CaseRow {
@@ -37,7 +40,11 @@ export interface CaseTable {
   groups: MetricGroup[]
   /** Numeric metrics whose largest value is far above the median; charts draw them on a symlog scale. */
   heavyTailed: Set<string>
+  /** Score the cases are read by (see {@link primaryScoreName}); `undefined` without one. */
+  primary?: string
   rows: CaseRow[]
+  /** Names of the scores among the numeric values, the primary score first; they are axes like any metric. */
+  scores: string[]
   /** Cases per value of each category axis, most frequent first (`state` in its usual order, passed first). */
   valueCounts: Record<string, Map<string, number>>
 }
@@ -82,7 +89,7 @@ export interface MetricSeparation {
    * metric alone (`0.5`: no better than a coin, `1`: every case of one outcome lies on one side).
    */
   auc: number
-  /** Which outcome has the larger values: a case state, or the lower / higher `exact` score. */
+  /** Which outcome has the larger values: a case state, or the lower / higher primary score. */
   higherIn: 'failed' | 'higher-scored' | 'lower-scored' | 'passed'
 }
 
@@ -98,13 +105,13 @@ export interface MetricSeparation {
  *   matched to cases by task id and case id
  *
  * Returns:
- * - one row per case with the value of its last round for every metric; built-in axes `duration_s`, `exact`,
- *   `judge` (when recorded), `state`, `group` (the name prefix before `/`, when names share that shape), and the
- *   eval `entry` and the run and eval matrix axes of the case's task (they split the cases only when several tasks
- *   are in the table)
+ * - one row per case with the value of its last round for every metric and every score (the counted value of
+ *   `metrics.scores`, else the recorded value of the case); built-in axes `duration_s`, `state`, `group` (the name
+ *   prefix before `/`, when names share that shape), and the eval `entry` and the run and eval matrix axes of the
+ *   case's task (they split the cases only when several tasks are in the table)
  * - metrics recorded as a list of numbers per case summarized as `<name>#mean` and `<name>#length`
  * - constant metrics split out, and correlated metrics (|r| ≥ 0.95 on `log1p` values) folded into the first of
- *   them in recording order, built-ins first
+ *   them in recording order: scores first (the primary leads), then the built-ins, then the recorded metrics
  */
 export function buildCaseTable(cases: readonly CaseNode[], metrics: MetricsResponse | undefined): CaseTable {
   // Keyed like `CaseNode.key` (task id, NUL, case id): one case id repeats in every task of a run (an OSWorld question
@@ -132,6 +139,32 @@ export function buildCaseTable(cases: readonly CaseNode[], metrics: MetricsRespo
       }
     }
   }
+  // Scores of the case's last round as the summaries count them (a failed case scores 0); a case left out of a score
+  // (errored, unscored) has no point and so no value.
+  const scoreNames: string[] = []
+  for (const series of metrics?.scores ?? []) {
+    scoreNames.push(series.name)
+    for (const point of series.points) {
+      const entry = entryOf(point.taskId, point.caseId)
+      const round = point.roundIndex ?? 0
+      if ((entry.rounds[series.name] ?? -1) <= round) {
+        entry.rounds[series.name] = round
+        entry.numbers[series.name] = point.value
+      }
+    }
+  }
+  // Without the metrics response (still loading), the cases' own recorded values stand in for the counted ones.
+  if (metrics == null) {
+    for (const node of cases) {
+      for (const [name, value] of Object.entries(node.scores.values)) {
+        if (!scoreNames.includes(name))
+          scoreNames.push(name)
+        entryOf(node.taskId, node.caseId).numbers[name] = value
+      }
+    }
+  }
+  const primary = primaryScoreName(metrics?.scoreDefinitions ?? [], scoreNames)
+  const scores = primary == null ? scoreNames : [primary, ...scoreNames.filter(name => name !== primary)]
   const derivedNames: string[] = []
   for (const series of metrics?.nonNumeric ?? []) {
     for (const point of series.values) {
@@ -166,10 +199,6 @@ export function buildCaseTable(cases: readonly CaseNode[], metrics: MetricsRespo
     const numbers: Record<string, number> = { ...recorded?.numbers }
     if (node.durationMs != null)
       numbers.duration_s = node.durationMs / 1000
-    if (node.scores.exact != null)
-      numbers.exact = node.scores.exact
-    if (node.scores.judge != null)
-      numbers.judge = node.scores.judge
     // Built-ins first (key order is the default axis order), and they win over a recorded metric of the same name.
     // The matrix axes of the case's task (`tts`, `asr`, …): one value per task, so they split only a run-wide table.
     // The eval entry too (`…/formula-speech/en`, by its last path segment): one entry per language or dataset split.
@@ -181,9 +210,8 @@ export function buildCaseTable(cases: readonly CaseNode[], metrics: MetricsRespo
     return { caseId: node.caseId, categories, key: node.key, name: node.name, numbers, state: node.state, taskId: node.taskId }
   })
 
-  const builtIns = ['duration_s', 'exact', 'judge']
   const recordedNames = [...(metrics?.numeric ?? []).map(series => series.name), ...derivedNames]
-  const numericNames = [...builtIns, ...recordedNames].filter((name, index, all) => all.indexOf(name) === index && rows.some(row => row.numbers[name] != null))
+  const numericNames = [...scores, 'duration_s', ...recordedNames].filter((name, index, all) => all.indexOf(name) === index && rows.some(row => row.numbers[name] != null))
 
   const constants: CaseTable['constants'] = []
   const groups: MetricGroup[] = []
@@ -246,12 +274,56 @@ export function buildCaseTable(cases: readonly CaseNode[], metrics: MetricsRespo
       : numericValues ? (value: string) => Number(value) : (value: string) => -counts.get(value)!
     valueCounts[name] = new Map([...counts].sort(([a], [b]) => order(a) - order(b)))
   }
-  return { categories, constants, duplicates, groups, heavyTailed, rows, valueCounts }
+  return { categories, constants, duplicates, groups, heavyTailed, primary, rows, scores: scores.filter(name => rows.some(row => row.numbers[name] != null)), valueCounts }
+}
+
+/**
+ * Definitions of every column of a case table by name: the metrics' `metric.defined`, and the scores' `score.defined`
+ * read as columns ({@link scoreColumn}); a score wins over a metric of the same name.
+ *
+ * Use when:
+ * - a chart, menu, or table labels and describes the columns of a {@link CaseTable} (`metricLabel`, `metricDefinition`)
+ */
+export function columnDefinitions(metricDefinitions: readonly MetricDefinition[], scoreDefinitions: readonly ScoreDefinition[]): Map<string, ColumnDefinition> {
+  return new Map<string, ColumnDefinition>([
+    ...metricDefinitions.map(definition => [definition.name, definition] as const),
+    ...scoreDefinitions.map(definition => [definition.name, scoreColumn(definition, scoreDefinitions)] as const),
+  ])
+}
+
+/**
+ * Formats one value of a column for people: a score as a percentage, a metric rounded with its unit.
+ *
+ * Before:
+ * - (0.875, score column), (48213, { unit: 'tokens' }), (1.23456, undefined)
+ *
+ * After:
+ * - "87.5%", "48.2K tokens", "1.23"
+ */
+export function formatColumnValue(value: number, definition: ColumnDefinition | undefined): string {
+  if (definition?.score != null)
+    return formatScore(value)
+  const text = Math.abs(value) >= 1000 ? new Intl.NumberFormat(undefined, { maximumFractionDigits: 1, notation: 'compact' }).format(value) : String(Math.round(value * 100) / 100)
+  return definition?.unit == null ? text : `${text} ${definition.unit}`
 }
 
 /** Separator between a list metric's name and its summary in a derived metric name; never part of a recorded name. */
 const DERIVED_SEPARATOR = '#'
 
+/**
+ * Stable offset in `-0.5..0.5` for one case, so a dot keeps its spot inside a band (or a line its place on a category
+ * axis) across redraws instead of jumping on every update. `salt` gives a case independent offsets per direction.
+ *
+ * Use when:
+ * - spreading many cases that share one value (a category, the free direction of a strip) so they stay countable
+ */
+export function caseOffset(key: string, salt = 0): number {
+  let hash = (salt + 1) * 2654435761
+  for (let index = 0; index < key.length; index++)
+    hash = Math.imul(hash ^ key.charCodeAt(index), 2654435761)
+  return ((hash >>> 0) % 1000) / 1000 - 0.5
+}
+
 /**
  * Pearson correlation of two metrics over the cases that recorded both, on `log1p` values so token counts that span
  * several orders of magnitude are compared by shape, not by their few largest cases. `0` when either is flat.
@@ -329,7 +401,7 @@ export function matchesFilters(row: CaseRow, filters: CaseFilters): boolean {
  * Definition of a metric for its hover card; a derived metric borrows the description of its list metric (and its
  * unit for the mean, not for the length).
  */
-export function metricDefinition(name: string, definitions: ReadonlyMap<string, MetricDefinition>): MetricDefinition | undefined {
+export function metricDefinition(name: string, definitions: ReadonlyMap<string, ColumnDefinition>): ColumnDefinition | undefined {
   const derived = parseDerivedMetric(name)
   const base = definitions.get(derived?.base ?? name)
   if (derived == null || base == null)
@@ -344,7 +416,7 @@ export function metricDefinition(name: string, definitions: ReadonlyMap<string,
  * Use when:
  * - any chart, menu, or table names a metric of a {@link CaseTable}
  */
-export function metricLabel(name: string, definitions: ReadonlyMap<string, MetricDefinition>): string {
+export function metricLabel(name: string, definitions: ReadonlyMap<string, ColumnDefinition>): string {
   const derived = parseDerivedMetric(name)
   if (derived != null)
     return `${metricLabel(derived.base, definitions)} · ${derived.summary}`
@@ -387,65 +459,6 @@ export function parseDerivedMetric(name: string): undefined | { base: string, su
   return at > 0 && (summary === 'mean' || summary === 'length') ? { base: name.slice(0, at), summary } : undefined
 }
 
-/**
- * Short label of a metric for a chart axis: the last dot segment, or more segments while two picked metrics would
- * share a label.
- *
- * Before:
- * - "osworld.model.usage.output_tokens_details.reasoning_tokens"
- *
- * After:
- * - "reasoning_tokens"
- */
-export function shortMetricName(name: string, all: readonly string[]): string {
-  const parts = name.split('.')
-  for (let take = 1; take < parts.length; take++) {
-    const label = parts.slice(-take).join('.')
-    if (all.every(other => other === name || (!other.endsWith(`.${label}`) && other !== label)))
-      return label
-  }
-  return name
-}
-
-/** The numbers of a recorded list value, or `undefined` when it is not a non-empty list of finite numbers. */
-function numbersOf(value: unknown): number[] | undefined {
-  if (!Array.isArray(value) || value.length === 0)
-    return undefined
-  return value.every(item => typeof item === 'number' && Number.isFinite(item)) ? value as number[] : undefined
-}
-
-/** Whether two categories split the cases the same way, value for value (`exact` 1/0 against `state` passed/failed). */
-function sameSplit(rows: readonly CaseRow[], left: string, right: string): boolean {
-  const forward = new Map<string, string>()
-  const backward = new Map<string, string>()
-  for (const row of rows) {
-    const a = row.categories[left] ?? ''
-    const b = row.categories[right] ?? ''
-    if ((forward.get(a) ?? b) !== b || (backward.get(b) ?? a) !== a)
-      return false
-    forward.set(a, b)
-    backward.set(b, a)
-  }
-  return true
-}
-
-/** Scores are the outcome itself, so ranking them by how they separate the outcome would explain nothing. */
-const OUTCOME_METRICS = new Set(['exact', 'judge'])
-
-/**
- * Stable offset in `-0.5..0.5` for one case, so a dot keeps its spot inside a band (or a line its place on a category
- * axis) across redraws instead of jumping on every update. `salt` gives a case independent offsets per direction.
- *
- * Use when:
- * - spreading many cases that share one value (a category, the free direction of a strip) so they stay countable
- */
-export function caseOffset(key: string, salt = 0): number {
-  let hash = (salt + 1) * 2654435761
-  for (let index = 0; index < key.length; index++)
-    hash = Math.imul(hash ^ key.charCodeAt(index), 2654435761)
-  return ((hash >>> 0) % 1000) / 1000 - 0.5
-}
-
 /**
  * Ranks numeric metrics by how well they separate failed from passed cases.
  *
@@ -456,19 +469,20 @@ export function caseOffset(key: string, salt = 0): number {
  * - finished cases (`passed`, `failed`, `errored`, `timeout`); running and skipped cases are left out
  *
  * Returns:
- * - one entry per non-constant metric with values in both outcomes, except the score metrics; ties count half
+ * - one entry per non-constant metric with values in both outcomes, except the scores; ties count half
  *   (Mann–Whitney U), so a metric that is mostly 0 in both outcomes stays near `0.5`
  * - when no finished case failed (a scored eval, where a score of 0 is a result, not a failure), the outcomes are the
- *   cases below the mean `exact` score against the rest, so the ranking still explains what lowers the score
+ *   cases below the mean primary score against the rest, so the ranking still explains what lowers the score
  */
 export function separationOf(table: CaseTable): Map<string, MetricSeparation> {
   const result = new Map<string, MetricSeparation>()
   const finished = table.rows.filter(row => row.state === 'passed' || row.state === 'failed' || row.state === 'errored' || row.state === 'timeout')
-  const outcome = outcomeOf(finished)
+  const outcome = outcomeOf(finished, table.primary)
   if (outcome == null)
     return result
   for (const name of table.groups.flatMap(group => [group.representative, ...group.members.map(member => member.name)])) {
-    if (OUTCOME_METRICS.has(name))
+    // Scores are the outcome itself, so ranking them by how they separate the outcome would explain nothing.
+    if (table.scores.includes(name))
       continue
     const low: number[] = []
     const high: number[] = []
@@ -502,14 +516,43 @@ export function separationOf(table: CaseTable): Map<string, MetricSeparation> {
   return result
 }
 
+/**
+ * Short label of a metric for a chart axis: the last dot segment, or more segments while two picked metrics would
+ * share a label.
+ *
+ * Before:
+ * - "osworld.model.usage.output_tokens_details.reasoning_tokens"
+ *
+ * After:
+ * - "reasoning_tokens"
+ */
+export function shortMetricName(name: string, all: readonly string[]): string {
+  const parts = name.split('.')
+  for (let take = 1; take < parts.length; take++) {
+    const label = parts.slice(-take).join('.')
+    if (all.every(other => other === name || (!other.endsWith(`.${label}`) && other !== label)))
+      return label
+  }
+  return name
+}
+
+/** The numbers of a recorded list value, or `undefined` when it is not a non-empty list of finite numbers. */
+function numbersOf(value: unknown): number[] | undefined {
+  if (!Array.isArray(value) || value.length === 0)
+    return undefined
+  return value.every(item => typeof item === 'number' && Number.isFinite(item)) ? value as number[] : undefined
+}
+
 /**
  * The two outcomes the separation compares: failed against passed cases, or, with no failed case, cases below the
- * mean `exact` score against the rest. `undefined` when neither split has two sides.
+ * mean primary score against the rest. `undefined` when neither split has two sides.
  */
-function outcomeOf(finished: readonly CaseRow[]): undefined | { high: MetricSeparation['higherIn'], low: MetricSeparation['higherIn'], sideOf: (row: CaseRow) => 'high' | 'low' | undefined } {
+function outcomeOf(finished: readonly CaseRow[], primary: string | undefined): undefined | { high: MetricSeparation['higherIn'], low: MetricSeparation['higherIn'], sideOf: (row: CaseRow) => 'high' | 'low' | undefined } {
   if (finished.some(row => row.state === 'failed') && finished.some(row => row.state === 'passed'))
     return { high: 'passed', low: 'failed', sideOf: row => (row.state === 'passed' ? 'high' : 'low') }
-  const scores = finished.map(row => row.numbers.exact).filter((value): value is number => value != null)
+  if (primary == null)
+    return undefined
+  const scores = finished.map(row => row.numbers[primary]).filter((value): value is number => value != null)
   if (new Set(scores).size < 2)
     return undefined
   const mean = scores.reduce((sum, value) => sum + value, 0) / scores.length
@@ -517,8 +560,23 @@ function outcomeOf(finished: readonly CaseRow[]): undefined | { high: MetricSepa
     high: 'higher-scored',
     low: 'lower-scored',
     sideOf: (row) => {
-      const value = row.numbers.exact
+      const value = row.numbers[primary]
       return value == null ? undefined : value < mean ? 'low' : 'high'
     },
   }
 }
+
+/** Whether two categories split the cases the same way, value for value (a 1/0 score against `state` passed/failed). */
+function sameSplit(rows: readonly CaseRow[], left: string, right: string): boolean {
+  const forward = new Map<string, string>()
+  const backward = new Map<string, string>()
+  for (const row of rows) {
+    const a = row.categories[left] ?? ''
+    const b = row.categories[right] ?? ''
+    if ((forward.get(a) ?? b) !== b || (backward.get(b) ?? a) !== a)
+      return false
+    forward.set(a, b)
+    backward.set(b, a)
+  }
+  return true
+}
diff --git a/packages/ui/client/src/components/AxisPicker.vue b/packages/ui/client/src/components/AxisPicker.vue
index ecc66c5..4133127 100644
--- a/packages/ui/client/src/components/AxisPicker.vue
+++ b/packages/ui/client/src/components/AxisPicker.vue
@@ -1,6 +1,6 @@
 <script setup lang="ts">
-import type { MetricDefinition } from '../../../src/shared/contract'
 import type { CaseTable, MetricSeparation } from '../caseMetrics'
+import type { ColumnDefinition } from '../scores'
 
 import { useMediaQuery } from '@vueuse/core'
 import { PopoverContent, PopoverPortal, PopoverRoot, PopoverTrigger } from 'reka-ui'
@@ -16,7 +16,7 @@ import AxisPickerList from './AxisPickerList.vue'
  */
 const props = defineProps<{
   /** Definition of a metric (label, description, unit) for its details. */
-  definitionOf: (name: string) => MetricDefinition | undefined
+  definitionOf: (name: string) => ColumnDefinition | undefined
   labelOf: (name: string) => string
   separation: ReadonlyMap<string, MetricSeparation>
   table: CaseTable
diff --git a/packages/ui/client/src/components/AxisPickerList.vue b/packages/ui/client/src/components/AxisPickerList.vue
index 380f268..650bdf1 100644
--- a/packages/ui/client/src/components/AxisPickerList.vue
+++ b/packages/ui/client/src/components/AxisPickerList.vue
@@ -1,6 +1,6 @@
 <script setup lang="ts">
-import type { MetricDefinition } from '../../../src/shared/contract'
 import type { CaseTable, MetricSeparation } from '../caseMetrics'
+import type { ColumnDefinition } from '../scores'
 import type { BesideTarget } from './MetricBesideCard.vue'
 
 import { DrawerContent, DrawerDescription, DrawerHandle, DrawerOverlay, DrawerPortal, DrawerRootNested, DrawerTitle } from 'vaul-vue'
@@ -17,7 +17,7 @@ import { localize } from '../localize'
  * desktop popover and the phone drawer, the same way AIRI's Live2D report shares its content.
  */
 const props = defineProps<{
-  definitionOf: (name: string) => MetricDefinition | undefined
+  definitionOf: (name: string) => ColumnDefinition | undefined
   /**
    * How a metric's details open: `hover` (desktop popover) floats them beside the row; `drawer` (phone) shows the
    * description as one line under the name and opens the details in a second drawer from an info button.
diff --git a/packages/ui/client/src/components/BarChart.vue b/packages/ui/client/src/components/BarChart.vue
index 4f15173..e77387c 100644
--- a/packages/ui/client/src/components/BarChart.vue
+++ b/packages/ui/client/src/components/BarChart.vue
@@ -36,7 +36,7 @@ const width = useElementWidth(root, 640)
 const labelWidth = computed(() => Math.min(260, Math.round(width.value * 0.3)))
 const plotWidth = computed(() => width.value - labelWidth.value - 70)
 
-/** Series with at least one value; an all-empty series (e.g. judge scores of an eval without a judge) is dropped. */
+/** Series with at least one value; an all-empty series (e.g. a score only the tasks of another eval record) is dropped. */
 const shown = computed(() => props.series.map((name, index) => ({ index, name })).filter(({ index }) => props.data.some(datum => datum.values[index] != null)))
 const rows = computed(() => {
   let y = 4
diff --git a/packages/ui/client/src/components/CaseCard.vue b/packages/ui/client/src/components/CaseCard.vue
index c448ce7..115dafe 100644
--- a/packages/ui/client/src/components/CaseCard.vue
+++ b/packages/ui/client/src/components/CaseCard.vue
@@ -20,6 +20,8 @@ const props = defineProps<{
   /** Axis under the pointer (or where the case was pinned); its row is emphasized and its value leads the card. */
   activeAxis?: string
   axes: readonly ParallelAxis[]
+  /** Formats a numeric value of an axis: a score as a percentage, a metric with its unit (`formatColumnValue`). */
+  formatOf: (name: string, value: number) => string
   /** Link to the case page. */
   href: string
   labelOf: (name: string) => string
@@ -35,7 +37,6 @@ const props = defineProps<{
   /** Every case of the charts, the population the value is placed in. */
   rows: readonly CaseRow[]
   scales: Readonly<Record<string, ScaleContinuousNumeric<number, number>>>
-  unitOf: (name: string) => string | undefined
   /** Cases per value of each category axis (`CaseTable.valueCounts`), for "46 / 361 cases". */
   valueCounts: Readonly<Record<string, ReadonlyMap<string, number>>>
 }>()
@@ -79,13 +80,11 @@ const lines = computed(() => props.axes.map((axis) => {
   const below = values.filter(item => item < value).length
   const tied = values.filter(item => item === value).length
   const percentile = Math.round(((below + tied / 2) / values.length) * 100)
-  const unit = props.unitOf(axis.name)
-  const number = Math.abs(value) >= 1000 ? Intl.NumberFormat(undefined, { maximumFractionDigits: 1, notation: 'compact' }).format(value) : String(Math.round(value * 100) / 100)
   const position = distribution?.position?.(value)
   const bin = position == null ? undefined : Math.min(BINS - 1, Math.max(0, Math.floor(position * BINS)))
   // How far the value sits from the middle of all cases: 0 at the median, 1 at either end.
   const notability = Math.abs(percentile - 50) / 50
-  return { bin, detail: `p${percentile}`, heights, kind: 'numeric' as const, name: axis.name, notability, position, text: unit == null ? number : `${number} ${unit}` }
+  return { bin, detail: `p${percentile}`, heights, kind: 'numeric' as const, name: axis.name, notability, position, text: props.formatOf(axis.name, value) }
 }))
 /** Small round icon action beside the case name; finger-sized on phones (the card opens in a drawer there). */
 const action = 'grid place-items-center size-9 md:size-6 shrink-0 rounded-full border border-neutral-200 dark:border-neutral-700/60 text-neutral-600 dark:text-neutral-300 transition-colors hover:(text-primary-700 dark:text-primary-200 border-primary-500/50 bg-primary-500/8) focus-visible:(outline-2 outline-offset-1 outline-primary-500)'
diff --git a/packages/ui/client/src/components/CaseMetricsPanel.vue b/packages/ui/client/src/components/CaseMetricsPanel.vue
index 9d1049b..edee06f 100644
--- a/packages/ui/client/src/components/CaseMetricsPanel.vue
+++ b/packages/ui/client/src/components/CaseMetricsPanel.vue
@@ -2,7 +2,7 @@
 import type { ScaleContinuousNumeric } from 'd3'
 import type { ComponentPublicInstance } from 'vue'
 
-import type { CaseNode, MetricDefinition } from '../../../src/shared/contract'
+import type { CaseNode } from '../../../src/shared/contract'
 import type { CaseFilters, CaseRow } from '../caseMetrics'
 import type { BesideTarget } from './MetricBesideCard.vue'
 import type { ParallelAxis } from './ParallelCoordinates.vue'
@@ -29,8 +29,8 @@ import SegmentedControl from './SegmentedControl.vue'
 import StateView from './StateView.vue'
 
 import { api } from '../api'
-import { buildCaseTable, matchesFilters, metricDefinition, metricLabel, metricScale, separationOf } from '../caseMetrics'
-import { formatTick } from '../format'
+import { buildCaseTable, columnDefinitions, formatColumnValue, matchesFilters, metricDefinition, metricLabel, metricScale, separationOf } from '../caseMetrics'
+import { formatScore, formatTick } from '../format'
 import { href, navigate } from '../router'
 
 /**
@@ -56,8 +56,8 @@ const metrics = useQuery({
 watch(() => props.refreshKey, () => void metrics.refetch())
 
 const table = computed(() => buildCaseTable(props.cases, metrics.data.value))
-// Labels, descriptions, and units the task announced with `metric.defined`.
-const definitions = computed(() => new Map<string, MetricDefinition>((metrics.data.value?.definitions ?? []).map(definition => [definition.name, definition])))
+// Labels, descriptions, and units the task announced with `metric.defined`, and its scores' `score.defined`.
+const definitions = computed(() => columnDefinitions(metrics.data.value?.definitions ?? [], metrics.data.value?.scoreDefinitions ?? []))
 const labelOf = (name: string) => metricLabel(name, definitions.value)
 /** Every case's value of one metric or category, for the distribution in its hover card. */
 const valuesOf = (name: string) => table.value.rows.map(row => row.numbers[name] ?? row.categories[name])
@@ -84,9 +84,10 @@ const separation = computed(() => separationOf(table.value))
 const pickedMetrics = shallowRef<string[]>()
 const pickedCategories = shallowRef<string[]>()
 const numericAxes = computed<string[]>({
-  // Ranked by separation; metrics without one (scores, one-outcome metrics) come after, in recording order.
+  // The primary score first (what the cases are read by), then ranked by separation; metrics without one (the other
+  // scores, one-outcome metrics) come after, in recording order.
   get: () => pickedMetrics.value ?? table.value.groups.map(group => group.representative)
-    .toSorted((a, b) => (separation.value.get(b)?.auc ?? 0) - (separation.value.get(a)?.auc ?? 0))
+    .toSorted((a, b) => Number(b === table.value.primary) - Number(a === table.value.primary) || (separation.value.get(b)?.auc ?? 0) - (separation.value.get(a)?.auc ?? 0))
     .slice(0, DEFAULT_METRIC_AXES),
   set: (names) => {
     pickedMetrics.value = names
@@ -299,31 +300,31 @@ watch(selection, (value) => {
 })
 
 const selectedRows = computed(() => table.value.rows.filter(row => selected.value.has(row.key)))
-/** Whether the cases carry an exact score: the selection is then summarized by its mean score, not its pass rate. */
-const scored = computed(() => table.value.rows.some(row => row.numbers.exact != null))
 /**
- * What the selection holds: the mean exact score of its scored cases in a scored eval (where a score of 0 is a result,
- * not a failure), else the share of finished cases that passed.
+ * What the selection holds: the mean primary score of its scored cases in a scored eval (where a score of 0 is a
+ * result, not a failure), else the share of finished cases that passed.
  */
 const selectionSummary = computed(() => {
   const rows = selectedRows.value
-  if (scored.value) {
-    const scores = rows.map(row => row.numbers.exact).filter((value): value is number => value != null)
-    return { label: 'exact', value: scores.length === 0 ? '—' : `${((scores.reduce((sum, value) => sum + value, 0) / scores.length) * 100).toFixed(1)}%` }
+  const primary = table.value.primary
+  if (primary != null) {
+    const scores = rows.map(row => row.numbers[primary]).filter((value): value is number => value != null)
+    return { label: labelOf(primary), value: formatScore(scores.length === 0 ? null : scores.reduce((sum, value) => sum + value, 0) / scores.length) }
   }
   const finished = rows.filter(row => row.state === 'passed' || row.state === 'failed' || row.state === 'errored' || row.state === 'timeout')
   return { label: 'pass', value: finished.length === 0 ? '—' : `${((finished.filter(row => row.state === 'passed').length / finished.length) * 100).toFixed(1)}%` }
 })
 
 /**
- * Color by the case state, a category, or a metric in steps. When every finished case ended the same way (a scored
- * eval: every case passed), the state paints one color, so the chart opens colored by the exact score instead.
+ * Color by the case state, a category, or a metric in steps. A task with a primary score opens colored by it (a
+ * failed case scores 0, so the score already tells wrong results apart, and it ranks the passed ones too); without
+ * one, by the state.
  */
 const pickedColor = shallowRef<string>()
 const colorOptions = computed(() => [...table.value.categories, ...numericAxes.value])
 const defaultColor = computed(() => {
-  const states = new Set(table.value.rows.filter(row => row.state !== 'running' && row.state !== 'skipped').map(row => row.state))
-  return states.size <= 1 && colorOptions.value.includes('exact') ? 'exact' : 'state'
+  const primary = table.value.primary
+  return primary != null && colorOptions.value.includes(primary) ? primary : 'state'
 })
 const colorBy = computed<string>({
   get: () => (pickedColor.value != null && colorOptions.value.includes(pickedColor.value) ? pickedColor.value : defaultColor.value),
@@ -345,8 +346,8 @@ function sequentialSlot(index: number, count: number, lowStandsOut = false) {
   const slot = count <= 1 ? SEQUENTIAL_STEPS : Math.round(1 + (index * (SEQUENTIAL_STEPS - 1)) / (count - 1))
   return lowStandsOut ? SEQUENTIAL_STEPS + 1 - slot : slot
 }
-/** Whether higher is better for a metric: the scores, and metrics defined with `better: 'higher'`. */
-const higherIsBetter = (name: string) => name === 'exact' || name === 'judge' || metricDefinition(name, definitions.value)?.better === 'higher'
+/** Whether higher is better for a metric: the scores (their columns say so), and metrics defined with `better: 'higher'`. */
+const higherIsBetter = (name: string) => metricDefinition(name, definitions.value)?.better === 'higher'
 
 /** One entry of the color legend; clicking it selects its cases (`select`), and `active` marks a selected one. */
 interface LegendItem { active?: boolean, color: string, count: number, label: string, select?: () => void }
@@ -367,11 +368,12 @@ const palette = computed<{ colorOf: (row: CaseRow) => string, legend: LegendItem
     return { colorOf: row => `var(--mark-${row.state})`, legend: (categoryValues.value.state ?? []).map(value => categoryItem(value, `var(--mark-${value})`)) }
   const values = categoryValues.value[name]
   if (values != null) {
-    // A two-valued score or count (`exact` 0 / 1) is ordered: low to high along the sequential ramp, not series hues.
+    // A two-valued score or count (a 0 / 1 score) is ordered: low to high along the sequential ramp, not series hues.
     if (values.every(value => value.trim() !== '' && Number.isFinite(Number(value)))) {
       const ordered = values.toSorted((a, b) => Number(a) - Number(b))
       const slots = new Map(ordered.map((value, index) => [value, `var(--seq-${sequentialSlot(index, ordered.length, higherIsBetter(name))})`]))
-      return { colorOf: row => slots.get(row.categories[name] ?? '') ?? 'var(--chart-grid)', legend: ordered.map(value => categoryItem(value, slots.get(value)!)) }
+      const definition = metricDefinition(name, definitions.value)
+      return { colorOf: row => slots.get(row.categories[name] ?? '') ?? 'var(--chart-grid)', legend: ordered.map(value => ({ ...categoryItem(value, slots.get(value)!), label: formatColumnValue(Number(value), definition) })) }
     }
     const slots = new Map(values.slice(0, SERIES_SLOTS).map((value, index) => [value, `var(--series-${index + 1})`]))
     const legend = [...slots].map(([label, color]) => categoryItem(label, color))
@@ -401,7 +403,7 @@ const palette = computed<{ colorOf: (row: CaseRow) => string, legend: LegendItem
         active: range != null && range[0] === band.from && range[1] === to,
         color: `var(--seq-${sequentialSlot(index, bands.length, higherIsBetter(name))})`,
         count: rows.filter(row => row.numbers[name] != null && bandOf(row.numbers[name]!) === index).length,
-        label: `${formatTick(band.from)}–${formatTick(band.to)}`,
+        label: metricDefinition(name, definitions.value)?.score == null ? `${formatTick(band.from)}–${formatTick(band.to)}` : `${formatScore(band.from)}–${formatScore(band.to)}`,
         select: () => onBrush(name, range != null && range[0] === band.from && range[1] === to ? null : [band.from, to]),
       }
     }),
@@ -509,7 +511,7 @@ onKeyStroke('Enter', (event) => {
   navigate({ caseId: row.caseId, name: 'case', runKey: props.runKey, taskId: row.taskId })
 })
 const caseHref = (row: CaseRow) => href({ caseId: row.caseId, name: 'case', runKey: props.runKey, taskId: row.taskId })
-const unitOf = (name: string) => definitions.value.get(name)?.unit
+const formatOf = (name: string, value: number) => formatColumnValue(value, metricDefinition(name, definitions.value))
 </script>
 
 <template>
@@ -596,14 +598,14 @@ const unitOf = (name: string) => definitions.value.get(name)?.unit
             </div>
             <ParallelSets
               v-else-if="view === 'sets'" key="sets"
-              :axes="setAxes" :color-of="colorOf" :filters="filters" :label-of="labelOf" :rows="table.rows" :selected="selected"
+              :axes="setAxes" :color-of="colorOf" :filters="filters" :label-of="labelOf" :rows="table.rows" :score="table.primary" :selected="selected"
               @pick="onPick"
               @reorder="names => axisOrder = [...names, ...axes.map(axis => axis.name).filter(name => !names.includes(name))]"
               @inspect="(name, box) => inspected = name == null || box == null ? undefined : { left: box.left, name, top: box.bottom + 6 }"
             />
             <GroupBars
               v-else-if="view === 'groups'" key="groups"
-              :categories="groupCategories" :color-of="colorOf" :definition-of="name => metricDefinition(name, definitions)" :filters="filters" :label-of="labelOf" :legend-colors="palette.legend.map(item => item.color)" :rows="table.rows" :values-of="valuesOf"
+              :categories="groupCategories" :color-of="colorOf" :definition-of="name => metricDefinition(name, definitions)" :filters="filters" :label-of="labelOf" :legend-colors="palette.legend.map(item => item.color)" :rows="table.rows" :score="table.primary" :values-of="valuesOf"
               @pick="onPick"
             />
             <div v-else-if="view === 'pairs'" key="pairs" :class="['grid gap-3']">
@@ -650,7 +652,7 @@ const unitOf = (name: string) => definitions.value.get(name)?.unit
         >
           <AnimatedContent>
             <div v-if="shownCard" :class="['w-80 max-w-[calc(100vw-2rem)]']">
-              <CaseCard :active-axis="shownCard.axis" :axes="axes" :href="caseHref(shownCard.row)" :label-of="labelOf" :pinned="shownCard.pinned" :row="shownCard.row" :rows="table.rows" :scales="scales" :unit-of="unitOf" :value-counts="table.valueCounts" @unpin="pinned = undefined" />
+              <CaseCard :active-axis="shownCard.axis" :axes="axes" :href="caseHref(shownCard.row)" :label-of="labelOf" :pinned="shownCard.pinned" :row="shownCard.row" :rows="table.rows" :scales="scales" :format-of="formatOf" :value-counts="table.valueCounts" @unpin="pinned = undefined" />
             </div>
             <PopoverArrow :class="['floating-arrow']" />
           </AnimatedContent>
@@ -673,7 +675,7 @@ const unitOf = (name: string) => definitions.value.get(name)?.unit
             The case's value on every axis and where it sits among all cases.
           </DrawerDescription>
           <div v-if="pinnedRow" :class="['overflow-y-auto min-h-0']">
-            <CaseCard :active-axis="pinned?.axis ?? undefined" :axes="axes" :href="caseHref(pinnedRow)" :label-of="labelOf" layout="drawer" pinned :row="pinnedRow" :rows="table.rows" :scales="scales" :unit-of="unitOf" :value-counts="table.valueCounts" @unpin="pinned = undefined" />
+            <CaseCard :active-axis="pinned?.axis ?? undefined" :axes="axes" :href="caseHref(pinnedRow)" :label-of="labelOf" layout="drawer" pinned :row="pinnedRow" :rows="table.rows" :scales="scales" :format-of="formatOf" :value-counts="table.valueCounts" @unpin="pinned = undefined" />
           </div>
         </DrawerContent>
       </DrawerPortal>
diff --git a/packages/ui/client/src/components/CrumbSwitcher.test.ts b/packages/ui/client/src/components/CrumbSwitcher.test.ts
index b80f033..f1112f5 100644
--- a/packages/ui/client/src/components/CrumbSwitcher.test.ts
+++ b/packages/ui/client/src/components/CrumbSwitcher.test.ts
@@ -8,9 +8,9 @@ import CrumbSwitcher from './CrumbSwitcher.vue'
 
 /** Experiments of one project as the breadcrumb of an experiment page offers them. */
 const options: SwitcherOption[] = [
-  { detail: '10 attempt(s)', exact: 0.438, href: '#/?experiment=matrix-run-model-gpt-5.5-repeat-1&project=p&workspace=w', id: 'matrix-run-model-gpt-5.5-repeat-1', label: 'model gpt-5.5 · repeat 1', state: 'ended' },
-  { chips: [{ axis: 'model', values: 'gpt-5.5' }], detail: '1 attempt(s)', exact: 0.496, href: '#/?experiment=auv-c759d695&project=p&workspace=w', id: 'auv-c759d695', label: 'auv-c759d695', state: 'ended' },
-  { detail: '1 attempt(s)', exact: 1, href: '#/?experiment=matrix-run-model-gpt-5.5-repeat-1--run-model-gpt-5.5-repeat-3&project=p&workspace=w', id: 'matrix-run-model-gpt-5.5-repeat-1--run-model-gpt-5.5-repeat-3', label: 'model gpt-5.5 · repeat 1–3', state: 'legacy' },
+  { detail: '10 attempt(s)', href: '#/?experiment=matrix-run-model-gpt-5.5-repeat-1&project=p&workspace=w', id: 'matrix-run-model-gpt-5.5-repeat-1', label: 'model gpt-5.5 · repeat 1', score: 0.438, scoreLabel: 'Pass', state: 'ended' },
+  { chips: [{ axis: 'model', values: 'gpt-5.5' }], detail: '1 attempt(s)', href: '#/?experiment=auv-c759d695&project=p&workspace=w', id: 'auv-c759d695', label: 'auv-c759d695', score: 0.496, scoreLabel: 'Pass', state: 'ended' },
+  { detail: '1 attempt(s)', href: '#/?experiment=matrix-run-model-gpt-5.5-repeat-1--run-model-gpt-5.5-repeat-3&project=p&workspace=w', id: 'matrix-run-model-gpt-5.5-repeat-1--run-model-gpt-5.5-repeat-3', label: 'model gpt-5.5 · repeat 1–3', score: 1, scoreLabel: 'Pass', state: 'unsupported' },
 ]
 
 describe('crumbSwitcher', () => {
diff --git a/packages/ui/client/src/components/CrumbSwitcher.vue b/packages/ui/client/src/components/CrumbSwitcher.vue
index fc771b6..816c3ad 100644
--- a/packages/ui/client/src/components/CrumbSwitcher.vue
+++ b/packages/ui/client/src/components/CrumbSwitcher.vue
@@ -99,7 +99,7 @@ function choose(id: unknown): void {
                   <span :class="['flex items-center gap-2 min-w-0']">
                     <StateBadge v-if="option.state" :state="option.state" icon-only />
                     <span :class="['font-medium truncate']">{{ option.label }}</span>
-                    <span v-if="option.exact !== undefined" :class="['ml-auto shrink-0 tabular-nums text-xs font-semibold']">{{ formatScore(option.exact) }}</span>
+                    <span v-if="option.score !== undefined" :class="['ml-auto shrink-0 tabular-nums text-xs font-semibold']" :title="option.scoreLabel">{{ formatScore(option.score) }}</span>
                   </span>
                   <span v-if="option.chips?.length" :class="['flex flex-wrap gap-1']">
                     <span v-for="chip in option.chips" :key="chip.axis" :class="['badge state-neutral font-normal']"><span class="muted">{{ chip.axis }}</span>{{ chip.values }}</span>
diff --git a/packages/ui/client/src/components/ErrorGroupsPanel.test.ts b/packages/ui/client/src/components/ErrorGroupsPanel.test.ts
index 12dbc46..4edae32 100644
--- a/packages/ui/client/src/components/ErrorGroupsPanel.test.ts
+++ b/packages/ui/client/src/components/ErrorGroupsPanel.test.ts
@@ -6,7 +6,7 @@ import { render } from 'vitest-browser-vue'
 import ErrorGroupsPanel from './ErrorGroupsPanel.vue'
 
 function caseOf(caseId: string, state: CaseNode['state'], errorMessage?: string): CaseNode {
-  return { artifactCount: 0, caseId, errorMessage, errorName: errorMessage == null ? undefined : 'APICallError', evidenceLinkCount: 0, identity: caseId, index: 0, key: `task\u0000${caseId}`, logLineCount: 0, metricCount: 0, name: caseId, rounds: [], scores: {}, startedAt: '2026-10-19T06:36:26.000Z', state, taskId: 'task' }
+  return { artifactCount: 0, caseId, errorMessage, errorName: errorMessage == null ? undefined : 'APICallError', evidenceLinkCount: 0, identity: caseId, index: 0, key: `task\u0000${caseId}`, logLineCount: 0, metricCount: 0, name: caseId, rounds: [], scores: { unscored: {}, values: {} }, startedAt: '2026-10-19T06:36:26.000Z', state, taskId: 'task' }
 }
 
 describe('errorGroupsPanel', () => {
diff --git a/packages/ui/client/src/components/ExecutionTree.vue b/packages/ui/client/src/components/ExecutionTree.vue
index 2f4c7ee..8989128 100644
--- a/packages/ui/client/src/components/ExecutionTree.vue
+++ b/packages/ui/client/src/components/ExecutionTree.vue
@@ -1,5 +1,5 @@
 <script setup lang="ts">
-import type { CaseCounts, CaseNode, CaseRound, RunSnapshot } from '../../../src/shared/contract'
+import type { CaseCounts, CaseNode, CaseRound, RunSnapshot, TaskNode } from '../../../src/shared/contract'
 import type { Segment } from './SegmentedControl.vue'
 
 import { useVirtualizer } from '@tanstack/vue-virtual'
@@ -14,6 +14,7 @@ import StateBadge from './StateBadge.vue'
 import { formatDuration, formatScore } from '../format'
 import { levelIcons, levelLabels } from '../hierarchy'
 import { href } from '../router'
+import { completionText, headlineOf, headlineView } from '../scores'
 
 /**
  * Live execution tree `run → project → task → round → case → try`, virtualized so tasks with hundreds of cases
@@ -69,6 +70,13 @@ function matches(state: string, name: string): boolean {
   return state === filter.value
 }
 
+/** A task's headline on its row: the primary score (or the passed cases) and, below 100 %, the completion rate. */
+function taskDetail(task: TaskNode): string {
+  const view = headlineView(headlineOf(task.scoring), props.snapshot.scoreDefinitions)
+  const incomplete = view.completion.mean != null && view.completion.mean < 1
+  return `${view.label} ${formatScore(view.summary.mean)}${incomplete ? ` · ${completionText(view.completion)}` : ''}`
+}
+
 const rows = computed<Row[]>(() => {
   const list: Row[] = []
   const snapshot = props.snapshot
@@ -81,7 +89,7 @@ const rows = computed<Row[]>(() => {
       continue
     for (const task of tasks) {
       const taskKey = `t:${task.id}`
-      list.push({ counts: { ...task.caseCounts, total: task.caseTotal ?? task.caseCounts.total }, depth: 1, detail: `exact ${formatScore(task.scores.exact)}`, expandable: true, href: href({ name: 'task', runKey: snapshot.runKey, taskId: task.id }), key: taskKey, kind: 'task', label: task.displayName, state: task.state })
+      list.push({ counts: { ...task.caseCounts, total: task.caseTotal ?? task.caseCounts.total }, depth: 1, detail: taskDetail(task), expandable: true, href: href({ name: 'task', runKey: snapshot.runKey, taskId: task.id }), key: taskKey, kind: 'task', label: task.displayName, state: task.state })
       if (!open.has(taskKey))
         continue
       const pushCase = (node: CaseNode, depth: number, round: CaseRound | undefined) => {
diff --git a/packages/ui/client/src/components/GroupBars.vue b/packages/ui/client/src/components/GroupBars.vue
index 4d12ed2..17a179c 100644
--- a/packages/ui/client/src/components/GroupBars.vue
+++ b/packages/ui/client/src/components/GroupBars.vue
@@ -1,8 +1,8 @@
 <script setup lang="ts">
 import type { ComponentPublicInstance } from 'vue'
 
-import type { MetricDefinition } from '../../../src/shared/contract'
 import type { CaseFilters, CaseRow } from '../caseMetrics'
+import type { ColumnDefinition } from '../scores'
 import type { BesideTarget } from './MetricBesideCard.vue'
 
 import { DropdownMenuContent, DropdownMenuItemIndicator, DropdownMenuPortal, DropdownMenuRadioGroup, DropdownMenuRadioItem, DropdownMenuRoot, DropdownMenuTrigger } from 'reka-ui'
@@ -15,7 +15,7 @@ import { formatScore } from '../format'
 
 /**
  * Cases grouped by one category (a stratum, a grading rule, a matrix axis): one row per value with a horizontal bar of
- * its cases in the chart's colors, the group's mean exact score (or pass rate without scores), and its case count.
+ * its cases in the chart's colors, the group's mean primary score (or pass rate without one), and its case count.
  * Rows are sorted by that score, lowest first, so the groups that pull the score down lead. Clicking a row picks its
  * value, like clicking it on a chart axis.
  *
@@ -27,12 +27,14 @@ const props = defineProps<{
   categories: readonly string[]
   colorOf: (row: CaseRow) => string
   /** Definition of a category (label, description) for the details beside the menu. */
-  definitionOf: (name: string) => MetricDefinition | undefined
+  definitionOf: (name: string) => ColumnDefinition | undefined
   filters: CaseFilters
   labelOf: (name: string) => string
   /** Colors of the legend in its order, so the segments of every bar stack the same way. */
   legendColors: readonly string[]
   rows: readonly CaseRow[]
+  /** Score the groups are ranked by (`CaseTable.primary`); without it, the pass rate. */
+  score?: string
   /** Every case's value of a category, for the distribution beside the menu. */
   valuesOf: (name: string) => readonly unknown[]
 }>()
@@ -41,7 +43,7 @@ const emit = defineEmits<{ pick: [name: string, value: string] }>()
 /** Group by the first category that is not the outcome itself; `state` only when nothing else exists. */
 const picked = shallowRef<string>()
 const groupBy = computed<string>({
-  get: () => (picked.value != null && props.categories.includes(picked.value) ? picked.value : props.categories.find(name => name !== 'state' && name !== 'exact') ?? props.categories[0] ?? 'state'),
+  get: () => (picked.value != null && props.categories.includes(picked.value) ? picked.value : props.categories.find(name => name !== 'state' && name !== props.score) ?? props.categories[0] ?? 'state'),
   set: (name) => {
     picked.value = name
   },
@@ -56,7 +58,7 @@ const highlighted = shallowRef<BesideTarget>()
 const menu = useTemplateRef<ComponentPublicInstance>('menu')
 const menuElement = computed(() => (menu.value?.$el instanceof HTMLElement ? menu.value.$el : undefined))
 
-const scored = computed(() => props.rows.some(row => row.numbers.exact != null))
+const scored = computed(() => props.score != null && props.rows.some(row => row.numbers[props.score!] != null))
 const groups = computed(() => {
   const name = groupBy.value
   const byValue = new Map<string, CaseRow[]>()
@@ -72,7 +74,7 @@ const groups = computed(() => {
   }
   const colorRank = new Map(props.legendColors.map((color, index) => [color, index]))
   return [...byValue].map(([value, rows]) => {
-    const scores = rows.map(row => row.numbers.exact).filter((score): score is number => score != null)
+    const scores = props.score == null ? [] : rows.map(row => row.numbers[props.score!]).filter((score): score is number => score != null)
     const finished = rows.filter(row => row.state === 'passed' || row.state === 'failed' || row.state === 'errored' || row.state === 'timeout')
     const score = scored.value
       ? (scores.length === 0 ? null : scores.reduce((sum, item) => sum + item, 0) / scores.length)
@@ -110,7 +112,7 @@ const groups = computed(() => {
         </DropdownMenuPortal>
       </DropdownMenuRoot>
       <MetricBesideCard :anchor="menuElement" :target="highlighted" :definition-of="props.definitionOf" :values-of="props.valuesOf" />
-      <span class="muted">{{ scored ? 'mean exact score' : 'pass rate' }} per group, lowest first · bars colored like the chart · click a group to select it</span>
+      <span class="muted">{{ scored ? `mean ${props.labelOf(props.score!)}` : 'pass rate' }} per group, lowest first · bars colored like the chart · click a group to select it</span>
     </div>
     <ul :class="['m-0 p-0 list-none grid']">
       <li v-for="group in groups" :key="group.value">
diff --git a/packages/ui/client/src/components/HeadlineScore.test.ts b/packages/ui/client/src/components/HeadlineScore.test.ts
new file mode 100644
index 0000000..0d0a6d0
--- /dev/null
+++ b/packages/ui/client/src/components/HeadlineScore.test.ts
@@ -0,0 +1,82 @@
+import type { HeadlineResult, ScoreDefinition } from '../../../src/shared/contract'
+
+import { describe, expect, it, vi } from 'vitest'
+import { render } from 'vitest-browser-vue'
+
+import HeadlineScore from './HeadlineScore.vue'
+
+const pass = { mean: 0.875, n: 119, name: 'pass', sum: 104.125, total: 121, unscored: { grader_failed: 2 } }
+const definitions: ScoreDefinition[] = [{ description: 'Written back correctly and read naturally.', label: { 'en': 'Pass', 'zh-Hans': '通过' }, name: 'pass', primary: true }]
+/** A run whose primary score is `pass`, with 4 of 125 cases errored and 2 the judge could not grade. */
+const result: HeadlineResult = {
+  completion: { mean: 0.968, n: 121, name: 'completion', sum: 121, total: 125, unscored: { errored: 4 } },
+  headline: { kind: 'score', score: pass },
+}
+
+describe('headlineScore', () => {
+  /**
+   * @example
+   * expect(screen.getByTestId('headline-label')).toHaveTextContent('Pass')
+   */
+  it('shows the value with the label of the primary score, the completion rate, and the unscored cases', async () => {
+    const screen = await render(HeadlineScore, { props: { definitions, result } })
+
+    await expect.element(screen.getByTestId('headline-value')).toHaveTextContent('87.5%')
+    await expect.element(screen.getByTestId('headline-label')).toHaveTextContent('Pass')
+    await expect.element(screen.getByTestId('headline-completion')).toHaveTextContent('completed 96.8% (4 errored)')
+    await expect.element(screen.getByTestId('headline-unscored')).toHaveTextContent('2 unscored')
+    await expect.element(screen.getByTestId('headline-unscored')).toHaveAttribute('title', '2 grader failed')
+  })
+
+  /**
+   * @example
+   * expect(onErrors).toHaveBeenCalledOnce()
+   */
+  it('leads from a completion rate below 100% to the cases without a result', async () => {
+    const onErrors = vi.fn()
+    const screen = await render(HeadlineScore, { props: { definitions, onErrors, result } })
+
+    await screen.getByRole('button', { name: /completed 96.8%/ }).click()
+    expect(onErrors).toHaveBeenCalledOnce()
+  })
+
+  /**
+   * @example
+   * expect(screen.getByTestId('headline-label')).toHaveTextContent('Passed cases')
+   */
+  it('reads a run without scores by its passed cases, and a list row hides a full completion rate', async () => {
+    const screen = await render(HeadlineScore, {
+      props: {
+        definitions: [],
+        result: { completion: { mean: 1, n: 10, name: 'completion', sum: 10, total: 10, unscored: {} }, headline: { kind: 'passed-cases', score: { mean: 0.6, n: 10, name: 'passed', sum: 6, total: 10, unscored: {} } } },
+        size: 'sm',
+      },
+    })
+
+    await expect.element(screen.getByTestId('headline-value')).toHaveTextContent('60.0%')
+    await expect.element(screen.getByTestId('headline-label')).toHaveTextContent('Passed cases')
+    expect(screen.container.querySelector('[data-testid="headline-completion"]')).toBeNull()
+  })
+
+  /**
+   * @example
+   * expect(screen.getByTestId('headline-tasks')).toHaveTextContent('osworld [en]')
+   */
+  it('lists each task\'s primary score under the completion rate when the tasks name different primary scores', async () => {
+    const screen = await render(HeadlineScore, {
+      props: {
+        definitions: [...definitions, { label: 'Success', name: 'osworld.success', primary: true }],
+        result: {
+          completion: result.completion,
+          headline: { kind: 'per-task', tasks: [{ score: pass, taskId: 'tts' }, { score: { mean: 0.4, n: 5, name: 'osworld.success', sum: 2, total: 5, unscored: {} }, taskId: 'osworld' }] },
+        },
+        taskLabel: (taskId: string) => `${taskId} [en]`,
+      },
+    })
+
+    await expect.element(screen.getByTestId('headline-value')).toHaveTextContent('96.8%')
+    await expect.element(screen.getByTestId('headline-label')).toHaveTextContent('completed')
+    await expect.element(screen.getByTestId('headline-tasks')).toHaveTextContent(/tts \[en\]\s*Pass\s*87\.5%/)
+    await expect.element(screen.getByTestId('headline-tasks')).toHaveTextContent(/osworld \[en\]\s*Success\s*40\.0%/)
+  })
+})
diff --git a/packages/ui/client/src/components/HeadlineScore.vue b/packages/ui/client/src/components/HeadlineScore.vue
new file mode 100644
index 0000000..220726d
--- /dev/null
+++ b/packages/ui/client/src/components/HeadlineScore.vue
@@ -0,0 +1,111 @@
+<script setup lang="ts">
+import type { HeadlineResult, ScoreDefinition } from '../../../src/shared/contract'
+import type { ColumnDefinition } from '../scores'
+
+import { computed } from 'vue'
+
+import MetricInfo from './MetricInfo.vue'
+
+import { formatScore } from '../format'
+import { completionParts, headlineView, scoreColumn, unscoredText } from '../scores'
+
+/**
+ * The headline of a run, a task, or a matrix cell, read the same everywhere: the value with the localized label of
+ * the primary score, and the completion rate next to it (`87.5% Pass · completed 96.8%`), since a score over the
+ * cases that produced a result says little without how many did. Hovering the value shows the score's description,
+ * formula, grader, and counted cases in the metric hover card.
+ *
+ * Without a primary score the headline is the share of passed cases (`Passed cases`). When the tasks of a run name
+ * different primary scores (`per-task`), the completion rate is the headline and each task's primary score is listed
+ * under it with the task's name.
+ *
+ * Use when:
+ * - a page, card, row, or switcher shows how a run, task, or matrix cell scored
+ */
+const props = defineProps<{
+  /**
+   * `end` right-aligns the lines, for a score column of a list.
+   * @default 'start'
+   */
+  align?: 'end' | 'start'
+  /** `score.defined` of the headline's scores (`headlineDefinitions`, `scoreDefinitions`). */
+  definitions: readonly ScoreDefinition[]
+  /**
+   * Called when the completion rate below 100 % is clicked; the page shows the cases without a result. Without it, the
+   * rate is plain text (inside a card that is a link itself, for example).
+   */
+  onErrors?: () => void
+  result: HeadlineResult | null | undefined
+  /**
+   * `lg`: the number of a score card; `md`: a page header; `sm`: a list row, where the completion rate is shown only
+   * below 100 % (the row's case bar says the rest).
+   * @default 'md'
+   */
+  size?: 'lg' | 'md' | 'sm'
+  /** Readable name of a task, for the tasks of a `per-task` headline. */
+  taskLabel?: (taskId: string) => string
+}>()
+
+const view = computed(() => (props.result == null ? undefined : headlineView(props.result, props.definitions)))
+const completion = computed(() => (view.value == null ? { rate: '' } : completionParts(view.value.completion)))
+/** What the hover card describes: the primary score, or the built-in passed share or completion rate. */
+const column = computed<ColumnDefinition | undefined>(() => {
+  if (view.value == null)
+    return undefined
+  if (view.value.definition != null)
+    return scoreColumn(view.value.definition, props.definitions)
+  return view.value.kind === 'passed-cases'
+    ? { better: 'higher', description: 'Share of the settled cases that passed. The tasks record no score.', label: view.value.label, name: 'passed' }
+    : { better: 'higher', description: 'Share of the settled cases that produced a result. The tasks name different primary scores, listed below.', label: 'Completion rate', name: 'completion' }
+})
+const unscoredCount = computed(() => Object.values(view.value?.summary.unscored ?? {}).reduce((sum, count) => sum + count, 0))
+const incomplete = computed(() => view.value != null && view.value.completion.mean != null && view.value.completion.mean < 1)
+const showCompletion = computed(() => view.value != null && view.value.kind !== 'per-task' && (props.size !== 'sm' || incomplete.value))
+const valueClass = computed(() => ({ lg: 'text-2xl', md: 'text-xl', sm: 'text-sm' })[props.size ?? 'md'])
+</script>
+
+<template>
+  <span v-if="view" :class="['inline-flex flex-col gap-1 min-w-0', props.align === 'end' && 'items-end text-right']" data-testid="headline">
+    <span :class="['flex items-baseline flex-wrap gap-x-1.5 min-w-0', props.align === 'end' && 'justify-end']">
+      <MetricInfo :name="column?.name ?? view.summary.name" :definition="column" :values="[]" :summary="view.summary" :completion="view.completion">
+        <span :class="['inline-flex items-baseline gap-1.5 min-w-0 cursor-help']">
+          <span :class="[valueClass, 'font-semibold tabular-nums leading-none']" data-testid="headline-value">{{ formatScore(view.summary.mean) }}</span>
+          <span :class="['text-xs muted truncate']" data-testid="headline-label">{{ view.label }}</span>
+        </span>
+      </MetricInfo>
+      <!-- Facts about the value itself (a change from the previous run), from the caller. -->
+      <slot />
+    </span>
+    <span v-if="showCompletion || unscoredCount > 0" :class="['flex flex-wrap gap-x-1.5 text-xs tabular-nums', props.align === 'end' && 'justify-end']">
+      <template v-if="showCompletion">
+        <button
+          v-if="incomplete && props.onErrors" type="button"
+          :class="['text-right underline decoration-dotted underline-offset-2 [color:var(--ink-errored)]']"
+          title="Show the cases without a result"
+          data-testid="headline-completion"
+          @click.stop.prevent="props.onErrors()"
+        >
+          <span :class="['whitespace-nowrap']">{{ completion.rate }}</span><template v-if="completion.reasons != null">
+            <!-- The space between the parts is the only place a narrow column may wrap the completion text. -->
+            {{ ' ' }}<span :class="['whitespace-nowrap']">{{ completion.reasons }}</span>
+          </template>
+        </button>
+        <span v-else :class="[incomplete ? '[color:var(--ink-errored)]' : 'muted']" title="Share of settled cases that produced a result" data-testid="headline-completion">
+          <span :class="['whitespace-nowrap']">{{ completion.rate }}</span><template v-if="completion.reasons != null">
+            <!-- The space between the parts is the only place a narrow column may wrap the completion text. -->
+            {{ ' ' }}<span :class="['whitespace-nowrap']">{{ completion.reasons }}</span>
+          </template>
+        </span>
+      </template>
+      <span v-if="unscoredCount > 0" :class="['muted whitespace-nowrap']" :title="unscoredText(view.summary.unscored)" data-testid="headline-unscored">· {{ unscoredCount }} unscored</span>
+    </span>
+    <ul v-if="view.tasks.length > 0" :class="['m-0 p-0 list-none grid gap-0.5 text-xs']" data-testid="headline-tasks">
+      <li v-for="task in view.tasks" :key="task.taskId" :class="['flex items-baseline gap-1.5 min-w-0']">
+        <span :class="['truncate muted']">{{ props.taskLabel?.(task.taskId) ?? task.taskId }}</span>
+        <span :class="['shrink-0']">{{ task.label }}</span>
+        <b :class="['ml-auto shrink-0 tabular-nums']">{{ formatScore(task.summary.mean) }}</b>
+      </li>
+    </ul>
+  </span>
+  <span v-else :class="['muted tabular-nums']" data-testid="headline">—</span>
+</template>
diff --git a/packages/ui/client/src/components/InferencePanel.vue b/packages/ui/client/src/components/InferencePanel.vue
index 46ddde8..93138a4 100644
--- a/packages/ui/client/src/components/InferencePanel.vue
+++ b/packages/ui/client/src/components/InferencePanel.vue
@@ -9,6 +9,7 @@ import StateBadge from './StateBadge.vue'
 
 import { groupInferenceCalls, modelKeyOf, summarizeByModel, tokenUsage } from '../../../src/shared/inference'
 import { formatCount, formatDuration, formatTime, previewValue } from '../format'
+import { isGraderCall } from '../scores'
 
 /**
  * Model calls of one case: what they cost in total (calls, tokens, model time, tool calls), then one row per call
@@ -16,10 +17,19 @@ import { formatCount, formatDuration, formatTime, previewValue } from '../format
  *
  * With more than one model, every model gets its own row, and the summary adds up the token counts each model
  * reported. That sum says how many tokens the case processed, but not what it cost: tokenizers and prices differ.
+ *
+ * With `grader`, only the calls of that grader (a score's judge) are shown and summed, with a chip to show them all.
  */
-const props = defineProps<{ events: CaseDetail['inference'] }>()
+const props = defineProps<{
+  events: CaseDetail['inference']
+  /** Grader `id` of a score; narrows the panel to its calls ({@link isGraderCall}). */
+  grader?: string
+}>()
+const emit = defineEmits<{ clearGrader: [] }>()
 
-const calls = computed(() => groupInferenceCalls(props.events).map(call => ({ ...call, tokens: tokenUsage(call.usage) })))
+const calls = computed(() => groupInferenceCalls(props.events)
+  .filter(call => props.grader == null || isGraderCall(call, props.grader))
+  .map(call => ({ ...call, tokens: tokenUsage(call.usage) })))
 const models = computed(() => summarizeByModel(calls.value).map(model => ({
   ...model,
   tokens: tokenUsage(model.usage),
@@ -60,6 +70,11 @@ function toggle(seq: number): void {
 
 <template>
   <div :class="['flex flex-col gap-4']" data-testid="inference-panel">
+    <div v-if="props.grader" :class="['flex items-center gap-2 text-xs']">
+      <button type="button" :class="['btn btn-active text-xs py-0.5!']" data-testid="inference-grader" @click="emit('clearGrader')">
+        <span class="i-mingcute-filter-line" aria-hidden="true" />calls of grader <span class="mono">{{ props.grader }}</span><span class="i-mingcute-close-line" aria-label="Show every call" />
+      </button>
+    </div>
     <section class="card">
       <h2 class="card-title">
         <span class="i-mingcute-coin-line" aria-hidden="true" />Usage
diff --git a/packages/ui/client/src/components/MatrixView.vue b/packages/ui/client/src/components/MatrixView.vue
index 0c46400..937d554 100644
--- a/packages/ui/client/src/components/MatrixView.vue
+++ b/packages/ui/client/src/components/MatrixView.vue
@@ -14,6 +14,7 @@ import { useCatalog } from '../composables/useCatalog'
 import { countOf, formatScore, shortTaskId } from '../format'
 import { attemptLabel, formatShortDateTime } from '../labels'
 import { href } from '../router'
+import { headlineOf, headlineView } from '../scores'
 
 /**
  * Matrix of a project or experiment (`GET /api/matrix`): one grid per eval entry, rows along the run axis with the
@@ -144,13 +145,22 @@ const visibleGrids = computed(() => grids.value.map((grid) => {
   return { ...grid, columns: kept.map(index => grid.columns[index]!), rows: grid.rows.map(row => ({ ...row, cells: kept.map(index => row.cells[index]) })) }
 }))
 
+/** Label of a cell's headline (its primary score, or the passed cases), for the trend and the tooltips. */
+function headlineLabel(cell: MatrixCell): string {
+  return headlineView(headlineOf(cell.latest.scoring), matrix.data.value?.scoreDefinitions ?? []).label
+}
 /**
  * Whether a cell is read by its score: every case of its run got a result and none failed an assertion, so the case
  * counts are all green and say nothing; how the scores spread does.
  */
 function isScored(cell: MatrixCell): boolean {
-  const { counts, exact, state } = cell.latest
-  return exact != null && counts.failed === 0 && state !== 'running' && state !== 'queued'
+  const { counts, scoring, state } = cell.latest
+  return scoring.primary != null && counts.failed === 0 && state !== 'running' && state !== 'queued'
+}
+/** Mean of the primary score of the cell's run, `null` without one. */
+function latestScore(cell: MatrixCell): null | number {
+  const { primary, scores } = cell.latest.scoring
+  return scores.find(summary => summary.name === primary)?.mean ?? null
 }
 /** When a point of a cell's history started: its run's start time, else the attempt label. */
 function pointTime(point: MatrixCell['history'][number]): string {
@@ -158,22 +168,23 @@ function pointTime(point: MatrixCell['history'][number]): string {
 }
 
 /**
- * Change of the exact score from the previous full run, in percentage points; `undefined` unless the cell's run and an
- * earlier run both covered the cell fully (a partial run or one without a result is no baseline).
+ * Change of the primary score from the previous full run, in percentage points; `undefined` unless the cell's run and
+ * an earlier run both covered the cell fully (a partial run or one without a result is no baseline).
  */
 function scoreDelta(cell: MatrixCell): number | undefined {
-  if (cell.latest.coverage !== 'full' || cell.latest.exact == null)
+  const latest = latestScore(cell)
+  if (cell.latest.coverage !== 'full' || latest == null)
     return undefined
   const index = cell.history.findIndex(point => point.runKey === cell.latest.runKey)
-  const previous = cell.history.slice(0, Math.max(0, index)).findLast(point => point.coverage === 'full' && point.exact != null)
-  return previous == null ? undefined : (cell.latest.exact - previous.exact!) * 100
+  const previous = cell.history.slice(0, Math.max(0, index)).findLast(point => point.coverage === 'full' && point.score != null)
+  return previous == null ? undefined : (latest - previous.score!) * 100
 }
 
 const COVERAGE_TEXT = { full: 'every case', none: 'no result', other: 'other cases', partial: 'some cases' } as const
-/** Tooltip of one history point: when, how much it measured, and its score. */
+/** Tooltip of one history point: when, how much it measured, its score, and its completion rate. */
 function pointTitle(cell: MatrixCell, point: MatrixCell['history'][number]): string {
   const measured = point.counts.passed + point.counts.failed
-  return `${pointTime(point)} · ${COVERAGE_TEXT[point.coverage]} (${measured} of ${cell.caseCount} cases with a result) · exact ${formatScore(point.exact)}`
+  return `${pointTime(point)} · ${COVERAGE_TEXT[point.coverage]} (${measured} of ${cell.caseCount} cases with a result) · ${headlineLabel(cell)} ${formatScore(point.score)} · completed ${formatScore(point.completion)}`
 }
 </script>
 
@@ -216,7 +227,8 @@ function pointTitle(cell: MatrixCell, point: MatrixCell['history'][number]): str
                 :class="['w-44 md:w-56', cell.superseded && 'opacity-60']"
                 compact
                 :href="href({ name: 'task', runKey: cell.latest.runKey, taskId: cell.latest.taskId })"
-                :score="cell.latest.exact"
+                :headline="headlineOf(cell.latest.scoring)"
+                :definitions="matrix.data.value?.scoreDefinitions ?? []"
                 :state="isScored(cell) && cell.latest.state === 'passed' ? undefined : cell.latest.state"
                 :counts="isScored(cell) ? undefined : { ...cell.latest.counts, total: Math.max(cell.latest.counts.total, cell.latest.caseTotal ?? 0) }"
                 :run="runOf(cell)"
@@ -238,8 +250,8 @@ function pointTitle(cell: MatrixCell, point: MatrixCell['history'][number]): str
                 <span :class="['flex items-center gap-1.5 min-w-0']" :title="`${attemptLabel(cell.latest.attemptId)} · ${cell.latest.runKey}`">
                   <span :class="['truncate']">{{ cell.latest.startedAt ? formatShortDateTime(new Date(cell.latest.startedAt)) : attemptLabel(cell.latest.attemptId) }} · {{ countOf(cell.latest.counts.total, 'case') }} · {{ countOf(cell.history.length, 'run') }}</span>
                   <Sparkline
-                    :class="['ml-auto']" :values="cell.history.map(point => point.exact)" :coverage="cell.history.map(point => point.coverage)" :titles="cell.history.map(point => pointTitle(cell, point))"
-                    :label="`exact score of ${cell.history.length} runs`" :width="48" :height="16"
+                    :class="['ml-auto']" :values="cell.history.map(point => point.score)" :coverage="cell.history.map(point => point.coverage)" :titles="cell.history.map(point => pointTitle(cell, point))"
+                    :label="`${headlineLabel(cell)} of ${cell.history.length} runs`" :width="48" :height="16"
                   />
                 </span>
                 <span v-if="cell.latest.coverage !== 'full' && cell.latest.state !== 'running' && cell.latest.state !== 'queued'" :class="['flex items-center gap-1.5 [color:var(--ink-errored)]']" data-testid="matrix-cell-partial">
diff --git a/packages/ui/client/src/components/MetricBesideCard.vue b/packages/ui/client/src/components/MetricBesideCard.vue
index 5c5ddcb..e12f860 100644
--- a/packages/ui/client/src/components/MetricBesideCard.vue
+++ b/packages/ui/client/src/components/MetricBesideCard.vue
@@ -1,5 +1,5 @@
 <script setup lang="ts">
-import type { MetricDefinition } from '../../../src/shared/contract'
+import type { ColumnDefinition } from '../scores'
 
 import { useMediaQuery } from '@vueuse/core'
 import { PopoverAnchor, PopoverContent, PopoverPortal, PopoverRoot } from 'reka-ui'
@@ -28,7 +28,7 @@ export interface BesideTarget {
 const props = defineProps<{
   /** The menu or list: the card sits right of its edge. */
   anchor: HTMLElement | null | undefined
-  definitionOf: (name: string) => MetricDefinition | undefined
+  definitionOf: (name: string) => ColumnDefinition | undefined
   /** The highlighted metric and its item: the card follows the item up and down. `undefined` while none is. */
   target: BesideTarget | undefined
   valuesOf: (name: string) => readonly unknown[]
diff --git a/packages/ui/client/src/components/MetricInfo.vue b/packages/ui/client/src/components/MetricInfo.vue
index e07fa24..8f7c0e0 100644
--- a/packages/ui/client/src/components/MetricInfo.vue
+++ b/packages/ui/client/src/components/MetricInfo.vue
@@ -1,5 +1,6 @@
 <script setup lang="ts">
-import type { MetricDefinition } from '../../../src/shared/contract'
+import type { ScoreSummary } from '../../../src/shared/contract'
+import type { ColumnDefinition } from '../scores'
 
 import { HoverCardArrow, HoverCardContent, HoverCardPortal, HoverCardRoot, HoverCardTrigger } from 'reka-ui'
 
@@ -10,9 +11,11 @@ import MetricSummary from './MetricSummary.vue'
  * Hover card of one metric around any trigger (a chip, a chart caption, a list row): shows {@link MetricSummary}.
  */
 const props = defineProps<{
+  completion?: ScoreSummary
   current?: number | string
-  definition?: MetricDefinition
+  definition?: ColumnDefinition
   name: string
+  summary?: ScoreSummary
   values: readonly unknown[]
 }>()
 </script>
@@ -25,7 +28,7 @@ const props = defineProps<{
     <HoverCardPortal>
       <HoverCardContent as-child :side-offset="6" :class="['floating p-2.5 text-xs select-text']">
         <AnimatedContent>
-          <MetricSummary :name="props.name" :definition="props.definition" :values="props.values" :current="props.current" />
+          <MetricSummary :name="props.name" :definition="props.definition" :values="props.values" :current="props.current" :summary="props.summary" :completion="props.completion" />
           <HoverCardArrow :class="['floating-arrow']" />
         </AnimatedContent>
       </HoverCardContent>
diff --git a/packages/ui/client/src/components/MetricSummary.vue b/packages/ui/client/src/components/MetricSummary.vue
index ee569bd..32e4c4b 100644
--- a/packages/ui/client/src/components/MetricSummary.vue
+++ b/packages/ui/client/src/components/MetricSummary.vue
@@ -1,23 +1,33 @@
 <script setup lang="ts">
-import type { MetricDefinition } from '../../../src/shared/contract'
+import type { ScoreSummary } from '../../../src/shared/contract'
+import type { ColumnDefinition } from '../scores'
 
 import { computed } from 'vue'
 
+import { formatColumnValue } from '../caseMetrics'
+import { formatScore } from '../format'
 import { localize } from '../localize'
+import { completionText, graderText, unscoredText } from '../scores'
 
 /**
- * What a metric is and how its values spread: label, name, description, unit and direction, and a small
- * distribution (a histogram for numbers, a stacked bar for categories), with the value of one case marked.
+ * What a metric or a score is and how its values spread: label, name, description, unit and direction, and a small
+ * distribution (a histogram for numbers, a stacked bar for categories), with the value of one case marked. A score
+ * (`definition.score`) adds its formula, its grader, and how errors count, and reads its values as percentages; with
+ * `summary`, the cases that counted toward its mean and the ones left unscored.
  *
  * Use as the body of a hover card ({@link MetricInfo}) or of a chart's floating tooltip.
  */
 const props = defineProps<{
+  /** Completion rate of the cases the summary covers, shown under it. */
+  completion?: ScoreSummary
   /** Value of the case being looked at, marked on the distribution. */
   // NOTICE: no `boolean` here: Vue casts an absent prop whose type includes Boolean to `false`, which would
   // mark a value on cards that have no current case. Callers pass flags as strings.
   current?: number | string
-  definition?: MetricDefinition
+  definition?: ColumnDefinition
   name: string
+  /** Summary of a score over a group of cases (a headline): mean, counted cases, unscored reasons. */
+  summary?: ScoreSummary
   /** Every recorded value of the metric in scope (one per case); `undefined` entries are cases without it. */
   values: readonly unknown[]
 }>()
@@ -25,6 +35,7 @@ const props = defineProps<{
 const label = computed(() => localize(props.definition?.label))
 const description = computed(() => localize(props.definition?.description))
 const unit = computed(() => props.definition?.unit)
+const score = computed(() => props.definition?.score)
 
 /** Histogram bins of numeric values; most metrics have a long tail, so a handful of bins reads best. */
 const BINS = 16
@@ -68,10 +79,7 @@ const categories = computed(() => {
   return { segments: rest > 0 ? [...shown, { color: 'var(--chart-text)', count: rest, value: `${ranked.length - TOP_CATEGORIES} more` }] : shown, total }
 })
 
-function format(value: number): string {
-  const text = Math.abs(value) >= 1000 ? Intl.NumberFormat(undefined, { maximumFractionDigits: 1, notation: 'compact' }).format(value) : String(Math.round(value * 100) / 100)
-  return unit.value == null ? text : `${text} ${unit.value}`
-}
+const format = (value: number) => formatColumnValue(value, props.definition)
 </script>
 
 <template>
@@ -87,11 +95,36 @@ function format(value: number): string {
     <p v-if="description" :class="['m-0 leading-snug opacity-90']">
       {{ description }}
     </p>
-    <div v-if="unit || props.definition?.better" :class="['flex flex-wrap gap-x-3 gap-y-0.5 opacity-80']">
+    <div v-if="unit || props.definition?.better || score" :class="['flex flex-wrap gap-x-3 gap-y-0.5 opacity-80']">
+      <span v-if="score?.primary" :class="['font-semibold']">primary score</span>
       <span v-if="unit">unit · {{ unit }}</span>
       <span v-if="props.definition?.better" :class="['inline-flex items-center gap-0.5']">
         <span :class="[props.definition.better === 'higher' ? 'i-mingcute-arrow-up-line' : 'i-mingcute-arrow-down-line']" aria-hidden="true" />{{ props.definition.better }} is better
       </span>
+      <span v-if="score?.errors === 'zero'">errored cases count as 0</span>
+    </div>
+    <dl v-if="props.definition?.formula || score?.grader" :class="['m-0 grid grid-cols-[auto_minmax(0,1fr)] gap-x-2 gap-y-0.5']">
+      <template v-if="props.definition?.formula">
+        <dt :class="['opacity-70']">
+          formula
+        </dt>
+        <dd :class="['m-0']" data-testid="metric-summary-formula">
+          {{ props.definition.formula }}
+        </dd>
+      </template>
+      <template v-if="score?.grader">
+        <dt :class="['opacity-70']">
+          grader
+        </dt>
+        <dd :class="['m-0']">
+          {{ graderText(score.grader) }}
+        </dd>
+      </template>
+    </dl>
+    <div v-if="props.summary" :class="['grid gap-0.5 tabular-nums']" data-testid="metric-summary-score">
+      <span><b>{{ formatScore(props.summary.mean) }}</b> over {{ props.summary.n }} of {{ props.summary.total }} cases</span>
+      <span v-if="unscoredText(props.summary.unscored)" :class="['opacity-80']">unscored: {{ unscoredText(props.summary.unscored) }}</span>
+      <span v-if="props.completion" :class="['opacity-80']">{{ completionText(props.completion) }}</span>
     </div>
 
     <figure v-if="numeric && histogram" :class="['m-0 grid gap-0.5']" :aria-label="`Distribution of ${props.values.length} values`">
@@ -123,7 +156,7 @@ function format(value: number): string {
         </li>
       </ul>
     </figure>
-    <div :class="['opacity-70']">
+    <div v-if="props.summary == null" :class="['opacity-70']">
       {{ props.values.filter(value => value != null).length }} value(s)
     </div>
   </div>
diff --git a/packages/ui/client/src/components/ParallelSets.vue b/packages/ui/client/src/components/ParallelSets.vue
index c5b73bc..80311f4 100644
--- a/packages/ui/client/src/components/ParallelSets.vue
+++ b/packages/ui/client/src/components/ParallelSets.vue
@@ -10,7 +10,7 @@ import { formatScore } from '../format'
  * Parallel sets of cases over category axes (matrix axes such as `tts` and `language`, string metrics such as a
  * stratum): every axis stacks one block per value, as tall as its share of the cases, and ribbons as wide as the cases
  * they carry join neighbor axes. Ribbons take the chart's case colors; cases outside the selection fade to gray.
- * Beside each value, its cases' mean exact score (of the selection, when one is made), so brushing `tts=official`
+ * Beside each value, its cases' mean primary score (of the selection, when one is made), so brushing `tts=official`
  * reads as "within official, ko is worst". Blocks are sorted by that score, best on top.
  *
  * Use when:
@@ -27,6 +27,8 @@ const props = defineProps<{
   filters: CaseFilters
   labelOf: (name: string) => string
   rows: readonly CaseRow[]
+  /** Score beside each value (`CaseTable.primary`); without it, the blocks show their case counts only. */
+  score?: string
   /** Keys of the selected cases. */
   selected: ReadonlySet<string>
 }>()
@@ -55,7 +57,9 @@ const filtering = computed(() => props.selected.size < props.rows.length)
 interface Block { count: number, from: number, height: number, mean: null | number, picked: boolean, value: string }
 
 function meanOf(rows: readonly CaseRow[]) {
-  const scores = rows.map(row => row.numbers.exact).filter((value): value is number => value != null)
+  if (props.score == null)
+    return null
+  const scores = rows.map(row => row.numbers[props.score!]).filter((value): value is number => value != null)
   return scores.length === 0 ? null : scores.reduce((sum, value) => sum + value, 0) / scores.length
 }
 
@@ -201,7 +205,7 @@ function startDrag(name: string, event: PointerEvent): void {
         >{{ props.labelOf(axis.name) }}</text>
         <text :x="axis.x" y="27" text-anchor="middle" font-size="10" fill="var(--chart-text)">click to pick · drag title</text>
         <g v-for="block in axis.blocks" :key="block.value" :class="['cursor-pointer']" @click="emit('pick', axis.name, block.value)">
-          <title>{{ block.value }} · {{ block.count }} case(s) · exact {{ formatScore(block.mean) }}</title>
+          <title>{{ block.value }} · {{ block.count }} case(s)<template v-if="props.score"> · {{ props.labelOf(props.score) }} {{ formatScore(block.mean) }}</template></title>
           <rect
             :x="axis.x - BLOCK_PX / 2" :y="block.from" :width="BLOCK_PX" :height="Math.max(1, block.height)" rx="2"
             :fill="block.picked ? 'var(--series-1)' : 'var(--chart-grid)'" :stroke="block.picked ? 'var(--series-1)' : 'none'"
diff --git a/packages/ui/client/src/components/ProgressPanel.test.ts b/packages/ui/client/src/components/ProgressPanel.test.ts
index 757b929..bcfa357 100644
--- a/packages/ui/client/src/components/ProgressPanel.test.ts
+++ b/packages/ui/client/src/components/ProgressPanel.test.ts
@@ -13,11 +13,13 @@ const snapshot = {
     waiting: [],
   },
   eventCounts: {},
+  headline: null,
   identity: { attemptId: 'a', experimentId: 'e', runId: 'run-1791747103580-dd3a8517', workspaceId: 'w' },
   parseErrors: 0,
   progress: { basis: 'cases', cases: { completed: 58, estimatedTotal: 345, knownTotal: 345, running: 8, waiting: 0 }, elapsedMs: 3_805_930, estimatedTotalMs: 22_638_722, etaMs: 18_832_792, history: [], lowerBound: false, ratio: 58 / 345, tasks: { completed: 0, total: 1 } },
   projects: [],
   runKey: 'run-1791747103580-dd3a8517',
+  scoreDefinitions: [],
   seq: 930,
   seqGaps: 0,
   source: 'live-tail',
diff --git a/packages/ui/client/src/components/ScoreBadge.vue b/packages/ui/client/src/components/ScoreBadge.vue
new file mode 100644
index 0000000..ac995be
--- /dev/null
+++ b/packages/ui/client/src/components/ScoreBadge.vue
@@ -0,0 +1,74 @@
+<script setup lang="ts">
+import type { ScoreRecordedData } from 'vieval/events'
+
+import type { ScoreDefinition } from '../../../src/shared/contract'
+
+import { useMediaQuery } from '@vueuse/core'
+import { PopoverArrow, PopoverContent, PopoverPortal, PopoverRoot, PopoverTrigger } from 'reka-ui'
+import { DrawerContent, DrawerDescription, DrawerHandle, DrawerOverlay, DrawerPortal, DrawerRoot, DrawerTitle } from 'vaul-vue'
+import { computed, shallowRef } from 'vue'
+
+import AnimatedContent from './AnimatedContent.vue'
+import ScoreDetail from './ScoreDetail.vue'
+
+import { scoreLabel } from '../scores'
+
+/**
+ * One score of a case as a badge (`Pass 100.0%`); opening it shows {@link ScoreDetail}: why the score has its value,
+ * what graded it, and the grader's model calls. A popover on desktop, a bottom drawer on phones, with the same body.
+ */
+const props = defineProps<{
+  callCount: number
+  callsHref?: string
+  definitions: readonly ScoreDefinition[]
+  name: string
+  record?: ScoreRecordedData
+  /** The value as shown: a percentage, an unscored reason, or `—`. */
+  text: string
+}>()
+
+const isDesktop = useMediaQuery('(min-width: 768px)')
+const open = shallowRef(false)
+const label = computed(() => scoreLabel(props.name, props.definitions))
+const unscored = computed(() => props.record != null && 'unscored' in props.record)
+const badge = computed(() => ['badge cursor-pointer transition-colors hover:bg-primary-500/10 focus-visible:(outline-2 outline-primary-500)', unscored.value ? 'state-interrupted' : 'state-neutral'])
+</script>
+
+<template>
+  <PopoverRoot v-if="isDesktop" v-model:open="open">
+    <PopoverTrigger :class="badge" :title="props.name" data-testid="score-badge">
+      {{ label }} {{ props.text }}<span class="i-mingcute-down-line opacity-60" aria-hidden="true" />
+    </PopoverTrigger>
+    <PopoverPortal>
+      <PopoverContent as-child align="end" :side-offset="6" :collision-padding="12" :class="['floating p-3 text-xs select-text outline-none']">
+        <AnimatedContent>
+          <!-- The content scrolls inside: `AnimatedContent` clips its own root. -->
+          <div :class="['w-80 max-w-[calc(100vw-2rem)] max-h-[min(32rem,var(--reka-popover-content-available-height))] overflow-y-auto']">
+            <ScoreDetail :call-count="props.callCount" :calls-href="props.callsHref" :definitions="props.definitions" :name="props.name" :record="props.record" :text="props.text" />
+          </div>
+          <PopoverArrow :class="['floating-arrow']" />
+        </AnimatedContent>
+      </PopoverContent>
+    </PopoverPortal>
+  </PopoverRoot>
+  <DrawerRoot v-else v-model:open="open">
+    <button type="button" :class="badge" :title="props.name" data-testid="score-badge" @click="open = true">
+      {{ label }} {{ props.text }}<span class="i-mingcute-down-line opacity-60" aria-hidden="true" />
+    </button>
+    <DrawerPortal>
+      <DrawerOverlay :class="['fixed inset-0 z-50 bg-black/40']" />
+      <DrawerContent :class="['floating fixed inset-x-0 bottom-0 max-h-[85dvh] flex flex-col gap-3 rounded-b-none! rounded-t-2xl! border-b-0! px-4 pt-3 pb-6 text-sm outline-none']">
+        <DrawerHandle :class="['bg-neutral-300! dark:bg-neutral-600!']" />
+        <DrawerTitle :class="['sr-only']">
+          {{ label }}
+        </DrawerTitle>
+        <DrawerDescription :class="['sr-only']">
+          Why the score has this value, and what graded it.
+        </DrawerDescription>
+        <div :class="['overflow-y-auto min-h-0']">
+          <ScoreDetail :call-count="props.callCount" :calls-href="props.callsHref" :definitions="props.definitions" :name="props.name" :record="props.record" :text="props.text" />
+        </div>
+      </DrawerContent>
+    </DrawerPortal>
+  </DrawerRoot>
+</template>
diff --git a/packages/ui/client/src/components/ScoreCard.vue b/packages/ui/client/src/components/ScoreCard.vue
index 0df895d..2de4bd7 100644
--- a/packages/ui/client/src/components/ScoreCard.vue
+++ b/packages/ui/client/src/components/ScoreCard.vue
@@ -1,17 +1,17 @@
 <script setup lang="ts">
-import type { CaseCounts } from '../../../src/shared/contract'
+import type { CaseCounts, HeadlineResult, ScoreDefinition } from '../../../src/shared/contract'
 
 import CountsBar from './CountsBar.vue'
+import HeadlineScore from './HeadlineScore.vue'
 import RunEta from './RunEta.vue'
 import StateBadge from './StateBadge.vue'
 
-import { formatScore } from '../format'
-
 /**
  * The standard card of a scored item (a project, a matrix cell, a compared column), laid out the same everywhere:
  *
  * ```
- * 66.1%  ↑2.3                 [running]
+ * 66.1% Pass  ↑2.3            [running]
+ * completed 96.8%
  * Title
  * meta (run counts, times)
  * Ends at 06:04      started 1h 12m ago   ← only while the run is running
@@ -29,19 +29,17 @@ const props = defineProps<{
   /** Tighter panel padding, for cards in a grid of many (matrix cells). */
   compact?: boolean
   counts?: CaseCounts
+  /** `score.defined` of the headline's scores, for their labels and descriptions. */
+  definitions: readonly ScoreDefinition[]
+  /** Headline of the item ({@link HeadlineScore}); `null` before any case settled. */
+  headline: HeadlineResult | null | undefined
   href?: string
   /** Run whose progress shows while `state` is `running`. */
   run?: { lastSeq: number, runKey: string, startedAt?: string }
-  /** Exact score, from 0 to 1; `null` before any case was scored. */
-  score: null | number | undefined
-  /**
-   * What the score is (for example "best exact score of 12 runs"); the tooltip and accessible name of the number.
-   *
-   * @default 'exact score'
-   */
-  scoreLabel?: string
   /** State of the item's newest run, in the top-right corner. */
   state?: string
+  /** Readable name of a task, for a headline that lists each task's primary score. */
+  taskLabel?: (taskId: string) => string
   title?: string
 }>()
 </script>
@@ -59,13 +57,10 @@ const props = defineProps<{
   >
     <span :class="['relative flex flex-col gap-2 flex-1 min-w-0 rounded-lg bg-white dark:bg-neutral-900', props.compact ? 'p-3 md:p-3.5' : 'p-4']">
       <span :class="['flex items-start gap-2']">
-        <span :class="['flex items-baseline flex-wrap gap-x-1.5 min-w-0']">
-          <span :class="['text-2xl font-semibold tabular-nums leading-none']" :title="props.scoreLabel ?? 'exact score'" data-testid="score-card-score">
-            {{ formatScore(props.score) }}<span :class="['sr-only']"> {{ props.scoreLabel ?? 'exact score' }}</span>
-          </span>
+        <HeadlineScore :class="['min-w-0']" :result="props.headline" :definitions="props.definitions" size="lg" :task-label="props.taskLabel" data-testid="score-card-score">
           <!-- Change from the previous run, a trend line, …: facts about the score itself. -->
           <slot name="score" />
-        </span>
+        </HeadlineScore>
         <StateBadge v-if="props.state" :class="['ml-auto shrink-0']" :state="props.state" />
       </span>
       <span v-if="props.title || $slots.title" :class="['font-semibold min-w-0 truncate']">
diff --git a/packages/ui/client/src/components/ScoreDetail.vue b/packages/ui/client/src/components/ScoreDetail.vue
new file mode 100644
index 0000000..ed58605
--- /dev/null
+++ b/packages/ui/client/src/components/ScoreDetail.vue
@@ -0,0 +1,118 @@
+<script setup lang="ts">
+import type { Grader, ScoreRecordedData } from 'vieval/events'
+
+import type { ScoreDefinition } from '../../../src/shared/contract'
+
+import { computed } from 'vue'
+
+import ObjectTree from './ObjectTree.vue'
+
+import { localize } from '../localize'
+import { formulaText, graderText, scoreDefinition, scoreLabel, unscoredReasonLabel } from '../scores'
+
+/**
+ * Why one score of one case has its value: the value, its definition (description, formula), what produced it (the
+ * grader, or for a derived score the graders of its inputs), the rationale and metadata the record carried, and a link
+ * to the grader's model calls.
+ *
+ * The body of the score badges on the case page, shared by the desktop popover and the phone drawer.
+ */
+const props = defineProps<{
+  /** Model calls of the grader in this case; the link is shown when there are any. */
+  callCount: number
+  /** Case page on its inference tab, narrowed to the grader's calls. */
+  callsHref?: string
+  definitions: readonly ScoreDefinition[]
+  name: string
+  /** The last `score.recorded` of the score in the case's last round; absent when the case never recorded it. */
+  record?: ScoreRecordedData
+  /** The value as shown on the badge (a percentage, an unscored reason, or `—`). */
+  text: string
+}>()
+
+const definition = computed(() => scoreDefinition(props.name, props.definitions))
+const label = computed(() => scoreLabel(props.name, props.definitions))
+const description = computed(() => localize(definition.value?.description))
+const formula = computed(() => formulaText(definition.value, props.definitions))
+const grader = computed<Grader | undefined>(() => props.record?.grader ?? definition.value?.grader)
+/** A derived score has no grader of its own; the graders of its inputs say what it rests on. */
+const inputGraders = computed(() => (definition.value?.combine?.inputs ?? []).flatMap((input) => {
+  const inputGrader = scoreDefinition(input.name, props.definitions)?.grader
+  return inputGrader == null ? [] : [{ label: scoreLabel(input.name, props.definitions), text: graderText(inputGrader) }]
+}))
+const unscored = computed(() => (props.record != null && 'unscored' in props.record ? unscoredReasonLabel(props.record.unscored.reason) : undefined))
+const derived = computed(() => props.record?.derived === true || definition.value?.combine != null)
+</script>
+
+<template>
+  <div :class="['grid gap-2 text-xs']" data-testid="score-detail">
+    <div :class="['flex items-start gap-2']">
+      <div :class="['min-w-0']">
+        <div :class="['text-sm font-semibold leading-snug']">
+          {{ label }}
+        </div>
+        <div v-if="label !== props.name" :class="['mono text-[11px] opacity-70 break-all']">
+          {{ props.name }}
+        </div>
+      </div>
+      <b :class="['ml-auto text-lg tabular-nums leading-none']" data-testid="score-detail-value">{{ props.text }}</b>
+    </div>
+    <div v-if="definition?.primary || derived || unscored" :class="['flex flex-wrap gap-1']">
+      <span v-if="definition?.primary" :class="['badge state-neutral']">primary</span>
+      <span v-if="derived" :class="['badge state-neutral']" data-testid="score-detail-derived">derived</span>
+      <span v-if="unscored" :class="['badge state-interrupted']" data-testid="score-detail-unscored">unscored · {{ unscored }}</span>
+    </div>
+    <p v-if="description" :class="['m-0 leading-snug opacity-90']">
+      {{ description }}
+    </p>
+    <dl :class="['m-0 grid grid-cols-[auto_minmax(0,1fr)] gap-x-3 gap-y-1']">
+      <template v-if="formula">
+        <dt :class="['opacity-70']">
+          formula
+        </dt>
+        <dd :class="['m-0']">
+          {{ formula }}
+        </dd>
+      </template>
+      <template v-if="grader">
+        <dt :class="['opacity-70']">
+          grader
+        </dt>
+        <dd :class="['m-0']" data-testid="score-detail-grader">
+          {{ graderText(grader) }}<span v-if="grader.id" :class="['mono opacity-70']"> ({{ grader.id }})</span>
+        </dd>
+      </template>
+      <template v-else-if="inputGraders.length > 0">
+        <dt :class="['opacity-70']">
+          graders
+        </dt>
+        <dd :class="['m-0 grid gap-0.5']">
+          <span v-for="input in inputGraders" :key="input.label">{{ input.label }}: {{ input.text }}</span>
+        </dd>
+      </template>
+    </dl>
+    <a v-if="props.callsHref && props.callCount > 0" :href="props.callsHref" :class="['link inline-flex items-center gap-1']" data-testid="score-detail-calls">
+      <span class="i-mingcute-robot-line" aria-hidden="true" />{{ props.callCount }} model {{ props.callCount === 1 ? 'call' : 'calls' }} of the grader
+    </a>
+    <div v-if="props.record?.rationale" :class="['grid gap-0.5']">
+      <div :class="['opacity-70']">
+        rationale
+      </div>
+      <!-- A judge's reasoning can run long; it scrolls inside the card. -->
+      <p :class="['m-0 max-h-48 overflow-y-auto whitespace-pre-wrap leading-snug']" data-testid="score-detail-rationale">
+        {{ props.record.rationale }}
+      </p>
+    </div>
+    <div v-if="props.record?.metadata" :class="['grid gap-0.5']">
+      <div :class="['opacity-70']">
+        metadata
+      </div>
+      <div :class="['max-h-48 overflow-auto']">
+        <ObjectTree :value="props.record.metadata" />
+      </div>
+    </div>
+    <p v-if="props.record == null" :class="['m-0 opacity-70']">
+      This case recorded no value of the score.
+    </p>
+  </div>
+</template>
diff --git a/packages/ui/client/src/components/ScoresPanel.vue b/packages/ui/client/src/components/ScoresPanel.vue
index fed1536..c4edf53 100644
--- a/packages/ui/client/src/components/ScoresPanel.vue
+++ b/packages/ui/client/src/components/ScoresPanel.vue
@@ -1,5 +1,5 @@
 <script setup lang="ts">
-import type { MetricDefinition, RunSnapshot } from '../../../src/shared/contract'
+import type { CaseScoring, RunSnapshot, ScoreSummary } from '../../../src/shared/contract'
 import type { CaseFilters, CaseRow } from '../caseMetrics'
 import type { BarDatum } from './BarChart.vue'
 import type { LinePoint } from './LineChart.vue'
@@ -11,6 +11,7 @@ import BarChart from './BarChart.vue'
 import CaseCartesian from './CaseCartesian.vue'
 import CaseMetricsPanel from './CaseMetricsPanel.vue'
 import CountsBar from './CountsBar.vue'
+import HeadlineScore from './HeadlineScore.vue'
 import LineChart from './LineChart.vue'
 import MetricInfo from './MetricInfo.vue'
 import PairsHeatmap from './PairsHeatmap.vue'
@@ -19,14 +20,16 @@ import SegmentedControl from './SegmentedControl.vue'
 import StateView from './StateView.vue'
 
 import { api } from '../api'
-import { buildCaseTable, metricLabel, metricScale, separationOf } from '../caseMetrics'
+import { buildCaseTable, columnDefinitions, metricLabel, metricScale, separationOf } from '../caseMetrics'
 import { formatScore, previewValue } from '../format'
 import { levelIcons } from '../hierarchy'
 import { href, navigate } from '../router'
+import { graderText, headlineOf, scoreColumn, scoreDefinition, scoreLabel, unscoredText } from '../scores'
 
 /**
- * Scores and metrics of a run: overall scores, scores per matrix row (task) and per auto-attempt round, numeric
- * metrics as small multiples, and the other recorded values.
+ * Scores and metrics of a run: the completion rate and every score over the whole run (label, mean, counted cases,
+ * unscored reasons, formula, grader; the primary score marked), scores per matrix row (task) and per auto-attempt
+ * round, numeric metrics as small multiples, and the other recorded values.
  */
 const props = defineProps<{
   /** Changes when the run advanced, at most once per refetch interval (`useRunLive().refreshKey`). */
@@ -45,8 +48,46 @@ watch(() => props.refreshKey, () => {
 })
 
 const matrixKeys = computed(() => [...new Set(props.snapshot.tasks.flatMap(task => Object.keys({ ...task.matrix?.run, ...task.matrix?.eval })))])
-const byTask = computed<BarDatum[]>(() => (scores.data.value?.byTask ?? []).map(task => ({ href: href({ name: 'task', runKey: runKey.value, taskId: task.taskId }), label: task.displayName, values: [task.exact, task.judge] })))
-const byRound = computed<BarDatum[]>(() => (scores.data.value?.byRound ?? []).map(round => ({ label: `${props.snapshot.tasks.find(task => task.id === round.taskId)?.displayName ?? round.taskId} · round ${round.roundIndex}`, values: [round.exact, round.judge] })))
+const scoreDefinitions = computed(() => scores.data.value?.definitions ?? props.snapshot.scoreDefinitions)
+/** Every score of the run in the order the summary lists them; the series of the bar charts and the table columns. */
+const scoreNames = computed(() => (scores.data.value?.overall.scores ?? []).map(summary => summary.name))
+/** Primary scores of the run: the one of the headline, or each task's under a `per-task` headline. */
+const primaries = computed(() => {
+  const headline = scores.data.value?.overall.headline?.headline
+  if (headline == null || headline.kind === 'passed-cases')
+    return new Set<string>()
+  return new Set(headline.kind === 'score' ? [headline.score.name] : headline.tasks.map(task => task.score.name))
+})
+/**
+ * Rows of the scores table: the completion rate first (how many cases produced a result, so a score is never read
+ * without it), then every score with its definition, formula, and grader.
+ */
+const scoreRows = computed(() => {
+  const overall = scores.data.value?.overall
+  if (overall == null)
+    return []
+  const completion: ScoreSummary | undefined = overall.headline?.completion
+  const rows = overall.scores.map((summary) => {
+    const definition = scoreDefinition(summary.name, scoreDefinitions.value)
+    return {
+      column: definition == null ? undefined : scoreColumn(definition, scoreDefinitions.value),
+      // Cases that counted toward the mean.
+      counted: summary.n,
+      grader: definition?.grader == null ? undefined : graderText(definition.grader),
+      label: scoreLabel(summary.name, scoreDefinitions.value),
+      primary: primaries.value.has(summary.name),
+      summary,
+    }
+  }).toSorted((a, b) => Number(b.primary) - Number(a.primary))
+  if (completion == null)
+    return rows
+  // Every settled case counts toward the completion rate as 1 or 0, so its count is the cases that produced a result.
+  return [{ column: undefined, counted: completion.sum, grader: undefined, label: 'Completion rate', primary: false, summary: completion }, ...rows]
+})
+const meanOf = (scoring: CaseScoring, name: string) => scoring.scores.find(summary => summary.name === name)?.mean ?? null
+const byTask = computed<BarDatum[]>(() => (scores.data.value?.byTask ?? []).map(task => ({ href: href({ name: 'task', runKey: runKey.value, taskId: task.taskId }), label: task.displayName, values: scoreNames.value.map(name => meanOf(task.scoring, name)) })))
+const byRound = computed<BarDatum[]>(() => (scores.data.value?.byRound ?? []).map(round => ({ label: `${props.snapshot.tasks.find(task => task.id === round.taskId)?.displayName ?? round.taskId} · round ${round.roundIndex}`, values: scoreNames.value.map(name => meanOf(round.scoring, name)) })))
+const taskLabel = (taskId: string) => props.snapshot.tasks.find(task => task.id === taskId)?.displayName ?? taskId
 /** Names of the "Other metrics" rows that are expanded; every row starts collapsed to one line. */
 const expanded = reactive(new Set<string>())
 /** One line for a collapsed row: the distinct values, each cut short. */
@@ -61,7 +102,7 @@ function toggleMetric(name: string): void {
   if (!expanded.delete(name))
     expanded.add(name)
 }
-const definitions = computed(() => new Map<string, MetricDefinition>((metrics.data.value?.definitions ?? []).map(definition => [definition.name, definition])))
+const definitions = computed(() => columnDefinitions(metrics.data.value?.definitions ?? [], metrics.data.value?.scoreDefinitions ?? []))
 const labelOf = (name: string) => metricLabel(name, definitions.value)
 
 /**
@@ -109,24 +150,64 @@ const numeric = computed(() => (metrics.data.value?.numeric ?? []).map(metric =>
         <h2 class="card-title">
           <span class="i-mingcute-target-line" aria-hidden="true" />Scores
         </h2>
-        <dl :class="['flex flex-wrap gap-x-8 gap-y-2 px-3 py-2 text-sm']">
-          <div v-for="kind in (['exact', 'judge', 'hybrid'] as const)" :key="kind">
-            <dt class="text-xs muted">
-              {{ kind }} (events)
-            </dt>
-            <dd class="text-xl font-semibold tabular-nums">
-              {{ formatScore(scores.data.value?.overall[kind]) }}
-            </dd>
-          </div>
-          <div v-if="scores.data.value?.summary">
-            <dt class="text-xs muted">
-              exact (run-summary.json)
-            </dt>
-            <dd class="text-xl font-semibold tabular-nums">
-              {{ formatScore(scores.data.value.summary.exact) }}
-            </dd>
-          </div>
-        </dl>
+        <div :class="['overflow-x-auto']">
+          <table :class="['w-full']" data-testid="score-rows">
+            <thead>
+              <tr :class="['border-b border-neutral-200/50 dark:border-neutral-800/50']">
+                <th class="th">
+                  Score
+                </th>
+                <th class="th text-right">
+                  Mean
+                </th>
+                <th class="th text-right">
+                  Counted
+                </th>
+                <th class="th">
+                  Not counted
+                </th>
+                <th class="th">
+                  Formula
+                </th>
+                <th class="th">
+                  Grader
+                </th>
+              </tr>
+            </thead>
+            <tbody>
+              <tr v-for="row in scoreRows" :key="row.summary.name" :class="['border-b border-neutral-100/70 dark:border-neutral-900']" data-testid="score-row">
+                <td :class="['td text-sm whitespace-nowrap']">
+                  <MetricInfo :name="row.summary.name" :definition="row.column" :values="[]" :summary="row.summary">
+                    <span :class="['cursor-help', row.column?.label == null && row.column != null && 'mono']">{{ row.label }}</span>
+                  </MetricInfo>
+                  <span v-if="row.primary" :class="['badge state-neutral ml-1.5']" data-testid="score-row-primary">primary</span>
+                </td>
+                <td :class="['td text-right tabular-nums font-semibold']">
+                  {{ formatScore(row.summary.mean) }}
+                </td>
+                <td :class="['td text-right mono text-xs whitespace-nowrap']">
+                  {{ row.counted }} / {{ row.summary.total }}
+                </td>
+                <td :class="['td text-xs muted']">
+                  {{ unscoredText(row.summary.unscored) || '—' }}
+                </td>
+                <td :class="['td text-xs']">
+                  {{ row.column?.formula ?? '—' }}
+                </td>
+                <td :class="['td text-xs mono']">
+                  {{ row.grader ?? '—' }}
+                </td>
+              </tr>
+            </tbody>
+          </table>
+        </div>
+        <p v-if="scoreRows.length <= 1" :class="['px-3 py-2 text-xs muted']">
+          No score recorded. Cases record one with context.recordScore(score, value); the headline is the share of passed cases.
+        </p>
+        <div v-if="scores.data.value?.summary" :class="['flex flex-wrap items-baseline gap-x-3 gap-y-1 px-3 py-2 text-xs']">
+          <span class="muted">Official summary (run-summary.json)</span>
+          <HeadlineScore :result="scores.data.value.summary" :definitions="scoreDefinitions" size="sm" :task-label="taskLabel" />
+        </div>
       </section>
 
       <!-- Every case of the run against the matrix axes and its metrics: where the score is lost across tasks. -->
@@ -151,10 +232,10 @@ const numeric = computed(() => (metrics.data.value?.numeric ?? []).map(metric =>
                   Cases
                 </th>
                 <th class="th text-right">
-                  Exact
+                  Headline
                 </th>
-                <th class="th text-right">
-                  Judge
+                <th v-for="name in scoreNames" :key="name" class="th text-right">
+                  {{ scoreLabel(name, scoreDefinitions) }}
                 </th>
                 <th class="th text-right">
                   Rounds
@@ -172,11 +253,11 @@ const numeric = computed(() => (metrics.data.value?.numeric ?? []).map(metric =>
                 <td :class="['td min-w-48']">
                   <CountsBar :counts="task.caseCounts" />
                 </td>
-                <td :class="['td text-right mono text-xs']">
-                  {{ formatScore(task.scores.exact) }}
+                <td :class="['td text-right']">
+                  <HeadlineScore :result="headlineOf(task.scoring)" :definitions="scoreDefinitions" size="sm" align="end" />
                 </td>
-                <td :class="['td text-right mono text-xs']">
-                  {{ formatScore(task.scores.judge) }}
+                <td v-for="name in scoreNames" :key="name" :class="['td text-right mono text-xs']">
+                  {{ formatScore(meanOf(task.scoring, name)) }}
                 </td>
                 <td :class="['td text-right mono text-xs']">
                   {{ task.roundCount }}
@@ -185,17 +266,17 @@ const numeric = computed(() => (metrics.data.value?.numeric ?? []).map(metric =>
             </tbody>
           </table>
         </div>
-        <div v-if="byTask.length > 0" class="p-3">
-          <BarChart :data="byTask" :series="['exact', 'judge']" />
+        <div v-if="byTask.length > 0 && scoreNames.length > 0" class="p-3">
+          <BarChart :data="byTask" :series="scoreNames.map(name => scoreLabel(name, scoreDefinitions))" />
         </div>
       </section>
 
-      <section v-if="byRound.length > 1" :class="['card']">
+      <section v-if="byRound.length > 1 && scoreNames.length > 0" :class="['card']">
         <h2 class="card-title">
           <span :class="levelIcons.round" aria-hidden="true" />Scores per auto-attempt round
         </h2>
         <div class="p-3">
-          <BarChart :data="byRound" :series="['exact', 'judge']" />
+          <BarChart :data="byRound" :series="scoreNames.map(name => scoreLabel(name, scoreDefinitions))" />
         </div>
       </section>
     </StateView>
diff --git a/packages/ui/client/src/components/Sparkline.vue b/packages/ui/client/src/components/Sparkline.vue
index 8e7ea68..9f457e3 100644
--- a/packages/ui/client/src/components/Sparkline.vue
+++ b/packages/ui/client/src/components/Sparkline.vue
@@ -18,7 +18,7 @@ const props = withDefaults(defineProps<{
   /** Coverage of each value, in the same order; every value counts as `full` without it. */
   coverage?: readonly SparklineCoverage[]
   height?: number
-  /** Accessible summary, for example "exact score of 10 runs". */
+  /** Accessible summary, for example "Pass of 10 runs" (the label of the primary score). */
   label: string
   /** Tooltip of each point (for example its time and score), in the same order as `values`. */
   titles?: readonly string[]
diff --git a/packages/ui/client/src/composables/useStatusFavicon.test.ts b/packages/ui/client/src/composables/useStatusFavicon.test.ts
index 9635ebf..9187092 100644
--- a/packages/ui/client/src/composables/useStatusFavicon.test.ts
+++ b/packages/ui/client/src/composables/useStatusFavicon.test.ts
@@ -46,7 +46,7 @@ describe('useStatusFavicon', () => {
    * expect(iconHrefs()).toEqual(['./vieval-logo.svg', './favicon-96x96.png'])
    */
   it('keeps the plain logo for a state without a clear outcome', async () => {
-    const { App } = shell('legacy')
+    const { App } = shell('unsupported')
     await render(App)
 
     await expect.poll(iconHrefs).toEqual(['./vieval-logo.svg', './favicon-96x96.png'])
diff --git a/packages/ui/client/src/composables/useStatusFavicon.ts b/packages/ui/client/src/composables/useStatusFavicon.ts
index 4e61439..2d3c20a 100644
--- a/packages/ui/client/src/composables/useStatusFavicon.ts
+++ b/packages/ui/client/src/composables/useStatusFavicon.ts
@@ -125,7 +125,7 @@ async function draw(status: FaviconStatus): Promise<string> {
  * Maps a run, task, or case state to the badge of the tab icon; states without a clear outcome get none.
  *
  * Before:
- * - "running", "stale", "queued" / "failed", "timeout", "interrupted", "unfinished" / "ended", "passed" / "legacy"
+ * - "running", "stale", "queued" / "failed", "timeout", "interrupted", "unfinished" / "ended", "passed" / "unsupported"
  *
  * After:
  * - "pending" / "failure" / "success" / undefined
diff --git a/packages/ui/client/src/errorGroups.test.ts b/packages/ui/client/src/errorGroups.test.ts
index 0c5163e..39952b8 100644
--- a/packages/ui/client/src/errorGroups.test.ts
+++ b/packages/ui/client/src/errorGroups.test.ts
@@ -15,7 +15,7 @@ function caseOf(fields: Partial<CaseNode> & Pick<CaseNode, 'caseId' | 'state'>):
     metricCount: 0,
     name: fields.caseId,
     rounds: [],
-    scores: {},
+    scores: { unscored: {}, values: {} },
     startedAt: '2026-10-19T06:36:26.000Z',
     taskId: 'task',
     ...fields,
diff --git a/packages/ui/client/src/pages/CasePage.vue b/packages/ui/client/src/pages/CasePage.vue
index 9293bde..8310c1d 100644
--- a/packages/ui/client/src/pages/CasePage.vue
+++ b/packages/ui/client/src/pages/CasePage.vue
@@ -1,5 +1,5 @@
 <script setup lang="ts">
-import type { MetricDefinition, RunListItem } from '../../../src/shared/contract'
+import type { RunListItem } from '../../../src/shared/contract'
 import type { ViewTab } from '../components/ViewTabs.vue'
 import type { Route } from '../router'
 
@@ -14,16 +14,19 @@ import LogView from '../components/LogView.vue'
 import MetricInfo from '../components/MetricInfo.vue'
 import ObjectTree from '../components/ObjectTree.vue'
 import PageHeader from '../components/PageHeader.vue'
+import ScoreBadge from '../components/ScoreBadge.vue'
 import StateBadge from '../components/StateBadge.vue'
 import StateView from '../components/StateView.vue'
 import ViewTabs from '../components/ViewTabs.vue'
 
+import { groupInferenceCalls } from '../../../src/shared/inference'
 import { api } from '../api'
-import { metricLabel } from '../caseMetrics'
+import { columnDefinitions, metricLabel } from '../caseMetrics'
 import { useRunLive } from '../composables/useRunLive'
 import { useStatusFavicon } from '../composables/useStatusFavicon'
-import { formatDateTime, formatDuration, formatScore, formatTime, previewValue } from '../format'
+import { formatDateTime, formatDuration, formatTime, previewValue } from '../format'
 import { href, navigate } from '../router'
+import { caseScoreText, isGraderCall, primaryScoreName, scoreDefinition } from '../scores'
 import { runSwitcherOptions } from '../switcher'
 
 /**
@@ -43,12 +46,41 @@ const taskMetrics = useQuery({
   key: () => ['runs', props.route.runKey, 'task', props.route.taskId, 'metrics'],
   query: () => api.metrics(props.route.runKey, { taskId: props.route.taskId }),
 })
-const definitions = computed(() => new Map<string, MetricDefinition>((detail.data.value?.metricDefinitions ?? []).map(definition => [definition.name, definition])))
+const definitions = computed(() => columnDefinitions(detail.data.value?.metricDefinitions ?? [], detail.data.value?.scoreDefinitions ?? []))
 const labelOf = (name: string) => metricLabel(name, definitions.value)
 function taskValuesOf(name: string): unknown[] {
   const data = taskMetrics.data.value
-  return [...(data?.numeric.find(series => series.name === name)?.points ?? []), ...(data?.nonNumeric.find(series => series.name === name)?.values ?? [])].map(entry => entry.value)
+  return [
+    ...(data?.numeric.find(series => series.name === name)?.points ?? []),
+    ...(data?.nonNumeric.find(series => series.name === name)?.values ?? []),
+    ...(data?.scores.find(series => series.name === name)?.points ?? []),
+  ].map(entry => entry.value)
 }
+
+/**
+ * Every score of the case, the primary first: the scores its task defined and the ones it recorded, each with the
+ * last record of the last round (rationale, grader, metadata) and the grader's model calls in this case.
+ */
+const caseScores = computed(() => {
+  const data = detail.data.value
+  if (data == null)
+    return []
+  const names = [...new Set([...data.scoreDefinitions.map(definition => definition.name), ...Object.keys(data.scores.values), ...Object.keys(data.scores.unscored)])]
+  const primary = primaryScoreName(data.scoreDefinitions, names)
+  const lastRound = data.rounds.at(-1)?.roundIndex ?? 0
+  const calls = groupInferenceCalls(data.inference)
+  return names.toSorted((a, b) => Number(b === primary) - Number(a === primary)).map((name) => {
+    const record = data.scoreRecords.findLast(entry => entry.record.name === name && (entry.roundIndex ?? 0) === lastRound)?.record
+    const graderId = record?.grader?.id ?? scoreDefinition(name, data.scoreDefinitions)?.grader?.id
+    return {
+      callCount: graderId == null ? 0 : calls.filter(call => isGraderCall(call, graderId)).length,
+      callsHref: graderId == null ? undefined : href({ ...props.route, grader: graderId, tab: 'inference' }),
+      name,
+      record,
+      text: caseScoreText(data.scores, name),
+    }
+  })
+})
 const timeline = useQuery({
   key: () => ['runs', props.route.runKey, 'case', props.route.taskId, props.route.caseId, 'timeline'],
   query: () => api.timeline(props.route.runKey, ref.value),
@@ -68,7 +100,8 @@ const tabs = computed<ViewTab<Tab>[]>(() => [
   // Calls, not events: every call writes a request, a response, and two events per tool call.
   { count: detail.data.value == null ? undefined : detail.data.value.inference.filter(entry => entry.event === 'inference.requested').length, icon: 'i-mingcute-robot-line', label: 'Inference', value: 'inference' },
 ])
-const tabRoute = (value: Tab): Route => ({ ...props.route, tab: value === 'overview' ? undefined : value })
+// Switching tabs drops the grader filter of the inference tab.
+const tabRoute = (value: Tab): Route => ({ ...props.route, grader: undefined, tab: value === 'overview' ? undefined : value })
 // An unknown `tab` in the address shows the overview.
 const tab = computed<Tab>({
   get: () => tabs.value.find(item => item.value === props.route.tab)?.value ?? 'overview',
@@ -106,8 +139,7 @@ const running = computed(() => detail.data.value?.state === 'running')
     <template #actions>
       <template v-if="detail.data.value">
         <StateBadge :state="detail.data.value.state" />
-        <span class="badge state-neutral">exact {{ formatScore(detail.data.value.scores.exact) }}</span>
-        <span v-if="detail.data.value.scores.judge != null" class="badge state-neutral">judge {{ formatScore(detail.data.value.scores.judge) }}</span>
+        <ScoreBadge v-for="score in caseScores" :key="score.name" :call-count="score.callCount" :calls-href="score.callsHref" :definitions="detail.data.value.scoreDefinitions" :name="score.name" :record="score.record" :text="score.text" />
         <span class="badge state-neutral">{{ formatDuration(detail.data.value.durationMs) }}</span>
         <span v-if="running" class="badge state-running"><span class="vv-pulse i-mingcute-airdrop-line" aria-hidden="true" />live</span>
       </template>
@@ -156,12 +188,29 @@ const running = computed(() => detail.data.value?.state === 'running')
           </div>
           <section class="card">
             <h2 class="card-title">
-              <span class="i-mingcute-dashboard-3-line" aria-hidden="true" />Metrics <span class="text-xs muted font-normal">{{ detail.data.value.metrics.length }}</span>
+              <span class="i-mingcute-dashboard-3-line" aria-hidden="true" />Scores and metrics <span class="text-xs muted font-normal">{{ caseScores.length + detail.data.value.metrics.length }}</span>
             </h2>
-            <StateView :empty="detail.data.value.metrics.length === 0" empty-text="No metrics recorded for this case">
+            <StateView :empty="detail.data.value.metrics.length === 0 && caseScores.length === 0" empty-text="No scores or metrics recorded for this case">
               <div :class="['overflow-auto max-h-96']">
                 <table class="w-full">
                   <tbody>
+                    <!-- Scores first, as percentages; the badges above open what graded them. -->
+                    <tr v-for="score in caseScores" :key="`score:${score.name}`" :class="['border-b border-neutral-100/70 dark:border-neutral-900', 'max-md:(flex flex-wrap items-baseline py-1)']" data-testid="case-score-row">
+                      <td :class="['td mono text-xs muted whitespace-nowrap max-md:hidden']">
+                        score
+                      </td>
+                      <td :class="['td text-xs whitespace-nowrap', definitions.get(score.name)?.label == null && 'mono', 'max-md:(flex-1 min-w-0 truncate py-0.5 font-medium)']">
+                        <MetricInfo :name="score.name" :definition="definitions.get(score.name)" :values="taskValuesOf(score.name)" :current="detail.data.value.scores.values[score.name]">
+                          <span :class="['cursor-help']">{{ labelOf(score.name) }}</span>
+                        </MetricInfo>
+                      </td>
+                      <td :class="['td mono text-xs break-all', 'max-md:(order-last basis-full py-0.5)']">
+                        {{ score.text }}
+                      </td>
+                      <td :class="['td mono muted text-xs whitespace-nowrap', 'max-md:py-0.5']">
+                        round {{ detail.data.value.rounds.at(-1)?.roundIndex ?? 0 }}
+                      </td>
+                    </tr>
                     <!-- Phones: name and round on one line, the value below it at full width. -->
                     <tr v-for="metric in detail.data.value.metrics" :key="metric.seq" :class="['border-b border-neutral-100/70 dark:border-neutral-900', 'max-md:(flex flex-wrap items-baseline py-1)']">
                       <td :class="['td mono text-xs muted whitespace-nowrap max-md:hidden']">
@@ -201,7 +250,7 @@ const running = computed(() => detail.data.value?.state === 'running')
         </template>
 
         <StateView v-else-if="tab === 'inference'" :empty="detail.data.value.inference.length === 0" empty-text="No inference events" empty-hint="Model calls made through vieval's inference executors appear here.">
-          <InferencePanel :events="detail.data.value.inference" />
+          <InferencePanel :events="detail.data.value.inference" :grader="props.route.grader" @clear-grader="navigate({ ...props.route, grader: undefined })" />
         </StateView>
       </ViewTabs>
     </template>
diff --git a/packages/ui/client/src/pages/ComparePage.vue b/packages/ui/client/src/pages/ComparePage.vue
index 1c0cf14..12e42b0 100644
--- a/packages/ui/client/src/pages/ComparePage.vue
+++ b/packages/ui/client/src/pages/ComparePage.vue
@@ -16,10 +16,10 @@ import StateView from '../components/StateView.vue'
 
 import { api } from '../api'
 import { useCatalog } from '../composables/useCatalog'
-import { formatScore } from '../format'
 import { levelIcons } from '../hierarchy'
 import { attemptLabel, experimentLabel, runLabel } from '../labels'
 import { href } from '../router'
+import { caseScoreText, gradersDiffer, graderText, headlineOf, headlineValue, headlineView, scoreLabel } from '../scores'
 
 /**
  * Comparisons: between experiments of a project, between attempts (runs) of an experiment, between chosen runs,
@@ -54,29 +54,33 @@ const matrixRuns = computed(() => catalog.runs.value.filter(run => run.counts.to
 /**
  * Matrix rows of one eval entry of a run as a {@link ComparisonResponse}: one column per task (matrix row such as
  * `model=…, repeat=…`), one row per case name. Only tasks of the same entry share cases, so each entry is compared
- * on its own.
+ * on its own. The rows compare the tasks' primary score when they share one (as the comparison endpoints do), else
+ * the case state alone.
  */
 function compareMatrixRows(snapshot: RunSnapshot, entryId: string | undefined): ComparisonResponse {
   const tasks = snapshot.tasks.filter(task => (task.entryId ?? task.id) === entryId)
   const columns = tasks.map(task => ({
     id: task.id,
     label: task.matrix == null ? task.displayName : Object.entries({ ...task.matrix.run, ...task.matrix.eval }).map(([key, value]) => `${key}=${value}`).join(', ') || task.displayName,
-    runs: [{ attemptId: snapshot.identity.attemptId, counts: task.caseCounts, exactAverage: task.scores.exact, experimentId: snapshot.identity.experimentId, hybridAverage: task.scores.hybrid, judgeAverage: task.scores.judge, runKey: snapshot.runKey, startedAt: task.startedAt, state: snapshot.state }],
+    runs: [{ attemptId: snapshot.identity.attemptId, counts: task.caseCounts, experimentId: snapshot.identity.experimentId, headline: headlineOf(task.scoring), runKey: snapshot.runKey, startedAt: task.startedAt, state: snapshot.state }],
+    scoreGraders: task.scoreGraders,
   }))
+  const primaries = new Set(tasks.map(task => task.scoring.primary))
+  const primary = primaries.size === 1 ? [...primaries][0] : undefined
   const rows = new Map<string, ComparisonResponse['rows'][number]>()
   for (const task of tasks) {
     for (const node of task.cases) {
       const row = rows.get(node.name) ?? { caseName: node.name, cells: {}, changed: false }
-      row.cells[task.id] = { exact: node.scores.exact, judge: node.scores.judge, roundCount: node.rounds.length, state: node.state }
+      row.cells[task.id] = { roundCount: node.rounds.length, scores: node.scores, state: node.state }
       rows.set(node.name, row)
     }
   }
   for (const row of rows.values()) {
     const cells = Object.values(row.cells)
-    row.changed = cells.length !== columns.length || new Set(cells.map(cell => `${cell.state}:${cell.exact}`)).size > 1
+    row.changed = cells.length !== columns.length || new Set(cells.map(cell => `${cell.state}:${primary == null ? '' : cell.scores.values[primary]}`)).size > 1
   }
   const list = [...rows.values()]
-  return { columns, rows: list, summary: { changed: list.filter(row => row.changed).length, rows: list.length }, view: 'runs' }
+  return { columns, primary, rows: list, scoreDefinitions: snapshot.scoreDefinitions, summary: { changed: list.filter(row => row.changed).length, rows: list.length }, view: 'runs' }
 }
 
 // Same entry as the run page's snapshot (`useRunLive`): a run opened there compares at once.
@@ -116,11 +120,35 @@ const onlyChanged = shallowRef(false)
 const search = shallowRef('')
 const rows = computed(() => (result.value?.rows ?? []).filter(row => (!onlyChanged.value || row.changed) && row.caseName.toLowerCase().includes(search.value.toLowerCase())))
 
+const scoreDefinitions = computed(() => result.value?.scoreDefinitions ?? [])
+/** Label of the score the columns are compared by: the shared primary score, else the headline of passed cases. */
+const primaryLabel = computed(() => (result.value?.primary == null ? 'Passed cases' : scoreLabel(result.value.primary, scoreDefinitions.value)))
+/** Each column's headline next to its completion rate: a score over fewer completed cases is not comparable. */
 const chart = computed<BarDatum[]>(() => (result.value?.columns ?? []).map((column) => {
   const run = column.runs.at(-1)
-  const passRate = run == null || run.counts.total === 0 ? null : run.counts.passed / run.counts.total
-  return { href: run && view.value !== 'matrix' ? href({ name: 'run', runKey: run.runKey }) : undefined, label: columnLabel(column), values: [run?.exactAverage ?? null, passRate] }
+  return { href: run && view.value !== 'matrix' ? href({ name: 'run', runKey: run.runKey }) : undefined, label: columnLabel(column), values: [headlineValue(run?.headline), run?.headline?.completion.mean ?? null] }
 }))
+const chartSeries = computed(() => {
+  const headline = result.value?.columns.map(column => column.runs.at(-1)?.headline).find(item => item != null)
+  return [headline == null ? primaryLabel.value : headlineView(headline, scoreDefinitions.value).label, 'completed']
+})
+
+/**
+ * Graders of the primary score per column, when they differ in kind, model, or version: the values then do not
+ * measure the same thing, so the comparison warns. Each column lists the graders its score records carried (a
+ * record's own grader, for example a judge model set by a matrix axis, else the definition's), so the matrix view
+ * warns about matrix rows judged by different models too.
+ */
+const graderWarning = computed(() => {
+  const primary = result.value?.primary
+  if (primary == null)
+    return undefined
+  const columns = (result.value?.columns ?? []).map(column => ({ graders: column.scoreGraders[primary] ?? [], label: columnLabel(column) }))
+  const all = columns.flatMap(column => column.graders)
+  if (!all.some(grader => gradersDiffer(grader, all[0])))
+    return undefined
+  return columns.filter(column => column.graders.length > 0).map(column => `${column.label}: ${column.graders.map(graderText).join(' / ')}`).join(' · ')
+})
 
 const title = computed(() => ({ attempts: 'Attempts of an experiment', experiments: 'Experiments of a project', matrix: 'Matrix rows of a run', runs: 'Chosen runs' } as Record<string, string>)[view.value ?? ''] ?? 'Comparisons')
 const position = computed(() => ({ experiment: query.value.experiment, project: query.value.project, runKey: query.value.run, workspace: query.value.workspace }))
@@ -179,7 +207,8 @@ const position = computed(() => ({ experiment: query.value.experiment, project:
         v-for="column in result?.columns"
         :key="column.id"
         :href="column.runs.at(-1) && view !== 'matrix' ? href({ name: 'run', runKey: column.runs.at(-1)!.runKey }) : undefined"
-        :score="column.runs.at(-1)?.exactAverage"
+        :headline="column.runs.at(-1)?.headline"
+        :definitions="scoreDefinitions"
         :state="column.runs.at(-1)?.state"
         :counts="column.runs.at(-1)?.counts"
         :title="columnLabel(column)"
@@ -191,12 +220,17 @@ const position = computed(() => ({ experiment: query.value.experiment, project:
       </ScoreCard>
     </section>
 
+    <div v-if="graderWarning" role="status" :class="['flex items-start gap-2 mb-4 rounded px-3 py-2 text-sm state-interrupted']" data-testid="compare-grader-warning">
+      <span class="i-mingcute-warning-line mt-0.5 shrink-0" aria-hidden="true" />
+      <span>The columns graded {{ primaryLabel }} differently, so its values do not measure the same thing. <span :class="['mono text-xs']">{{ graderWarning }}</span></span>
+    </div>
+
     <section :class="['card mb-4']">
       <h2 class="card-title">
-        <span class="i-mingcute-chart-horizontal-line" aria-hidden="true" />Exact score and pass rate per column
+        <span class="i-mingcute-chart-horizontal-line" aria-hidden="true" />Headline and completion rate per column
       </h2>
       <div class="p-3">
-        <BarChart :data="chart" :series="['exact score', 'pass rate']" />
+        <BarChart :data="chart" :series="chartSeries" />
       </div>
     </section>
 
@@ -212,7 +246,7 @@ const position = computed(() => ({ experiment: query.value.experiment, project:
             <thead :class="['sticky top-0 bg-white dark:bg-neutral-900 z-1']">
               <tr :class="['border-b border-neutral-200/50 dark:border-neutral-800/50']">
                 <th class="th">
-                  Case
+                  Case <span :class="['font-normal muted']">· {{ primaryLabel }}</span><span v-if="graderWarning" class="i-mingcute-warning-line text-amber-600 ml-1 align-[-0.15em]" :title="`Graders differ: ${graderWarning}`" data-testid="compare-grader-warning-icon" />
                 </th>
                 <th v-for="column in result?.columns" :key="column.id" :class="['th text-xs max-w-40 truncate']" :title="column.label">
                   {{ columnLabel(column) }}
@@ -227,7 +261,7 @@ const position = computed(() => ({ experiment: query.value.experiment, project:
                 <td v-for="column in result?.columns" :key="column.id" class="td">
                   <span v-if="row.cells[column.id]" :class="['inline-flex items-center gap-1']">
                     <StateBadge :state="row.cells[column.id]!.state" icon-only />
-                    <span class="text-xs mono">{{ formatScore(row.cells[column.id]!.exact) }}</span>
+                    <span v-if="result?.primary" class="text-xs mono">{{ caseScoreText(row.cells[column.id]!.scores, result.primary) }}</span>
                     <span v-if="row.cells[column.id]!.roundCount > 1" class="text-xs muted">×{{ row.cells[column.id]!.roundCount }}</span>
                   </span>
                   <span v-else class="text-xs muted">not run</span>
diff --git a/packages/ui/client/src/pages/ExperimentPage.vue b/packages/ui/client/src/pages/ExperimentPage.vue
index d92ed6d..21f3c09 100644
--- a/packages/ui/client/src/pages/ExperimentPage.vue
+++ b/packages/ui/client/src/pages/ExperimentPage.vue
@@ -6,6 +6,7 @@ import { computed, nextTick, shallowRef, watch } from 'vue'
 
 import CountsBar from '../components/CountsBar.vue'
 import ExperimentName from '../components/ExperimentName.vue'
+import HeadlineScore from '../components/HeadlineScore.vue'
 import IdLabel from '../components/IdLabel.vue'
 import IdMenu from '../components/IdMenu.vue'
 import MatrixView from '../components/MatrixView.vue'
@@ -17,10 +18,11 @@ import TimeAgo from '../components/TimeAgo.vue'
 import ViewTabs from '../components/ViewTabs.vue'
 
 import { useCatalog } from '../composables/useCatalog'
-import { formatDuration, formatScore } from '../format'
+import { formatDuration } from '../format'
 import { levelIcons } from '../hierarchy'
 import { attemptDate, attemptLabel, differingAxes, experimentLabel, formatAxisValues, mergeMatrices, shortRunHash } from '../labels'
 import { href, navigate } from '../router'
+import { headlineValue, headlineView } from '../scores'
 
 /**
  * One experiment: its attempts newest first as list rows (one per run: state, duration, cases, score) and the matrix
@@ -46,7 +48,12 @@ const scope = computed(() => ({ experiment: props.experiment, project: props.pro
 const runs = computed(() => catalog.runsIn(scope.value))
 const matrix = computed(() => mergeMatrices(runs.value.map(run => run.matrix)))
 const title = computed(() => experimentLabel(props.experiment, matrix.value).text)
-const trend = computed(() => runs.value.map(run => run.exactAverage).toReversed())
+const trend = computed(() => runs.value.map(run => headlineValue(run.headline)).toReversed())
+/** What the trend shows: the headline label of the newest run. */
+const trendLabel = computed(() => {
+  const newest = runs.value[0]
+  return newest?.headline == null ? 'score' : headlineView(newest.headline, newest.headlineDefinitions).label
+})
 
 /** Attempts newest first, each with its runs (one per attempt unless a run was repeated under the same label). */
 const attempts = computed(() => {
@@ -71,13 +78,13 @@ const expectedCounts = (run: (typeof runs.value)[number]) => ({ ...run.counts, t
 const isPartial = (run: (typeof runs.value)[number]) => run.state !== 'running' && run.caseTotal != null && run.counts.total < run.caseTotal
 
 /**
- * Grid of a run row. Wide screens: one line with select, attempt, run, duration, cases, exact score, menu (the header
- * uses the same columns). Narrow screens: select, attempt, exact score, and menu on the first line, the run and its
+ * Grid of a run row. Wide screens: one line with select, attempt, run, duration, cases, headline score, menu (the
+ * header uses the same columns). Narrow screens: select, attempt, score, and menu on the first line, the run and its
  * duration below the attempt, then the cases.
  */
 const rowGrid = [
   'grid items-center gap-x-3 gap-y-1.5 grid-cols-[1rem_minmax(0,1fr)_auto_1.75rem]',
-  'md:(gap-x-4 grid-cols-[1rem_minmax(14rem,1.6fr)_minmax(9rem,1fr)_5rem_minmax(12rem,2fr)_4rem_1.75rem])',
+  'md:(gap-x-4 grid-cols-[1rem_minmax(14rem,1.6fr)_minmax(9rem,1fr)_5rem_minmax(12rem,2fr)_8rem_1.75rem])',
 ]
 const selected = shallowRef<string[]>([])
 function duration(run: { endedAt?: string, startedAt?: string }): number | undefined {
@@ -106,7 +113,7 @@ watch([() => props.attempt, () => attempts.value.length], async ([attempt]) => {
     </template>
     <template #actions>
       <!-- Phones need the width for the name; every attempt row below shows its score. -->
-      <Sparkline :class="['max-md:hidden']" :values="trend" :label="`exact score of ${runs.length} runs`" :width="140" :height="32" />
+      <Sparkline :class="['max-md:hidden']" :values="trend" :label="`${trendLabel} of ${runs.length} runs`" :width="140" :height="32" />
     </template>
   </PageHeader>
 
@@ -122,7 +129,7 @@ watch([() => props.attempt, () => attempts.value.length], async ([attempt]) => {
         <section :class="['card']">
           <div :class="[...rowGrid, 'max-md:hidden px-3 py-1.5 border-b border-neutral-200/50 dark:border-neutral-800/50', 'text-xs font-medium muted']" aria-hidden="true">
             <span /><span>Attempt</span><span>Run</span><span>Duration</span><span>Cases</span><span class="text-right">
-              Exact
+              Score
             </span><span />
           </div>
           <ol>
@@ -164,7 +171,7 @@ watch([() => props.attempt, () => attempts.value.length], async ([attempt]) => {
                   <CountsBar :class="['flex-1']" :counts="expectedCounts(run)" />
                   <span v-if="isPartial(run)" :class="['badge state-interrupted shrink-0']" :title="`${run.counts.total} of ${run.caseTotal} expected cases ran`" data-testid="attempt-partial">{{ run.counts.total }} / {{ run.caseTotal }}</span>
                 </span>
-                <span :class="['text-right tabular-nums font-semibold max-md:(row-start-1 col-start-3)']">{{ formatScore(run.exactAverage) }}</span>
+                <HeadlineScore :class="['relative z-1 justify-self-end max-md:(row-start-1 col-start-3)']" :result="run.headline" :definitions="run.headlineDefinitions" size="sm" align="end" />
                 <!-- IdMenu renders no element of its own; the wrapper carries the grid placement. -->
                 <span :class="['flex max-md:(row-start-1 col-start-4)']"><IdMenu :id="run.runKey" kind="run" /></span>
               </div>
diff --git a/packages/ui/client/src/pages/HomePage.vue b/packages/ui/client/src/pages/HomePage.vue
index 61c40c4..8761f78 100644
--- a/packages/ui/client/src/pages/HomePage.vue
+++ b/packages/ui/client/src/pages/HomePage.vue
@@ -30,9 +30,9 @@ const runCount = computed(() => workspaces.value.reduce((sum, workspace) => sum
 /**
  * Card data of a project: its matrix axes, with the score of its newest run.
  *
- * The score is the newest run's own, like the state and the case bar beside it, and empty when that run scored no case.
- * It previously was the best exact score of any run, so a 3-case smoke run that got everything right showed 100% above
- * a newer full run's 94%.
+ * The score is the newest run's own headline, like the state and the case bar beside it, and empty when that run
+ * scored no case. It previously was the best score of any run, so a 3-case smoke run that got everything right showed
+ * 100% above a newer full run's 94%.
  */
 function toCard(project: ProjectEntry) {
   const matrix = mergeMatrices(project.runs.map(run => run.matrix))
@@ -61,8 +61,8 @@ const sections = computed(() => workspaces.value.map(workspace => ({ id: workspa
           v-for="project in section.projects"
           :key="project.id"
           :href="href({ name: 'scope', project: project.id, workspace: section.id })"
-          :score="project.latest.exactAverage"
-          score-label="exact score of the newest run"
+          :headline="project.latest.headline"
+          :definitions="project.latest.headlineDefinitions"
           :state="project.latest.state"
           :counts="project.latest.counts"
           :run="project.latest"
diff --git a/packages/ui/client/src/pages/ProjectPage.vue b/packages/ui/client/src/pages/ProjectPage.vue
index 7ed0df2..c2122af 100644
--- a/packages/ui/client/src/pages/ProjectPage.vue
+++ b/packages/ui/client/src/pages/ProjectPage.vue
@@ -7,6 +7,7 @@ import { computed } from 'vue'
 
 import CountsBar from '../components/CountsBar.vue'
 import ExperimentName from '../components/ExperimentName.vue'
+import HeadlineScore from '../components/HeadlineScore.vue'
 import MatrixView from '../components/MatrixView.vue'
 import PageHeader from '../components/PageHeader.vue'
 import Sparkline from '../components/Sparkline.vue'
@@ -16,10 +17,10 @@ import TimeAgo from '../components/TimeAgo.vue'
 import ViewTabs from '../components/ViewTabs.vue'
 
 import { useCatalog } from '../composables/useCatalog'
-import { formatScore } from '../format'
 import { levelIcons } from '../hierarchy'
 import { mergeMatrices } from '../labels'
 import { href, navigate } from '../router'
+import { headlineValue, headlineView } from '../scores'
 
 /**
  * One project: its experiments as list rows (what was measured, how its latest run went, the trend over attempts) and
@@ -41,23 +42,28 @@ const tab = computed<Tab>({
 const catalog = useCatalog()
 const project = computed(() => catalog.workspaces.value.find(workspace => workspace.id === props.workspace)?.projects.find(entry => entry.id === props.project))
 
-/** Row data of an experiment: its matrix, the trend of exact scores (oldest run first), and its newest run. */
+/**
+ * Row data of an experiment: its matrix, the trend of its headline scores (oldest run first) named by the newest run's
+ * headline, and its newest run.
+ */
 function toRow(experiment: ExperimentEntry) {
+  const latest = experiment.latest
   return {
     ...experiment,
     matrix: mergeMatrices(experiment.runs.map(run => run.matrix)),
-    trend: experiment.runs.map(run => run.exactAverage).toReversed(),
+    trend: experiment.runs.map(run => headlineValue(run.headline)).toReversed(),
+    trendLabel: latest.headline == null ? 'score' : headlineView(latest.headline, latest.headlineDefinitions).label,
   }
 }
 const rows = computed(() => (project.value?.experiments ?? []).map(toRow))
 /**
- * Grid of a row. Wide screens: one line with name, latest run, trend, cases, exact score, attempts (the header uses
- * the same columns). Narrow screens: the name on top, the latest run with the exact score below it, then the cases;
+ * Grid of a row. Wide screens: one line with name, latest run, trend, cases, headline score, attempts (the header
+ * uses the same columns). Narrow screens: the name on top, the latest run with the score below it, then the cases;
  * the trend and the attempt count are left out (both are on the experiment page).
  */
 const rowGrid = [
   'grid items-center gap-x-4 gap-y-1.5 grid-cols-[minmax(0,1fr)_auto]',
-  'md:grid-cols-[minmax(14rem,2fr)_minmax(12rem,1.2fr)_4.5rem_minmax(16rem,1.8fr)_4rem_4rem]',
+  'md:grid-cols-[minmax(14rem,2fr)_minmax(12rem,1.2fr)_4.5rem_minmax(16rem,1.8fr)_8rem_4rem]',
 ]
 </script>
 
@@ -73,7 +79,7 @@ const rowGrid = [
       <section v-if="tab === 'experiments'" :class="['card']">
         <div :class="[...rowGrid, 'max-md:hidden px-3 py-1.5 border-b border-neutral-200/50 dark:border-neutral-800/50', 'text-xs font-medium muted']" aria-hidden="true">
           <span>Experiment</span><span>Latest run</span><span>Trend</span><span>Cases (latest)</span><span class="text-right">
-            Exact
+            Score
           </span><span class="text-right">
             Attempts
           </span>
@@ -94,9 +100,9 @@ const rowGrid = [
               <TimeAgo :time="row.latest.startedAt" :class="['text-xs muted truncate']" />
             </span>
             <!-- The cell stays even when the trend draws nothing (fewer than two scored runs), so later columns keep their place. -->
-            <span :class="['flex max-md:hidden']"><Sparkline :values="row.trend" :label="`exact score of ${row.runs.length} runs`" :width="72" /></span>
+            <span :class="['flex max-md:hidden']"><Sparkline :values="row.trend" :label="`${row.trendLabel} of ${row.runs.length} runs`" :width="72" /></span>
             <CountsBar :class="['max-md:(row-start-3 col-span-2)']" :counts="row.latest.counts" />
-            <span :class="['text-right tabular-nums font-semibold max-md:(row-start-2 col-start-2)']">{{ formatScore(row.latest.exactAverage) }}</span>
+            <HeadlineScore :class="['relative z-1 justify-self-end max-md:(row-start-2 col-start-2)']" :result="row.latest.headline" :definitions="row.latest.headlineDefinitions" size="sm" align="end" />
             <span :class="['text-right tabular-nums muted max-md:hidden']">{{ row.attempts.length }}</span>
           </li>
         </ul>
diff --git a/packages/ui/client/src/pages/RunPage.vue b/packages/ui/client/src/pages/RunPage.vue
index 2d2d9e2..d17e077 100644
--- a/packages/ui/client/src/pages/RunPage.vue
+++ b/packages/ui/client/src/pages/RunPage.vue
@@ -1,14 +1,17 @@
 <script setup lang="ts">
+import type { ComponentPublicInstance } from 'vue'
+
 import type { LogLine, RunListItem } from '../../../src/shared/contract'
 import type { ViewTab } from '../components/ViewTabs.vue'
 import type { Route } from '../router'
 
-import { computed, toRef } from 'vue'
+import { computed, toRef, useTemplateRef } from 'vue'
 
 import CountsBar from '../components/CountsBar.vue'
 import ErrorGroupsPanel from '../components/ErrorGroupsPanel.vue'
 import ExecutionTree from '../components/ExecutionTree.vue'
 import ExperimentName from '../components/ExperimentName.vue'
+import HeadlineScore from '../components/HeadlineScore.vue'
 import IdLabel from '../components/IdLabel.vue'
 import LogView from '../components/LogView.vue'
 import OtelPanel from '../components/OtelPanel.vue'
@@ -19,12 +22,11 @@ import ScoresPanel from '../components/ScoresPanel.vue'
 import StateView from '../components/StateView.vue'
 import ViewTabs from '../components/ViewTabs.vue'
 
-import { scoreRounds } from '../../../src/shared/reduce'
 import { api } from '../api'
 import { useCatalog } from '../composables/useCatalog'
 import { useRunLive } from '../composables/useRunLive'
 import { useStatusFavicon } from '../composables/useStatusFavicon'
-import { formatDateTime, formatScore } from '../format'
+import { formatDateTime } from '../format'
 import { levelIcons } from '../hierarchy'
 import { attemptLabel, runLabel } from '../labels'
 import { href, navigate } from '../router'
@@ -65,22 +67,28 @@ const loadLogs = computed(() => {
   return (offset: number, limit: number) => api.logs(key, { limit, offset })
 })
 /**
- * Exact score and case counts of the whole run from the live snapshot, so they move with the run (the catalog entry
- * of the run is read once per page). The score follows the task scores: every settled round of every case with a
- * result counts; rounds without one lower the coverage instead.
+ * Case counts of the whole run from the live snapshot, so they move with the run (the catalog entry of the run is read
+ * once per page), beside the snapshot's headline (`resolveHeadline` over the tasks' events).
  */
-const score = computed(() => {
+const counts = computed(() => {
   const snapshot = live.snapshot.value
   if (snapshot == null)
     return undefined
-  const counts = { errored: 0, failed: 0, passed: 0, requeued: 0, running: 0, skipped: 0, timeout: 0, total: 0 }
+  const sum = { errored: 0, failed: 0, passed: 0, requeued: 0, running: 0, skipped: 0, timeout: 0, total: 0 }
   for (const task of snapshot.tasks) {
-    for (const key of Object.keys(counts) as (keyof typeof counts)[])
-      counts[key] += task.caseCounts[key]
+    for (const key of Object.keys(sum) as (keyof typeof sum)[])
+      sum[key] += task.caseCounts[key]
   }
-  const { coverage, exact } = scoreRounds(snapshot.tasks.flatMap(task => task.cases.flatMap(node => node.rounds)))
-  return { counts, coverage: coverage.total === 0 ? undefined : coverage.measured / coverage.total, exact }
+  return sum
 })
+const taskLabel = (taskId: string) => live.snapshot.value?.tasks.find(task => task.id === taskId)?.displayName ?? taskId
+// A completion rate below 100% leads to the cases without a result, grouped by their error.
+const errorGroups = useTemplateRef<ComponentPublicInstance>('errorGroups')
+function showErrors(): void {
+  const element = errorGroups.value?.$el
+  if (element instanceof Element)
+    element.scrollIntoView({ behavior: 'smooth', block: 'start' })
+}
 useStatusFavicon(() => live.snapshot.value?.state)
 const label = computed(() => runLabel({ runKey: props.route.runKey, startedAt: live.snapshot.value?.startedAt }))
 /** The tab of a running run leads with its progress, so it can be followed from a background tab. */
@@ -117,17 +125,12 @@ const errorCaseHref = (target: { caseId: string, taskId: string }) => href({ ...
       <RunDetails :run-id="props.route.runKey" :report-path="props.run?.reportPath" :report-root="props.run?.reportRoot" />
     </template>
     <template #actions>
-      <!-- The run's score at a glance, on the title's line: exact score and the case bar (counts in its tooltip). -->
-      <span v-if="score" :class="['flex items-center gap-3 mr-1', 'max-md:(w-full mr-0)']" data-testid="run-score">
-        <span :class="['flex items-baseline gap-1.5']">
-          <span :class="['text-xl font-semibold tabular-nums leading-none']">{{ formatScore(score.exact) }}</span>
-          <span :class="['text-xs muted']">exact</span>
-        </span>
-        <!-- A score over part of the cases is not the score of the run, so a coverage below 100% stands next to it. -->
-        <span v-if="score.coverage != null && score.coverage < 1" :class="['text-xs tabular-nums', '[color:var(--ink-errored)]']" title="Share of settled cases that produced a result" data-testid="run-coverage">
-          {{ (score.coverage * 100).toFixed(1) }}% coverage
-        </span>
-        <CountsBar :class="['w-32 max-md:flex-1']" :counts="score.counts" />
+      <!-- The run's score at a glance, on the title's line: the headline with its completion rate, and the case bar
+           (counts in its tooltip). A score over part of the cases is not the score of the run, so the completion rate
+           always stands next to it. -->
+      <span v-if="counts && live.snapshot.value" :class="['flex items-center gap-3 mr-1', 'max-md:(w-full mr-0)']" data-testid="run-score">
+        <HeadlineScore :result="live.snapshot.value.headline" :definitions="live.snapshot.value.scoreDefinitions" :task-label="taskLabel" @errors="showErrors" />
+        <CountsBar :class="['w-32 max-md:flex-1']" :counts="counts" />
       </span>
       <a v-if="position.workspace && position.project && position.experiment" :class="['btn']" :href="href({ name: 'compare', query: { experiment: position.experiment, project: position.project, view: 'attempts', workspace: position.workspace } })">
         <span class="i-mingcute-git-compare-line" aria-hidden="true" />Compare attempts
@@ -141,7 +144,7 @@ const errorCaseHref = (target: { caseId: string, taskId: string }) => href({ ...
       <div v-if="live.error.value" role="alert" :class="['mt-2 text-xs rounded state-failed px-2 py-1']">
         Refresh failed: {{ live.error.value.message }} — showing the last snapshot.
       </div>
-      <ErrorGroupsPanel :tasks="live.snapshot.value.tasks" :case-href="errorCaseHref" />
+      <ErrorGroupsPanel ref="errorGroups" :tasks="live.snapshot.value.tasks" :case-href="errorCaseHref" />
       <div :class="['mt-4']">
         <ViewTabs v-model="tab" :items="tabs" :href="value => href(tabRoute(value))" label="Run views">
           <ExecutionTree v-if="tab === 'execution'" :snapshot="live.snapshot.value" />
diff --git a/packages/ui/client/src/pages/TaskPage.vue b/packages/ui/client/src/pages/TaskPage.vue
index 44662be..e1c90ad 100644
--- a/packages/ui/client/src/pages/TaskPage.vue
+++ b/packages/ui/client/src/pages/TaskPage.vue
@@ -16,6 +16,7 @@ import AnimatedContent from '../components/AnimatedContent.vue'
 import BarChart from '../components/BarChart.vue'
 import CaseMetricsPanel from '../components/CaseMetricsPanel.vue'
 import CountsBar from '../components/CountsBar.vue'
+import HeadlineScore from '../components/HeadlineScore.vue'
 import MetricBesideCard from '../components/MetricBesideCard.vue'
 import PageHeader from '../components/PageHeader.vue'
 import ScoreBar from '../components/ScoreBar.vue'
@@ -23,16 +24,18 @@ import StateBadge from '../components/StateBadge.vue'
 import StateView from '../components/StateView.vue'
 
 import { api } from '../api'
-import { buildCaseTable, metricDefinition, metricLabel } from '../caseMetrics'
+import { buildCaseTable, columnDefinitions, formatColumnValue, metricDefinition, metricLabel } from '../caseMetrics'
 import { useRunLive } from '../composables/useRunLive'
 import { useStatusFavicon } from '../composables/useStatusFavicon'
-import { formatDuration, formatScore, formatTick, formatTime } from '../format'
+import { formatDuration, formatScore, formatTime } from '../format'
 import { levelIcons } from '../hierarchy'
 import { href } from '../router'
+import { headlineOf, scoreLabel, unscoredReasonLabel } from '../scores'
 import { runSwitcherOptions } from '../switcher'
 
 /**
- * One task (a matrix row of an eval entry): matrix values, scores per auto-attempt round, and all its cases.
+ * One task (a matrix row of an eval entry): matrix values, its headline and scores, scores per auto-attempt round, and
+ * all its cases, sorted by the primary score (lowest first) until another order is picked.
  */
 const props = defineProps<{ route: Extract<Route, { name: 'task' }>, run?: RunListItem }>()
 
@@ -52,6 +55,8 @@ const roundChart = computed<BarDatum[]>(() => rounds.value.map(round => ({ label
 const filter = shallowRef('')
 /** Cases picked in the case metrics charts; `undefined` while nothing is picked there. Clearing it clears the charts. */
 const picked = shallowRef<ReadonlySet<string>>()
+/** Only the cases without a result (errored, timed out): set from the headline's completion rate. */
+const onlyErrored = shallowRef(false)
 
 // The same query as the case metrics panel (one cache entry): the list shows and sorts by the cases' metrics.
 const metrics = useQuery({
@@ -61,11 +66,16 @@ const metrics = useQuery({
 watch(live.refreshKey, () => void metrics.refetch())
 const table = computed(() => buildCaseTable(task.value?.cases ?? [], metrics.data.value))
 const rowsByKey = computed(() => new Map(table.value.rows.map(row => [row.key, row])))
-const definitions = computed(() => new Map((metrics.data.value?.definitions ?? []).map(definition => [definition.name, definition])))
+const definitions = computed(() => columnDefinitions(metrics.data.value?.definitions ?? [], metrics.data.value?.scoreDefinitions ?? []))
 const labelOf = (name: string) => metricLabel(name, definitions.value)
+const scoreDefinitions = computed(() => metrics.data.value?.scoreDefinitions ?? live.snapshot.value?.scoreDefinitions ?? [])
+/** The score the task is read by, as the server resolved it; the list sorts and the charts color by it. */
+const primary = computed(() => task.value?.scoring.primary)
+/** The task's scores other than the primary one, with their means, beside the headline. */
+const otherScores = computed(() => (task.value?.scoring.scores ?? []).filter(summary => summary.name !== primary.value))
 
-/** Built-in values the list always shows (state, exact, duration) or that no column needs. */
-const BUILT_IN_COLUMNS = new Set(['duration_s', 'exact', 'group', 'judge', 'state'])
+/** Built-in values the list always shows (state, the primary score, duration) or that no column needs. */
+const BUILT_IN_COLUMNS = new Set(['duration_s', 'group', 'state'])
 /**
  * Metrics the list can show as columns: every string metric (a stratum, but also a case id with a value per case, too
  * many values for a chart axis) and every numeric one, as long as it is not the same for every case (a matrix value of
@@ -74,7 +84,7 @@ const BUILT_IN_COLUMNS = new Set(['duration_s', 'exact', 'group', 'judge', 'stat
 const columnOptions = computed(() => [
   ...new Set(table.value.rows.flatMap(row => Object.keys(row.categories))),
   ...table.value.groups.flatMap(group => [group.representative, ...group.members.map(member => member.name)]),
-].filter(name => !BUILT_IN_COLUMNS.has(name) && new Set(table.value.rows.map(row => row.numbers[name] ?? row.categories[name])).size > 1))
+].filter(name => !BUILT_IN_COLUMNS.has(name) && name !== primary.value && new Set(table.value.rows.map(row => row.numbers[name] ?? row.categories[name])).size > 1))
 const pickedColumns = shallowRef<string[]>([])
 /** The column option under the pointer or the keyboard, described beside the open menu. */
 const highlightedColumn = shallowRef<BesideTarget>()
@@ -91,7 +101,7 @@ const rowMinWidth = computed(() => (columns.value.length === 0 ? undefined : `${
  * Order of the list: as recorded, by score, by duration, or by a numeric column. Cases without the value go last
  * either way, so "lowest first" never opens on cases that have no score.
  */
-type SortKey = 'duration:desc' | 'exact:asc' | 'exact:desc' | 'index' | `metric:${string}:asc` | `metric:${string}:desc`
+type SortKey = 'duration:desc' | 'index' | 'score:asc' | 'score:desc' | `metric:${string}:asc` | `metric:${string}:desc`
 /** Phones put the picked columns on the line under the name, after the score. */
 function columnLine(key: string): string {
   return columns.value.map(name => `${labelOf(name)} ${columnValue(key, name)}`).join(' · ')
@@ -100,13 +110,25 @@ function columnLine(key: string): string {
 function columnValue(key: string, name: string): string {
   const row = rowsByKey.value.get(key)
   const value = row?.numbers[name] ?? row?.categories[name]
-  return value == null ? '—' : typeof value === 'number' ? formatTick(value) : value
+  return value == null ? '—' : typeof value === 'number' ? formatColumnValue(value, metricDefinition(name, definitions.value)) : value
 }
-const sortKey = shallowRef<SortKey>('index')
+// `undefined` until the viewer picks an order: a task with a primary score opens on its lowest-scored cases.
+const pickedSort = shallowRef<SortKey>()
+const sortKey = computed<SortKey>({
+  get: () => pickedSort.value ?? (primary.value == null ? 'index' : 'score:asc'),
+  set: (value) => {
+    pickedSort.value = value
+  },
+})
+const primaryLabel = computed(() => (primary.value == null ? undefined : scoreLabel(primary.value, scoreDefinitions.value)))
 const sortOptions = computed<{ label: string, value: SortKey }[]>(() => [
   { label: 'Recorded order', value: 'index' },
-  { label: 'Score, lowest first', value: 'exact:asc' },
-  { label: 'Score, highest first', value: 'exact:desc' },
+  ...(primaryLabel.value == null
+    ? []
+    : [
+        { label: `${primaryLabel.value}, lowest first`, value: 'score:asc' as const },
+        { label: `${primaryLabel.value}, highest first`, value: 'score:desc' as const },
+      ]),
   { label: 'Duration, longest first', value: 'duration:desc' },
   ...columns.value.filter(name => table.value.groups.some(group => group.representative === name || group.members.some(member => member.name === name))).flatMap(name => [
     { label: `${labelOf(name)}, lowest first`, value: `metric:${name}:asc` as const },
@@ -114,10 +136,10 @@ const sortOptions = computed<{ label: string, value: SortKey }[]>(() => [
   ]),
 ])
 const sortLabel = computed(() => sortOptions.value.find(option => option.value === sortKey.value)?.label ?? 'Recorded order')
-function sortValue(node: { durationMs?: number, key: string, scores: { exact?: number } }): number | undefined {
+function sortValue(node: { durationMs?: number, key: string }): number | undefined {
   const key = sortKey.value
-  if (key.startsWith('exact'))
-    return node.scores.exact
+  if (key.startsWith('score:'))
+    return primary.value == null ? undefined : rowsByKey.value.get(node.key)?.numbers[primary.value]
   if (key.startsWith('duration'))
     return node.durationMs
   if (key.startsWith('metric:'))
@@ -126,7 +148,9 @@ function sortValue(node: { durationMs?: number, key: string, scores: { exact?: n
 }
 
 const cases = computed(() => {
-  const list = (task.value?.cases ?? []).filter(node => (picked.value == null || picked.value.has(node.key)) && node.name.toLowerCase().includes(filter.value.toLowerCase()))
+  const list = (task.value?.cases ?? []).filter(node => (picked.value == null || picked.value.has(node.key))
+    && (!onlyErrored.value || node.state === 'errored' || node.state === 'timeout')
+    && node.name.toLowerCase().includes(filter.value.toLowerCase()))
   if (sortKey.value === 'index')
     return list
   const direction = sortKey.value.endsWith(':desc') ? -1 : 1
@@ -138,6 +162,17 @@ const cases = computed(() => {
     return (left - right) * direction || a.index - b.index
   })
 })
+/**
+ * A case's primary score on its row: the counted value (a failed case scores 0), the unscored reason, or `—`; with
+ * the score's label once, in the column title, on wide screens.
+ */
+function caseScore(node: { key: string, scores: { unscored: Record<string, string> } }): string {
+  if (primary.value == null)
+    return ''
+  const reason = node.scores.unscored[primary.value]
+  return reason == null ? formatScore(rowsByKey.value.get(node.key)?.numbers[primary.value]) : unscoredReasonLabel(reason)
+}
+
 // Once per second: running cases show how long they have run, in whole seconds.
 const now = useTimestamp({ scheduler: callback => useIntervalFn(callback, 1000) })
 
@@ -182,16 +217,28 @@ const position = computed(() => ({
         </div>
         <div>
           <div class="text-xs muted">
-            {{ task.scores.judge == null ? 'Exact' : 'Exact / judge / hybrid' }}
-          </div><b class="tabular-nums">{{ formatScore(task.scores.exact) }}<template v-if="task.scores.judge != null"> / {{ formatScore(task.scores.judge) }} / {{ formatScore(task.scores.hybrid) }}</template></b>
+            Headline
+          </div>
+          <!-- A completion rate below 100% narrows the case list to the cases without a result. -->
+          <HeadlineScore :result="headlineOf(task.scoring)" :definitions="scoreDefinitions" @errors="onlyErrored = true" />
+        </div>
+        <div v-if="otherScores.length > 0">
+          <div class="text-xs muted">
+            Other scores
+          </div>
+          <ul :class="['m-0 p-0 list-none grid gap-0.5']" data-testid="task-other-scores">
+            <li v-for="summary in otherScores" :key="summary.name" :class="['flex items-baseline gap-2']">
+              <span>{{ scoreLabel(summary.name, scoreDefinitions) }}</span><b :class="['tabular-nums ml-auto']">{{ formatScore(summary.mean) }}</b>
+            </li>
+          </ul>
         </div>
         <!-- Where the mean comes from: how many cases scored what. -->
-        <div v-if="task.cases.some(node => node.scores.exact != null)" :class="['min-w-40 max-md:(min-w-0 w-full)']">
+        <div v-if="primary && table.rows.some(row => row.numbers[primary!] != null)" :class="['min-w-40 max-md:(min-w-0 w-full)']">
           <div class="text-xs muted">
-            Scores per case
+            {{ primaryLabel }} per case
           </div>
           <div :class="['flex items-center h-6']">
-            <ScoreBar :class="['w-full']" :scores="task.cases.map(node => node.scores.exact)" />
+            <ScoreBar :class="['w-full']" :scores="table.rows.map(row => row.numbers[primary!])" />
           </div>
         </div>
         <div :class="['min-w-64 max-md:(min-w-0 w-full)']">
@@ -248,6 +295,9 @@ const position = computed(() => ({
           <button v-if="picked" type="button" :class="['btn btn-active text-xs py-0.5! font-normal']" data-testid="case-list-picked" @click="picked = undefined">
             <span class="i-mingcute-filter-line" aria-hidden="true" />{{ picked.size }} picked in the charts<span class="i-mingcute-close-line" aria-label="Clear the selection" />
           </button>
+          <button v-if="onlyErrored" type="button" :class="['btn btn-active text-xs py-0.5! font-normal']" data-testid="case-list-errored" @click="onlyErrored = false">
+            <span class="i-mingcute-filter-line" aria-hidden="true" />without a result<span class="i-mingcute-close-line" aria-label="Show every case" />
+          </button>
           <span :class="['flex flex-wrap items-center gap-2 ml-auto font-normal', 'max-md:(w-full ml-0)']">
             <DropdownMenuRoot>
               <DropdownMenuTrigger :class="['option-trigger text-xs! py-1!']" data-testid="case-list-sort">
@@ -303,7 +353,7 @@ const position = computed(() => ({
             <div v-if="columns.length > 0" :class="['sticky top-0 z-1 flex items-center gap-x-2 px-3 py-1 text-xs muted bg-white dark:bg-neutral-900 border-b border-neutral-100/70 dark:border-neutral-900', 'max-md:hidden']" :style="{ minWidth: rowMinWidth }">
               <span :class="['flex-1']" />
               <span v-for="name in columns" :key="name" :class="['shrink-0 truncate text-right']" :style="{ width: `${COLUMN_PX}px` }" :title="name">{{ labelOf(name) }}</span>
-              <span :class="['w-36 shrink-0 text-right']">exact · duration</span>
+              <span :class="['w-36 shrink-0 text-right truncate']">{{ primaryLabel ? `${primaryLabel} · duration` : 'duration' }}</span>
             </div>
             <div :style="{ height: `${virtualizer.getTotalSize()}px`, position: 'relative', minWidth: wide ? rowMinWidth : undefined }">
               <a
@@ -321,7 +371,7 @@ const position = computed(() => ({
                 </span>
                 <span v-for="name in columns" :key="name" :class="['shrink-0 truncate text-right text-xs mono', 'max-md:hidden', name === columns[0] && 'ml-auto']" :style="{ width: `${COLUMN_PX}px` }" :title="columnValue(cases[item.index]!.key, name)">{{ columnValue(cases[item.index]!.key, name) }}</span>
                 <!-- Phones: under the name (third grid column), aligned with it, with the picked columns after the score. -->
-                <span :class="['text-xs muted mono whitespace-nowrap', columns.length === 0 ? 'ml-auto' : 'w-36 shrink-0 text-right', 'max-md:(col-start-3 ml-0 w-auto text-left truncate)']" :title="columns.length > 0 ? columnLine(cases[item.index]!.key) : undefined"><template v-if="cases[item.index]!.state === 'running'"><span class="[color:var(--ink-running)]">running {{ formatDuration(Math.max(0, now - Date.parse(cases[item.index]!.rounds.at(-1)?.startedAt ?? cases[item.index]!.startedAt))) }}</span></template><template v-else>exact {{ formatScore(cases[item.index]!.scores.exact) }} · {{ formatDuration(cases[item.index]!.durationMs) }}<span v-if="!wide && columns.length > 0"> · {{ columnLine(cases[item.index]!.key) }}</span></template></span>
+                <span :class="['text-xs muted mono whitespace-nowrap', columns.length === 0 ? 'ml-auto' : 'w-36 shrink-0 text-right', 'max-md:(col-start-3 ml-0 w-auto text-left truncate)']" :title="columns.length > 0 ? columnLine(cases[item.index]!.key) : undefined"><template v-if="cases[item.index]!.state === 'running'"><span class="[color:var(--ink-running)]">running {{ formatDuration(Math.max(0, now - Date.parse(cases[item.index]!.rounds.at(-1)?.startedAt ?? cases[item.index]!.startedAt))) }}</span></template><template v-else><template v-if="primary">{{ wide && columns.length > 0 ? '' : `${primaryLabel} ` }}{{ caseScore(cases[item.index]!) }} · </template>{{ formatDuration(cases[item.index]!.durationMs) }}<span v-if="!wide && columns.length > 0"> · {{ columnLine(cases[item.index]!.key) }}</span></template></span>
               </a>
             </div>
           </div>
diff --git a/packages/ui/client/src/router.test.ts b/packages/ui/client/src/router.test.ts
index b6e377c..7c9d933 100644
--- a/packages/ui/client/src/router.test.ts
+++ b/packages/ui/client/src/router.test.ts
@@ -19,6 +19,17 @@ describe('router', () => {
     expect(parseRoute(href(route))).toEqual(route)
   })
 
+  /**
+   * @example
+   * expect(parseRoute(href(route)).grader).toBe('judge')
+   */
+  it('keeps the grader a case\'s inference tab is narrowed to', () => {
+    const route = { caseId: '0:a', grader: 'judge', name: 'case', runKey: 'run-1-a', tab: 'inference', taskId: 't' } as const
+
+    expect(href(route)).toBe('#/runs/run-1-a/case?caseId=0%3Aa&grader=judge&tab=inference&taskId=t')
+    expect(parseRoute(href(route))).toEqual(route)
+  })
+
   /**
    * @example
    * expect(parseRoute('#/?workspace=w&project=p')).toEqual({ name: 'scope', workspace: 'w', project: 'p', … })
diff --git a/packages/ui/client/src/router.ts b/packages/ui/client/src/router.ts
index 9bdf734..37a9034 100644
--- a/packages/ui/client/src/router.ts
+++ b/packages/ui/client/src/router.ts
@@ -3,11 +3,12 @@ import { computed, shallowRef } from 'vue'
 /**
  * One screen of the viewer, decoded from the location hash.
  *
- * Ids travel as query parameters because task and case ids contain `/` and `%` themselves.
+ * Ids travel as query parameters because task and case ids contain `/` and `%` themselves. `grader` on a case narrows
+ * its inference tab to one grader's model calls (a score's grader `id`).
  */
 export type Route
   = | { attempt?: string, experiment?: string, name: 'scope', project?: string, tab?: string, workspace?: string }
-    | { caseId: string, name: 'case', runKey: string, tab?: string, taskId: string }
+    | { caseId: string, grader?: string, name: 'case', runKey: string, tab?: string, taskId: string }
     | { name: 'compare', query: Record<string, string> }
     | { name: 'run', runKey: string, tab?: string }
     | { name: 'task', runKey: string, taskId: string }
@@ -27,7 +28,7 @@ export function href(route: Route): string {
   }
   switch (route.name) {
     case 'case':
-      return `#/runs/${encodeURIComponent(route.runKey)}/case${query({ caseId: route.caseId, tab: route.tab, taskId: route.taskId })}`
+      return `#/runs/${encodeURIComponent(route.runKey)}/case${query({ caseId: route.caseId, grader: route.grader, tab: route.tab, taskId: route.taskId })}`
     case 'compare':
       return `#/compare${query(route.query)}`
     case 'run':
@@ -58,7 +59,7 @@ export function parseRoute(hash: string): Route {
   if (segments[0] === 'runs' && segments[1] != null) {
     const runKey = segments[1]
     if (segments[2] === 'case' && query.taskId != null && query.caseId != null)
-      return { caseId: query.caseId, name: 'case', runKey, tab: query.tab, taskId: query.taskId }
+      return { caseId: query.caseId, grader: query.grader, name: 'case', runKey, tab: query.tab, taskId: query.taskId }
     if (segments[2] === 'task' && query.taskId != null)
       return { name: 'task', runKey, taskId: query.taskId }
     return { name: 'run', runKey, tab: query.tab }
diff --git a/packages/ui/client/src/scores.test.ts b/packages/ui/client/src/scores.test.ts
new file mode 100644
index 0000000..c82a0ae
--- /dev/null
+++ b/packages/ui/client/src/scores.test.ts
@@ -0,0 +1,165 @@
+import type { HeadlineResult, ScoreDefinition, ScoreSummary } from '../../src/shared/contract'
+
+import { describe, expect, it } from 'vitest'
+
+import { caseScoreText, completionText, formulaText, gradersDiffer, graderText, headlineOf, headlineValue, headlineView, isGraderCall, primaryScoreName, unscoredText } from './scores'
+
+/** The scores of airi's `tts-speech-formula` eval, the first eval migrated to named scores. */
+const definitions: ScoreDefinition[] = [
+  { label: { 'en': 'Written back correctly', 'zh-Hans': '听写正确' }, name: 'tts_speech_formula.correct' },
+  { grader: { id: 'judge', kind: 'model', model: 'gpt-5.5', reducer: 'median', votes: 3 }, label: { 'en': 'Natural reading', 'zh-Hans': '读法自然' }, name: 'tts_speech_formula.natural' },
+  {
+    combine: { inputs: [{ name: 'tts_speech_formula.correct' }, { atLeast: 0.5, name: 'tts_speech_formula.natural' }], kind: 'all' },
+    description: { 'en': 'Written back correctly and read naturally.', 'zh-Hans': '听写正确，并且读法自然。' },
+    label: { 'en': 'Pass', 'zh-Hans': '通过' },
+    name: 'tts_speech_formula.pass',
+    primary: true,
+  },
+]
+
+function summary(name: string, mean: null | number, unscored: Record<string, number> = {}): ScoreSummary {
+  return { mean, n: 40, name, sum: (mean ?? 0) * 40, total: 42, unscored }
+}
+
+const completion = summary('completion', 0.968, { errored: 4 })
+
+describe('headlineView', () => {
+  /**
+   * @example
+   * expect(headlineView(result, definitions).label).toBe('Pass')
+   */
+  it('names a score headline by the localized label of the primary score', () => {
+    const result: HeadlineResult = { completion, headline: { kind: 'score', score: summary('tts_speech_formula.pass', 0.875) } }
+    const view = headlineView(result, definitions)
+
+    expect(view.label).toBe('Pass')
+    expect(view.summary.mean).toBe(0.875)
+    expect(view.definition?.primary).toBe(true)
+    expect(view.tasks).toEqual([])
+  })
+
+  /**
+   * @example
+   * expect(headlineView(result, []).label).toBe('Passed cases')
+   */
+  it('labels the passed share when no score was recorded', () => {
+    const result: HeadlineResult = { completion, headline: { kind: 'passed-cases', score: summary('passed', 0.5) } }
+
+    expect(headlineView(result, []).label).toBe('Passed cases')
+    expect(headlineValue(result)).toBe(0.5)
+  })
+
+  /**
+   * @example
+   * expect(view.tasks.map(task => task.label)).toEqual(['Pass', 'Natural reading'])
+   */
+  it('lists each task\'s primary score and keeps the completion rate as the value of a per-task headline', () => {
+    const result: HeadlineResult = {
+      completion,
+      headline: { kind: 'per-task', tasks: [{ score: summary('tts_speech_formula.pass', 0.8), taskId: 'en' }, { score: summary('tts_speech_formula.natural', 0.6), taskId: 'zh' }] },
+    }
+    const view = headlineView(result, definitions)
+
+    expect(view.label).toBe('completed')
+    expect(view.summary.mean).toBe(0.968)
+    expect(view.tasks.map(task => task.label)).toEqual(['Pass', 'Natural reading'])
+    expect(view.tasks.map(task => task.taskId)).toEqual(['en', 'zh'])
+    expect(headlineValue(result)).toBeNull()
+  })
+})
+
+describe('headlineOf', () => {
+  /**
+   * @example
+   * expect(headlineOf(scoring).headline.kind).toBe('score')
+   */
+  it('reads a task\'s primary score as its headline, and the passed cases without one', () => {
+    const scoring = { completion, passed: summary('passed', 0.9), primary: 'tts_speech_formula.pass', scores: [summary('tts_speech_formula.pass', 0.875)] }
+
+    expect(headlineOf(scoring)).toEqual({ completion, headline: { kind: 'score', score: scoring.scores[0] } })
+    expect(headlineOf({ ...scoring, primary: undefined })).toEqual({ completion, headline: { kind: 'passed-cases', score: scoring.passed } })
+  })
+})
+
+describe('score texts', () => {
+  /**
+   * @example
+   * expect(completionText(completion)).toBe('completed 96.8% (4 errored)')
+   */
+  it('formats the completion rate with the cases that produced no result', () => {
+    expect(completionText(completion)).toBe('completed 96.8% (4 errored)')
+    expect(completionText(summary('completion', 1))).toBe('completed 100.0%')
+    expect(completionText(summary('completion', 0.9, { errored: 1, timeout: 2 }))).toBe('completed 90.0% (1 errored, 2 timed out)')
+  })
+
+  /**
+   * @example
+   * expect(formulaText(definitions[2], definitions)).toBe('all of: Written back correctly, Natural reading ≥ 50%')
+   */
+  it('spells out the formula of a derived score with the labels of its inputs', () => {
+    expect(formulaText(definitions[2], definitions)).toBe('all of: Written back correctly, Natural reading ≥ 50%')
+    expect(formulaText({ combine: { inputs: [{ name: 'a', weight: 2 }, { name: 'b', weight: 1 }], kind: 'weighted' }, name: 'w' }, [])).toBe('weighted: a ×2, b ×1')
+    expect(formulaText({ combine: { inputs: [{ name: 'a' }, { name: 'b' }], kind: 'mean' }, name: 'm' }, [])).toBe('mean of: a, b')
+    expect(formulaText(definitions[0], definitions)).toBeUndefined()
+  })
+
+  /**
+   * @example
+   * expect(graderText({ kind: 'model', model: 'gpt-5.5', reducer: 'median', votes: 3 })).toBe('model · gpt-5.5 · 3 votes, median')
+   */
+  it('formats what produced a score', () => {
+    expect(graderText({ kind: 'model', model: 'gpt-5.5', reducer: 'median', votes: 3 })).toBe('model · gpt-5.5 · 3 votes, median')
+    expect(graderText({ kind: 'rule', version: '2' })).toBe('rule · version 2')
+    expect(graderText({ kind: 'external' })).toBe('external')
+  })
+
+  /**
+   * @example
+   * expect(unscoredText({ grader_failed: 2, refusal: 1 })).toBe('2 grader failed, 1 refusal')
+   */
+  it('labels the suggested unscored reasons and keeps other reasons as written', () => {
+    expect(unscoredText({ grader_failed: 2, refusal: 1 })).toBe('2 grader failed, 1 refusal')
+    expect(unscoredText({ 'input_unscored': 3, 'judge timed out': 1 })).toBe('3 input unscored, 1 judge timed out')
+    expect(caseScoreText({ unscored: { natural: 'invalid_response' }, values: { pass: 1 } }, 'natural')).toBe('invalid response')
+    expect(caseScoreText({ unscored: {}, values: { pass: 1 } }, 'pass')).toBe('100.0%')
+    expect(caseScoreText({ unscored: {}, values: {} }, 'pass')).toBe('—')
+  })
+})
+
+describe('graders', () => {
+  /**
+   * @example
+   * expect(gradersDiffer({ kind: 'model', model: 'a' }, { kind: 'model', model: 'b' })).toBe(true)
+   */
+  it('tells graders apart by kind, model, and version, not by votes', () => {
+    expect(gradersDiffer({ kind: 'model', model: 'a' }, { kind: 'model', model: 'b' })).toBe(true)
+    expect(gradersDiffer({ kind: 'rule', version: '1' }, { kind: 'rule', version: '2' })).toBe(true)
+    expect(gradersDiffer({ kind: 'model', model: 'a', votes: 3 }, { kind: 'model', model: 'a', votes: 5 })).toBe(false)
+    expect(gradersDiffer(undefined, { kind: 'rule' })).toBe(false)
+  })
+
+  /**
+   * @example
+   * expect(isGraderCall({ provider: { id: 'judge' } }, 'judge')).toBe(true)
+   */
+  it('matches a grader\'s model calls by provider id or request role', () => {
+    expect(isGraderCall({ provider: { id: 'judge', model: 'gpt-5.5' } }, 'judge')).toBe(true)
+    expect(isGraderCall({ provider: { id: 'openai' }, request: { role: 'judge' } }, 'judge')).toBe(true)
+    expect(isGraderCall({ provider: { id: 'openai' }, request: { step: 1 } }, 'judge')).toBe(false)
+  })
+})
+
+describe('primaryScoreName', () => {
+  /**
+   * @example
+   * expect(primaryScoreName(definitions, names)).toBe('tts_speech_formula.pass')
+   */
+  it('picks the primary score, else the only score, else none', () => {
+    const names = definitions.map(definition => definition.name)
+
+    expect(primaryScoreName(definitions, names)).toBe('tts_speech_formula.pass')
+    expect(primaryScoreName([], ['only'])).toBe('only')
+    expect(primaryScoreName([], ['a', 'b'])).toBeUndefined()
+    expect(primaryScoreName([{ name: 'a', primary: true }, { name: 'b', primary: true }], ['a', 'b'])).toBeUndefined()
+  })
+})
diff --git a/packages/ui/client/src/scores.ts b/packages/ui/client/src/scores.ts
new file mode 100644
index 0000000..73a66bf
--- /dev/null
+++ b/packages/ui/client/src/scores.ts
@@ -0,0 +1,285 @@
+import type { Grader } from 'vieval/events'
+
+import type { CaseScores, CaseScoring, HeadlineResult, MetricDefinition, ScoreDefinition, ScoreSummary } from '../../src/shared/contract'
+import type { InferenceCall } from '../../src/shared/inference'
+
+import { formatScore } from './format'
+import { localize } from './localize'
+
+/**
+ * Definition of one numeric column of the case charts and tables: a metric's `metric.defined`, or a score's
+ * `score.defined` read as a metric (higher is better, no unit) with the score definition kept in `score`.
+ */
+export interface ColumnDefinition extends MetricDefinition {
+  /** How a derived score combines its inputs, with their labels ({@link formulaText}). */
+  formula?: string
+  /** Present when the column is a score: its values are `0..1` and are shown as a percentage. */
+  score?: ScoreDefinition
+}
+
+/** One task's headline score of a `per-task` headline, with its label resolved. */
+export interface HeadlineTask {
+  definition?: ScoreDefinition
+  label: string
+  summary: ScoreSummary
+  taskId: string
+}
+
+/** What a headline shows, resolved for display: the label and value of the headline, and the completion rate beside it. */
+export interface HeadlineView {
+  /** Share of settled cases that produced a result; shown next to the headline everywhere. */
+  completion: ScoreSummary
+  /** Definition of the headline score; absent for `passed-cases` and `per-task`. */
+  definition?: ScoreDefinition
+  kind: HeadlineResult['headline']['kind']
+  /** Localized label of the primary score, `Passed cases`, or, for `per-task`, `completed`. */
+  label: string
+  /** Summary of the headline value; the completion rate for `per-task`. */
+  summary: ScoreSummary
+  /** Each task's primary score, for `per-task` only. */
+  tasks: HeadlineTask[]
+}
+
+/**
+ * Labels of the suggested unscored reasons (`recordUnscored`), and of the reasons a case leaves the completion rate
+ * (`errored`, `timeout`); any other reason is shown as written.
+ */
+const UNSCORED_REASONS: Record<string, string> = {
+  errored: 'errored',
+  grader_failed: 'grader failed',
+  input_unscored: 'input unscored',
+  invalid_response: 'invalid response',
+  no_response: 'no response',
+  refusal: 'refusal',
+  timeout: 'timed out',
+}
+
+/** Shown instead of a score label when a group of cases records no score: its headline is the share of passed cases. */
+const PASSED_CASES_LABEL = 'Passed cases'
+
+/**
+ * Formats the value of one score of one case: the value as a percentage, the unscored reason, or `—`.
+ *
+ * Before:
+ * - ({ values: { pass: 1 }, unscored: {} }, 'pass'), ({ values: {}, unscored: { natural: 'grader_failed' } }, 'natural')
+ *
+ * After:
+ * - "100.0%", "grader failed"
+ */
+export function caseScoreText(scores: CaseScores, name: string): string {
+  const reason = scores.unscored[name]
+  if (reason != null)
+    return unscoredReasonLabel(reason)
+  return formatScore(scores.values[name])
+}
+
+/**
+ * The two parts of {@link completionText}, so a narrow column can wrap between them and never inside one.
+ *
+ * Before:
+ * - { mean: 0.968, unscored: { errored: 4 } }
+ *
+ * After:
+ * - { rate: "completed 96.8%", reasons: "(4 errored)" }
+ */
+export function completionParts(completion: ScoreSummary): { rate: string, reasons?: string } {
+  const parts = [
+    completion.unscored.errored ? `${completion.unscored.errored} errored` : undefined,
+    completion.unscored.timeout ? `${completion.unscored.timeout} timed out` : undefined,
+  ].filter(Boolean)
+  const rate = `completed ${formatScore(completion.mean)}`
+  return parts.length > 0 ? { rate, reasons: `(${parts.join(', ')})` } : { rate }
+}
+
+/**
+ * Formats the completion rate of a headline with the cases that produced no result.
+ *
+ * Before:
+ * - { mean: 0.968, unscored: { errored: 4 } }, { mean: 1, unscored: {} }
+ *
+ * After:
+ * - "completed 96.8% (4 errored)", "completed 100.0%"
+ */
+export function completionText(completion: ScoreSummary): string {
+  const { rate, reasons } = completionParts(completion)
+  return reasons == null ? rate : `${rate} ${reasons}`
+}
+
+/**
+ * Formats how a derived score combines other scores, with their labels.
+ *
+ * Before:
+ * - { kind: 'all', inputs: [{ name: 'correct' }, { atLeast: 0.5, name: 'natural' }] }
+ * - { kind: 'weighted', inputs: [{ name: 'a', weight: 2 }, { name: 'b', weight: 1 }] }
+ *
+ * After:
+ * - "all of: correct, natural ≥ 50%"
+ * - "weighted: a ×2, b ×1"
+ */
+export function formulaText(definition: ScoreDefinition | undefined, definitions: readonly ScoreDefinition[]): string | undefined {
+  const combine = definition?.combine
+  if (combine == null)
+    return undefined
+  const inputs = combine.inputs.map((input) => {
+    const label = scoreLabel(input.name, definitions)
+    if (combine.kind === 'all')
+      return input.atLeast == null || input.atLeast >= 1 ? label : `${label} ≥ ${Math.round(input.atLeast * 100)}%`
+    if (combine.kind === 'weighted')
+      return `${label} ×${input.weight ?? 1}`
+    return label
+  })
+  const prefix = { all: 'all of', mean: 'mean of', weighted: 'weighted' }[combine.kind]
+  return `${prefix}: ${inputs.join(', ')}`
+}
+
+/**
+ * Whether two graders measure differently: their kind, model, or version differ. Two values of one score from such
+ * graders do not measure the same thing, so a comparison warns about them.
+ */
+export function gradersDiffer(left: Grader | undefined, right: Grader | undefined): boolean {
+  if (left == null || right == null)
+    return false
+  return left.kind !== right.kind || left.model !== right.model || left.version !== right.version
+}
+
+/**
+ * Formats what produced a score value.
+ *
+ * Before:
+ * - { kind: 'model', model: 'gpt-5.5', reducer: 'median', votes: 3 }, { kind: 'rule', version: '2' }
+ *
+ * After:
+ * - "model · gpt-5.5 · 3 votes, median", "rule · version 2"
+ */
+export function graderText(grader: Grader): string {
+  const votes = grader.votes == null ? undefined : `${grader.votes} ${grader.votes === 1 ? 'vote' : 'votes'}${grader.reducer == null ? '' : `, ${grader.reducer}`}`
+  return [grader.kind, grader.model, grader.version == null ? undefined : `version ${grader.version}`, votes ?? grader.reducer].filter(Boolean).join(' · ')
+}
+
+/**
+ * Headline of one group of cases (a task, a matrix cell) from its scores: the primary score, or the passed cases when
+ * it has none.
+ *
+ * Use when:
+ * - a view holds a task's `CaseScoring` (task nodes, matrix cells) and shows it like a run headline
+ *
+ * Expects:
+ * - `scoring.primary` resolved by the server (`scoreCases`); this does not pick a primary itself
+ *
+ * Returns:
+ * - a `score` headline when the primary score has a summary, else `passed-cases`
+ */
+export function headlineOf(scoring: CaseScoring): HeadlineResult {
+  const primary = scoring.primary == null ? undefined : scoring.scores.find(summary => summary.name === scoring.primary)
+  return {
+    completion: scoring.completion,
+    headline: primary == null ? { kind: 'passed-cases', score: scoring.passed } : { kind: 'score', score: primary },
+  }
+}
+
+/**
+ * The single value of a headline for a trend or a sort: the mean of its score; `null` for a `per-task` headline,
+ * whose tasks have different primary scores.
+ */
+export function headlineValue(result: HeadlineResult | null | undefined): null | number {
+  if (result == null || result.headline.kind === 'per-task')
+    return null
+  return result.headline.score.mean
+}
+
+/**
+ * Resolves a headline for display: the localized label of its score and the completion rate.
+ *
+ * Use when:
+ * - any page shows a run's, task's, or matrix cell's headline (`HeadlineScore.vue`, switchers, chart series)
+ *
+ * Expects:
+ * - `definitions` hold the `score.defined` of the headline's scores (`headlineDefinitions`, `scoreDefinitions`)
+ *
+ * Returns:
+ * - for `per-task`, the completion rate as the value, labelled `completed`, and each task's primary score
+ */
+export function headlineView(result: HeadlineResult, definitions: readonly ScoreDefinition[]): HeadlineView {
+  const { completion, headline } = result
+  if (headline.kind === 'per-task') {
+    const tasks = headline.tasks.map(task => ({ definition: scoreDefinition(task.score.name, definitions), label: scoreLabel(task.score.name, definitions), summary: task.score, taskId: task.taskId }))
+    return { completion, kind: headline.kind, label: 'completed', summary: completion, tasks }
+  }
+  if (headline.kind === 'passed-cases')
+    return { completion, kind: headline.kind, label: PASSED_CASES_LABEL, summary: headline.score, tasks: [] }
+  return { completion, definition: scoreDefinition(headline.score.name, definitions), kind: headline.kind, label: scoreLabel(headline.score.name, definitions), summary: headline.score, tasks: [] }
+}
+
+/**
+ * Whether a model call was made by a grader: its provider id, or the `role` of the request data the eval attached,
+ * is the grader's `id` (a judge's calls report usage through the same `inference.*` events as the case's own calls).
+ *
+ * Use when:
+ * - a score links to the calls of its grader, and the inference panel narrows to them
+ */
+export function isGraderCall(call: Pick<InferenceCall, 'provider' | 'request'>, graderId: string): boolean {
+  if (call.provider?.id === graderId)
+    return true
+  const request = call.request
+  return typeof request === 'object' && request != null && 'role' in request && request.role === graderId
+}
+
+/**
+ * Picks the headline score among the score names of a group of cases: the one defined `primary`, else the only score.
+ *
+ * Use when:
+ * - a client-side view of cases (the case charts, a run-wide table) needs the score its outcome is read by
+ *
+ * Returns:
+ * - `undefined` when no score is primary and there is not exactly one, or when the cases' tasks name different
+ *   primary scores (a run-wide table of several evals)
+ */
+export function primaryScoreName(definitions: readonly ScoreDefinition[], names: readonly string[]): string | undefined {
+  const primaries = [...new Set(definitions.filter(definition => definition.primary === true && names.includes(definition.name)).map(definition => definition.name))]
+  if (primaries.length === 1)
+    return primaries[0]
+  return primaries.length === 0 && names.length === 1 ? names[0] : undefined
+}
+
+/**
+ * Reads a score's `score.defined` as a column of the case charts and hover cards: higher is better, values shown as a
+ * percentage, and the formula of a derived score spelled out with the labels of `definitions`.
+ */
+export function scoreColumn(definition: ScoreDefinition, definitions: readonly ScoreDefinition[]): ColumnDefinition {
+  return { better: 'higher', description: definition.description, formula: formulaText(definition, definitions), label: definition.label, name: definition.name, score: definition }
+}
+
+export function scoreDefinition(name: string, definitions: readonly ScoreDefinition[]): ScoreDefinition | undefined {
+  return definitions.findLast(definition => definition.name === name)
+}
+
+/** Readable name of a score: its localized label, else its name. */
+export function scoreLabel(name: string, definitions: readonly ScoreDefinition[]): string {
+  return localize(scoreDefinition(name, definitions)?.label) ?? name
+}
+
+/**
+ * Normalizes an unscored reason code into its label; unknown reasons stay as written.
+ *
+ * Before:
+ * - "grader_failed", "judge timed out"
+ *
+ * After:
+ * - "grader failed", "judge timed out"
+ */
+export function unscoredReasonLabel(reason: string): string {
+  return UNSCORED_REASONS[reason] ?? reason
+}
+
+/**
+ * Formats the cases that recorded a score as unscored, per reason, largest first.
+ *
+ * Before:
+ * - { refusal: 1, grader_failed: 2 }
+ *
+ * After:
+ * - "2 grader failed, 1 refusal"
+ */
+export function unscoredText(unscored: Readonly<Record<string, number>>): string {
+  return Object.entries(unscored).filter(([, count]) => count > 0).sort((a, b) => b[1] - a[1]).map(([reason, count]) => `${count} ${unscoredReasonLabel(reason)}`).join(', ')
+}
diff --git a/packages/ui/client/src/switcher.test.ts b/packages/ui/client/src/switcher.test.ts
index 09ee916..10b41b2 100644
--- a/packages/ui/client/src/switcher.test.ts
+++ b/packages/ui/client/src/switcher.test.ts
@@ -12,16 +12,22 @@ import { runSwitcherOptions } from './switcher'
 // Removal condition: a shared snapshot fixture builder for client tests.
 const snapshot = {
   runKey: 'run-1-a',
+  scoreDefinitions: [{ label: { 'en': 'Pass', 'zh-Hans': '通过' }, name: 'pass', primary: true }],
   tasks: [
     {
       caseCounts: { failed: 1, passed: 1, running: 0, skipped: 0, timeout: 0, total: 2 },
       cases: [
-        { caseId: '0:chrome%2Fa', durationMs: 90_000, index: 0, name: 'chrome/a', scores: { exact: 1 }, state: 'passed', taskId: 't::1' },
-        { caseId: '1:chrome%2Fb', index: 1, name: 'chrome/b', scores: {}, state: 'running', taskId: 't::1' },
+        { caseId: '0:chrome%2Fa', durationMs: 90_000, index: 0, name: 'chrome/a', scores: { unscored: {}, values: { pass: 1 } }, state: 'passed', taskId: 't::1' },
+        { caseId: '1:chrome%2Fb', index: 1, name: 'chrome/b', scores: { unscored: {}, values: {} }, state: 'running', taskId: 't::1' },
       ],
       displayName: 'osworld [model=gpt-5.5]',
       id: 't::1',
-      scores: { exact: 0.5 },
+      scoring: {
+        completion: { mean: 1, n: 1, name: 'completion', sum: 1, total: 1, unscored: {} },
+        passed: { mean: 1, n: 1, name: 'passed', sum: 1, total: 1, unscored: {} },
+        primary: 'pass',
+        scores: [{ mean: 0.5, n: 2, name: 'pass', sum: 1, total: 2, unscored: {} }],
+      },
       startedAt: '2026-10-12T15:47:59.000Z',
       state: 'running',
     },
@@ -41,17 +47,20 @@ describe('runSwitcherOptions', () => {
     expect(options?.[1]?.detail).toBe('#2')
     expect(options?.[1]?.state).toBe('running')
     expect(options?.[0]?.href).toBe('#/runs/run-1-a/case?caseId=0%3Achrome%252Fa&taskId=t%3A%3A1')
+    expect(options?.[0]?.score).toBe(1)
+    expect(options?.[1]?.score).toBeNull()
   })
 
   /**
    * @example
-   * expect(runSwitcherOptions(snapshot, 'task')?.[0]?.label).toBe('osworld [model=gpt-5.5]')
+   * expect(runSwitcherOptions(snapshot, 'task')?.[0]?.score).toBe(0.5)
    */
-  it('offers the tasks of the run as siblings of a task', () => {
+  it('offers the tasks of the run as siblings of a task, with the primary score as their headline', () => {
     const options = runSwitcherOptions(snapshot, 'task')
 
     expect(options?.[0]?.label).toBe('osworld [model=gpt-5.5]')
-    expect(options?.[0]?.exact).toBe(0.5)
+    expect(options?.[0]?.score).toBe(0.5)
+    expect(options?.[0]?.scoreLabel).toBe('Pass')
     expect(options?.[0]?.counts?.total).toBe(2)
   })
 
diff --git a/packages/ui/client/src/switcher.ts b/packages/ui/client/src/switcher.ts
index 2cd1bbe..292f7a0 100644
--- a/packages/ui/client/src/switcher.ts
+++ b/packages/ui/client/src/switcher.ts
@@ -1,10 +1,11 @@
-import type { CaseCounts, RunListItem, RunSnapshot } from '../../src/shared/contract'
+import type { CaseCounts, HeadlineResult, RunListItem, RunSnapshot, ScoreDefinition } from '../../src/shared/contract'
 import type { HierarchyPosition } from './hierarchy'
 import type { MatrixChip } from './labels'
 
 import { formatDuration } from './format'
 import { attemptDate, attemptLabel, experimentLabel, mergeMatrices, runLabel } from './labels'
 import { href } from './router'
+import { headlineOf, headlineView, scoreLabel } from './scores'
 
 /** Levels whose breadcrumb segment can switch to a sibling. */
 export type SwitchableLevel = 'attempt' | 'experiment' | 'project' | 'run' | 'workspace'
@@ -15,12 +16,15 @@ export interface SwitcherOption {
   counts?: CaseCounts
   /** Short facts, for example "3 experiments · 12 runs". */
   detail?: string
-  exact?: null | number
   href: string
   /** Full id; searchable and shown as the tooltip. */
   id: string
   /** Readable name (`labels.ts`). */
   label: string
+  /** Headline value of the option (the primary score, the passed cases, or the completion rate), from 0 to 1. */
+  score?: null | number
+  /** What `score` is, for example `Pass`; its tooltip. */
+  scoreLabel?: string
   /** State of the newest run below the option. */
   state?: string
   /** Start of the newest run below the option (ISO). */
@@ -56,15 +60,16 @@ export function filterOptions(options: readonly SwitcherOption[], query: string)
  * - the snapshot the page already holds (`useRunLive`); no request is made
  *
  * Returns:
- * - tasks in run order, or the cases of `taskId` in case order; `undefined` without a snapshot
+ * - tasks in run order with their headline, or the cases of `taskId` in case order with their value of the task's
+ *   primary score; `undefined` without a snapshot
  */
 export function runSwitcherOptions(snapshot: RunSnapshot | undefined, level: 'case' | 'task', taskId?: string): SwitcherOption[] | undefined {
   if (snapshot == null)
     return undefined
   if (level === 'task') {
     return snapshot.tasks.map(task => ({
+      ...headlineScore(headlineOf(task.scoring), snapshot.scoreDefinitions),
       counts: task.caseCounts,
-      exact: task.scores.exact,
       href: href({ name: 'task', runKey: snapshot.runKey, taskId: task.id }),
       id: task.id,
       label: task.displayName,
@@ -72,9 +77,12 @@ export function runSwitcherOptions(snapshot: RunSnapshot | undefined, level: 'ca
       time: task.startedAt,
     }))
   }
-  return snapshot.tasks.find(task => task.id === taskId)?.cases.map(node => ({
+  const task = snapshot.tasks.find(item => item.id === taskId)
+  const primary = task?.scoring.primary
+  return task?.cases.map(node => ({
     detail: [`#${node.index + 1}`, node.durationMs == null ? undefined : formatDuration(node.durationMs)].filter(Boolean).join(' · '),
-    exact: node.scores.exact,
+    // The case's own recorded value of the task's primary score.
+    ...(primary == null ? {} : { score: node.scores.values[primary] ?? null, scoreLabel: scoreLabel(primary, snapshot.scoreDefinitions) }),
     href: href({ caseId: node.caseId, name: 'case', runKey: snapshot.runKey, taskId: node.taskId }),
     id: node.caseId,
     label: node.name,
@@ -113,7 +121,7 @@ export function switcherOptions(position: HierarchyPosition, runs: readonly RunL
   if (workspace != null && project != null) {
     result.project = group(inWorkspace, run => run.projectSegment).map(([id, list]) => ({
       detail: `${new Set(list.map(run => run.experimentId)).size} experiment(s) · ${list.length} run(s)`,
-      exact: list[0]!.exactAverage,
+      ...runScore(list[0]!),
       href: href({ name: 'scope', project: id, workspace }),
       id,
       label: id,
@@ -128,7 +136,7 @@ export function switcherOptions(position: HierarchyPosition, runs: readonly RunL
       return {
         chips: label.textIsMatrix ? undefined : label.chips,
         detail: `${new Set(list.map(run => run.attemptId)).size} attempt(s)`,
-        exact: list[0]!.exactAverage,
+        ...runScore(list[0]!),
         href: href({ experiment: id, name: 'scope', project, workspace }),
         id,
         label: label.text,
@@ -141,7 +149,7 @@ export function switcherOptions(position: HierarchyPosition, runs: readonly RunL
   if (workspace != null && project != null && experiment != null && attempt != null) {
     result.attempt = group(inExperiment, run => run.attemptId).map(([id, list]) => ({
       counts: list[0]!.counts,
-      exact: list[0]!.exactAverage,
+      ...runScore(list[0]!),
       href: href({ attempt: id, experiment, name: 'scope', project, workspace }),
       id,
       label: attemptLabel(id),
@@ -155,7 +163,7 @@ export function switcherOptions(position: HierarchyPosition, runs: readonly RunL
     result.run = siblings.map(run => ({
       counts: run.counts,
       detail: attemptLabel(run.attemptId),
-      exact: run.exactAverage,
+      ...runScore(run),
       href: href({ name: 'run', runKey: run.runKey }),
       id: run.runKey,
       label: runLabel(run),
@@ -178,3 +186,15 @@ function group(runs: readonly RunListItem[], key: (run: RunListItem) => string):
   }
   return [...groups.entries()]
 }
+
+/** The headline value and label of an option; `per-task` headlines show the completion rate (see `headlineView`). */
+function headlineScore(result: HeadlineResult | null, definitions: readonly ScoreDefinition[]): Pick<SwitcherOption, 'score' | 'scoreLabel'> {
+  if (result == null)
+    return { score: null }
+  const view = headlineView(result, definitions)
+  return { score: view.summary.mean, scoreLabel: view.label }
+}
+
+function runScore(run: RunListItem): Pick<SwitcherOption, 'score' | 'scoreLabel'> {
+  return headlineScore(run.headline, run.headlineDefinitions)
+}
diff --git a/packages/ui/src/live.test.ts b/packages/ui/src/live.test.ts
index 92ac3e9..47690c8 100644
--- a/packages/ui/src/live.test.ts
+++ b/packages/ui/src/live.test.ts
@@ -47,11 +47,13 @@ export default defineConfig({
 import process from 'node:process'
 import { existsSync } from 'node:fs'
 import { setTimeout as delay } from 'node:timers/promises'
-import { caseOf, describeTask } from '${vievalEntry}'
+import { caseOf, defineScore, describeTask } from '${vievalEntry}'
 
 // The test creates this file once it has seen the run in progress; the slow case waits for it.
 const gate = ${JSON.stringify(join(projectDirectory, 'release-slow-case'))}
 
+const quality = defineScore('demo.quality', { grader: { id: 'judge', kind: 'model' }, label: 'Quality' })
+
 const runs = new Map<string, number>()
 const count = (name: string) => {
   const value = (runs.get(name) ?? 0) + 1
@@ -71,7 +73,7 @@ describeTask('demo', () => {
     await context.recordArtifact('page.html', { body: '<h1>report</h1><script>alert(1)</script>', contentType: 'text/html' })
     await context.recordArtifact('config.yaml', { body: 'model: a\\n', contentType: 'application/yaml' })
     context.recordMetric('latency_ms', 42)
-    context.recordScore(0.75)
+    context.recordScore(quality, 0.75, { rationale: 'clear enough' })
     process.stderr.write('slow case done\\n')
   }, { autoAttempt: 1 })
 
@@ -158,7 +160,10 @@ describe('--with-viewer: in-process run on the event bus', () => {
       const byName = Object.fromEntries(task.cases.map(node => [node.name, node]))
       expect(byName['flaky/attempt'].rounds.map(round => [round.roundIndex, round.state, round.roundIndexInferred])).toEqual([[0, 'errored', false], [1, 'passed', false]])
       expect(byName['flaky/retry'].rounds.map(round => round.tries.map(entry => `${entry.retryIndex}:${entry.state}`))).toEqual([['0:retried', '1:passed'], ['0:retried', '1:passed']])
-      expect(byName['slow/artifacts'].scores).toEqual({ exact: 0.75 })
+      expect(byName['slow/artifacts'].scores).toEqual({ unscored: {}, values: { 'demo.quality': 0.75 } })
+      expect(task.scoring.primary).toBe('demo.quality')
+      expect(snapshot.scoreDefinitions).toEqual([expect.objectContaining({ grader: { id: 'judge', kind: 'model' }, label: 'Quality', name: 'demo.quality' })])
+      expect(snapshot.headline?.headline).toMatchObject({ kind: 'score', score: { name: 'demo.quality' } })
 
       // The run is listed once, from the bus, even though its report directory exists too.
       const runs = await api<RunListItem[]>('/api/runs')
@@ -173,6 +178,7 @@ describe('--with-viewer: in-process run on the event bus', () => {
 
       const slow = byName['slow/artifacts']
       const detail = v.parse(caseDetailSchema, await api<CaseDetail>(`/api/runs/${runKey}/case?taskId=${encodeURIComponent(task.id)}&caseId=${encodeURIComponent(slow.caseId)}`))
+      expect(detail.scoreRecords.map(entry => entry.record)).toEqual([{ name: 'demo.quality', rationale: 'clear enough', value: 0.75 }, { name: 'demo.quality', rationale: 'clear enough', value: 0.75 }])
       expect(detail.artifacts.map(artifact => [artifact.name, artifact.previewKind]).sort()).toEqual(
         [['config.yaml', 'yaml'], ['config.yaml', 'yaml'], ['page.html', 'html'], ['page.html', 'html'], ['rows.jsonl', 'jsonl'], ['rows.jsonl', 'jsonl']],
       )
diff --git a/packages/ui/src/server/app.test.ts b/packages/ui/src/server/app.test.ts
index ce9208f..6b528e6 100644
--- a/packages/ui/src/server/app.test.ts
+++ b/packages/ui/src/server/app.test.ts
@@ -49,6 +49,8 @@ const RUNNING = 'run-1791747103580-dd3a8517'
 const ARCHIVED = 'run-1791746045991-a77ba0a4'
 const LEGACY = 'run-1791656450009-5299b70e'
 const FAILED_CASE = '1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122'
+/** The one score the OSWorld runs record, without a `score.defined`: the primary score as the task's only score. */
+const SCORE = 'osworld.success'
 /** Shortly after the last event of the running fixture (19:39:57Z), so the run is `running`, not `stale`. */
 const NOW = Date.parse('2026-10-11T19:45:00.000Z')
 
@@ -77,26 +79,36 @@ describe('viewer API on the OSWorld report root (real data)', () => {
     expect(project.experiments.map(experiment => experiment.id)).toEqual(['matrix-run-model-gpt-5.5-repeat-1'])
     const attempts = project.experiments[0].attempts
     expect(attempts.map(attempt => [attempt.id, attempt.runs.map(run => [run.runKey, run.source, run.state])])).toEqual([
-      ['attempt-2026-10-10T18-20-50-009Z', [[LEGACY, 'legacy', 'legacy']]],
+      ['attempt-2026-10-10T18-20-50-009Z', [[LEGACY, 'unsupported', 'unsupported']]],
       ['attempt-2026-10-11T19-14-05-991Z', [[ARCHIVED, 'offline', 'interrupted']]],
       ['attempt-2026-10-11T19-31-43-580Z', [[RUNNING, 'live-tail', 'running']]],
     ])
   })
 
-  it('lists runs with counts and scores that match run-summary.json', async () => {
+  /**
+   * @example
+   * expect(byKey[ARCHIVED].headline?.headline).toEqual({ kind: 'score', score: { mean: 11 / 18, n: 18, name: SCORE, ... } })
+   */
+  it('lists runs with counts and the headline of run-summary.json, else of their events', async () => {
     const { json } = await setup()
     const runs = (await json<unknown[]>('/api/runs')).map(item => expectSchema(runListItemSchema, item))
     const byKey = Object.fromEntries(runs.map(run => [run.runKey, run]))
     // An archived report written before `errored` existed counts every failure as `failed`.
     expect(byKey[ARCHIVED].counts).toEqual({ errored: 0, failed: 7, passed: 11, requeued: 0, running: 0, skipped: 0, timeout: 0, total: 18 })
-    expect(byKey[ARCHIVED].exactAverage).toBeCloseTo(0.6111111111111112, 12)
+    expect(byKey[ARCHIVED].headline).toEqual({
+      completion: { mean: 1, n: 18, name: 'completion', sum: 18, total: 18, unscored: {} },
+      headline: { kind: 'score', score: { mean: 11 / 18, n: 18, name: SCORE, sum: 11, total: 18, unscored: {} } },
+    })
+    // The score has no `score.defined`, so there is no definition to show with it.
+    expect(byKey[ARCHIVED].headlineDefinitions).toEqual([])
     expect(byKey[ARCHIVED].reportPath).toBe(OSWORLD_RUNS.archived)
-    expect(byKey[LEGACY].counts).toMatchObject({ failed: 1, passed: 2, total: 3 })
-    expect(byKey[LEGACY].exactAverage).toBeCloseTo(2 / 3, 12)
     expect(byKey[RUNNING].counts).toEqual({ errored: 0, failed: 2, passed: 4, requeued: 0, running: 8, skipped: 0, timeout: 0, total: 14 })
+    // No run-summary.json yet: the headline comes from the six settled cases.
+    expect(byKey[RUNNING].headline?.headline).toMatchObject({ kind: 'score', score: { n: 6, name: SCORE, total: 6 } })
     expect(byKey[RUNNING].lastSeq).toBe(150)
     expect(byKey[RUNNING].matrix).toEqual({ model: ['gpt-5.5'], repeat: ['1'] })
-    expect(byKey[LEGACY].matrix).toEqual({ model: ['gpt-5.5'], repeat: ['1'] })
+    // A run written by an older vieval is listed, without reading its events.
+    expect(byKey[LEGACY]).toMatchObject({ counts: { total: 0 }, headline: null, matrix: {}, source: 'unsupported', state: 'unsupported' })
     expect((await json<unknown[]>('/api/runs?state=running')).length).toBe(1)
     expect((await json<unknown[]>('/api/runs?attempt=attempt-2026-10-11T19-14-05-991Z')).length).toBe(1)
   })
@@ -143,9 +155,22 @@ describe('viewer API on the OSWorld report root (real data)', () => {
     for (let index = 1; index < history.length; index += 1) {
       expect(history[index].elapsedMs).toBeGreaterThanOrEqual(history[index - 1].elapsedMs)
     }
+  })
 
-    const legacy = expectSchema(runSnapshotSchema, await json(`/api/runs/${LEGACY}/snapshot`)).progress
-    expect(legacy.history.at(-1)).toEqual({ completed: 3, elapsedMs: legacy.elapsedMs })
+  /**
+   * @example
+   * expect(response.status).toBe(422)
+   */
+  it('lists a run of an older event schema version as unsupported and does not open it', async () => {
+    const { get, json } = await setup()
+    const item = expectSchema(runListItemSchema, await json(`/api/runs/${LEGACY}`))
+    expect([item.source, item.state]).toEqual(['unsupported', 'unsupported'])
+    for (const path of [`/api/runs/${LEGACY}/snapshot`, `/api/runs/${LEGACY}/scores`, `/api/runs/${LEGACY}/events`, `/api/compare/runs?runs=${LEGACY},${ARCHIVED}`]) {
+      const response = await get(path)
+      expect([path, response.status]).toEqual([path, 422])
+      const error = expectSchema(errorResponseSchema, await response.json())
+      expect(error.message).toBe(`This report was written by an older vieval. Run it again. (Run "${LEGACY}" has event schema version 1.)`)
+    }
   })
 
   /**
@@ -155,7 +180,8 @@ describe('viewer API on the OSWorld report root (real data)', () => {
   it('returns the matrix of a project: one cell per matrix row, latest run and history across attempts', async () => {
     const { json } = await setup()
     const matrix = expectSchema(matrixResponseSchema, await json('/api/matrix?workspace=default-workspace&project=osworld-kubernetes'))
-    expect(matrix.runCount).toBe(3)
+    // The run of an older vieval is not read.
+    expect(matrix.runCount).toBe(2)
     expect(matrix.axes.filter(axis => axis.kind === 'run')).toEqual([
       { kind: 'run', name: 'model', values: ['gpt-5.5'] },
       { kind: 'run', name: 'repeat', values: ['1'] },
@@ -165,12 +191,15 @@ describe('viewer API on the OSWorld report root (real data)', () => {
     expect(cell.run).toEqual({ model: 'gpt-5.5', repeat: '1' })
     expect(cell.latest).toMatchObject({ attemptId: 'attempt-2026-10-11T19-31-43-580Z', runKey: RUNNING, state: 'running', taskId: OSWORLD_TASK_ID })
     expect(cell.latest.counts).toMatchObject({ failed: 2, passed: 4, running: 8, total: 14 })
-    expect(cell.history.map(point => point.runKey)).toEqual([LEGACY, ARCHIVED, RUNNING])
-    expect(cell.history[1].exact).toBeCloseTo(0.6111111111111112, 12)
+    expect(cell.history.map(point => point.runKey)).toEqual([ARCHIVED, RUNNING])
+    expect(cell.history[0].score).toBeCloseTo(11 / 18, 12)
+    expect(cell.history[0].completion).toBe(1)
+    expect(cell.latest.scoring.primary).toBe(SCORE)
+    expect(matrix.scoreDefinitions).toEqual([])
 
     // Filters narrow the scope like `/api/runs`; an unknown workspace is an empty matrix, no filter is every run.
     expect(expectSchema(matrixResponseSchema, await json('/api/matrix?workspace=other')).cells).toEqual([])
-    expect(expectSchema(matrixResponseSchema, await json('/api/matrix')).runCount).toBe(3)
+    expect(expectSchema(matrixResponseSchema, await json('/api/matrix')).runCount).toBe(2)
   })
 
   it('returns task, case detail with input, metrics, scores, and evidence of a finished run', async () => {
@@ -179,6 +208,7 @@ describe('viewer API on the OSWorld report root (real data)', () => {
     expect(snapshot.state).toBe('interrupted')
     expect(snapshot.interruptSignal).toBe('SIGINT')
     expect(snapshot.summary).toMatchObject({ caseSummary: { failed: 7, passed: 11, total: 18 } })
+    expect(snapshot.summary?.headline).toEqual(snapshot.headline)
     expect(snapshot.progress.basis).toBe('ended')
 
     const task = expectSchema(taskNodeSchema, await json(`/api/runs/${ARCHIVED}/task?taskId=${q(OSWORLD_TASK_ID)}`))
@@ -194,6 +224,9 @@ describe('viewer API on the OSWorld report root (real data)', () => {
     expect(detail.durationMs).toBe(Date.parse('2026-10-11T19:16:32.903Z') - Date.parse('2026-10-11T19:14:05.993Z'))
     expect((detail.input as { task: { id: string } }).task.id).toBe('06fe7178-4491-4589-810f-2e2bc9502122')
     expect(detail.metrics.find(metric => metric.name === 'osworld.pod')?.value).toBe('osworld-06fe7178-624029')
+    expect(detail.scores).toEqual({ unscored: {}, values: { [SCORE]: 0 } })
+    expect(detail.scoreRecords).toEqual([{ record: { name: SCORE, value: 0 }, roundIndex: 0, seq: 32 }])
+    expect(detail.scoreDefinitions).toEqual([])
     expect(detail.evidence).toHaveLength(1)
     expect(detail.evidence[0]).toMatchObject({ adapter: 'osworld', allowed: true, exists: true, kind: 'directory', metricName: 'osworld.evidence' })
     expect(detail.evidence[0].summary).toMatchObject({ podName: 'osworld-06fe7178-624029', status: 'evaluated', taskId: '06fe7178-4491-4589-810f-2e2bc9502122' })
@@ -246,14 +279,20 @@ describe('viewer API on the OSWorld report root (real data)', () => {
   it('returns scores, metrics, timeline, events, and logs', async () => {
     const { json } = await setup()
     const scores = expectSchema(scoresResponseSchema, await json(`/api/runs/${ARCHIVED}/scores`))
-    expect(scores.overall.exact).toBeCloseTo(11 / 18, 12)
-    expect(scores.summary?.exact).toBeCloseTo(0.6111111111111112, 12)
+    expect(scores.overall.scores).toEqual([{ mean: 11 / 18, n: 18, name: SCORE, sum: 11, total: 18, unscored: {} }])
+    expect(scores.overall.headline).toEqual(scores.summary)
     expect(scores.byRound).toHaveLength(1)
+    expect(scores.byRound[0].scoring.primary).toBe(SCORE)
     expect(scores.cases).toHaveLength(18)
 
     const metrics = expectSchema(metricsResponseSchema, await json(`/api/runs/${ARCHIVED}/metrics`))
     expect(metrics.numeric.find(metric => metric.name === 'osworld.agent.steps')?.summary.count).toBe(15)
     expect(metrics.nonNumeric.map(metric => metric.name)).toContain('osworld.evidence')
+    // Case 2 recorded 0.94 but failed, so it counts 0, like in the task's mean.
+    const [success] = metrics.scores
+    expect(success.summary).toEqual(scores.overall.scores[0])
+    expect(success.points).toHaveLength(18)
+    expect(success.points.find(point => point.caseId.startsWith('2:'))?.value).toBe(0)
 
     const timeline = expectSchema(timelineResponseSchema, await json(`/api/runs/${ARCHIVED}/timeline`))
     expect(timeline.entries).toHaveLength(18)
@@ -271,13 +310,16 @@ describe('viewer API on the OSWorld report root (real data)', () => {
   it('compares attempts of one experiment and experiments of one project', async () => {
     const { json } = await setup()
     const attempts = expectSchema(comparisonResponseSchema, await json('/api/compare/attempts?workspace=default-workspace&project=osworld-kubernetes&experiment=matrix-run-model-gpt-5.5-repeat-1'))
-    expect(attempts.columns.map(column => column.id)).toEqual([LEGACY, ARCHIVED, RUNNING])
+    // The run of an older vieval is no column.
+    expect(attempts.columns.map(column => column.id)).toEqual([ARCHIVED, RUNNING])
+    expect(attempts.primary).toBe(SCORE)
     const row = attempts.rows.find(candidate => candidate.caseName === 'chrome/06fe7178-4491-4589-810f-2e2bc9502122')
-    expect(row?.cells[ARCHIVED]).toEqual({ exact: 0, roundCount: 1, state: 'failed' })
+    expect(row?.cells[ARCHIVED]).toEqual({ roundCount: 1, scores: { unscored: {}, values: { [SCORE]: 0 } }, state: 'failed' })
+    expect(attempts.columns[0].runs[0].headline?.headline.kind).toBe('score')
     expect(attempts.summary.rows).toBeGreaterThan(18)
 
     const experiments = expectSchema(comparisonResponseSchema, await json('/api/compare/experiments?workspace=default-workspace&project=osworld-kubernetes'))
-    expect(experiments.columns.map(column => [column.id, column.runs.length])).toEqual([['matrix-run-model-gpt-5.5-repeat-1', 3]])
+    expect(experiments.columns.map(column => [column.id, column.runs.length])).toEqual([['matrix-run-model-gpt-5.5-repeat-1', 2]])
 
     const runs = expectSchema(comparisonResponseSchema, await json(`/api/compare/runs?runs=${ARCHIVED},${RUNNING}`))
     expect(runs.columns.map(column => column.id)).toEqual([ARCHIVED, RUNNING])
@@ -293,6 +335,8 @@ describe('viewer API on the OSWorld report root (real data)', () => {
     expect(logs.records.filter(record => record.severity === 'ERROR')).toHaveLength(7)
     const metrics = expectSchema(otelMetricsResponseSchema, await json(`/api/runs/${ARCHIVED}/otel/metrics`))
     expect(metrics.metrics.map(metric => metric.name)).toContain('vieval.case.duration_ms')
+    expect(metrics.metrics.find(metric => metric.name === `vieval.score.${SCORE}`)?.dataPoints).toHaveLength(15)
+    expect(traces.spans.find(span => span.kind === 'task')?.attributes).toMatchObject({ 'vieval.score.osworld.success': 11 / 18, 'vieval.task.primary_score': SCORE })
     const raw = await get(`/api/runs/${ARCHIVED}/otel/raw/traces`)
     expect(raw.status).toBe(200)
     expect((await raw.json() as { resourceSpans: unknown[] }).resourceSpans).toHaveLength(1)
diff --git a/packages/ui/src/server/app.ts b/packages/ui/src/server/app.ts
index 9f9e083..cb1f363 100644
--- a/packages/ui/src/server/app.ts
+++ b/packages/ui/src/server/app.ts
@@ -17,7 +17,7 @@ import { ViewerError } from './errors'
 import { contentTypeFromName, listTree, readJsonlPage, resolveInsideRoot, serveFile } from './files'
 import { buildOtelLogs, buildOtelMetrics, buildOtelTraces } from './otel'
 import { createImagePreviews } from './preview'
-import { toRunListItem } from './store'
+import { toRunListItem, UNSUPPORTED_RUN_MESSAGE } from './store'
 import { buildCaseDetail, buildLogs, buildMetrics, buildProgress, buildScores, buildSnapshot, buildTask, buildTimeline, findCase, listArtifacts, MAX_LOG_LIMIT } from './views'
 
 /** Most events one `/events` page returns. */
@@ -82,12 +82,19 @@ export function createViewerApp(options: CreateViewerAppOptions): H3 {
     }
   })
 
-  async function runFrom(event: H3Event): Promise<RunHandle> {
+  /**
+   * The run a route reads, or 404. A run written by an older vieval is listed (`/api/runs/:runKey`) but not opened:
+   * every route that reads its contents answers 422.
+   */
+  async function runFrom(event: H3Event, options: { listOnly?: boolean } = {}): Promise<RunHandle> {
     const runKey = param(event, 'runKey')
     const handle = await store.get(runKey)
     if (handle == null) {
       throw new ViewerError(404, `Run "${runKey}" not found.`)
     }
+    if (options.listOnly !== true) {
+      assertSupported(handle)
+    }
     return handle
   }
 
@@ -155,7 +162,7 @@ export function createViewerApp(options: CreateViewerAppOptions): H3 {
       .sort((left, right) => (right.startedAt ?? '').localeCompare(left.startedAt ?? ''))
   })
 
-  app.get('/api/runs/:runKey', async event => toRunListItem(await runFrom(event), now()))
+  app.get('/api/runs/:runKey', async event => toRunListItem(await runFrom(event, { listOnly: true }), now()))
   app.get('/api/runs/:runKey/snapshot', async event => buildSnapshot(await runFrom(event), evidence, now()))
   app.get('/api/runs/:runKey/progress', async event => buildProgress(await runFrom(event), now()))
 
@@ -333,10 +340,10 @@ export function createViewerApp(options: CreateViewerAppOptions): H3 {
   app.get('/api/matrix', async (event) => {
     const query = getQuery(event) as Record<string, string | undefined>
     const at = now()
-    const runs = (await store.list()).map((handle) => {
+    const runs = (await store.list()).filter(handle => handle.unsupportedVersion() == null).map((handle) => {
       // Built once and shared by the list item and the matrix: building task nodes walks every case.
       const tasks = buildTaskNodes(handle.reducer(), () => 0)
-      return { item: toRunListItem(handle, at, tasks), tasks }
+      return { item: toRunListItem(handle, at, tasks), scoreDefinitions: handle.reducer().scoreDefinitions, tasks }
     })
     return buildMatrix(runs.filter(run => inScope(run.item, query)))
   })
@@ -364,6 +371,7 @@ export function createViewerApp(options: CreateViewerAppOptions): H3 {
       if (handle == null) {
         throw new ViewerError(404, `Run "${key}" not found.`)
       }
+      assertSupported(handle)
       return handle
     }))
     return compareRuns(handles, now())
@@ -404,6 +412,13 @@ function artifactsRootOf(handle: RunHandle): string {
   return root
 }
 
+function assertSupported(handle: RunHandle): void {
+  const version = handle.unsupportedVersion()
+  if (version != null) {
+    throw new ViewerError(422, `${UNSUPPORTED_RUN_MESSAGE} (Run "${handle.runKey}" has event schema version ${version}.)`)
+  }
+}
+
 function getOrCreate<K, V>(map: Map<K, V>, key: K, create: () => V): V {
   let value = map.get(key)
   if (value === undefined) {
diff --git a/packages/ui/src/server/compare.test.ts b/packages/ui/src/server/compare.test.ts
index 5b8b31c..5f91c0d 100644
--- a/packages/ui/src/server/compare.test.ts
+++ b/packages/ui/src/server/compare.test.ts
@@ -1,31 +1,47 @@
-import type { RunListItem, TaskNode } from '../shared/contract'
+import type { RunListItem, ScoreDefinition, TaskNode } from '../shared/contract'
+import type { RunReducer } from '../shared/reduce'
 
 import { describe, expect, it } from 'vitest'
 
-import { createEnvelopeFactory } from '../../test/helpers'
+import { createEnvelopeFactory, createReducerHandle } from '../../test/helpers'
 import { buildTaskNodes, createRunReducer } from '../shared/reduce'
-import { buildMatrix } from './compare'
+import { buildMatrix, compareRuns } from './compare'
 
 const TASK = 'evals%2Fdemo::default::run=tts%3Dofficial::eval=asr%3Dwhisper-1'
 const TASK_WITH_JUDGE = 'evals%2Fdemo::default::run=tts%3Dofficial::eval=judge%3Dgpt-5.5'
 
+/** Score every case of the test runs records, unless a case names other values. */
+const SCORE = 'demo.score'
+
 interface CaseResult {
-  exact?: number
+  /** Recorded score values by name. */
+  scores?: Record<string, number>
   state: 'errored' | 'failed' | 'passed'
 }
 
+interface TestRun {
+  item: RunListItem
+  reducer: RunReducer
+  scoreDefinitions: ReadonlyMap<string, ScoreDefinition>
+  tasks: TaskNode[]
+}
+
 /** One ended run of {@link TASK}: the given cases (named `<prefix>-<index>`) settle, out of `total` expected ones. */
-function runOf(runKey: string, startedAt: string, total: number, results: readonly CaseResult[], options: { prefix?: string, taskId?: string } = {}): { item: RunListItem, tasks: TaskNode[] } {
+function runOf(runKey: string, startedAt: string, total: number, results: readonly CaseResult[], options: { definitions?: ScoreDefinition[], prefix?: string, taskId?: string } = {}): TestRun {
   const { prefix = 'case', taskId = TASK } = options
   const envelope = createEnvelopeFactory(runKey)
   const reducer = createRunReducer()
   const taskScope = { projectId: 'demo', taskId }
   reducer.apply(envelope('task.started', {}, taskScope))
+  for (const definition of options.definitions ?? []) {
+    reducer.apply(envelope('score.defined', definition, taskScope))
+  }
   results.forEach((result, index) => {
     const scope = { caseId: `${index}:${prefix}-${index}`, projectId: 'demo', taskId }
     reducer.apply(envelope('case.started', { index, name: `${prefix}-${index}`, total }, scope))
-    if (result.exact != null)
-      reducer.apply(envelope('score.recorded', { kind: 'exact', score: result.exact }, scope))
+    for (const [name, value] of Object.entries(result.scores ?? {})) {
+      reducer.apply(envelope('score.recorded', { name, value }, scope))
+    }
     reducer.apply(envelope('case.ended', { index, name: `${prefix}-${index}`, state: result.state, total }, scope))
   })
   reducer.apply(envelope('task.ended', { state: 'passed' }, taskScope))
@@ -33,10 +49,9 @@ function runOf(runKey: string, startedAt: string, total: number, results: readon
     attemptId: `attempt-${runKey}`,
     counts: { errored: 0, failed: 0, passed: 0, requeued: 0, running: 0, skipped: 0, timeout: 0, total: 0 },
     evalMatrix: {},
-    exactAverage: null,
     experimentId: 'exp-a',
-    hybridAverage: null,
-    judgeAverage: null,
+    headline: null,
+    headlineDefinitions: [],
     lastSeq: 0,
     matrix: {},
     projectIds: ['demo'],
@@ -50,10 +65,10 @@ function runOf(runKey: string, startedAt: string, total: number, results: readon
     state: 'ended',
     workspaceId: 'ws-a',
   }
-  return { item, tasks: buildTaskNodes(reducer, () => 0) }
+  return { item, reducer, scoreDefinitions: reducer.scoreDefinitions, tasks: buildTaskNodes(reducer, () => 0) }
 }
 
-const scored = (count: number, exact: number): CaseResult[] => Array.from({ length: count }, () => ({ exact, state: 'passed' }))
+const scored = (count: number, value: number): CaseResult[] => Array.from({ length: count }, () => ({ scores: { [SCORE]: value }, state: 'passed' }))
 
 describe('buildMatrix', () => {
   // ROOT CAUSE:
@@ -179,4 +194,91 @@ describe('buildMatrix', () => {
     expect(cell.caseCount).toBe(3)
     expect(cell.history.map(point => point.coverage)).toEqual(['full', 'other', 'full'])
   })
+
+  /**
+   * @example
+   * expect(cell.latest.caseScores).toEqual([1, 0, null])
+   */
+  it('scores each run of a cell by its primary score, with the completion rate next to it', () => {
+    const definitions = [{ label: 'Correct', name: 'demo.correct' }, { name: SCORE, primary: true }]
+    const old = runOf('run-old', '2026-10-19T06:36:00.000Z', 2, scored(2, 0.5), { definitions })
+    const fresh = runOf('run-new', '2026-10-19T07:00:00.000Z', 3, [
+      { scores: { 'demo.correct': 1, [SCORE]: 1 }, state: 'passed' },
+      { scores: { [SCORE]: 1 }, state: 'failed' },
+      { state: 'errored' },
+    ], { definitions })
+
+    const [cell] = buildMatrix([old, fresh]).cells
+    expect(cell.history.map(point => [point.runKey, point.score, point.completion])).toEqual([['run-old', 0.5, 1], ['run-new', 0.5, 2 / 3]])
+    expect(cell.latest.scoring.primary).toBe(SCORE)
+    // Per case, the value it counts with: the failed case 0, the errored one none.
+    expect(cell.latest.caseScores).toEqual([1, 0, null])
+  })
+
+  /**
+   * @example
+   * expect(buildMatrix(runs).scoreDefinitions.map(definition => definition.name)).toEqual(['demo.correct', 'demo.score'])
+   */
+  it('returns the newest definition of each score in the cells', () => {
+    const old = runOf('run-old', '2026-10-19T06:36:00.000Z', 1, scored(1, 1), { definitions: [{ label: 'Old', name: SCORE }] })
+    const fresh = runOf('run-new', '2026-10-19T07:00:00.000Z', 1, scored(1, 1), { definitions: [{ label: 'Correct', name: 'demo.correct' }, { label: 'New', name: SCORE }] })
+
+    expect(buildMatrix([old, fresh]).scoreDefinitions).toEqual([{ label: 'Correct', name: 'demo.correct' }, { label: 'New', name: SCORE }])
+  })
+})
+
+describe('compareRuns', () => {
+  /**
+   * @example
+   * expect(comparison.rows.map(row => [row.caseName, row.changed])).toEqual([['case-0', false], ['case-1', true], ['case-2', false]])
+   */
+  it('marks a row changed when the state or the value it counts with in the shared primary score differs', () => {
+    const left = runOf('run-a', '2026-10-19T06:36:00.000Z', 3, [
+      { scores: { 'demo.other': 0, [SCORE]: 1 }, state: 'passed' },
+      { scores: { [SCORE]: 0.5 }, state: 'passed' },
+      { scores: { [SCORE]: 1 }, state: 'failed' },
+    ], { definitions: [{ name: SCORE, primary: true }] })
+    const right = runOf('run-b', '2026-10-19T07:00:00.000Z', 3, [
+      // Another value of a score that is not primary does not change the row.
+      { scores: { 'demo.other': 1, [SCORE]: 1 }, state: 'passed' },
+      { scores: { [SCORE]: 1 }, state: 'passed' },
+      // Both failed: both count 0, whatever they recorded.
+      { scores: { [SCORE]: 0 }, state: 'failed' },
+    ], { definitions: [{ name: SCORE, primary: true }] })
+
+    const comparison = compareRuns([createReducerHandle(left.reducer), createReducerHandle(right.reducer)], Date.parse('2026-10-20T00:00:00.000Z'))
+    expect(comparison.primary).toBe(SCORE)
+    expect(comparison.rows.map(row => [row.caseName, row.changed])).toEqual([['case-0', false], ['case-1', true], ['case-2', false]])
+    expect(comparison.rows[0].cells['run-a']).toEqual({ roundCount: 1, scores: { unscored: {}, values: { 'demo.other': 0, [SCORE]: 1 } }, state: 'passed' })
+    expect(comparison.columns.map(column => column.runs[0].headline?.headline.kind)).toEqual(['score', 'score'])
+    expect(comparison.scoreDefinitions).toEqual([{ name: SCORE, primary: true }])
+  })
+
+  /**
+   * @example
+   * run-a judged by judge-a (definition), run-b by judge-b (definition) -> each column lists its own grader.
+   */
+  it('lists the graders of each column, so a comparison can tell columns judged by different models apart', () => {
+    const left = runOf('run-a', '2026-10-19T06:36:00.000Z', 1, scored(1, 1), { definitions: [{ grader: { kind: 'model', model: 'judge-a' }, name: SCORE, primary: true }] })
+    const right = runOf('run-b', '2026-10-19T07:00:00.000Z', 1, scored(1, 1), { definitions: [{ grader: { kind: 'model', model: 'judge-b' }, name: SCORE, primary: true }] })
+
+    const comparison = compareRuns([createReducerHandle(left.reducer), createReducerHandle(right.reducer)], Date.parse('2026-10-20T00:00:00.000Z'))
+    expect(comparison.columns.map(column => column.scoreGraders)).toEqual([
+      { [SCORE]: [{ kind: 'model', model: 'judge-a' }] },
+      { [SCORE]: [{ kind: 'model', model: 'judge-b' }] },
+    ])
+  })
+
+  /**
+   * @example
+   * expect(comparison.primary).toBeUndefined()
+   */
+  it('compares by state only when the columns have different primary scores', () => {
+    const left = runOf('run-a', '2026-10-19T06:36:00.000Z', 1, [{ scores: { 'demo.a': 1 }, state: 'passed' }])
+    const right = runOf('run-b', '2026-10-19T07:00:00.000Z', 1, [{ scores: { 'demo.b': 0 }, state: 'passed' }])
+
+    const comparison = compareRuns([createReducerHandle(left.reducer), createReducerHandle(right.reducer)], Date.parse('2026-10-20T00:00:00.000Z'))
+    expect(comparison.primary).toBeUndefined()
+    expect(comparison.rows.map(row => row.changed)).toEqual([false])
+  })
 })
diff --git a/packages/ui/src/server/compare.ts b/packages/ui/src/server/compare.ts
index 2427730..6062fc3 100644
--- a/packages/ui/src/server/compare.ts
+++ b/packages/ui/src/server/compare.ts
@@ -1,7 +1,7 @@
-import type { CellCoverage, ComparedRun, ComparisonCell, ComparisonResponse, MatrixCell, MatrixResponse, RunListItem, TaskNode } from '../shared/contract'
+import type { CellCoverage, ComparedRun, ComparisonCell, ComparisonResponse, MatrixCell, MatrixResponse, RunListItem, ScoreDefinition, ScoreGraders, TaskNode } from '../shared/contract'
 import type { RunHandle } from './store'
 
-import { axisValues, buildTaskNodes, compareNatural } from '../shared/reduce'
+import { addScoreGrader, axisValues, buildTaskNodes, compareNatural, countedScores, countingDefinitions, pickScoreDefinitions } from '../shared/reduce'
 import { toRunListItem } from './store'
 
 /** One run of a matrix cell: its result, and the cases it ran (by `CaseNode.identity`). */
@@ -23,7 +23,7 @@ interface Column {
  * - `GET /api/matrix` shows how every matrix combination did, across experiments and attempts
  *
  * Expects:
- * - each run with the task nodes it was listed with (built once per run by the caller)
+ * - each run with the task nodes it was listed with (built once per run by the caller), and its score definitions
  *
  * Returns:
  * - cells ordered by entry, then run row and eval row in natural order; `history` is every run with a task in the
@@ -32,31 +32,42 @@ interface Column {
  *   smoke or limited run, another case set) is `other`, not `full`: its score is not comparable. Tasks without a
  *   parsable matrix are skipped.
  *   Tasks of one run that share a cell (several inference executors) keep the last one in queue order.
+ *   Each run is scored by its task's primary score (`score`, and `caseScores` per case by the rule of `scoreCases`),
+ *   with its completion rate next to it.
  */
-export function buildMatrix(runs: readonly { item: RunListItem, tasks: readonly TaskNode[] }[]): MatrixResponse {
+export function buildMatrix(runs: readonly { item: RunListItem, scoreDefinitions: ReadonlyMap<string, ScoreDefinition>, tasks: readonly TaskNode[] }[]): MatrixResponse {
   const oldestFirst = [...runs].sort((left, right) => (left.item.startedAt ?? '').localeCompare(right.item.startedAt ?? ''))
   // Runs of each cell, oldest first, with the cases each one ran; whether a run is `full` depends on the cases of the
   // cell's other runs, so it is settled after every run was read.
   const cells = new Map<string, { entryId: string, eval: Record<string, string>, evalRowId: string, run: Record<string, string>, runRowId: string, runs: CellRun[] }>()
   const runRows: Record<string, string>[] = []
   const evalRows: Record<string, string>[] = []
-  for (const { item, tasks } of oldestFirst) {
+  // Newest definition of each score, since runs are read oldest first.
+  const definitions = new Map<string, ScoreDefinition>()
+  for (const { item, scoreDefinitions, tasks } of oldestFirst) {
+    for (const [name, definition] of scoreDefinitions) {
+      definitions.set(name, definition)
+    }
     for (const task of tasks) {
       if (task.matrix == null || task.entryId == null) {
         continue
       }
       runRows.push(task.matrix.run)
       evalRows.push(task.matrix.eval)
+      const primary = task.scoring.primary
+      const taskDefinitions = countingDefinitions(task.scoring, scoreDefinitions)
       const result: MatrixCell['latest'] = {
         attemptId: item.attemptId,
-        caseScores: task.cases.map(node => node.scores.exact ?? null),
+        caseScores: task.cases.map((node) => {
+          const round = node.rounds.at(-1)
+          return primary == null || round == null ? null : countedScores(round, taskDefinitions)[primary] ?? null
+        }),
         caseTotal: task.caseTotal,
         counts: task.caseCounts,
         coverage: coverageOf(task),
-        exact: task.scores.exact,
         experimentId: item.experimentId,
-        judge: task.scores.judge,
         runKey: item.runKey,
+        scoring: task.scoring,
         startedAt: item.startedAt,
         state: task.state,
         taskId: task.id,
@@ -105,7 +116,16 @@ export function buildMatrix(runs: readonly { item: RunListItem, tasks: readonly
     return {
       ...cell,
       caseCount: reference?.cases.size ?? Math.max(latest.counts.total, latest.caseTotal ?? 0),
-      history: settled.map(run => ({ attemptId: run.attemptId, counts: run.counts, coverage: run.coverage, exact: run.exact, runKey: run.runKey, startedAt: run.startedAt, state: run.state })),
+      history: settled.map(run => ({
+        attemptId: run.attemptId,
+        completion: run.scoring.completion.mean,
+        counts: run.counts,
+        coverage: run.coverage,
+        runKey: run.runKey,
+        score: primaryMean(run.scoring),
+        startedAt: run.startedAt,
+        state: run.state,
+      })),
       latest,
       superseded: newestOf(cellRuns) < (rowNewest.get(rowKey(cell)) ?? ''),
     }
@@ -120,6 +140,7 @@ export function buildMatrix(runs: readonly { item: RunListItem, tasks: readonly
       || compareNatural(left.runRowId, right.runRowId)
       || compareNatural(left.evalRowId, right.evalRowId)),
     runCount: runs.length,
+    scoreDefinitions: pickScoreDefinitions(definitions, output.flatMap(cell => cell.latest.scoring.scores.map(score => score.name))),
   }
 }
 
@@ -160,9 +181,16 @@ export function compareRuns(handles: readonly RunHandle[], now: number): Compari
   }), now)
 }
 
+/**
+ * Case rows of the columns; a row is `changed` when its state differs between columns, or, when every scored task
+ * shares one primary score, the value the case counts with in it (a failed case counts 0).
+ */
 function buildComparison(view: ComparisonResponse['view'], columns: readonly Column[], now: number): ComparisonResponse {
-  const rows = new Map<string, Record<string, ComparisonCell>>()
+  // Per case name: the cell of each column, and the primary value the cell counts with.
+  const rows = new Map<string, { cells: Record<string, ComparisonCell>, counted: Record<string, Record<string, number>> }>()
   const outputColumns: ComparisonResponse['columns'] = []
+  const definitions = new Map<string, ScoreDefinition>()
+  const primaries = new Set<string>()
   for (const column of columns) {
     const runs: ComparedRun[] = column.handles.map((handle) => {
       const item = toRunListItem(handle, now)
@@ -170,36 +198,51 @@ function buildComparison(view: ComparisonResponse['view'], columns: readonly Col
         attemptId: item.attemptId,
         counts: item.counts,
         durationMs: item.startedAt != null && item.endedAt != null ? Date.parse(item.endedAt) - Date.parse(item.startedAt) : undefined,
-        exactAverage: item.exactAverage,
         experimentId: item.experimentId,
-        hybridAverage: item.hybridAverage,
-        judgeAverage: item.judgeAverage,
+        headline: item.headline,
         runKey: item.runKey,
         startedAt: item.startedAt,
         state: item.state,
       }
     })
-    outputColumns.push({ id: column.id, label: column.label, runs })
+    const scoreGraders: ScoreGraders = {}
+    outputColumns.push({ id: column.id, label: column.label, runs, scoreGraders })
     // Older runs first, so a newer run overwrites the cells of the same case.
     for (const handle of column.handles) {
-      for (const task of buildTaskNodes(handle.reducer(), () => 0)) {
+      const reducer = handle.reducer()
+      for (const [name, definition] of reducer.scoreDefinitions) {
+        definitions.set(name, definition)
+      }
+      for (const task of buildTaskNodes(reducer, () => 0)) {
+        for (const [name, graders] of Object.entries(task.scoreGraders)) {
+          graders.forEach(grader => addScoreGrader(scoreGraders, name, grader))
+        }
+        if (task.scoring.primary != null) {
+          primaries.add(task.scoring.primary)
+        }
+        const taskDefinitions = countingDefinitions(task.scoring, reducer.tasks.get(task.id)?.scoreDefinitions ?? new Map())
         for (const node of task.cases) {
-          const cells = rows.get(node.name) ?? {}
-          cells[column.id] = { exact: node.scores.exact, judge: node.scores.judge, roundCount: node.rounds.length, state: node.state }
-          rows.set(node.name, cells)
+          const row = rows.get(node.name) ?? { cells: {}, counted: {} }
+          const round = node.rounds.at(-1)
+          row.cells[column.id] = { roundCount: node.rounds.length, scores: node.scores, state: node.state }
+          row.counted[column.id] = round == null ? {} : countedScores(round, taskDefinitions)
+          rows.set(node.name, row)
         }
       }
     }
   }
-  const outputRows = [...rows].sort((left, right) => left[0].localeCompare(right[0])).map(([caseName, cells]) => {
+  const primary = primaries.size === 1 ? [...primaries][0] : undefined
+  const outputRows = [...rows].sort((left, right) => left[0].localeCompare(right[0])).map(([caseName, { cells, counted }]) => {
     const values = Object.values(cells)
     const states = new Set(values.map(cell => cell.state))
-    const scores = new Set(values.map(cell => cell.exact ?? (cell.state === 'passed' ? 1 : 0)))
+    const scores = new Set(Object.values(counted).map(value => primary == null ? null : value[primary] ?? null))
     return { caseName, cells, changed: values.length > 1 && (states.size > 1 || scores.size > 1) }
   })
   return {
     columns: outputColumns,
+    ...(primary == null ? {} : { primary }),
     rows: outputRows,
+    scoreDefinitions: [...definitions.values()],
     summary: { changed: outputRows.filter(row => row.changed).length, rows: outputRows.length },
     view,
   }
@@ -228,10 +271,18 @@ function isStrictSubset(inner: ReadonlySet<string>, outer: ReadonlySet<string>):
 }
 
 function matches(handle: RunHandle, filter: { project: string, workspace: string }, now: number): boolean {
+  // A run written by an older vieval cannot be read, so it is no column.
+  if (handle.unsupportedVersion() != null) {
+    return false
+  }
   const item = toRunListItem(handle, now)
   return item.workspaceId === filter.workspace && (item.projectSegment === filter.project || item.projectIds.includes(filter.project))
 }
 
+function primaryMean(scoring: TaskNode['scoring']): null | number {
+  return scoring.scores.find(score => score.name === scoring.primary)?.mean ?? null
+}
+
 function sortByStart(list: readonly RunHandle[], now: number): RunHandle[] {
   // `toRunListItem` rebuilds every task node of a run; compute each start time once, not once per comparison.
   const startedAt = new Map(list.map(handle => [handle, toRunListItem(handle, now).startedAt ?? '']))
diff --git a/packages/ui/src/server/event-tail.test.ts b/packages/ui/src/server/event-tail.test.ts
index 3900949..5489124 100644
--- a/packages/ui/src/server/event-tail.test.ts
+++ b/packages/ui/src/server/event-tail.test.ts
@@ -68,29 +68,48 @@ describe('createEventTail', () => {
     expect(afterReplace.envelopes.map(item => item.seq)).toEqual([1, 2])
   })
 
-  it('counts invalid lines, detects legacy schema version 1, and tolerates a missing file', async () => {
+  /**
+   * @example
+   * expect(read.unsupportedVersion).toBe(1)
+   */
+  it('counts invalid lines, reports the schema version of an older vieval, and tolerates a missing file', async () => {
     const directory = await createTemporaryDirectory()
     const filePath = join(directory, 'events.jsonl')
     const tail = createEventTail(filePath)
-    expect(await tail.read()).toEqual({ envelopes: [], legacy: false, parseErrors: 0, partialBytes: 0, reset: false })
+    expect(await tail.read()).toEqual({ envelopes: [], parseErrors: 0, partialBytes: 0, reset: false })
 
     await writeFile(filePath, [
       '{not json',
       JSON.stringify({ event: 'x', schemaVersion: 3 }),
+      JSON.stringify({ event: 'x', schemaVersion: 4 }),
       JSON.stringify({ attemptId: 'a', data: {}, event: 'RunStarted', experimentId: 'e', runId: 'r', schemaVersion: 1, timestamp: 't', workspaceId: 'w' }),
       JSON.stringify({ event: 'run.started', schemaVersion: 2 }),
       '',
     ].join('\n'))
     const read = await tail.read()
-    expect(read.legacy).toBe(true)
+    // The first older line names the version; a newer or malformed line is only a parse error.
+    expect(read.unsupportedVersion).toBe(1)
     expect(read.parseErrors).toBe(3)
     expect(read.envelopes).toEqual([])
   })
 })
 
 describe('parseEventLine', () => {
-  it('parses valid version 2 envelopes', () => {
+  /**
+   * @example
+   * expect(parseEventLine(JSON.stringify(envelope))).toEqual(envelope)
+   */
+  it('parses valid version 3 envelopes', () => {
     const envelope = createEnvelopeFactory()('run.started', { totalTasks: 2 })
     expect(parseEventLine(JSON.stringify(envelope))).toEqual(envelope)
   })
+
+  /**
+   * @example
+   * expect(parseEventLine('{"schemaVersion":2,...}')).toBe(2)
+   */
+  it('returns the schema version of an envelope written by an older vieval', () => {
+    const envelope = { ...createEnvelopeFactory()('score.recorded', { kind: 'exact', score: 1 }), schemaVersion: 2 }
+    expect(parseEventLine(JSON.stringify(envelope))).toBe(2)
+  })
 })
diff --git a/packages/ui/src/server/event-tail.ts b/packages/ui/src/server/event-tail.ts
index 16f81cb..a68053b 100644
--- a/packages/ui/src/server/event-tail.ts
+++ b/packages/ui/src/server/event-tail.ts
@@ -24,14 +24,17 @@ export interface EventTail {
 export interface EventTailRead {
   /** Valid envelopes of the current schema version, in file order. */
   envelopes: RunEvent[]
-  /** A line of event schema version 1 was seen: the file is a legacy report. */
-  legacy: boolean
   /** Complete lines that were not valid envelopes. */
   parseErrors: number
   /** Bytes after the last newline, kept for the next read (a line still being written). */
   partialBytes: number
   /** The file shrank or was replaced: the caller must discard state built from earlier reads. */
   reset: boolean
+  /**
+   * Event schema version of a line older than {@link RUN_EVENT_SCHEMA_VERSION}, when one was seen: an older vieval
+   * wrote the file, and its events cannot be read.
+   */
+  unsupportedVersion?: number
 }
 
 /**
@@ -59,7 +62,7 @@ export function createEventTail(filePath: string): EventTail {
   return {
     offset: () => offset,
     async read() {
-      const result: EventTailRead = { envelopes: [], legacy: false, parseErrors: 0, partialBytes: 0, reset: false }
+      const result: EventTailRead = { envelopes: [], parseErrors: 0, partialBytes: 0, reset: false }
       const info = await stat(filePath).catch(() => undefined)
       if (info == null) {
         return result
@@ -105,8 +108,8 @@ export function createEventTail(filePath: string): EventTail {
           continue
         }
         const parsed = parseEventLine(line)
-        if (parsed === 'legacy') {
-          result.legacy = true
+        if (typeof parsed === 'number') {
+          result.unsupportedVersion ??= parsed
         }
         else if (parsed == null) {
           result.parseErrors += 1
@@ -125,10 +128,10 @@ export function createEventTail(filePath: string): EventTail {
  *
  * Returns:
  * - the envelope for the current schema version
- * - `'legacy'` for schema version 1 lines
+ * - the schema version of a line written by an older vieval (below {@link RUN_EVENT_SCHEMA_VERSION})
  * - `undefined` for anything else
  */
-export function parseEventLine(line: string): 'legacy' | RunEvent | undefined {
+export function parseEventLine(line: string): number | RunEvent | undefined {
   let value: unknown
   try {
     value = JSON.parse(line)
@@ -137,8 +140,8 @@ export function parseEventLine(line: string): 'legacy' | RunEvent | undefined {
     return undefined
   }
   const version = (value as null | { schemaVersion?: unknown })?.schemaVersion
-  if (version === 1) {
-    return 'legacy'
+  if (typeof version === 'number' && version < RUN_EVENT_SCHEMA_VERSION) {
+    return version
   }
   if (version !== RUN_EVENT_SCHEMA_VERSION) {
     return undefined
diff --git a/packages/ui/src/server/otel.ts b/packages/ui/src/server/otel.ts
index a0f7f99..d2fcbe1 100644
--- a/packages/ui/src/server/otel.ts
+++ b/packages/ui/src/server/otel.ts
@@ -39,6 +39,9 @@ export function buildOtelLogs(handle: RunHandle): OtelLogsResponse {
 
 /**
  * Gauges: every numeric metric, case scores, and case durations.
+ *
+ * A case score gauge is `vieval.score.<name>` with the value each round recorded: the same name as in the CLI's
+ * `otlp/metrics.json` and as the score attributes of the task and case spans.
  */
 export function buildOtelMetrics(handle: RunHandle): OtelMetricsResponse {
   const reducer = handle.reducer()
@@ -63,11 +66,8 @@ export function buildOtelMetrics(handle: RunHandle): OtelMetricsResponse {
       if (round.endedAt != null) {
         push('vieval.case.duration_ms', { attributes, time: round.endedAt, value: Date.parse(round.endedAt) - Date.parse(round.startedAt) })
       }
-      for (const kind of ['exact', 'judge'] as const) {
-        const value = round.scores[kind]
-        if (value != null) {
-          push(`vieval.case.score.${kind}`, { attributes, time: round.endedAt ?? round.startedAt, value })
-        }
+      for (const [name, value] of Object.entries(round.scores.values)) {
+        push(`vieval.score.${name}`, { attributes, time: round.endedAt ?? round.startedAt, value })
       }
     }
   }
@@ -124,7 +124,12 @@ export function buildOtelTraces(handle: RunHandle): OtelTracesResponse {
   for (const task of buildTaskNodes(reducer, () => 0)) {
     const taskSpanId = hash(`task:${runId}:${task.id}`, 16)
     spans.push({
-      attributes: { 'vieval.task.display_name': task.displayName, 'vieval.task.id': task.id, 'vieval.task.score.exact': task.scores.exact },
+      attributes: {
+        'vieval.task.display_name': task.displayName,
+        'vieval.task.id': task.id,
+        ...(task.scoring.primary == null ? {} : { 'vieval.task.primary_score': task.scoring.primary }),
+        ...Object.fromEntries(task.scoring.scores.map(score => [`vieval.score.${score.name}`, score.mean])),
+      },
       endTime: task.endedAt,
       kind: 'task',
       name: 'vieval.task',
@@ -147,7 +152,14 @@ export function buildOtelTraces(handle: RunHandle): OtelTracesResponse {
         }
         const caseSpanId = hash(`case:${runId}:${node.key}:${round.roundIndex}`, 16)
         spans.push({
-          attributes: { 'vieval.case.id': node.caseId, 'vieval.case.name': node.name, 'vieval.case.round_index': round.roundIndex, 'vieval.case.state': round.state, ...(round.scores.exact == null ? {} : { 'vieval.case.score.exact': round.scores.exact }) },
+          attributes: {
+            'vieval.case.id': node.caseId,
+            'vieval.case.name': node.name,
+            'vieval.case.round_index': round.roundIndex,
+            'vieval.case.state': round.state,
+            ...Object.fromEntries(Object.entries(round.scores.values).map(([name, value]) => [`vieval.score.${name}`, value])),
+            ...Object.fromEntries(Object.entries(round.scores.unscored).map(([name, reason]) => [`vieval.score.${name}.unscored`, reason])),
+          },
           endTime: round.endedAt,
           kind: 'case',
           name: 'vieval.case',
diff --git a/packages/ui/src/server/store.test.ts b/packages/ui/src/server/store.test.ts
index e5c2ae9..f79f009 100644
--- a/packages/ui/src/server/store.test.ts
+++ b/packages/ui/src/server/store.test.ts
@@ -3,8 +3,8 @@ import { join } from 'node:path'
 
 import { afterEach, describe, expect, it } from 'vitest'
 
-import { cleanupTemporaryDirectories, createTemporaryDirectory } from '../../test/helpers'
-import { findRunDirectories } from './store'
+import { cleanupTemporaryDirectories, createEnvelopeFactory, createTemporaryDirectory } from '../../test/helpers'
+import { createRunStore, findRunDirectories, readRunSummary, toRunListItem } from './store'
 
 afterEach(cleanupTemporaryDirectories)
 
@@ -43,3 +43,101 @@ describe('findRunDirectories', () => {
     expect(await findRunDirectories(join(base, 'missing'))).toEqual([])
   })
 })
+
+const HEADLINE = {
+  completion: { mean: 1, n: 2, name: 'completion', sum: 2, total: 2, unscored: {} },
+  headline: { kind: 'score', score: { mean: 0.5, n: 2, name: 'demo.pass', sum: 1, total: 2, unscored: {} } },
+}
+
+/** Writes one run directory under `<root>/w/p/e/a/<runKey>` with the given event lines and summary. */
+async function writeRun(root: string, runKey: string, lines: readonly unknown[], summary?: unknown): Promise<string> {
+  const directory = join(root, 'w', 'p', 'e', 'a', runKey)
+  await mkdir(directory, { recursive: true })
+  await writeFile(join(directory, 'events.jsonl'), lines.map(line => `${JSON.stringify(line)}\n`).join(''))
+  if (summary != null) {
+    await writeFile(join(directory, 'run-summary.json'), JSON.stringify(summary))
+  }
+  return directory
+}
+
+describe('readRunSummary', () => {
+  /**
+   * @example
+   * expect(summary?.headline).toEqual(HEADLINE)
+   */
+  it('reads the headline and the score definitions of run-summary.json schema version 2', async () => {
+    const root = await createTemporaryDirectory()
+    const directory = await writeRun(root, 'run-1', [], {
+      headline: HEADLINE,
+      projects: [{ caseSummary: { failed: 1, passed: 1, total: 2 } }],
+      schemaVersion: 2,
+      scoreDefinitions: [{ label: 'Pass', name: 'demo.pass', primary: true }],
+    })
+
+    const summary = await readRunSummary(join(directory, 'run-summary.json'))
+    expect(summary?.headline).toEqual(HEADLINE)
+    expect(summary?.scoreDefinitions).toEqual([{ label: 'Pass', name: 'demo.pass', primary: true }])
+    expect(summary?.caseSummary).toEqual({ errored: 0, failed: 1, passed: 1, requeued: 0, running: 0, skipped: 0, timeout: 0, total: 2 })
+  })
+
+  /**
+   * @example
+   * expect(summary?.headline).toBeUndefined()
+   */
+  it('reads a summary without a headline, so the run is scored from its events', async () => {
+    const root = await createTemporaryDirectory()
+    const directory = await writeRun(root, 'run-1', [], { interruptSignal: 'SIGINT', projects: [] })
+
+    const summary = await readRunSummary(join(directory, 'run-summary.json'))
+    expect(summary).toEqual({ caseSummary: undefined, headline: undefined, interruptSignal: 'SIGINT', scoreDefinitions: undefined })
+  })
+})
+
+describe('createRunStore', () => {
+  /**
+   * @example
+   * expect(handle?.unsupportedVersion()).toBe(2)
+   */
+  it('lists a run of event schema version 2 as unsupported without reading its events', async () => {
+    const root = await createTemporaryDirectory()
+    const envelope = createEnvelopeFactory('run-old')
+    const old = { ...envelope('score.recorded', { kind: 'exact', score: 1 }, { caseId: '0:a', projectId: 'p', taskId: 't' }), schemaVersion: 2 }
+    await writeRun(root, 'run-old', [old])
+    const store = createRunStore({ reportRoots: [root] })
+
+    const handle = await store.get('run-old')
+    expect(handle?.unsupportedVersion()).toBe(2)
+    expect(handle?.source()).toBe('unsupported')
+    expect(handle?.state(Date.now())).toBe('unsupported')
+    expect(handle?.reducer().events).toEqual([])
+    const item = toRunListItem(handle!, Date.now())
+    expect(item).toMatchObject({ attemptId: 'a', experimentId: 'e', headline: null, runKey: 'run-old', workspaceId: 'w' })
+    store.close()
+  })
+
+  /**
+   * @example
+   * expect(item.headlineDefinitions).toEqual([{ label: 'Pass', name: 'demo.pass', primary: true }])
+   */
+  it('lists a finished run with the headline of its summary and the definition of the headline score', async () => {
+    const root = await createTemporaryDirectory()
+    const envelope = createEnvelopeFactory('run-1')
+    const scope = { caseId: '0:a', projectId: 'p', taskId: 't' }
+    await writeRun(root, 'run-1', [
+      envelope('run.started', { totalTasks: 1 }),
+      envelope('score.defined', { name: 'demo.other' }, { projectId: 'p', taskId: 't' }),
+      envelope('score.defined', { label: 'Pass', name: 'demo.pass', primary: true }, { projectId: 'p', taskId: 't' }),
+      envelope('case.started', { index: 0, name: 'a', total: 1 }, scope),
+      envelope('score.recorded', { name: 'demo.pass', value: 1 }, scope),
+      envelope('case.ended', { index: 0, name: 'a', state: 'passed', total: 1 }, scope),
+      envelope('run.ended', { failedTasks: 0, passedTasks: 1, skippedTasks: 0, totalTasks: 1 }),
+    ], { headline: HEADLINE, projects: [], schemaVersion: 2 })
+    const store = createRunStore({ reportRoots: [root] })
+
+    const item = toRunListItem((await store.get('run-1'))!, Date.now())
+    // The summary's headline wins over the one the events give (mean 1 of one case).
+    expect(item.headline).toEqual(HEADLINE)
+    expect(item.headlineDefinitions).toEqual([{ label: 'Pass', name: 'demo.pass', primary: true }])
+    store.close()
+  })
+})
diff --git a/packages/ui/src/server/store.ts b/packages/ui/src/server/store.ts
index 5a26087..8f4dff1 100644
--- a/packages/ui/src/server/store.ts
+++ b/packages/ui/src/server/store.ts
@@ -2,7 +2,7 @@ import type { FSWatcher } from 'node:fs'
 
 import type { RunReadyContext } from 'vieval/cli'
 
-import type { CaseCounts, RunEvent, RunListItem, RunSource, RunState, TaskNode } from '../shared/contract'
+import type { CaseCounts, HeadlineResult, RunEvent, RunListItem, RunSource, RunState, ScoreDefinition, TaskNode } from '../shared/contract'
 import type { RunReducer } from '../shared/reduce'
 
 import { watch } from 'node:fs'
@@ -10,11 +10,12 @@ import { readFile, stat } from 'node:fs/promises'
 import { basename, dirname, join, relative, resolve, sep } from 'node:path'
 
 import { glob } from 'tinyglobby'
-import { hybridAverage } from 'vieval/core/scoring'
+import { scoreDefinedDataSchema } from 'vieval/events'
 
 import * as v from 'valibot'
 
-import { average, axisValues, buildTaskNodes, countCases, createRunReducer, expectedCases } from '../shared/reduce'
+import { headlineResultSchema } from '../shared/contract'
+import { axisValues, buildTaskNodes, countCases, createRunReducer, expectedCases, pickScoreDefinitions, resolveTaskHeadline } from '../shared/reduce'
 import { createEventTail } from './event-tail'
 import { parseJsonText } from './files'
 
@@ -24,26 +25,18 @@ export const DEFAULT_STALE_AFTER_MS = 60 * 60 * 1000
 /** Interval of the polling fallback that complements `fs.watch` for tailed runs. */
 const TAIL_POLL_MS = 1000
 
-/** Identity fields that both `run-summary.json` and every event envelope carry. */
-const reportIdentityEntries = {
-  attemptId: v.optional(v.string()),
-  experimentId: v.optional(v.string()),
-  runId: v.optional(v.string()),
-  workspaceId: v.optional(v.string()),
-}
-
 /**
- * Fields of `run-summary.json` (vieval's `CliRunOutput`) that the viewer reads.
+ * Fields of `run-summary.json` (vieval's `RunSummaryFile`) that the viewer reads.
  *
- * Loose and mostly optional, because the file may come from any vieval version, including event schema version 1
- * reports. Missing case counts read as 0, as in the reports of older versions.
+ * Loose and mostly optional: the viewer reads only the counts, the interrupt, and the resolved headline, and a run
+ * whose summary lacks the headline is scored from its events instead.
  */
 const runSummaryFileSchema = v.looseObject({
-  ...reportIdentityEntries,
+  /** Headline of the run, resolved once by the runner (`run-summary.json` schema version 2). */
+  headline: v.optional(headlineResultSchema),
   interruptSignal: v.optional(v.string()),
   projects: v.optional(v.array(v.looseObject({
     caseSummary: v.nullish(v.looseObject({
-      // Reports from before vieval split errored cases out of failed ones have no `errored` count.
       errored: v.optional(v.number(), 0),
       failed: v.optional(v.number(), 0),
       passed: v.optional(v.number(), 0),
@@ -51,37 +44,13 @@ const runSummaryFileSchema = v.looseObject({
       timeout: v.optional(v.number(), 0),
       total: v.optional(v.number(), 0),
     })),
-    result: v.nullish(v.looseObject({
-      overall: v.optional(v.looseObject({
-        exactAverage: v.nullish(v.number()),
-        judgeAverage: v.nullish(v.number()),
-        runCount: v.optional(v.number()),
-      })),
-    })),
-    taskCount: v.optional(v.number()),
   })), []),
+  /** Definitions of the run's scores (`run-summary.json` schema version 2). */
+  scoreDefinitions: v.optional(v.array(scoreDefinedDataSchema)),
 })
 
-/** One line of a legacy (event schema version 1) `cases.jsonl`. */
-const legacyCaseRecordSchema = v.looseObject({
-  caseId: v.string(),
-  caseName: v.optional(v.string()),
-  endedAt: v.optional(v.string()),
-  errorMessage: v.nullish(v.string()),
-  input: v.optional(v.unknown()),
-  metrics: v.optional(v.record(v.string(), v.unknown()), {}),
-  projectName: v.optional(v.string()),
-  scores: v.optional(v.record(v.string(), v.unknown()), {}),
-  startedAt: v.optional(v.string()),
-  state: v.optional(v.string()),
-  taskId: v.optional(v.string()),
-})
-
-/** First line of a legacy `events.jsonl`: only its identity and time are used. */
-const legacyEventSchema = v.looseObject({
-  ...reportIdentityEntries,
-  timestamp: v.optional(v.string()),
-})
+/** Answer to a request that opens a run written by an older vieval. */
+export const UNSUPPORTED_RUN_MESSAGE = 'This report was written by an older vieval. Run it again.'
 
 export interface CreateRunStoreOptions {
   reportRoots: readonly string[]
@@ -109,6 +78,11 @@ export interface RunHandle {
   /** Subscribes to new envelopes. Tailed runs are watched while at least one listener exists. */
   subscribe: (listener: RunListener) => () => void
   summary: () => RunSummaryInfo | undefined
+  /**
+   * Event schema version of the run's `events.jsonl` when an older vieval wrote it. Such a run is listed (source and
+   * state `unsupported`) but not opened.
+   */
+  unsupportedVersion: () => number | undefined
 }
 
 /**
@@ -132,10 +106,11 @@ export interface RunStore {
  */
 export interface RunSummaryInfo {
   caseSummary?: CaseCounts
-  exactAverage: null | number
-  hybridAverage: null | number
+  /** Headline resolved by the runner, when the summary holds one. */
+  headline?: HeadlineResult
   interruptSignal?: string
-  judgeAverage: null | number
+  /** Score definitions written with the headline. */
+  scoreDefinitions?: ScoreDefinition[]
 }
 
 /**
@@ -222,72 +197,6 @@ export async function findRunDirectories(root: string): Promise<string[]> {
   return [...new Set(files.map(file => resolve(dirname(file))))].sort()
 }
 
-/**
- * Rebuilds a legacy (event schema version 1) run from `cases.jsonl` and `run-summary.json`.
- *
- * The events of version 1 use other names and payloads; instead of replaying them, every case record is turned
- * into the version 2 events the reducer understands. Rounds, retries, logs, and inference events are not
- * available for such runs.
- */
-export async function loadLegacyRun(runDirectory: string): Promise<RunReducer> {
-  const reducer = createRunReducer()
-  const records = (await readFile(join(runDirectory, 'cases.jsonl'), 'utf-8').catch(() => ''))
-    .split('\n')
-    .filter(line => line.trim().length > 0)
-    .flatMap((line) => {
-      const record = parseJsonText(legacyCaseRecordSchema, line)
-      return record == null ? [] : [record]
-    })
-  const summary = parseJsonText(runSummaryFileSchema, await readFile(join(runDirectory, 'run-summary.json'), 'utf-8').catch(() => ''))
-  const firstLine = (await readFile(join(runDirectory, 'events.jsonl'), 'utf-8').catch(() => '')).split('\n', 1)[0]
-  const first = parseJsonText(legacyEventSchema, firstLine)
-  const identity = {
-    attemptId: summary?.attemptId ?? first?.attemptId ?? 'unknown-attempt',
-    experimentId: summary?.experimentId ?? first?.experimentId ?? 'unknown-experiment',
-    runId: summary?.runId ?? first?.runId ?? basename(runDirectory),
-    workspaceId: summary?.workspaceId ?? first?.workspaceId ?? 'unknown-workspace',
-  }
-  let seq = 0
-  const startedTimes = records.flatMap(record => record.startedAt ?? []).sort()
-  const endedTimes = records.flatMap(record => record.endedAt ?? []).sort()
-  const runStartedAt = first?.timestamp ?? startedTimes[0] ?? new Date(0).toISOString()
-  const runEndedAt = endedTimes.at(-1) ?? runStartedAt
-  const emit = (event: string, data: unknown, time: string, scope: { caseId?: string, projectId?: string, taskId?: string } = {}) => {
-    seq += 1
-    reducer.apply({ ...identity, ...scope, data, event, schemaVersion: 2, seq, time })
-  }
-  const projects = summary?.projects ?? []
-  emit('run.started', { totalTasks: projects.reduce((sum, project) => sum + (project.taskCount ?? 0), 0) }, runStartedAt)
-
-  const byTask = Map.groupBy(records, record => record.taskId ?? 'unknown-task')
-  for (const [taskId, taskRecords] of byTask) {
-    const projectId = taskRecords[0].projectName ?? 'unknown-project'
-    const scope = { projectId, taskId }
-    const taskStartedAt = taskRecords.map(record => record.startedAt ?? runStartedAt).sort()[0]
-    emit('task.queued', { displayName: parseTaskDisplayName(taskId) }, runStartedAt, scope)
-    emit('task.started', {}, taskStartedAt, scope)
-    for (const record of taskRecords) {
-      const caseScope = { ...scope, caseId: record.caseId }
-      const index = Number.parseInt(record.caseId.split(':')[0], 10) || 0
-      const name = record.caseName ?? record.caseId
-      const startedAt = record.startedAt ?? runStartedAt
-      const endedAt = record.endedAt ?? startedAt
-      emit('case.started', { index, input: record.input, name, total: taskRecords.length }, startedAt, caseScope)
-      for (const [metricName, value] of Object.entries(record.metrics)) {
-        emit('metric.recorded', { name: metricName, value }, endedAt, caseScope)
-      }
-      for (const [kind, score] of Object.entries(record.scores)) {
-        emit('score.recorded', { kind, score }, endedAt, caseScope)
-      }
-      emit('case.ended', { errorMessage: record.errorMessage ?? undefined, index, name, state: record.state, total: taskRecords.length }, endedAt, caseScope)
-    }
-    const failed = taskRecords.some(record => record.state !== 'passed' && record.state !== 'skipped')
-    emit('task.ended', { state: failed ? 'failed' : 'passed' }, taskRecords.map(record => record.endedAt ?? runEndedAt).sort().at(-1)!, scope)
-  }
-  emit('run.ended', { failedTasks: 0, passedTasks: 0, skippedTasks: 0, totalTasks: byTask.size }, runEndedAt)
-  return reducer
-}
-
 /**
  * Reads the official numbers of `run-summary.json`.
  */
@@ -298,7 +207,6 @@ export async function readRunSummary(filePath: string): Promise<RunSummaryInfo |
   }
   const caseSummary: CaseCounts = { errored: 0, failed: 0, passed: 0, requeued: 0, running: 0, skipped: 0, timeout: 0, total: 0 }
   let hasCaseSummary = false
-  const weighted = { exact: [0, 0], judge: [0, 0] }
   for (const project of parsed.projects) {
     const counts = project.caseSummary
     if (counts != null) {
@@ -307,29 +215,19 @@ export async function readRunSummary(filePath: string): Promise<RunSummaryInfo |
         caseSummary[key] += counts[key]
       }
     }
-    const overall = project.result?.overall
-    const weight = overall?.runCount ?? 1
-    for (const kind of ['exact', 'judge'] as const) {
-      const value = overall?.[`${kind}Average`]
-      if (value != null) {
-        weighted[kind][0] += value * weight
-        weighted[kind][1] += weight
-      }
-    }
   }
-  const exactAverage = weighted.exact[1] === 0 ? null : weighted.exact[0] / weighted.exact[1]
-  const judgeAverage = weighted.judge[1] === 0 ? null : weighted.judge[0] / weighted.judge[1]
   return {
     caseSummary: hasCaseSummary ? caseSummary : undefined,
-    exactAverage,
-    hybridAverage: hybridAverage(exactAverage, judgeAverage),
+    headline: parsed.headline,
     interruptSignal: parsed.interruptSignal,
-    judgeAverage,
+    scoreDefinitions: parsed.scoreDefinitions,
   }
 }
 
 /**
- * Builds the list item of one run (hierarchy labels, state, counts, scores).
+ * Builds the list item of one run (hierarchy labels, state, counts, headline).
+ *
+ * The headline is the one `run-summary.json` holds once the run ended, else the one resolved from the run's events.
  *
  * Expects:
  * - `tasks` only when the caller already built the run's task nodes (building them walks every case)
@@ -344,10 +242,10 @@ export function toRunListItem(handle: RunHandle, now: number, tasks: readonly Ta
   }
   const counts = countCases(tasks.flatMap(task => task.cases))
   const summary = handle.summary()
-  const taskExact = tasks.map(task => task.scores.exact).filter((value): value is number => value != null)
-  const taskJudge = tasks.map(task => task.scores.judge).filter((value): value is number => value != null)
-  const exactAverage = summary?.exactAverage ?? average(taskExact)
-  const judgeAverage = summary?.judgeAverage ?? average(taskJudge)
+  const headline = summary?.headline ?? resolveTaskHeadline(tasks)
+  const definitions = summary?.scoreDefinitions == null
+    ? reducer.scoreDefinitions
+    : new Map(summary.scoreDefinitions.map(definition => [definition.name, definition]))
   const run = reducer.run()
   return {
     ...identity,
@@ -355,9 +253,8 @@ export function toRunListItem(handle: RunHandle, now: number, tasks: readonly Ta
     counts,
     endedAt: run.endedAt,
     evalMatrix: axisValues(tasks.flatMap(task => task.matrix == null ? [] : [task.matrix.eval])),
-    exactAverage,
-    hybridAverage: hybridAverage(exactAverage, judgeAverage),
-    judgeAverage,
+    headline,
+    headlineDefinitions: pickScoreDefinitions(definitions, headlineScoreNames(headline)),
     lastEventAt: run.lastEvent?.time,
     lastSeq: reducer.seq(),
     matrix: axisValues(tasks.flatMap(task => task.matrix == null ? [] : [task.matrix.run])),
@@ -382,6 +279,7 @@ function createDirectoryRun(args: { reportRoot: string, runDirectory: string, st
   let reducer = createRunReducer()
   let source: RunSource = 'live-tail'
   let summary: RunSummaryInfo | undefined
+  let unsupportedVersion: number | undefined
   let parseErrors = 0
   let finished = false
   let refreshing: Promise<void> | undefined
@@ -397,10 +295,11 @@ function createDirectoryRun(args: { reportRoot: string, runDirectory: string, st
       return
     }
     const read = await tail.read()
-    if (read.legacy && source !== 'legacy') {
-      source = 'legacy'
-      reducer = await loadLegacyRun(args.runDirectory)
-      await loadSummary()
+    if (read.unsupportedVersion != null) {
+      // An older vieval wrote the events: they are not read, and the run is listed without opening it.
+      unsupportedVersion = read.unsupportedVersion
+      source = 'unsupported'
+      reducer = createRunReducer()
       finished = true
       stopWatching()
       listeners.emit({ envelopes: [], reset: true })
@@ -478,8 +377,8 @@ function createDirectoryRun(args: { reportRoot: string, runDirectory: string, st
     runKey: basename(args.runDirectory),
     source: () => source,
     state(now) {
-      if (source === 'legacy') {
-        return 'legacy'
+      if (source === 'unsupported') {
+        return 'unsupported'
       }
       const terminal = stateFromReducer(reducer, summary)
       if (terminal != null) {
@@ -502,6 +401,7 @@ function createDirectoryRun(args: { reportRoot: string, runDirectory: string, st
       }
     },
     summary: () => summary,
+    unsupportedVersion: () => unsupportedVersion,
   }
 
   return handle
@@ -567,6 +467,7 @@ function createLiveRun(context: RunReadyContext, staleAfterMs: number): { dispos
       return () => listeners.listeners.delete(listener)
     },
     summary: () => summary,
+    unsupportedVersion: () => undefined,
   }
 
   return {
@@ -582,14 +483,15 @@ function createLiveRun(context: RunReadyContext, staleAfterMs: number): { dispos
   }
 }
 
-function parseTaskDisplayName(taskId: string): string {
-  const [entry] = taskId.split('::')
-  try {
-    return decodeURIComponent(entry).split('/').at(-1) ?? taskId
+/** Names of the scores a headline shows. */
+function headlineScoreNames(headline: HeadlineResult | null): string[] {
+  if (headline == null || headline.headline.kind === 'passed-cases') {
+    return []
   }
-  catch {
-    return taskId
+  if (headline.headline.kind === 'score') {
+    return [headline.headline.score.name]
   }
+  return headline.headline.tasks.map(task => task.score.name)
 }
 
 function stateFromReducer(reducer: RunReducer, summary: RunSummaryInfo | undefined): RunState | undefined {
diff --git a/packages/ui/src/server/views.test.ts b/packages/ui/src/server/views.test.ts
index 54a2894..ee8dde1 100644
--- a/packages/ui/src/server/views.test.ts
+++ b/packages/ui/src/server/views.test.ts
@@ -1,30 +1,30 @@
-import type { RunReducer } from '../shared/reduce'
-import type { RunHandle } from './store'
-
 import { describe, expect, it } from 'vitest'
 
-import { createEnvelopeFactory } from '../../test/helpers'
+import { createEnvelopeFactory, createReducerHandle } from '../../test/helpers'
 import { createRunReducer } from '../shared/reduce'
-import { buildMetrics } from './views'
+import { createEvidenceService } from './evidence'
+import { buildMetrics, buildScores, buildSnapshot } from './views'
 
 const TASK = 'evals%2Fdemo::default::run=default::eval=default'
+const OTHER_TASK = 'evals%2Fdemo::default::run=model%3Db::eval=default'
 
-function handleOf(reducer: RunReducer): RunHandle {
-  return {
-    artifactsRoot: () => undefined,
-    parseErrors: () => 0,
-    projectSegment: 'demo',
-    reducer: () => reducer,
-    refresh: async () => {},
-    reportDirectory: undefined,
-    reportPath: undefined,
-    reportRoot: undefined,
-    runKey: 'run-test',
-    source: () => 'live-bus',
-    state: () => 'running',
-    subscribe: () => () => {},
-    summary: () => undefined,
+function createReducer() {
+  const envelope = createEnvelopeFactory()
+  const reducer = createRunReducer()
+  const apply = (event: string, data: unknown, scope: object = {}) => {
+    reducer.apply(envelope(event, data, scope))
   }
+  return { apply, reducer }
+}
+
+/** Settles one case of `taskId` with the given state and recorded score values. */
+function settleCase(apply: (event: string, data: unknown, scope: object) => void, taskId: string, index: number, state: string, values: Record<string, number> = {}): void {
+  const scope = { caseId: `${index}:c${index}`, projectId: 'demo', taskId }
+  apply('case.started', { index, name: `c${index}`, total: 1 }, scope)
+  for (const [name, value] of Object.entries(values)) {
+    apply('score.recorded', { name, value }, scope)
+  }
+  apply('case.ended', { index, name: `c${index}`, state, total: 1 }, scope)
 }
 
 describe('buildMetrics', () => {
@@ -49,7 +49,109 @@ describe('buildMetrics', () => {
       reducer.apply(envelope('metric.recorded', { name: 'steps', value }, scope, '2026-01-01T00:00:00.000Z'))
     }
 
-    const [series] = buildMetrics(handleOf(reducer), {}).numeric
+    const [series] = buildMetrics(createReducerHandle(reducer), {}).numeric
     expect(series.summary).toMatchObject({ count: 200_000, max: 199_999, min: 0, sum: 199_999 * 100_000 })
   })
+
+  /**
+   * @example
+   * expect(score.points.map(point => point.value)).toEqual([0.5, 0])
+   */
+  it('puts each case\'s counted score beside the metrics, with its definition and the summary of the selection', () => {
+    const { apply, reducer } = createReducer()
+    apply('score.defined', { label: 'Correct', name: 'demo.correct' }, { projectId: 'demo', taskId: TASK })
+    settleCase(apply, TASK, 0, 'passed', { 'demo.correct': 0.5 })
+    // A failed case counts 0, whatever it recorded; an errored one does not count.
+    settleCase(apply, TASK, 1, 'failed', { 'demo.correct': 1 })
+    settleCase(apply, TASK, 2, 'errored')
+    settleCase(apply, TASK, 3, 'failed')
+
+    const metrics = buildMetrics(createReducerHandle(reducer), {})
+    expect(metrics.scoreDefinitions).toEqual([{ label: 'Correct', name: 'demo.correct' }])
+    expect(metrics.scores).toHaveLength(1)
+    const [score] = metrics.scores
+    expect(score.name).toBe('demo.correct')
+    expect(score.points.map(point => [point.caseId, point.value])).toEqual([['0:c0', 0.5], ['1:c1', 0], ['3:c3', 0]])
+    expect(score.summary).toEqual({ mean: 0.5 / 3, n: 3, name: 'demo.correct', sum: 0.5, total: 4, unscored: {} })
+
+    const single = buildMetrics(createReducerHandle(reducer), { caseId: '0:c0', taskId: TASK })
+    expect(single.scores[0].summary).toMatchObject({ mean: 0.5, n: 1, total: 1 })
+  })
+
+  // ROOT CAUSE:
+  //
+  // A case that failed before it recorded a score had no point, while the task's mean counted it as 0: one round was
+  // scored on its own, so only the names it recorded were its scores.
+  //
+  // countedScores(round)   // {} for a failed round without records
+  //
+  // We fixed this by counting a round with every score of its task (`countingDefinitions`).
+  /**
+   * @example
+   * expect(failed.scores[0].points).toEqual([{ caseId: '1:c1', roundIndex: 0, taskId: TASK, value: 0 }])
+   */
+  it('counts a failed case in a score that only other cases of its task recorded', () => {
+    const { apply, reducer } = createReducer()
+    settleCase(apply, TASK, 0, 'passed', { 'demo.score': 1 })
+    settleCase(apply, TASK, 1, 'failed')
+
+    const failed = buildMetrics(createReducerHandle(reducer), { caseId: '1:c1', taskId: TASK })
+    expect(failed.scores.map(score => score.name)).toEqual(['demo.score'])
+    expect(failed.scores[0].points).toEqual([{ caseId: '1:c1', roundIndex: 0, taskId: TASK, value: 0 }])
+    expect(failed.scores[0].summary).toMatchObject({ mean: 0, n: 1, total: 1 })
+  })
+})
+
+describe('buildScores', () => {
+  // ROOT CAUSE:
+  //
+  // The overall score of a run averaged task means without weights, so a task of one case weighed as much as a task
+  // of a hundred:
+  //
+  // const exact = average(tasks.map(task => task.scores.exact))
+  //
+  // We fixed this by pooling the cases of every task (`poolScoreSummaries`).
+  /**
+   * @example
+   * expect(scores.overall.scores[0].mean).toBe(0.25)
+   */
+  it('pools the cases of every task instead of averaging task means', () => {
+    const { apply, reducer } = createReducer()
+    settleCase(apply, TASK, 0, 'passed', { 'demo.score': 1 })
+    settleCase(apply, OTHER_TASK, 0, 'passed', { 'demo.score': 0 })
+    settleCase(apply, OTHER_TASK, 1, 'passed', { 'demo.score': 0 })
+    settleCase(apply, OTHER_TASK, 2, 'passed', { 'demo.score': 0 })
+
+    const scores = buildScores(createReducerHandle(reducer))
+    // The mean of task means would be 0.5.
+    expect(scores.overall.scores).toEqual([{ mean: 0.25, n: 4, name: 'demo.score', sum: 1, total: 4, unscored: {} }])
+    expect(scores.overall.headline?.headline).toEqual({ kind: 'score', score: scores.overall.scores[0] })
+    expect(scores.byTask.map(task => task.scoring.scores[0].mean)).toEqual([1, 0])
+    expect(scores.byRound.map(round => [round.taskId, round.roundIndex, round.scoring.completion.n])).toEqual([[TASK, 0, 1], [OTHER_TASK, 0, 3]])
+    expect(scores.summary).toBeUndefined()
+  })
+})
+
+describe('buildSnapshot', () => {
+  /**
+   * @example
+   * expect(snapshot.summary?.headline).toEqual(summaryHeadline)
+   */
+  it('returns the headline from the events, and the one run-summary.json holds next to it', () => {
+    const { apply, reducer } = createReducer()
+    apply('score.defined', { name: 'demo.pass', primary: true }, { projectId: 'demo', taskId: TASK })
+    settleCase(apply, TASK, 0, 'passed', { 'demo.pass': 1 })
+    const summaryHeadline = {
+      completion: { mean: 1, n: 1, name: 'completion', sum: 1, total: 1, unscored: {} },
+      headline: { kind: 'score' as const, score: { mean: 1, n: 1, name: 'demo.pass', sum: 1, total: 1, unscored: {} } },
+    }
+    const evidence = createEvidenceService({ evidenceRoots: [], logDirectories: [] })
+
+    const snapshot = buildSnapshot(createReducerHandle(reducer, { summary: { headline: summaryHeadline } }), evidence, Date.now())
+    // The events resolve the same headline the runner wrote.
+    expect(snapshot.headline).toEqual(summaryHeadline)
+    expect(snapshot.scoreDefinitions).toEqual([{ name: 'demo.pass', primary: true }])
+    expect(snapshot.summary?.headline).toEqual(summaryHeadline)
+    expect(snapshot.tasks[0].scoring.primary).toBe('demo.pass')
+  })
 })
diff --git a/packages/ui/src/server/views.ts b/packages/ui/src/server/views.ts
index 80f1b13..dcbeab7 100644
--- a/packages/ui/src/server/views.ts
+++ b/packages/ui/src/server/views.ts
@@ -16,10 +16,10 @@ import type { RunHandle } from './store'
 import { readdir, stat } from 'node:fs/promises'
 import { join } from 'node:path'
 
-import { hybridAverage } from 'vieval/core/scoring'
+import { poolScoreSummaries } from 'vieval/core/scoring'
 
 import { sumInference } from '../shared/inference'
-import { average, buildProjectNodes, buildTaskNodes, caseKeyOf, currentPosition, estimateProgress, scoreRounds, toCaseNode } from '../shared/reduce'
+import { buildProjectNodes, buildTaskNodes, caseKeyOf, countedScores, countingDefinitions, currentPosition, estimateProgress, pickScoreDefinitions, resolveTaskHeadline, scoreRounds, toCaseNode } from '../shared/reduce'
 import { ViewerError } from './errors'
 import { classifyPreview, contentTypeFromName } from './files'
 
@@ -27,7 +27,7 @@ import { classifyPreview, contentTypeFromName } from './files'
 export const MAX_LOG_LIMIT = 5000
 
 /**
- * Case detail: tree node plus input, output, metrics, artifacts, evidence links, inference events.
+ * Case detail: tree node plus input, output, metrics, score records, artifacts, evidence links, inference events.
  */
 export async function buildCaseDetail(handle: RunHandle, evidence: EvidenceService, taskId: string, caseId: string): Promise<CaseDetail> {
   const reducer = handle.reducer()
@@ -44,6 +44,8 @@ export async function buildCaseDetail(handle: RunHandle, evidence: EvidenceServi
     metricDefinitions: definitionsOf(reducer, reduced.metrics.map(metric => metric.name)),
     metrics: [...reduced.metrics],
     output: lastRound?.output,
+    scoreDefinitions: [...(reducer.tasks.get(taskId)?.scoreDefinitions.values() ?? [])],
+    scoreRecords: reduced.scoreRecords.map(entry => ({ ...entry })),
     taskDisplayName: reducer.tasks.get(taskId)?.displayName ?? taskId,
   }
 }
@@ -63,7 +65,10 @@ export function buildLogs(handle: RunHandle, query: { caseId?: string, limit: nu
 }
 
 /**
- * Metrics of a run grouped by name: numeric series with summaries, other values as lists.
+ * Metrics of a run grouped by name: numeric series with summaries, other values as lists, and the scores beside them.
+ *
+ * Scores are one point per case: the value its last round counts with (the rule of `scoreCases`), so a failed case
+ * shows 0 and an unscored one shows nothing. Each score's summary pools the selected cases of every task.
  */
 export function buildMetrics(handle: RunHandle, filter: { caseId?: string, taskId?: string }): MetricsResponse {
   const reducer = handle.reducer()
@@ -80,6 +85,31 @@ export function buildMetrics(handle: RunHandle, filter: { caseId?: string, taskI
   }
   const cases = [...reducer.cases.values()].filter(reduced => (filter.taskId == null || reduced.taskId === filter.taskId) && (filter.caseId == null || reduced.caseId === filter.caseId))
   const inference = sumInference(cases.flatMap(reduced => reduced.inference))
+
+  const terminated = reducer.run().terminated
+  const scorePoints = new Map<string, MetricsResponse['scores'][number]['points']>()
+  const scoreSummaries: MetricsResponse['scores'][number]['summary'][] = []
+  for (const [taskId, taskCases] of Map.groupBy(cases, reduced => reduced.taskId)) {
+    const task = reducer.tasks.get(taskId)
+    const taskDefinitions = task?.scoreDefinitions ?? new Map()
+    // Every score of the task, also those only unselected cases recorded, so the selection counts like the task does.
+    const taskRounds = (task?.caseKeys ?? []).flatMap(key => toCaseNode(reducer.cases.get(key)!, { evidenceLinkCount: 0, terminated }).rounds)
+    const definitions = countingDefinitions(scoreRounds(taskRounds, [...taskDefinitions.values()]), taskDefinitions)
+    const lastRounds = taskCases.flatMap((reduced) => {
+      const round = toCaseNode(reduced, { evidenceLinkCount: 0, terminated }).rounds.at(-1)
+      return round == null ? [] : [{ reduced, round }]
+    })
+    scoreSummaries.push(...scoreRounds(lastRounds.map(({ round }) => round), definitions).scores)
+    for (const { reduced, round } of lastRounds) {
+      for (const [name, value] of Object.entries(countedScores(round, definitions))) {
+        const points = scorePoints.get(name) ?? []
+        points.push({ caseId: reduced.caseId, roundIndex: round.roundIndex, taskId, value })
+        scorePoints.set(name, points)
+      }
+    }
+  }
+  const scores = poolScoreSummaries(scoreSummaries).map(summary => ({ name: summary.name, points: scorePoints.get(summary.name) ?? [], summary }))
+
   return {
     definitions: definitionsOf(reducer, [...numeric.keys(), ...nonNumeric.keys()]),
     inference,
@@ -101,6 +131,8 @@ export function buildMetrics(handle: RunHandle, filter: { caseId?: string, taskI
         summary: { avg: sum / points.length, count: points.length, max, min, sum },
       }
     }).sort((left, right) => left.name.localeCompare(right.name)),
+    scoreDefinitions: pickScoreDefinitions(reducer.scoreDefinitions, scores.map(score => score.name)),
+    scores,
   }
 }
 
@@ -120,28 +152,31 @@ export function buildProgress(handle: RunHandle, now: number): ProgressResponse
 
 /**
  * Scores per task, per round, per case, and overall.
+ *
+ * Overall scores pool the cases of every task (`poolScoreSummaries`); they are not a mean of task means, so a large
+ * task weighs more than a small one.
  */
 export function buildScores(handle: RunHandle): ScoresResponse {
   const reducer = handle.reducer()
   const tasks = buildTaskNodes(reducer, () => 0)
   const byRound: ScoresResponse['byRound'] = []
   for (const task of tasks) {
+    const definitions = countingDefinitions(task.scoring, reducer.tasks.get(task.id)?.scoreDefinitions ?? new Map())
     const rounds = Map.groupBy(task.cases.flatMap(node => node.rounds), round => round.roundIndex)
     for (const [roundIndex, list] of [...rounds].sort((left, right) => left[0] - right[0])) {
-      byRound.push({ roundIndex, taskId: task.id, ...scoreRounds(list) })
+      byRound.push({ roundIndex, scoring: scoreRounds(list, definitions), taskId: task.id })
     }
   }
-  const exactValues = tasks.map(task => task.scores.exact).filter((value): value is number => value != null)
-  const judgeValues = tasks.map(task => task.scores.judge).filter((value): value is number => value != null)
-  const exact = average(exactValues)
-  const judge = average(judgeValues)
-  const summary = handle.summary()
   return {
     byRound,
-    byTask: tasks.map(task => ({ displayName: task.displayName, matrix: task.matrix, taskId: task.id, ...task.scores })),
+    byTask: tasks.map(task => ({ displayName: task.displayName, matrix: task.matrix, scoring: task.scoring, taskId: task.id })),
     cases: tasks.flatMap(task => task.cases.map(node => ({ caseId: node.caseId, name: node.name, scores: node.scores, state: node.state, taskId: node.taskId }))),
-    overall: { exact, hybrid: hybridAverage(exact, judge), judge },
-    summary: summary == null ? undefined : { exact: summary.exactAverage, hybrid: summary.hybridAverage, judge: summary.judgeAverage },
+    definitions: [...reducer.scoreDefinitions.values()],
+    overall: {
+      headline: resolveTaskHeadline(tasks),
+      scores: poolScoreSummaries(tasks.flatMap(task => task.scoring.scores)),
+    },
+    summary: handle.summary()?.headline,
   }
 }
 
@@ -153,16 +188,19 @@ export function buildSnapshot(handle: RunHandle, evidence: EvidenceService, now:
   const counts = evidence.countByCase(reducer)
   const run = reducer.run()
   const summary = handle.summary()
+  const tasks = buildTaskNodes(reducer, key => counts.get(key) ?? 0)
   return {
     current: currentPosition(reducer, now),
     endedAt: run.endedAt,
     eventCounts: { ...reducer.eventCounts },
+    headline: resolveTaskHeadline(tasks),
     identity: reducer.identity() ?? { attemptId: 'unknown-attempt', experimentId: 'unknown-experiment', runId: handle.runKey, workspaceId: 'unknown-workspace' },
     interruptSignal: run.interruptSignal ?? summary?.interruptSignal,
     parseErrors: handle.parseErrors(),
     progress: estimateProgress(reducer, now),
     projects: buildProjectNodes(reducer),
     runKey: handle.runKey,
+    scoreDefinitions: [...reducer.scoreDefinitions.values()],
     seq: reducer.seq(),
     seqGaps: reducer.seqGaps(),
     source: handle.source(),
@@ -172,11 +210,9 @@ export function buildSnapshot(handle: RunHandle, evidence: EvidenceService, now:
       ? undefined
       : {
           caseSummary: summary.caseSummary,
-          exactAverage: summary.exactAverage,
-          hybridAverage: summary.hybridAverage,
-          judgeAverage: summary.judgeAverage,
+          headline: summary.headline,
         },
-    tasks: buildTaskNodes(reducer, key => counts.get(key) ?? 0),
+    tasks,
   }
 }
 
diff --git a/packages/ui/src/shared/contract.ts b/packages/ui/src/shared/contract.ts
index 7bcbd70..c821fd3 100644
--- a/packages/ui/src/shared/contract.ts
+++ b/packages/ui/src/shared/contract.ts
@@ -13,7 +13,7 @@
  * - `retryIndex` is an immediate retry of one case inside one round (`autoRetry`).
  */
 
-import { localizedTextSchema, metricDefinedDataSchema } from 'vieval/events'
+import { graderSchema, localizedTextSchema, metricDefinedDataSchema, scoreDefinedDataSchema, scoreRecordedDataSchema } from 'vieval/events'
 
 import * as v from 'valibot'
 
@@ -22,6 +22,57 @@ export const VIEWER_API_VERSION = 1
 
 const nullableNumber = v.nullable(v.number())
 
+// ---------------------------------------------------------------------------------------------------------------
+// Scores (the shapes of `vieval/core/scoring`)
+// ---------------------------------------------------------------------------------------------------------------
+
+/** One named value over a group of cases (`ScoreSummary` of `vieval/core/scoring`). */
+export const scoreSummarySchema = v.object({
+  /** `sum / n`, or `null` when no case counted. */
+  mean: nullableNumber,
+  /** Cases that counted toward `mean`. */
+  n: v.number(),
+  /** Score name; `completion` and `passed` for the built-in summaries. */
+  name: v.string(),
+  sum: v.number(),
+  /** Settled cases that were not skipped. */
+  total: v.number(),
+  /** Cases that did not count, per reason (unscored reasons; `errored` / `timeout` for `completion`). */
+  unscored: v.record(v.string(), v.number()),
+})
+
+/** Scores of one group of cases, for example one task (`CaseScoring` of `vieval/core/scoring`). */
+export const caseScoringSchema = v.object({
+  completion: scoreSummarySchema,
+  passed: scoreSummarySchema,
+  /** Name of the headline score of the group, when it has one. */
+  primary: v.optional(v.string()),
+  scores: v.array(scoreSummarySchema),
+})
+
+/** Resolved headline and completion rate (`HeadlineResult` of `vieval/core/scoring`). */
+export const headlineResultSchema = v.object({
+  completion: scoreSummarySchema,
+  headline: v.variant('kind', [
+    v.object({ kind: v.literal('score'), score: scoreSummarySchema }),
+    v.object({ kind: v.literal('passed-cases'), score: scoreSummarySchema }),
+    v.object({ kind: v.literal('per-task'), tasks: v.array(v.object({ score: scoreSummarySchema, taskId: v.string() })) }),
+  ]),
+})
+
+/**
+ * Graders that produced each score of a group of cases, by score name: every distinct kind, model, and version seen in
+ * the score records (a record's own grader, else the definition's). Two groups whose graders of one score differ do
+ * not measure the same thing.
+ */
+export const scoreGradersSchema = v.record(v.string(), v.array(graderSchema))
+
+/** Score values of one case round: recorded or derived values, and unscored records with their reason. */
+export const caseScoresSchema = v.object({
+  unscored: v.record(v.string(), v.string()),
+  values: v.record(v.string(), v.number()),
+})
+
 // ---------------------------------------------------------------------------------------------------------------
 // Runs and the report hierarchy
 // ---------------------------------------------------------------------------------------------------------------
@@ -32,17 +83,18 @@ const nullableNumber = v.nullable(v.number())
  * - `live-bus`: a run of this process (`vieval run --with-viewer`), fed by its in-process event bus.
  * - `live-tail`: a run written by another process; its `events.jsonl` is tailed incrementally.
  * - `offline`: a finished run read from its report directory.
- * - `legacy`: a report written with event schema version 1; only `run-summary.json` / `cases.jsonl` are read.
+ * - `unsupported`: a report written by an older vieval (event schema version below 3). It is listed, but its events are not
+ *   read and every route that opens it answers 422 ("This report was written by an older vieval. Run it again.").
  */
-export const runSourceSchema = v.picklist(['live-bus', 'live-tail', 'offline', 'legacy'])
+export const runSourceSchema = v.picklist(['live-bus', 'live-tail', 'offline', 'unsupported'])
 
 /**
  * Run state.
  *
  * `stale` means: no terminal event, and the last event is older than the stale threshold (the writer probably
- * died). It is shown, not guessed as failed.
+ * died). It is shown, not guessed as failed. `unsupported` is the state of an `unsupported` source: a run that cannot be read.
  */
-export const runStateSchema = v.picklist(['running', 'stale', 'ended', 'interrupted', 'legacy'])
+export const runStateSchema = v.picklist(['running', 'stale', 'ended', 'interrupted', 'unsupported'])
 
 export const runIdentitySchema = v.object({
   attemptId: v.string(),
@@ -77,9 +129,10 @@ export const runListItemSchema = v.object({
   endedAt: v.optional(v.string()),
   /** Eval-matrix axes of the run's tasks and their distinct values, sorted, like `matrix` for the run matrix. */
   evalMatrix: v.record(v.string(), v.array(v.string())),
-  exactAverage: nullableNumber,
-  hybridAverage: nullableNumber,
-  judgeAverage: nullableNumber,
+  /** Headline of the run: from `run-summary.json` when written, else from its events. `null` before any case settled. */
+  headline: v.nullable(headlineResultSchema),
+  /** Latest `score.defined` of each score in `headline`, for its label and description. */
+  headlineDefinitions: v.array(scoreDefinedDataSchema),
   lastEventAt: v.optional(v.string()),
   lastSeq: v.number(),
   /**
@@ -141,11 +194,6 @@ export const tryNodeSchema = v.object({
   state: v.union([caseStateSchema, v.literal('retried')]),
 })
 
-export const scoreValuesSchema = v.object({
-  exact: v.optional(v.number()),
-  judge: v.optional(v.number()),
-})
-
 export const caseRoundSchema = v.object({
   endedAt: v.optional(v.string()),
   errorMessage: v.optional(v.string()),
@@ -159,8 +207,8 @@ export const caseRoundSchema = v.object({
   requeueDelayMs: v.optional(v.number()),
   roundIndex: v.number(),
   roundIndexInferred: v.boolean(),
-  /** Scores recorded with `recordScore` during this round. */
-  scores: scoreValuesSchema,
+  /** Scores of the round's last try: recorded with `recordScore` / `recordUnscored`, or derived. */
+  scores: caseScoresSchema,
   startedAt: v.string(),
   state: caseStateSchema,
   tries: v.array(tryNodeSchema),
@@ -189,7 +237,7 @@ export const caseNodeSchema = v.object({
   projectId: v.optional(v.string()),
   rounds: v.array(caseRoundSchema),
   /** Scores of the last round. */
-  scores: scoreValuesSchema,
+  scores: caseScoresSchema,
   startedAt: v.string(),
   /** State of the last round (the final state). */
   state: caseStateSchema,
@@ -221,12 +269,10 @@ export const taskNodeSchema = v.object({
   matrix: v.optional(taskMatrixSchema),
   projectId: v.string(),
   roundCount: v.number(),
-  /** Score of the task: every round's case outcome, averaged (the rule of `describeTask`). */
-  scores: v.object({
-    exact: nullableNumber,
-    hybrid: nullableNumber,
-    judge: nullableNumber,
-  }),
+  /** Graders that produced each score of the task. */
+  scoreGraders: scoreGradersSchema,
+  /** Scores of the task over every round's case outcome (`scoreCases`, the rule of `describeTask`). */
+  scoring: caseScoringSchema,
   startedAt: v.optional(v.string()),
   state: taskStateSchema,
 })
@@ -310,6 +356,8 @@ export const runSnapshotSchema = v.object({
   current: currentPositionSchema,
   endedAt: v.optional(v.string()),
   eventCounts: v.record(v.string(), v.number()),
+  /** Headline of the run from its events (`resolveHeadline` over the tasks), `null` before any case settled. */
+  headline: v.nullable(headlineResultSchema),
   identity: runIdentitySchema,
   interruptSignal: v.optional(v.string()),
   /** Lines of `events.jsonl` that were not valid envelopes (live tail only). */
@@ -317,6 +365,8 @@ export const runSnapshotSchema = v.object({
   progress: runProgressSchema,
   projects: v.array(projectNodeSchema),
   runKey: v.string(),
+  /** Latest `score.defined` of each score of the run. */
+  scoreDefinitions: v.array(scoreDefinedDataSchema),
   /** Highest applied `seq`. Subscribe to the stream with `?after=<seq>` to continue from this snapshot. */
   seq: v.number(),
   /** Missing `seq` numbers seen while reading; duplicates are dropped silently. */
@@ -327,9 +377,7 @@ export const runSnapshotSchema = v.object({
   /** Official summary written after the run (`run-summary.json`), when present. */
   summary: v.optional(v.object({
     caseSummary: v.optional(caseCountsSchema),
-    exactAverage: nullableNumber,
-    hybridAverage: nullableNumber,
-    judgeAverage: nullableNumber,
+    headline: v.optional(headlineResultSchema),
   })),
   tasks: v.array(taskNodeSchema),
 })
@@ -432,6 +480,14 @@ export const caseDetailSchema = v.object({
   metricDefinitions: v.array(metricDefinedDataSchema),
   metrics: v.array(metricEntrySchema),
   output: v.optional(v.unknown()),
+  /** Latest `score.defined` of each score of the case's task (label, description, combination, grader). */
+  scoreDefinitions: v.array(scoreDefinedDataSchema),
+  /** Every `score.recorded` of the case, with its rationale, grader, and metadata, in event order. */
+  scoreRecords: v.array(v.object({
+    record: scoreRecordedDataSchema,
+    roundIndex: v.optional(v.number()),
+    seq: v.number(),
+  })),
   taskDisplayName: v.string(),
 })
 
@@ -509,37 +565,44 @@ export const metricsResponseSchema = v.object({
       sum: v.number(),
     }),
   })),
-})
-
-const scoreTripleSchema = v.object({
-  exact: nullableNumber,
-  hybrid: nullableNumber,
-  judge: nullableNumber,
+  /** Latest `score.defined` of each score in `scores`. */
+  scoreDefinitions: v.array(scoreDefinedDataSchema),
+  /** Scores beside the metrics: each case's counted value (the last round), with the summary of the selection. */
+  scores: v.array(v.object({
+    name: v.string(),
+    points: v.array(v.object({ caseId: v.string(), roundIndex: v.optional(v.number()), taskId: v.string(), value: v.number() })),
+    summary: scoreSummarySchema,
+  })),
 })
 
 export const scoresResponseSchema = v.object({
   byRound: v.array(v.object({
     roundIndex: v.number(),
-    ...scoreTripleSchema.entries,
+    scoring: caseScoringSchema,
     taskId: v.string(),
   })),
   byTask: v.array(v.object({
     displayName: v.string(),
     matrix: v.optional(taskMatrixSchema),
-    ...scoreTripleSchema.entries,
+    scoring: caseScoringSchema,
     taskId: v.string(),
   })),
   cases: v.array(v.object({
     caseId: v.string(),
     name: v.string(),
-    scores: scoreValuesSchema,
+    scores: caseScoresSchema,
     state: caseStateSchema,
     taskId: v.string(),
   })),
-  /** From the run's events (all rounds, `describeTask` rule). */
-  overall: scoreTripleSchema,
+  /** Latest `score.defined` of each score of the run. */
+  definitions: v.array(scoreDefinedDataSchema),
+  /** From the run's events (all rounds, `describeTask` rule): the headline, and each score pooled over the tasks. */
+  overall: v.object({
+    headline: v.nullable(headlineResultSchema),
+    scores: v.array(scoreSummarySchema),
+  }),
   /** From `run-summary.json`, once written. */
-  summary: v.optional(scoreTripleSchema),
+  summary: v.optional(headlineResultSchema),
 })
 
 // ---------------------------------------------------------------------------------------------------------------
@@ -616,10 +679,13 @@ export const matrixCellSchema = v.object({
   /** Every run that executed this cell, oldest first (one point per run for the cell's sparkline). */
   history: v.array(v.object({
     attemptId: v.string(),
+    /** Completion rate of the run's task. */
+    completion: nullableNumber,
     counts: caseCountsSchema,
     coverage: cellCoverageSchema,
-    exact: nullableNumber,
     runKey: v.string(),
+    /** Mean of the cell's primary score in this run, or `null`. */
+    score: nullableNumber,
     startedAt: v.optional(v.string()),
     state: v.string(),
   })),
@@ -629,16 +695,16 @@ export const matrixCellSchema = v.object({
    */
   latest: v.object({
     attemptId: v.string(),
-    /** Exact score of every case of the run (`null`: no score), for the spread of the score under the mean. */
+    /** Primary score of every case of the run (`null`: no value), for the spread of the score under the mean. */
     caseScores: v.array(nullableNumber),
     /** Expected cases of the task (`case.started.total`); `undefined` until its first case started. */
     caseTotal: v.optional(v.number()),
     counts: caseCountsSchema,
     coverage: cellCoverageSchema,
-    exact: nullableNumber,
     experimentId: v.string(),
-    judge: nullableNumber,
     runKey: v.string(),
+    /** Scores of the task in this run. */
+    scoring: caseScoringSchema,
     startedAt: v.optional(v.string()),
     /** Task state (see `TaskNode.state`). */
     state: v.string(),
@@ -666,6 +732,8 @@ export const matrixResponseSchema = v.object({
   cells: v.array(matrixCellSchema),
   /** Runs that were read to build the cells. */
   runCount: v.number(),
+  /** Latest `score.defined` of each score in the cells. */
+  scoreDefinitions: v.array(scoreDefinedDataSchema),
 })
 
 export const screenshotTimelineSchema = v.object({
@@ -751,19 +819,16 @@ export const comparedRunSchema = v.object({
   attemptId: v.string(),
   counts: caseCountsSchema,
   durationMs: v.optional(v.number()),
-  exactAverage: nullableNumber,
   experimentId: v.string(),
-  hybridAverage: nullableNumber,
-  judgeAverage: nullableNumber,
+  headline: v.nullable(headlineResultSchema),
   runKey: v.string(),
   startedAt: v.optional(v.string()),
   state: runStateSchema,
 })
 
 export const comparisonCellSchema = v.object({
-  exact: v.optional(v.number()),
-  judge: v.optional(v.number()),
   roundCount: v.number(),
+  scores: caseScoresSchema,
   state: caseStateSchema,
 })
 
@@ -773,14 +838,20 @@ export const comparisonResponseSchema = v.object({
     id: v.string(),
     label: v.string(),
     runs: v.array(comparedRunSchema),
+    /** Graders that produced each score in the column's runs. */
+    scoreGraders: scoreGradersSchema,
   })),
+  /** Primary score the rows compare, when the columns share one. */
+  primary: v.optional(v.string()),
   /** Rows: one per case name; `cells[columnId]` is the case result in that column (latest run of the column). */
   rows: v.array(v.object({
     caseName: v.string(),
     cells: v.record(v.string(), comparisonCellSchema),
-    /** `true` when the case state or exact score differs between columns. */
+    /** `true` when the case state or the primary score differs between columns. */
     changed: v.boolean(),
   })),
+  /** Latest `score.defined` of each score in the cells. */
+  scoreDefinitions: v.array(scoreDefinedDataSchema),
   summary: v.object({
     changed: v.number(),
     rows: v.number(),
@@ -798,6 +869,8 @@ export type CaseCounts = v.InferOutput<typeof caseCountsSchema>
 export type CaseDetail = v.InferOutput<typeof caseDetailSchema>
 export type CaseNode = v.InferOutput<typeof caseNodeSchema>
 export type CaseRound = v.InferOutput<typeof caseRoundSchema>
+export type CaseScores = v.InferOutput<typeof caseScoresSchema>
+export type CaseScoring = v.InferOutput<typeof caseScoringSchema>
 export type CaseState = v.InferOutput<typeof caseStateSchema>
 export type CellCoverage = v.InferOutput<typeof cellCoverageSchema>
 export type ComparedRun = v.InferOutput<typeof comparedRunSchema>
@@ -809,6 +882,7 @@ export type EvidenceLink = v.InferOutput<typeof evidenceLinkSchema>
 export type ExternalLogsResponse = v.InferOutput<typeof externalLogsResponseSchema>
 export type FileEntry = v.InferOutput<typeof fileEntrySchema>
 export type FileTreeResponse = v.InferOutput<typeof fileTreeResponseSchema>
+export type HeadlineResult = v.InferOutput<typeof headlineResultSchema>
 export type ImagePreview = v.InferOutput<typeof imagePreviewSchema>
 export type JsonlPage = v.InferOutput<typeof jsonlPageSchema>
 export type LogLine = v.InferOutput<typeof logLineSchema>
@@ -832,7 +906,10 @@ export type RunProgress = v.InferOutput<typeof runProgressSchema>
 export type RunSnapshot = v.InferOutput<typeof runSnapshotSchema>
 export type RunSource = v.InferOutput<typeof runSourceSchema>
 export type RunState = v.InferOutput<typeof runStateSchema>
+export type ScoreDefinition = v.InferOutput<typeof scoreDefinedDataSchema>
+export type ScoreGraders = v.InferOutput<typeof scoreGradersSchema>
 export type ScoresResponse = v.InferOutput<typeof scoresResponseSchema>
+export type ScoreSummary = v.InferOutput<typeof scoreSummarySchema>
 export type ScreenshotTimeline = v.InferOutput<typeof screenshotTimelineSchema>
 export type TaskMatrix = v.InferOutput<typeof taskMatrixSchema>
 export type TaskNode = v.InferOutput<typeof taskNodeSchema>
diff --git a/packages/ui/src/shared/reduce.test.ts b/packages/ui/src/shared/reduce.test.ts
index 5f7cefd..1f2cd55 100644
--- a/packages/ui/src/shared/reduce.test.ts
+++ b/packages/ui/src/shared/reduce.test.ts
@@ -1,7 +1,9 @@
+import type { CaseRound } from './contract'
+
 import { describe, expect, it } from 'vitest'
 
 import { createEnvelopeFactory } from '../../test/helpers'
-import { axisValues, buildTaskNodes, createRunReducer, currentPosition, estimateProgress, parseTaskId, scoreRounds } from './reduce'
+import { axisValues, buildTaskNodes, countedScores, createRunReducer, currentPosition, estimateProgress, parseTaskId, resolveTaskHeadline, scoreRounds } from './reduce'
 
 const TASK = 'evals%2Fdemo::default::run=model%3Dm%26repeat%3D1::eval=default'
 const scope = (caseId?: string) => ({ caseId, projectId: 'demo', taskId: TASK })
@@ -16,7 +18,7 @@ describe('createRunReducer', () => {
     reducer.apply(envelope('task.started', {}, scope()))
     reducer.apply(envelope('case.started', { index: 0, input: { id: 'a' }, name: 'chrome/a', total: 2 }, scope('0:chrome%2Fa')))
     reducer.apply(envelope('metric.recorded', { name: 'osworld.pod', value: 'pod-a' }, scope('0:chrome%2Fa')))
-    reducer.apply(envelope('score.recorded', { kind: 'exact', score: 0.5 }, scope('0:chrome%2Fa')))
+    reducer.apply(envelope('score.recorded', { name: 'osworld.success', value: 0.5 }, scope('0:chrome%2Fa')))
     reducer.apply(envelope('case.ended', { index: 0, name: 'chrome/a', state: 'passed', total: 2 }, scope('0:chrome%2Fa')))
 
     const [task] = buildTaskNodes(reducer, () => 0)
@@ -26,9 +28,10 @@ describe('createRunReducer', () => {
     expect(task.matrix).toEqual({ eval: {}, evalRowId: 'default', run: { model: 'm', repeat: '1' }, runRowId: 'model=m&repeat=1' })
     expect(task.cases).toHaveLength(1)
     expect(task.cases[0].state).toBe('passed')
-    expect(task.cases[0].scores).toEqual({ exact: 0.5 })
+    expect(task.cases[0].scores).toEqual({ unscored: {}, values: { 'osworld.success': 0.5 } })
     expect(task.cases[0].metricCount).toBe(1)
-    expect(task.scores.exact).toBe(0.5)
+    expect(task.scoring.primary).toBe('osworld.success')
+    expect(task.scoring.scores).toEqual([{ mean: 0.5, n: 1, name: 'osworld.success', sum: 0.5, total: 1, unscored: {} }])
     expect([...reducer.projects.keys()]).toEqual(['demo'])
   })
 
@@ -47,7 +50,8 @@ describe('createRunReducer', () => {
     let [task] = buildTaskNodes(reducer, () => 0)
     expect(task.cases[0].state).toBe('requeued')
     expect(task.caseCounts).toMatchObject({ errored: 0, requeued: 1 })
-    expect(task.scores.exact).toBeNull()
+    // A requeued round has not ended, so it does not count toward completion yet.
+    expect(task.scoring.completion.total).toBe(0)
     expect(currentPosition(reducer, 0).waiting).toEqual([expect.objectContaining({ delayMs: 30_000, name: 'a', requeue: 2, requeueIndex: 1 })])
     expect(estimateProgress(reducer, 0).cases.completed).toBe(0)
 
@@ -62,7 +66,8 @@ describe('createRunReducer', () => {
     expect(round.tries[0]!.errorMessage).toBe('Remote sent 429 response')
     expect(round.errorMessage).toBeUndefined()
     expect(task.cases[0].state).toBe('passed')
-    expect(task.scores.exact).toBe(1)
+    expect(task.scoring.passed.mean).toBe(1)
+    expect(task.scoring.completion.mean).toBe(1)
     expect(estimateProgress(reducer, 0).cases.completed).toBe(1)
     expect(currentPosition(reducer, 0).waiting).toEqual([])
   })
@@ -101,7 +106,7 @@ describe('createRunReducer', () => {
     expect(task.cases[0].errorName).toBe('APICallError')
     expect(task.cases[0].rounds[0]?.errorName).toBe('APICallError')
     expect(task.caseCounts.errored).toBe(1)
-    expect(task.scores.exact).toBeNull()
+    expect(task.scoring.completion).toMatchObject({ mean: 0, n: 1, total: 1, unscored: { errored: 1 } })
   })
 
   /**
@@ -144,8 +149,8 @@ describe('createRunReducer', () => {
     expect(attempt.state).toBe('passed')
     expect(retry.rounds[0].tries.map(entry => [entry.retryIndex, entry.state])).toEqual([[0, 'retried'], [1, 'passed']])
     expect(retry.rounds[1].tries.map(entry => [entry.retryIndex, entry.state])).toEqual([[0, 'passed']])
-    // describeTask averages every round: failed (0) + passed (1) + passed (1) + passed (1).
-    expect(task.scores.exact).toBe(0.75)
+    // The passed share counts each case once by its last round: both cases end passed.
+    expect(task.scoring.passed).toMatchObject({ mean: 1, n: 2 })
     expect(estimateProgress(reducer, Date.UTC(2026, 0, 1, 0, 1)).lowerBound).toBe(true)
   })
 
@@ -355,33 +360,175 @@ describe('axisValues', () => {
   })
 })
 
-describe('scoreRounds / parseTaskId', () => {
-  it('follows scoreCaseOutcomes: failed → 0, passed without score → 1, judge averaged separately', () => {
-    expect(scoreRounds([
-      { scores: {}, state: 'failed' },
-      { scores: {}, state: 'passed' },
-      { scores: { exact: 0.5, judge: 0.8 }, state: 'passed' },
-      { scores: {}, state: 'running' },
-    ])).toEqual({ coverage: { measured: 3, total: 3 }, exact: 0.5, hybrid: 0.65, judge: 0.8, strictExact: 0.5 })
+describe('named scores', () => {
+  const CORRECT = { description: { en: 'Matches the reference.' }, label: { 'en': 'Correct', 'zh-Hans': '正确' }, name: 'formula.correct' }
+  const PASS = { combine: { inputs: [{ name: 'formula.correct' }], kind: 'all' as const }, name: 'formula.pass', primary: true }
+
+  /**
+   * @example
+   * expect(reducer.scoreDefinitions.get('formula.correct')?.label).toEqual({ en: 'Correct answer' })
+   */
+  it('keeps the latest definition per name, in the order the task first announced them', () => {
+    const envelope = createEnvelopeFactory()
+    const reducer = createRunReducer()
+    reducer.apply(envelope('score.defined', CORRECT, scope()))
+    reducer.apply(envelope('score.defined', PASS, scope()))
+    reducer.apply(envelope('score.defined', { ...CORRECT, label: { en: 'Correct answer' } }, scope()))
+    reducer.apply(envelope('score.defined', { name: 42 }, scope()))
+
+    expect(reducer.scoreDefinitions.get('formula.correct')?.label).toEqual({ en: 'Correct answer' })
+    expect([...reducer.tasks.get(TASK)!.scoreDefinitions.keys()]).toEqual(['formula.correct', 'formula.pass'])
+    const [task] = buildTaskNodes(reducer, () => 0)
+    // A declared score is known, with its primary flag, before any case records it.
+    expect(task.scoring.primary).toBe('formula.pass')
+    expect(task.scoring.scores.map(score => score.name)).toEqual(['formula.correct', 'formula.pass'])
   })
 
   /**
    * @example
-   * errored and unfinished rounds stay out of exact and lower the coverage, as in the run summary of vieval.
+   * expect(round.scores).toEqual({ unscored: { 'formula.natural': 'grader_failed' }, values: { 'formula.correct': 1 } })
    */
-  it('leaves rounds without a result out of exact and counts them in coverage', () => {
-    const scores = scoreRounds([
-      { scores: {}, state: 'passed' },
-      { scores: {}, state: 'errored' },
-      { scores: {}, state: 'timeout' },
-      { scores: {}, state: 'unfinished' },
+  it('keeps the last record per name of the last try of a round, and every record for the case detail', () => {
+    const envelope = createEnvelopeFactory()
+    const reducer = createRunReducer()
+    const caseScope = scope('0:a')
+    reducer.apply(envelope('case.started', { autoRetry: 1, index: 0, name: 'a', retryIndex: 0, total: 1 }, caseScope))
+    reducer.apply(envelope('score.recorded', { name: 'formula.correct', value: 0 }, caseScope))
+    reducer.apply(envelope('case.started', { autoRetry: 1, index: 0, name: 'a', retryIndex: 1, total: 1 }, caseScope))
+    reducer.apply(envelope('score.recorded', { name: 'formula.natural', value: 0.5 }, caseScope))
+    reducer.apply(envelope('score.recorded', { name: 'formula.natural', rationale: 'no verdict', unscored: { reason: 'grader_failed' } }, caseScope))
+    reducer.apply(envelope('score.recorded', { grader: { id: 'listener', kind: 'model', model: 'm' }, name: 'formula.correct', value: 1 }, caseScope))
+    reducer.apply(envelope('score.recorded', { name: 'formula.broken', value: 'high' }, caseScope))
+    reducer.apply(envelope('case.ended', { index: 0, name: 'a', state: 'passed', total: 1 }, caseScope))
+
+    const [task] = buildTaskNodes(reducer, () => 0)
+    const [round] = task.cases[0].rounds
+    // The failed try's value is gone: the round's scores are those of its last try.
+    expect(round.scores).toEqual({ unscored: { 'formula.natural': 'grader_failed' }, values: { 'formula.correct': 1 } })
+    expect(task.cases[0].scores).toEqual(round.scores)
+    const records = reducer.cases.get(`${TASK}\u00000:a`)!.scoreRecords
+    expect(records.map(entry => [entry.seq, entry.roundIndex, entry.record.name])).toEqual([
+      [2, 0, 'formula.correct'],
+      [4, 0, 'formula.natural'],
+      [5, 0, 'formula.natural'],
+      [6, 0, 'formula.correct'],
+    ])
+    expect(records[2].record).toEqual({ name: 'formula.natural', rationale: 'no verdict', unscored: { reason: 'grader_failed' } })
+    expect(records[3].record).toMatchObject({ grader: { id: 'listener', kind: 'model', model: 'm' } })
+    expect(task.scoring.scores).toEqual([
+      { mean: 1, n: 1, name: 'formula.correct', sum: 1, total: 1, unscored: {} },
+      { mean: null, n: 0, name: 'formula.natural', sum: 0, total: 1, unscored: { grader_failed: 1 } },
     ])
+  })
+
+  /**
+   * @example
+   * expect(scoring.scores[0]).toMatchObject({ mean: 0.25, n: 4, total: 6 })
+   */
+  it('counts rounds with the rule of scoreCases: failed 0, errored excluded, unfinished as errored, running left out', () => {
+    const rounds: Pick<CaseRound, 'scores' | 'state'>[] = [
+      { scores: { unscored: {}, values: { s: 1 } }, state: 'passed' },
+      { scores: { unscored: {}, values: { s: 1 } }, state: 'failed' },
+      { scores: { unscored: {}, values: {} }, state: 'passed' },
+      { scores: { unscored: { s: 'refusal' }, values: {} }, state: 'passed' },
+      { scores: { unscored: {}, values: { s: 1 } }, state: 'errored' },
+      { scores: { unscored: {}, values: {} }, state: 'unfinished' },
+      { scores: { unscored: {}, values: { s: 1 } }, state: 'running' },
+      { scores: { unscored: {}, values: { s: 1 } }, state: 'skipped' },
+    ]
+    const scoring = scoreRounds(rounds)
+    // passed 1, failed 0; the passed round without a value and the unscored one do not count.
+    expect(scoring.scores).toEqual([{ mean: 0.5, n: 2, name: 's', sum: 1, total: 6, unscored: { refusal: 1 } }])
+    expect(scoring.completion).toMatchObject({ mean: 4 / 6, n: 6, total: 6, unscored: { errored: 2 } })
+    expect(scoring.passed.mean).toBe(3 / 6)
+
+    // `errors: 'zero'` counts the errored and the unfinished round as 0.
+    const zero = scoreRounds(rounds, [{ errors: 'zero', name: 's' }])
+    expect(zero.scores[0]).toMatchObject({ mean: 0.25, n: 4, sum: 1 })
+  })
+
+  /**
+   * @example
+   * expect(countedScores({ scores, state: 'failed' })).toEqual({ s: 0 })
+   */
+  it('returns the value one round counts with', () => {
+    const scores = { unscored: { u: 'refusal' }, values: { s: 0.5 } }
+    expect(countedScores({ scores, state: 'passed' })).toEqual({ s: 0.5 })
+    expect(countedScores({ scores, state: 'failed' })).toEqual({ s: 0, u: 0 })
+    expect(countedScores({ scores, state: 'errored' })).toEqual({})
+    expect(countedScores({ scores, state: 'errored' }, [{ errors: 'zero', name: 's' }])).toEqual({ s: 0 })
+  })
+
+  /**
+   * @example
+   * expect(resolveTaskHeadline(tasks)?.headline.kind).toBe('per-task')
+   */
+  it('resolves the run headline over the tasks, by eval entry', () => {
+    const envelope = createEnvelopeFactory()
+    const reducer = createRunReducer()
+    const other = 'evals%2Fother::default::run=model%3Dm::eval=default'
+    const otherScope = (caseId?: string) => ({ caseId, projectId: 'demo', taskId: other })
+    reducer.apply(envelope('case.started', { index: 0, name: 'a', total: 1 }, scope('0:a')))
+    expect(resolveTaskHeadline(buildTaskNodes(reducer, () => 0))).toBeNull()
+
+    reducer.apply(envelope('score.recorded', { name: 'demo.score', value: 1 }, scope('0:a')))
+    reducer.apply(envelope('case.ended', { index: 0, name: 'a', state: 'passed', total: 1 }, scope('0:a')))
+    expect(resolveTaskHeadline(buildTaskNodes(reducer, () => 0))?.headline).toEqual({ kind: 'score', score: { mean: 1, n: 1, name: 'demo.score', sum: 1, total: 1, unscored: {} } })
+
+    reducer.apply(envelope('case.started', { index: 0, name: 'b', total: 1 }, otherScope('0:b')))
+    reducer.apply(envelope('score.recorded', { name: 'other.score', value: 0 }, otherScope('0:b')))
+    reducer.apply(envelope('case.ended', { index: 0, name: 'b', state: 'passed', total: 1 }, otherScope('0:b')))
+    const headline = resolveTaskHeadline(buildTaskNodes(reducer, () => 0))
+    expect(headline?.completion).toMatchObject({ mean: 1, n: 2 })
+    expect(headline?.headline.kind === 'per-task' ? headline.headline.tasks.map(entry => [entry.taskId, entry.score.name]) : []).toEqual([['evals/demo', 'demo.score'], ['evals/other', 'other.score']])
+  })
+
+  /**
+   * @example
+   * expect(resolveTaskHeadline(tasks)?.headline).toMatchObject({ kind: 'passed-cases' })
+   */
+  it('falls back to the passed share when no task records a score', () => {
+    const envelope = createEnvelopeFactory()
+    const reducer = createRunReducer()
+    reducer.apply(envelope('case.started', { index: 0, name: 'a', total: 2 }, scope('0:a')))
+    reducer.apply(envelope('case.ended', { index: 0, name: 'a', state: 'passed', total: 2 }, scope('0:a')))
+    reducer.apply(envelope('case.started', { index: 1, name: 'b', total: 2 }, scope('1:b')))
+    reducer.apply(envelope('case.ended', { index: 1, name: 'b', state: 'failed', total: 2 }, scope('1:b')))
+    expect(resolveTaskHeadline(buildTaskNodes(reducer, () => 0))?.headline).toEqual({ kind: 'passed-cases', score: { mean: 0.5, n: 2, name: 'passed', sum: 1, total: 2, unscored: {} } })
+  })
+})
 
-    expect(scores.exact).toBe(1)
-    expect(scores.coverage).toEqual({ measured: 1, total: 4 })
-    expect(scores.strictExact).toBe(0.25)
+describe('score graders', () => {
+  /**
+   * @example
+   * definition grader judge-a; one record overrides with judge-b; a derived record is skipped ->
+   * task.scoreGraders['formula.natural'] holds judge-a and judge-b once each.
+   */
+  it('lists each distinct grader of a score: the record\'s own, else the definition\'s', () => {
+    const envelope = createEnvelopeFactory()
+    const reducer = createRunReducer()
+    reducer.apply(envelope('score.defined', { grader: { id: 'judge', kind: 'model', model: 'judge-a' }, name: 'formula.natural' }, scope()))
+    reducer.apply(envelope('score.defined', { combine: { inputs: [{ name: 'formula.natural' }], kind: 'all' }, name: 'formula.pass' }, scope()))
+    for (const [index, grader] of [undefined, { id: 'judge', kind: 'model', model: 'judge-b' }, undefined].entries()) {
+      const caseScope = scope(`${index}:c${index}`)
+      reducer.apply(envelope('case.started', { index, name: `c${index}`, total: 3 }, caseScope))
+      reducer.apply(envelope('score.recorded', { ...(grader == null ? {} : { grader }), name: 'formula.natural', value: 1 }, caseScope))
+      reducer.apply(envelope('score.recorded', { derived: true, name: 'formula.pass', value: 1 }, caseScope))
+      reducer.apply(envelope('case.ended', { index, name: `c${index}`, state: 'passed', total: 3 }, caseScope))
+    }
+
+    const [task] = buildTaskNodes(reducer, () => 0)
+    expect(task.scoreGraders).toEqual({
+      'formula.natural': [{ id: 'judge', kind: 'model', model: 'judge-a' }, { id: 'judge', kind: 'model', model: 'judge-b' }],
+    })
   })
+})
 
+describe('parseTaskId', () => {
+  /**
+   * @example
+   * expect(parseTaskId('custom')).toEqual({})
+   */
   it('returns no matrix for ids that are not scheduled task ids', () => {
     expect(parseTaskId('custom')).toEqual({})
   })
diff --git a/packages/ui/src/shared/reduce.ts b/packages/ui/src/shared/reduce.ts
index 1d17dac..e7a4554 100644
--- a/packages/ui/src/shared/reduce.ts
+++ b/packages/ui/src/shared/reduce.ts
@@ -1,10 +1,12 @@
-import type { CaseOutcomeScores, CaseOutcomeState } from 'vieval/core/scoring'
-import type { LocalizedText } from 'vieval/events'
+import type { CaseOutcomeState, HeadlineResult } from 'vieval/core/scoring'
+import type { Grader, LocalizedText, ScoreRecordedData } from 'vieval/events'
 
 import type {
   CaseCounts,
   CaseNode,
   CaseRound,
+  CaseScores,
+  CaseScoring,
   CaseState,
   CurrentPosition,
   LogLine,
@@ -14,12 +16,17 @@ import type {
   RunEvent,
   RunIdentity,
   RunProgress,
+  ScoreDefinition,
+  ScoreGraders,
   TaskMatrix,
   TaskNode,
   TryNode,
 } from './contract'
 
-import { scoreCaseOutcomes } from 'vieval/core/scoring'
+import { countCase, resolveHeadline, scoreCases } from 'vieval/core/scoring'
+import { scoreDefinedDataSchema, scoreRecordedDataSchema } from 'vieval/events'
+
+import * as v from 'valibot'
 
 /**
  * Result of {@link RunReducer.apply}.
@@ -63,6 +70,8 @@ export interface ReducedCase {
   name: string
   projectId?: string
   rounds: MutableRound[]
+  /** Every `score.recorded` of the case, in event order, with the round it was recorded in. */
+  scoreRecords: ReducedScoreRecord[]
   taskId: string
 }
 
@@ -77,6 +86,15 @@ export interface ReducedInferenceEvent {
   time: string
 }
 
+/**
+ * One `score.recorded` event, attached to its case.
+ */
+export interface ReducedScoreRecord {
+  record: ScoreRecordedData
+  roundIndex?: number
+  seq: number
+}
+
 /**
  * Reduced state of one task.
  */
@@ -88,6 +106,8 @@ export interface ReducedTask {
   endedAt?: string
   id: string
   projectId: string
+  /** Latest `score.defined` of the task per score name, in the order the task first announced them. */
+  scoreDefinitions: Map<string, ScoreDefinition>
   startedAt?: string
   state: TaskNode['state']
 }
@@ -117,6 +137,8 @@ export interface RunReducer {
     terminated: boolean
     totalTasks?: number
   }
+  /** Latest `score.defined` per score name over every task of the run. */
+  readonly scoreDefinitions: ReadonlyMap<string, ScoreDefinition>
   readonly seq: () => number
   readonly seqGaps: () => number
   /**
@@ -148,7 +170,8 @@ interface MutableRound {
   requeueDelayMs?: number
   roundIndex: number
   roundIndexInferred: boolean
-  scores: { exact?: number, judge?: number }
+  /** Scores of the round's current (last) try; a new try of the round starts empty. */
+  scores: CaseScores
   startedAt: string
   state: CaseState
   tries: MutableTry[]
@@ -175,13 +198,17 @@ interface ReducedProject {
 }
 
 /**
- * Arithmetic mean, or `null` for no values (a score that was never recorded is not 0).
+ * Adds one grader of a score to `into`, unless a grader of the same kind, model, and version is there already.
+ *
+ * Use when:
+ * - graders of several task nodes are merged into one group, for example a comparison column
  */
-export function average(values: readonly number[]): null | number {
-  if (values.length === 0) {
-    return null
+export function addScoreGrader(into: ScoreGraders, name: string, grader: Grader): void {
+  const graders = into[name] ?? []
+  if (!graders.some(known => known.kind === grader.kind && known.model === grader.model && known.version === grader.version)) {
+    graders.push(grader)
   }
-  return values.reduce((sum, value) => sum + value, 0) / values.length
+  into[name] = graders
 }
 
 /**
@@ -233,7 +260,7 @@ export function buildTaskNodes(reducer: RunReducer, evidenceCount: (caseKey: str
       .map(key => reducer.cases.get(key)!)
       .map(reduced => toCaseNode(reduced, { evidenceLinkCount: evidenceCount(reduced.key), terminated }))
       .sort((left, right) => left.index - right.index)
-    const allRounds = cases.flatMap(node => node.rounds)
+    const allRounds = cases.flatMap(node => node.rounds.map(round => ({ ...round, caseId: node.caseId })))
     // A loop instead of `Math.max(...rounds)`: a task with many cases exceeds the engine's argument limit.
     let roundCount = 0
     for (const round of allRounds) {
@@ -253,7 +280,8 @@ export function buildTaskNodes(reducer: RunReducer, evidenceCount: (caseKey: str
       id: task.id,
       projectId: task.projectId,
       roundCount,
-      scores: scoreRounds(allRounds),
+      scoreGraders: collectScoreGraders(task.caseKeys.flatMap(key => reducer.cases.get(key)!.scoreRecords.map(({ record }) => record)), task.scoreDefinitions),
+      scoring: scoreRounds(allRounds, [...task.scoreDefinitions.values()]),
       startedAt: task.startedAt,
       state,
       ...parseTaskId(task.id),
@@ -268,6 +296,34 @@ export function caseKeyOf(taskId: string, caseId: string): string {
   return `${taskId}\u0000${caseId}`
 }
 
+/**
+ * Adds the graders of score records to `into`, by score name, each distinct grader once.
+ *
+ * Use when:
+ * - a task node or a comparison column says what produced its scores, so a comparison can tell when two columns
+ *   measured one score with different graders
+ *
+ * Expects:
+ * - `definitions` of the task the records belong to; a record without its own grader uses the definition's
+ *
+ * Returns:
+ * - `into`, with graders told apart by kind, model, and version (the fields that change what a value means);
+ *   derived records are skipped, because a derived score has no grader of its own
+ */
+export function collectScoreGraders(
+  records: Iterable<ScoreRecordedData>,
+  definitions: ReadonlyMap<string, ScoreDefinition>,
+  into: ScoreGraders = {},
+): ScoreGraders {
+  for (const record of records) {
+    const grader = record.derived === true ? undefined : record.grader ?? definitions.get(record.name)?.grader
+    if (grader != null) {
+      addScoreGrader(into, record.name, grader)
+    }
+  }
+  return into
+}
+
 /**
  * Orders strings with digit runs compared as numbers, so `repeat=2` sorts before `repeat=10`.
  */
@@ -300,6 +356,43 @@ export function countCases(nodes: readonly Pick<CaseNode, 'state'>[]): CaseCount
   return counts
 }
 
+/**
+ * Values one round counts with in each score of its task, by the rule of {@link scoreRounds}.
+ *
+ * Use when:
+ * - a per-case value must agree with the task's means, for example the spread under a matrix cell or a metric panel
+ *
+ * Expects:
+ * - `definitions` from {@link countingDefinitions}, so the round counts in the scores other rounds recorded
+ *
+ * Returns:
+ * - by score name: the recorded value of a passed round, 0 for a failed round (and for an errored one with
+ *   `errors: 'zero'`); a name the round does not count in (unscored, missing, excluded, still running) is absent
+ */
+export function countedScores(round: Pick<CaseRound, 'scores' | 'state'>, definitions: readonly ScoreDefinition[] = []): Record<string, number> {
+  // A running or requeued round has not ended, so it counts in no score.
+  if (round.state === 'running' || round.state === 'requeued')
+    return {}
+  // The round's own names count too, as `scoreCases` counts a name recorded without a definition.
+  const names = new Set(definitions.map(definition => definition.name))
+  const recorded = [...Object.keys(round.scores.values), ...Object.keys(round.scores.unscored)].filter(name => !names.has(name)).map(name => ({ name }))
+  return countCase({ scores: round.scores.values, state: toOutcomeState(round.state), unscored: round.scores.unscored }, [...definitions, ...recorded])
+}
+
+/**
+ * Definitions that count a part of a task's rounds in every score of the task.
+ *
+ * Use when:
+ * - one round or a subset of rounds is scored apart from its task ({@link countedScores}, one auto attempt, one
+ *   selection of cases): a failed round counts 0 even in a score only other rounds of the task recorded
+ *
+ * Returns:
+ * - one definition per score of `scoring`, in its order: the task's `score.defined`, or a bare `{ name }`
+ */
+export function countingDefinitions(scoring: Pick<CaseScoring, 'scores'>, definitions: ReadonlyMap<string, ScoreDefinition>): ScoreDefinition[] {
+  return scoring.scores.map(summary => definitions.get(summary.name) ?? { name: summary.name })
+}
+
 /**
  * Creates an empty reducer.
  *
@@ -320,6 +413,7 @@ export function createRunReducer(): RunReducer {
   const logs: LogLine[] = []
   const metrics: MetricEntry[] = []
   const metricDefinitions = new Map<string, MetricDefinition>()
+  const scoreDefinitions = new Map<string, ScoreDefinition>()
   let identity: RunIdentity | undefined
   let lastSeq = 0
   let seqGaps = 0
@@ -355,6 +449,7 @@ export function createRunReducer(): RunReducer {
         displayName: taskId,
         id: taskId,
         projectId: projectId ?? 'unknown-project',
+        scoreDefinitions: new Map(),
         state: 'queued',
       }
       tasks.set(taskId, task)
@@ -384,6 +479,7 @@ export function createRunReducer(): RunReducer {
         name: name ?? parseCaseName(envelope.caseId),
         projectId: envelope.projectId,
         rounds: [],
+        scoreRecords: [],
         taskId: envelope.taskId,
       }
       cases.set(key, reduced)
@@ -443,7 +539,7 @@ export function createRunReducer(): RunReducer {
       round = {
         roundIndex: attemptIndex ?? reduced.rounds.length,
         roundIndexInferred: attemptIndex == null,
-        scores: {},
+        scores: emptyCaseScores(),
         startedAt: envelope.time,
         state: 'running',
         tries: [],
@@ -473,6 +569,8 @@ export function createRunReducer(): RunReducer {
     }
     round.requeuedAt = undefined
     round.requeueDelayMs = undefined
+    // Scores belong to one try: a retry, or the run after a requeue, records its own.
+    round.scores = emptyCaseScores()
     round.state = 'running'
     round.tries.push({ autoRetry, requeueIndex, retryIndex, startedAt: envelope.time, state: 'running' })
   }
@@ -516,7 +614,7 @@ export function createRunReducer(): RunReducer {
       round = {
         roundIndex: attemptIndex ?? reduced.rounds.length,
         roundIndexInferred: attemptIndex == null,
-        scores: {},
+        scores: emptyCaseScores(),
         startedAt: envelope.time,
         state: 'running',
         tries: [{ autoRetry: 0, requeueIndex: 0, retryIndex: 0, startedAt: envelope.time, state: 'running' }],
@@ -713,12 +811,37 @@ export function createRunReducer(): RunReducer {
         startedAt ??= envelope.time
         totalTasks = asNumber(data.totalTasks) ?? totalTasks
         break
+      case 'score.defined': {
+        const parsed = v.safeParse(scoreDefinedDataSchema, envelope.data)
+        if (!parsed.success) {
+          break
+        }
+        scoreDefinitions.set(parsed.output.name, parsed.output)
+        if (envelope.taskId != null) {
+          ensureTask(envelope.taskId, envelope.projectId).scoreDefinitions.set(parsed.output.name, parsed.output)
+        }
+        break
+      }
       case 'score.recorded': {
+        const parsed = v.safeParse(scoreRecordedDataSchema, envelope.data)
         const reduced = ensureCase(envelope)
-        const round = reduced?.rounds.at(-1)
-        const score = asNumber(data.score)
-        if (round != null && score != null && (data.kind === 'exact' || data.kind === 'judge')) {
-          round.scores[data.kind] = score
+        if (!parsed.success || reduced == null) {
+          break
+        }
+        const record = parsed.output
+        const round = reduced.rounds.at(-1)
+        reduced.scoreRecords.push({ record, roundIndex: round?.roundIndex, seq: envelope.seq })
+        if (round == null) {
+          break
+        }
+        // The last record of a name in a try wins, whether it is a value or an unscored record.
+        delete round.scores.values[record.name]
+        delete round.scores.unscored[record.name]
+        if ('value' in record) {
+          round.scores.values[record.name] = record.value
+        }
+        else {
+          round.scores.unscored[record.name] = record.unscored.reason
         }
         break
       }
@@ -768,6 +891,7 @@ export function createRunReducer(): RunReducer {
     metrics,
     projects,
     run: () => ({ endedAt, interruptSignal, lastEvent, startedAt, terminated, totalTasks }),
+    scoreDefinitions,
     seq: () => lastSeq,
     seqGaps: () => seqGaps,
     settlements,
@@ -955,15 +1079,57 @@ export function parseTaskId(taskId: string): { entryId?: string, inferenceExecut
 }
 
 /**
- * Task score with the rule of `describeTask` (`scoreCaseOutcomes` of `vieval/core/scoring`), so the viewer and the
- * run summary agree: every settled round of every case contributes; errored, timed-out, and unfinished rounds stay out
- * of `exact` and lower `coverage`.
+ * Definitions of the named scores that have one, in the order of `names`, each name once.
+ *
+ * Use when:
+ * - a response carries the definitions of the scores it shows (labels, descriptions, combinations, graders)
  */
-export function scoreRounds(rounds: readonly Pick<CaseRound, 'scores' | 'state'>[]): CaseOutcomeScores {
-  // Running and requeued rounds have not ended, so they do not count yet.
-  return scoreCaseOutcomes(rounds.flatMap(round => round.state === 'running' || round.state === 'requeued'
+export function pickScoreDefinitions(definitions: ReadonlyMap<string, ScoreDefinition>, names: Iterable<string>): ScoreDefinition[] {
+  return [...new Set(names)].flatMap((name) => {
+    const definition = definitions.get(name)
+    return definition == null ? [] : [definition]
+  })
+}
+
+/**
+ * Headline of a group of tasks (a run), resolved with `resolveHeadline` of `vieval/core/scoring`.
+ *
+ * Use when:
+ * - a run's headline is shown before, or without, the one `run-summary.json` holds
+ *
+ * Expects:
+ * - task nodes from {@link buildTaskNodes}; tasks of one eval entry (matrix rows, executors) share their entry id, so
+ *   a run whose entries use different primary scores lists one headline per entry
+ *
+ * Returns:
+ * - `null` while no case settled, else the headline and the pooled completion rate
+ */
+export function resolveTaskHeadline(tasks: readonly Pick<TaskNode, 'entryId' | 'id' | 'scoring'>[]): HeadlineResult | null {
+  if (tasks.every(task => task.scoring.completion.total === 0)) {
+    return null
+  }
+  return resolveHeadline(tasks.map(task => ({ ...task.scoring, taskId: task.entryId ?? task.id })))
+}
+
+/**
+ * Scores of a group of rounds with the rule of `describeTask` (`scoreCases` of `vieval/core/scoring`), so the viewer
+ * and the run summary agree.
+ *
+ * Use when:
+ * - a task, a round of a task, or a selection of cases is shown with its scores
+ *
+ * Expects:
+ * - `definitions` of the task the rounds belong to, in declaration order
+ *
+ * Returns:
+ * - one summary per score, plus completion and passed; every settled round counts in the scores, an unfinished round
+ *   counts as errored, and running or requeued rounds are left out because they have not ended
+ * - completion and passed count the last round of each case once, when rounds carry their `caseId`
+ */
+export function scoreRounds(rounds: readonly (Pick<CaseRound, 'scores' | 'state'> & { caseId?: string })[], definitions: readonly ScoreDefinition[] = []): CaseScoring {
+  return scoreCases(rounds.flatMap(round => round.state === 'running' || round.state === 'requeued'
     ? []
-    : [{ scores: round.scores, state: toOutcomeState(round.state) }]))
+    : [{ ...(round.caseId == null ? {} : { caseId: round.caseId }), scores: round.scores.values, state: toOutcomeState(round.state), unscored: round.scores.unscored }]), definitions)
 }
 
 /**
@@ -990,7 +1156,7 @@ export function toCaseNode(reduced: ReducedCase, options: { evidenceLinkCount: n
     name: reduced.name,
     projectId: reduced.projectId,
     rounds,
-    scores: lastRound?.scores ?? {},
+    scores: lastRound?.scores ?? emptyCaseScores(),
     startedAt: firstStartedAt,
     state: lastRound?.state ?? (options.terminated ? 'unfinished' : 'running'),
     taskId: reduced.taskId,
@@ -1029,6 +1195,10 @@ function asString(value: unknown): string | undefined {
   return typeof value === 'string' ? value : undefined
 }
 
+function emptyCaseScores(): CaseScores {
+  return { unscored: {}, values: {} }
+}
+
 /**
  * Identity of a case across runs (see `CaseNode.identity`), resolved in the order of `vieval report compare`.
  *
@@ -1156,7 +1326,7 @@ function toRound(round: MutableRound, terminated: boolean): CaseRound {
     ...(unfinished || round.requeuedAt == null ? {} : { requeuedAt: round.requeuedAt }),
     roundIndex: round.roundIndex,
     roundIndexInferred: round.roundIndexInferred,
-    scores: { ...round.scores },
+    scores: { unscored: { ...round.scores.unscored }, values: { ...round.scores.values } },
     startedAt: round.startedAt,
     state: unfinished ? 'unfinished' : round.state,
     tries: round.tries.map(entry => ({
diff --git a/packages/ui/test/fixtures/README.md b/packages/ui/test/fixtures/README.md
index 96089d6..ef683e3 100644
--- a/packages/ui/test/fixtures/README.md
+++ b/packages/ui/test/fixtures/README.md
@@ -8,7 +8,7 @@ OSWorld run was executing on Kubernetes.
 | `reports/…/attempt-2026-10-11T19-31-43-580Z/run-1791747103580-dd3a8517/events.jsonl` | The run that was still executing: only `events.jsonl` existed. First 150 lines (seq 1–150). |
 | `../osworld-running-continuation.jsonl` | The next 100 lines of that file (seq 151–250), appended by the live-tail tests. |
 | `reports/…/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/` | A finished run interrupted by SIGINT: `events.jsonl`, `run-summary.json`, `cases.jsonl`, `metrics-summary.json`, `otlp/`. |
-| `reports/…/attempt-2026-10-10T18-20-50-009Z/run-1791656450009-5299b70e/` | A report with event schema version 1 (legacy). |
+| `reports/…/attempt-2026-10-10T18-20-50-009Z/run-1791656450009-5299b70e/` | A report with event schema version 1: written by an older vieval, so the viewer lists it as unsupported and does not open it. |
 | `episodes/06fe7178-…/2026-10-11T19-14-06.064Z-gpt-5.5-r1/` | The evidence directory of case `1:chrome/06fe7178-…` of the interrupted run: `result.json` (agent steps cut to 3), `auv-runs/records.jsonl` and one screenshot, the first 4 kB of `bridge-evaluate.log`. |
 | `logs/run-20261012-mcp-agent-b.jsonl` | The 24 command log lines of that case, plus two lines marked `"$fixture"` that must be filtered out (another pod; outside the case window). |
 
@@ -16,3 +16,11 @@ Changes to the copied data:
 
 - The absolute `.vieval` path is replaced with `__VIEVAL_DIR__`; `test/helpers.ts` writes the temporary copy's path.
 - Home directories and kubeconfig paths are replaced with `/home/user/…`.
+- The two runs of event schema version 2 were rewritten to version 3 (named scores) without adding or removing
+  lines, so every `seq` is the one the real run wrote:
+  - every envelope has `schemaVersion: 3`
+  - `score.recorded` `{ kind: 'exact', score }` became `{ name: 'osworld.success', value }`; no `score.defined` was
+    added, so the score has no label and is the primary score as the task's only score
+  - the interrupted run's `run-summary.json` is schema version 2: the `headline` and `scoreDefinitions` at the top,
+    and `result` in the named-score shape; its `cases.jsonl` is schema version 2 (a failed case counts 0), and its
+    `otlp/metrics.json` gauge is `vieval.score.osworld.success`
diff --git a/packages/ui/test/fixtures/osworld-running-continuation.jsonl b/packages/ui/test/fixtures/osworld-running-continuation.jsonl
index 4f76f73..1ea3742 100644
--- a/packages/ui/test/fixtures/osworld-running-continuation.jsonl
+++ b/packages/ui/test/fixtures/osworld-running-continuation.jsonl
@@ -1,100 +1,100 @@
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.pod","value":"osworld-fc6d8143-2b9873"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":151,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/fc6d8143-9452-4171-9459-7f515143419a/2026-10-11T19-31-43.724Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":152,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":153,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.agent.last_action","value":"none"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":154,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"auv.run.ids","value":["4c7fb02c-b09a-96f7-656a-d7925fbefc48","f3e0fc7b-4e4e-af5f-411f-279a5b1e6096","b8d582ce-ae92-8843-138f-52b7ed970190","922308f8-64bb-d641-1ba5-65e782f70b9f","054718a2-c95a-3461-3bfd-da04bb6b5f9e","1b03773e-d980-2bb4-8ec5-80c8b9fa45ab","aa1318e7-9c69-e668-e411-06daabedd81d","a9712163-546d-b15c-ec7b-8f42141e110c","e20101e2-c737-a562-a3dd-2bb0d5d2fc91","19103644-51a5-5107-cd20-0a76db76ee99","5eb63ca1-114c-cb90-c83e-6d45ac509f7c","a4996611-bb2f-9a49-0004-ee3e093f41a1","b7becb5c-6773-a570-51d9-cec3f7c94bfe","d131b483-b84d-a5f1-ce5c-f6785896f16a","fd5f8ef0-88e6-859a-5f58-6c9147557351","a029f2fe-c6ea-e21d-01be-f80df857960c","1e22dca8-6265-42f1-f34d-63941ab6ee01","671e7509-c78e-8415-e8df-cd26615d92d6","fe2bc6d5-8d0c-50b8-ae45-6ce874835510","32971742-b8e6-84ed-f832-7df728084f2f","412418ec-36b4-71bf-e564-24209140c040","0ac6d148-f237-a8ee-19e7-25e8d12f5aac","d6d65b34-eff4-9ce5-e0e5-40df9ffb1e34","d748fb95-8056-1b3b-5338-ec7fdc98071e","ab5b27e9-55bf-a636-db47-2e8ba1d7fbd4","49e59dd8-97aa-98bf-cd89-48e9b9508525","03a3cdfc-6f9d-8c98-2585-3c285eb29bd8","5fb05de0-3b4d-dcc1-a1e2-99cf9abae7a4","493928c9-d28b-0aa9-bd8f-ee25655e0f72","86aadbd3-0613-67f6-08b6-b00c3ba3bdc9","b1940dea-4a30-08e6-85bf-8b9c148c800b","beff49c4-41f0-cc1b-314e-9cff1488dba1","52aeb3d5-c170-bded-39d5-585c0fdeb33d","c669e804-ad48-d4e8-eab6-92679497d0f8","386919a9-8322-c6ba-f720-27e1cb6eea19","ad3ebba1-5007-522a-ec20-e19a5548e930","099907b8-db0d-f4bd-ca16-9eff6b33d2e9","a3a6da80-2cc7-ef0c-bc71-6c6261901ad5","9ab588e7-b6c5-4f66-26df-0cfc5aea9cd1","18f791c4-884a-cf37-6835-0e36426fed84","5c394a87-95fa-34e8-05f0-b8ffb0321ca3","8d6460d0-9c21-705d-ffe6-db65257ec782","024177e2-0c21-4a99-867c-8d8b0e54080d","92bbd75f-c044-d99e-99c1-426b351f1e87","119297bc-9fcc-53c4-1939-33e1f838f813","37dd7851-449f-3079-6d84-3af196d00992","e8172ef6-1530-9d48-6067-b060f2e01572","48890dcd-0d5c-6845-6481-2ba8b008e63d","c0bc0b86-8e4b-4bed-8ac9-49fda8472d8e","74973c55-4426-b6ec-2ee9-aeabc459be4b"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":155,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":156,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"kind":"exact","score":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":157,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":3,"name":"chrome/fc6d8143-9452-4171-9459-7f515143419a","state":"failed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":158,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.970Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"index":14,"input":{"domain":"chrome","file":"tasks/chrome/9f3f70fc-5afc-4958-a7b7-3bb4fcb01805.json","task":{"id":"9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","snapshot":"chrome","instruction":"Browse the list of women's Nike jerseys over $60.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.nba.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"conj":"or","func":["check_url_and_content_include","check_url_and_content_include"],"result":[{"type":"active_tab_url_and_body_text","goto_prefix":"https://www."},{"type":"active_tab_url_and_body_text","goto_prefix":"https://www."}],"expected":[{"type":"rule","rules":{"url_must_include":["women","nike","jerseys"],"body_must_include":["$60"],"body_must_exclude":[]}},{"type":"rule","rules":{"url_must_include":[],"body_must_include":["Women","Nike","Jerseys","$60"],"body_must_exclude":[]}}]},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":159,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":160,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.task.file","value":"tasks/chrome/9f3f70fc-5afc-4958-a7b7-3bb4fcb01805.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":161,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.evaluator.func","value":"[\"check_url_and_content_include\",\"check_url_and_content_include\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":162,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":163,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":164,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":165,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.pod","value":"osworld-1704f00f-cf69f9"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":166,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/1704f00f-79e6-43a7-961b-cedd3724d5fd/2026-10-11T19-31-43.745Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":167,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":168,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":169,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"auv.run.ids","value":["ecc428ea-8272-4c81-eb82-0c3c60a39ddd","72c671da-1544-dddc-6b23-7c48c0e3f1b5","fb5772c2-7bed-614f-0440-c5356a72cccf","9ef4c6e5-ffbe-9be2-d4fa-1bac8c50a118","4a485e88-071f-507f-ef26-4181eb6c957b","5da43328-5447-d884-dbfb-ceb36f22b3ca","9ad2d207-2866-b5ee-e94c-f16985bcb9cd","9c945d9e-81a9-0555-285b-4c5db586d676","365b1d63-faa2-a6a2-492b-616bc26d9b8f","0f93bd58-9ffb-6dc7-1125-c2593e8320ac","2db2e4d8-fbd9-f974-e2de-8281540fa28a","7f42fd8d-693a-848c-e55c-31b900838bb7","942827e1-cf4f-1e2b-cde4-0462fff7179e","6b85e4b5-f2c5-bf17-915b-16eb377c5125","7f176dbb-ef2b-2a9b-3eee-ddc4c0cd510f","a35145db-64e0-8eff-4cda-1c23bec723b9","60efaa3f-fbeb-b70e-b28a-ebbcd572fbdc","ca2912be-cc5c-5b99-9454-96eb38797a36","665deedf-4d12-b74e-06b6-1a1fb9281c73","83986397-be9f-b1a9-662e-1678f7bc5c6e","4e3dff58-3f6f-1d7a-63fb-7c45ba0555ac","60e21617-bbc3-4734-e658-892c34e8a2b6","3a23b189-75db-99f5-832e-06a644a29214","62ea20bb-5dcc-3ba5-0df2-cedb076ae3ec","b12c8ec7-636d-50ab-78c7-c78243f39d10","d4f9214a-b72d-af21-b7cd-1a5cd1b0564a","eb646ce3-11c0-8b59-3c9c-c7d776026e25","2a2d0a74-7a7e-62e6-aef8-19ae2bc0caae","72f7a686-bdd7-d6a3-5781-fc2d17f71725","ea693f64-2534-190c-f2b6-fbab078bbe65","5dfd84e8-aad2-90c0-5967-2830622f183b","48442f6f-9d27-92fa-cd5a-831acbd97873","e1a93ac1-2a52-18a3-61e4-72f9d58490de","b4524d01-f46f-ff83-ed01-7c211b13edc1","cb3d9a97-5f0c-948a-5760-0c4e073c511c","ea0fa507-7c19-de31-8182-29eedc898dcb","015f0433-1a63-c2d1-b8d7-e6745148537a","bcb23c82-b2e7-8944-0e12-cd395139ec91","95244657-1618-5933-d2f3-700a5e826043","f0792b88-c280-0b7d-7b9e-a1c44d29549c","5af8276d-4381-b3cd-b430-96a3a7bde7bc","7f27a920-1084-c1cd-eaa9-9e50e97ec9b3","5561a286-29e0-f697-e285-1e77346bda81","5b9400aa-8703-a624-cf6a-9786a18892fa","33737b77-5b5b-9593-706a-54c1b6ca221d","61ea26e9-0ee5-251e-376b-33579b6fe426","983eba9e-aed0-5916-3e11-64ae473e9a47","c8746c0c-3c72-2516-4660-61fe9391a54d","e6709bc3-425a-2b2f-f3e4-e2c3b24d79f0"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":170,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":171,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":172,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"index":5,"name":"chrome/1704f00f-79e6-43a7-961b-cedd3724d5fd","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":173,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"index":15,"input":{"domain":"chrome","file":"tasks/chrome/7f52cab9-535c-4835-ac8c-391ee64dc930.json","task":{"id":"7f52cab9-535c-4835-ac8c-391ee64dc930","snapshot":"chrome","instruction":"On Google Shopping, search for drip coffee makers and filter the results to show only items that are on sale, priced between $25-60, and have a black finish. Stay on the search results page.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://shopping.google.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"conj":"and","func":["check_direct_json_object","check_url_and_content_include"],"result":[{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["q"]},{"type":"active_tab_url_and_body_text","goto_prefix":"https://www."}],"expected":[{"type":"rule","rules":{"expected":{"q":"drip coffee maker"},"expect_in_result":true}},{"type":"rule","rules":{"url_must_include":["google"],"body_must_include":["Black","On sale","$25","$60"],"body_must_exclude":[]}}]},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/7f52cab9-535c-4835-ac8c-391ee64dc930","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":174,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":175,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.task.file","value":"tasks/chrome/7f52cab9-535c-4835-ac8c-391ee64dc930.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":176,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.evaluator.func","value":"[\"check_direct_json_object\",\"check_url_and_content_include\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":177,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":178,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":179,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":180,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.pod","value":"osworld-c1fa57f3-195cbd"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":181,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.378Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/c1fa57f3-c3db-4596-8f09-020701085416/2026-10-11T19-36-09.021Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":182,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.378Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.agent.steps","value":17},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":183,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":184,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"auv.run.ids","value":["e76a6573-bb6b-76e1-f97b-03a9c20fb234","de99164c-4cc6-0e57-3542-6c6df025d065","321e0a55-1031-7b6e-1869-3931cf1c0e61","c96b9ec9-523c-cde3-8fe8-4ebf642702d2","5bb929a6-4b83-3366-8505-d80a6fe74f28","5740bf99-5897-2b0e-cdd9-648b59ec8752","69ae83a8-c2f7-249b-76b1-c2483a945cab","ab4cbee1-19c2-d201-d46d-b05ff877463c","3f8db494-7855-0281-e467-9cf0e3d4c611","1a8c4510-ca4e-5f6a-9bda-cf88bd3d549a","24ef703b-f82a-67ae-dfad-126f3896e360","8f7f131c-0095-8b1a-1223-d0491a9ec1cf","180f445e-88a8-bea9-396d-6b2e0b10b0bc","d5edecbf-0782-58ad-ee6c-0dd1445f7f8b","4778cbb6-a960-a38e-0612-2e8b879332bc","b9679b1d-5e5a-d64a-7676-dd7480cf7b7c"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":185,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":186,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":187,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"index":9,"name":"chrome/c1fa57f3-c3db-4596-8f09-020701085416","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":188,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"index":16,"input":{"domain":"chrome","file":"tasks/chrome/82279c77-8fc6-46f6-9622-3ba96f61b477.json","task":{"id":"82279c77-8fc6-46f6-9622-3ba96f61b477","snapshot":"chrome","instruction":"Find electric cars with a maximum price of $50,000 within 50 miles of 10001.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.cars.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["list_price_max","maximum_distance","zip","fuel_slugs[]"]},"expected":{"type":"rule","rules":{"expected":{"list_price_max":"50000","maximum_distance":"50","zip":"10001","fuel_slugs[]":"electric"}}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/82279c77-8fc6-46f6-9622-3ba96f61b477","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":189,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":190,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.task.file","value":"tasks/chrome/82279c77-8fc6-46f6-9622-3ba96f61b477.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":191,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":192,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":193,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":194,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":195,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.pod","value":"osworld-b7895e80-315300"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":196,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.099Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/b7895e80-f4d1-4648-bee0-4eb45a6f1fa8/2026-10-11T19-39-57.878Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":197,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.099Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.agent.steps","value":22},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":198,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.099Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":199,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.099Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"auv.run.ids","value":["a780e6d4-7751-f83b-9921-943dbabe12e2","f9394b12-1dde-eea3-e158-1405ecc8f6d1","06ed7869-1c26-b7dd-afc5-c41faaf7af81","fba25e69-f7b5-fedb-6e8d-646148cfc8be","710c806d-6372-6604-c25c-1f88bcb3f2d0","650f5bc0-0417-0557-d091-646213b18c78","0a848667-196e-d3d4-84b4-d8a576f60100","aa2e7aa9-21bd-c50a-1079-037276b3ad03","de037340-2ca8-e2d1-2670-3c88fbd16bbf","f7c20aea-0382-833e-6a73-46139fadd6f7","6f05d132-5379-5347-60e8-1fd2624db15f","67b67541-f70b-cd12-6466-1593df536398","eab81856-50ec-b4b1-7164-8d3a0b23e9a2","783f6c92-816a-a015-b9c2-a4f49684a639","f6865d7c-1f73-7bfc-ef46-84aca151852c","d0023d16-70c4-3b90-29da-80c98b449333","2360ec8f-a465-3d06-59ee-840cbdeacff1","4e7f22b2-beb7-7b2a-cd15-325bac3a4acc","81184f9e-e749-2c7c-067e-790062afde56","e2b6cbeb-86e3-6a98-7326-02372e65c173","857213c1-298b-2a26-661b-d9d1ba860d3e"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":200,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.100Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":201,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.100Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":202,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.100Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"index":13,"name":"chrome/b7895e80-f4d1-4648-bee0-4eb45a6f1fa8","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":203,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.100Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"index":17,"input":{"domain":"chrome","file":"tasks/chrome/2888b4e6-5b47-4b57-8bf5-c73827890774.json","task":{"id":"2888b4e6-5b47-4b57-8bf5-c73827890774","snapshot":"chrome","instruction":"Show me all men's large-size short-sleeve shirts with a discount of 50% or more.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.macys.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"url_path_parse","goto_prefix":"https://www.","parse_keys":["mens_clothing","shirts","Men_regular_size_t","Price_discount_range","short_sleeve"]},"expected":{"type":"rule","rules":{"expected":{"mens_clothing":true,"shirts":true,"Men_regular_size_t":"L","Price_discount_range":"50_PERCENT_ off & more","short_sleeve":true}}}},"proxy":true,"possibility_of_env_change":"medium","fixed_ip":false}},"name":"chrome/2888b4e6-5b47-4b57-8bf5-c73827890774","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":204,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.100Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":205,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.task.file","value":"tasks/chrome/2888b4e6-5b47-4b57-8bf5-c73827890774.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":206,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":207,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":208,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":209,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":210,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.pod","value":"osworld-47543840-57071c"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":211,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.104Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/47543840-672a-467d-80df-8f7c3b9788c9/2026-10-11T19-34-36.356Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":212,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.agent.steps","value":40},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":213,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":214,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"auv.run.ids","value":["e8dea926-65f9-1fba-2b47-1c086f564939","6037d52c-9356-77e3-7a31-44b8adb6b170","c127e04d-b0ab-3b77-d3b2-6a533616d4c7","7c81d50c-f890-5943-81e8-d24b4d712c1e","a332da55-4d95-9de4-c6f2-800c5978b00c","91db45c6-b587-0ea3-fbfd-7ecd7ff75c3c","071bcd09-5ef5-8729-391b-ac87667a5f98","00239632-98e5-9ccf-b76f-b8e9ca959554","2fc168c7-dac7-9046-f3f1-c9f5003b45b5","5f161b5f-4ef0-af45-0fb3-9f8d1a9d2d5d","5632a61a-9a80-f267-bd05-84e3773f775e","b217437a-ef66-1ae9-e120-777b8a146ff7","8db8fc74-4539-4852-0937-9f027c92638f","4c5b114e-f075-9e00-f961-538e495029dc","4d38f613-6928-8511-9e08-ceda9ba048cf","af89d4f6-0c84-2c56-bdf0-0c8f2a2897e2","89200d9e-3359-77f7-8203-af3dfd53c58c","f1b76b72-374f-95aa-2064-fb64854948ac","553c4e62-7525-406a-ce60-db161d61474e","3f28335e-9f44-cddd-a9fd-937c6591540c","7dca1206-0d06-78f0-f0ff-f4b44ce4a85f","6f807e51-6d18-79d6-aef0-a17ce8ec911d","d7b3c026-b9b3-6a41-964c-b62f464be8ff","03950f0b-1113-8019-c0ad-5dfa670cefe5","0a0652a1-30d7-6d22-3b7b-81a894b67c4b","02152cc2-2f06-0cc2-e344-0d94f6f65a64","77b4fb01-6de7-c009-57d9-4419b73ef361","d604b7c7-cc97-ab65-3b93-ad6b48efa13f","96ad0c50-6916-561b-a7dd-492f260bc661","27bc182b-09f6-9dbf-69cc-0fbed92136ee","f0192cd6-879e-9ee1-e52a-9cf072277e2b","0e2cefa9-bdaf-7fce-3fd8-5faea8b0ef86","f3199d5a-76c6-1aba-a5fc-b97b4fd6ad8e","06b4ee57-2eb2-4312-1488-f6d69438e965","b1fc97f9-05d0-bbb5-199e-2ca38ff116c7","b0531e70-28fb-862a-040c-44dff04f23be","872255c6-cead-0049-b016-6e43d39b9d77","45ecfe57-2ac7-9db7-88d1-a22597f28842","81f43ead-b3aa-4a71-b6e3-3cb41d8b845b"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":215,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":216,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":217,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"index":8,"name":"chrome/47543840-672a-467d-80df-8f7c3b9788c9","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":218,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"index":18,"input":{"domain":"chrome","file":"tasks/chrome/b4f95342-463e-4179-8c3f-193cd7241fb2.json","task":{"id":"b4f95342-463e-4179-8c3f-193cd7241fb2","snapshot":"chrome","instruction":"Find the Next Available dates for Diamond.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.recreation.gov/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"class","class_singleObject":{},"class_multiObject":{"camp-sortable-column-header":{"2":"camp-sortable-column-header"}}},"expected":{"type":"gotoRecreationPage_and_get_html_content","selector":"class","class":"camp-sortable-column-header","order":"2"}},"proxy":false,"possibility_of_env_change":"high","fixed_ip":false}},"name":"chrome/b4f95342-463e-4179-8c3f-193cd7241fb2","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":219,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":220,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.task.file","value":"tasks/chrome/b4f95342-463e-4179-8c3f-193cd7241fb2.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":221,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":222,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":223,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":224,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":225,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.pod","value":"osworld-da46d875-0f2ebf"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":226,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/da46d875-6b82-4681-9284-653b0c7ae241/2026-10-11T19-37-52.633Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":227,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":228,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.agent.last_action","value":"none"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":229,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"auv.run.ids","value":["667ef41f-aff1-a90e-4baa-99b5c1cb7919","2d6f7e27-0bb8-e096-318f-c9372167dede","737a36b7-3e74-c5f3-d9f4-56890089bfce","30c43115-3885-1224-3b2c-94a271760e4e","0175e0c9-24c0-09fd-c11f-c14b5289ca6b","6c9862fe-f0f9-618e-469c-e0af939192a7","6da4b2e7-f8ac-e2d4-4ae3-2e9d8b1ae032","f268e396-e016-529b-7940-a106e0dfce0b","4ea7936c-2b5c-d647-3ef2-2bfb225d1358","ec58d524-7c7f-f023-a21e-d9b66cf30953","f3e5ba77-936d-4d0f-e67e-79f4500b4e4e","72bbf084-fc8e-04da-4381-cd707ee7f2e7","0cd0ba91-2598-1bf7-9c10-28ab3a183b54","96e2f246-423b-6724-e303-b0406f507684","c9c69cb6-20a0-2084-9703-bd2d3fed9591","413f2adf-8d0a-4628-0ba5-71ca6a63f3a7","febc7757-1a85-f9bf-c0ee-26b51f084b88","22e844af-1bb2-2913-8d54-6b31d99ba025","148cde6b-0fc7-b46a-fae1-ae335cecc6d8","d335e8cb-fb9b-70a7-ad7b-deace87c0a0e","86176196-fa0b-3c87-f5ed-9cd63975bdf0","87616c17-8224-6717-7205-37f3db02d27c","10466902-ddac-f6bc-e91d-05798956cbcc","e645bc21-04c3-f6d6-2ed1-cc1a461f14ef","a556eddb-da4a-4abf-0cb3-241e8b3fbe79","7b7af846-7cdc-9fc1-55f2-7deee2a1fdc8","7d888edd-fdfa-e92f-4bb9-851d0fe4abb7","eb51f97d-5c6c-2fdd-9a9c-5191469e137b","8ffe308d-3777-3b05-681b-44db8d4d7edd","2b337564-be38-1bea-e46e-efe546aece3b","4b46aa54-a3dc-4107-abe0-9edc28227441","3c3a6f4f-dc4c-a851-4dfa-6305b52efd7d","36f97f29-836d-e55e-5dec-43b22a950172","8a7f924b-ccea-8d80-9d33-819074291446","c7545e76-71ef-38ef-891b-6af3929156e2","182252ac-f9c8-0d58-c0d7-3d475d7e7e89","271eda00-a771-427e-6f94-6e82b08c8b25","fb988d72-deff-3f2c-293b-e0bd0a6102fe","326433d4-c3ae-a0f8-b850-04bfaf2cb439","6dfae1a9-bf89-84ca-e486-72f6d4f51276","73b0f574-cf0f-b2d1-195c-83cb3c148fc1","d0ee0a65-f541-3faa-5ede-985e91d6b967","f1063fc9-4897-7487-c74f-696a141a51d8","a84df944-5ccb-ff32-cfb6-b7cfbc1f5669","666f6ed7-1472-0a17-925d-8963368d27c9","5f23cfc6-6cbf-29ee-8661-4e4b7cdc3ae0","5bb6b97c-3700-f299-2a6b-7558ef731947","4b23c4b7-7b9b-99ac-f4e6-58f8323f48c3","5d0f3821-81bb-7347-c473-850caa9e0ac0","f9b315b2-4489-82c8-b860-e36022af1e9f"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":230,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":231,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"kind":"exact","score":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":232,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":10,"name":"chrome/da46d875-6b82-4681-9284-653b0c7ae241","state":"failed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":233,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.322Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"index":19,"input":{"domain":"chrome","file":"tasks/chrome/f5d96daf-83a8-4c86-9686-bada31fc66ab.json","task":{"id":"f5d96daf-83a8-4c86-9686-bada31fc66ab","snapshot":"chrome","instruction":"Compare iPhone 15 Pro Max with iPhone 14 Pro Max and iPhone 13 Pro Max","source":"Mind2Web","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.apple.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["modelList"],"split_list":true},"expected":{"type":"rule","rules":{"expected":{"modelList":["iphone-15-pro-max","iphone-14-pro-max","iphone-13-pro-max"]},"ignore_list_order":true}}},"proxy":true,"possibility_of_env_change":"medium","fixed_ip":false}},"name":"chrome/f5d96daf-83a8-4c86-9686-bada31fc66ab","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":234,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":235,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.task.file","value":"tasks/chrome/f5d96daf-83a8-4c86-9686-bada31fc66ab.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":236,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":237,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":238,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":239,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":240,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.pod","value":"osworld-f5d96daf-a4fee3"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":241,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/f5d96daf-83a8-4c86-9686-bada31fc66ab/2026-10-11T19-47-40.346Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":242,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.agent.steps","value":9},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":243,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":244,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"auv.run.ids","value":["37ae7073-ca2b-3479-dbfb-178fb020c512","8668c8a7-8c23-45d0-2767-1a90c130b5a4","4d4486f6-8e08-5a8f-c107-8321427ab14b","a7ef81c1-c0b3-9be7-b077-868b6095b9eb","5406c743-1b54-4618-bf0a-faa2b8ea60d3","51b0ad72-0cbe-c770-79a0-b2047ba40a5b","dff25941-664d-2291-c7c9-8647fe04facd"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":245,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":246,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"kind":"exact","score":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":247,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.480Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":19,"name":"chrome/f5d96daf-83a8-4c86-9686-bada31fc66ab","state":"failed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":248,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.481Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"20:chrome%2F121ba48f-9e17-48ce-9bc6-a4fb17a7ebba","data":{"index":20,"input":{"domain":"chrome","file":"tasks/chrome/121ba48f-9e17-48ce-9bc6-a4fb17a7ebba.json","task":{"id":"121ba48f-9e17-48ce-9bc6-a4fb17a7ebba","snapshot":"chrome","instruction":"Find Dota 2 game and add all DLC to cart.","source":"Mind2Web","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.dota2.com/home","https://store.steampowered.com"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_added_to_steam_cart","result":{"type":"page_info","url":"https://store.steampowered.com/cart/"},"expected":{"type":"rule","rules":{"items":["The Dota 2 Official Soundtrack"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/121ba48f-9e17-48ce-9bc6-a4fb17a7ebba","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":249,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.481Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"20:chrome%2F121ba48f-9e17-48ce-9bc6-a4fb17a7ebba","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":250,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.482Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.pod","value":"osworld-fc6d8143-2b9873"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":151,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/fc6d8143-9452-4171-9459-7f515143419a/2026-10-11T19-31-43.724Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":152,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":153,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.agent.last_action","value":"none"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":154,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"auv.run.ids","value":["4c7fb02c-b09a-96f7-656a-d7925fbefc48","f3e0fc7b-4e4e-af5f-411f-279a5b1e6096","b8d582ce-ae92-8843-138f-52b7ed970190","922308f8-64bb-d641-1ba5-65e782f70b9f","054718a2-c95a-3461-3bfd-da04bb6b5f9e","1b03773e-d980-2bb4-8ec5-80c8b9fa45ab","aa1318e7-9c69-e668-e411-06daabedd81d","a9712163-546d-b15c-ec7b-8f42141e110c","e20101e2-c737-a562-a3dd-2bb0d5d2fc91","19103644-51a5-5107-cd20-0a76db76ee99","5eb63ca1-114c-cb90-c83e-6d45ac509f7c","a4996611-bb2f-9a49-0004-ee3e093f41a1","b7becb5c-6773-a570-51d9-cec3f7c94bfe","d131b483-b84d-a5f1-ce5c-f6785896f16a","fd5f8ef0-88e6-859a-5f58-6c9147557351","a029f2fe-c6ea-e21d-01be-f80df857960c","1e22dca8-6265-42f1-f34d-63941ab6ee01","671e7509-c78e-8415-e8df-cd26615d92d6","fe2bc6d5-8d0c-50b8-ae45-6ce874835510","32971742-b8e6-84ed-f832-7df728084f2f","412418ec-36b4-71bf-e564-24209140c040","0ac6d148-f237-a8ee-19e7-25e8d12f5aac","d6d65b34-eff4-9ce5-e0e5-40df9ffb1e34","d748fb95-8056-1b3b-5338-ec7fdc98071e","ab5b27e9-55bf-a636-db47-2e8ba1d7fbd4","49e59dd8-97aa-98bf-cd89-48e9b9508525","03a3cdfc-6f9d-8c98-2585-3c285eb29bd8","5fb05de0-3b4d-dcc1-a1e2-99cf9abae7a4","493928c9-d28b-0aa9-bd8f-ee25655e0f72","86aadbd3-0613-67f6-08b6-b00c3ba3bdc9","b1940dea-4a30-08e6-85bf-8b9c148c800b","beff49c4-41f0-cc1b-314e-9cff1488dba1","52aeb3d5-c170-bded-39d5-585c0fdeb33d","c669e804-ad48-d4e8-eab6-92679497d0f8","386919a9-8322-c6ba-f720-27e1cb6eea19","ad3ebba1-5007-522a-ec20-e19a5548e930","099907b8-db0d-f4bd-ca16-9eff6b33d2e9","a3a6da80-2cc7-ef0c-bc71-6c6261901ad5","9ab588e7-b6c5-4f66-26df-0cfc5aea9cd1","18f791c4-884a-cf37-6835-0e36426fed84","5c394a87-95fa-34e8-05f0-b8ffb0321ca3","8d6460d0-9c21-705d-ffe6-db65257ec782","024177e2-0c21-4a99-867c-8d8b0e54080d","92bbd75f-c044-d99e-99c1-426b351f1e87","119297bc-9fcc-53c4-1939-33e1f838f813","37dd7851-449f-3079-6d84-3af196d00992","e8172ef6-1530-9d48-6067-b060f2e01572","48890dcd-0d5c-6845-6481-2ba8b008e63d","c0bc0b86-8e4b-4bed-8ac9-49fda8472d8e","74973c55-4426-b6ec-2ee9-aeabc459be4b"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":155,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":156,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.success","value":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":157,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.969Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":3,"name":"chrome/fc6d8143-9452-4171-9459-7f515143419a","state":"failed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":158,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.970Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"index":14,"input":{"domain":"chrome","file":"tasks/chrome/9f3f70fc-5afc-4958-a7b7-3bb4fcb01805.json","task":{"id":"9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","snapshot":"chrome","instruction":"Browse the list of women's Nike jerseys over $60.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.nba.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"conj":"or","func":["check_url_and_content_include","check_url_and_content_include"],"result":[{"type":"active_tab_url_and_body_text","goto_prefix":"https://www."},{"type":"active_tab_url_and_body_text","goto_prefix":"https://www."}],"expected":[{"type":"rule","rules":{"url_must_include":["women","nike","jerseys"],"body_must_include":["$60"],"body_must_exclude":[]}},{"type":"rule","rules":{"url_must_include":[],"body_must_include":["Women","Nike","Jerseys","$60"],"body_must_exclude":[]}}]},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":159,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":160,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.task.file","value":"tasks/chrome/9f3f70fc-5afc-4958-a7b7-3bb4fcb01805.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":161,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.evaluator.func","value":"[\"check_url_and_content_include\",\"check_url_and_content_include\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":162,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":163,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":164,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"14:chrome%2F9f3f70fc-5afc-4958-a7b7-3bb4fcb01805","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":165,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:46.971Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.pod","value":"osworld-1704f00f-cf69f9"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":166,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/1704f00f-79e6-43a7-961b-cedd3724d5fd/2026-10-11T19-31-43.745Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":167,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":168,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":169,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"auv.run.ids","value":["ecc428ea-8272-4c81-eb82-0c3c60a39ddd","72c671da-1544-dddc-6b23-7c48c0e3f1b5","fb5772c2-7bed-614f-0440-c5356a72cccf","9ef4c6e5-ffbe-9be2-d4fa-1bac8c50a118","4a485e88-071f-507f-ef26-4181eb6c957b","5da43328-5447-d884-dbfb-ceb36f22b3ca","9ad2d207-2866-b5ee-e94c-f16985bcb9cd","9c945d9e-81a9-0555-285b-4c5db586d676","365b1d63-faa2-a6a2-492b-616bc26d9b8f","0f93bd58-9ffb-6dc7-1125-c2593e8320ac","2db2e4d8-fbd9-f974-e2de-8281540fa28a","7f42fd8d-693a-848c-e55c-31b900838bb7","942827e1-cf4f-1e2b-cde4-0462fff7179e","6b85e4b5-f2c5-bf17-915b-16eb377c5125","7f176dbb-ef2b-2a9b-3eee-ddc4c0cd510f","a35145db-64e0-8eff-4cda-1c23bec723b9","60efaa3f-fbeb-b70e-b28a-ebbcd572fbdc","ca2912be-cc5c-5b99-9454-96eb38797a36","665deedf-4d12-b74e-06b6-1a1fb9281c73","83986397-be9f-b1a9-662e-1678f7bc5c6e","4e3dff58-3f6f-1d7a-63fb-7c45ba0555ac","60e21617-bbc3-4734-e658-892c34e8a2b6","3a23b189-75db-99f5-832e-06a644a29214","62ea20bb-5dcc-3ba5-0df2-cedb076ae3ec","b12c8ec7-636d-50ab-78c7-c78243f39d10","d4f9214a-b72d-af21-b7cd-1a5cd1b0564a","eb646ce3-11c0-8b59-3c9c-c7d776026e25","2a2d0a74-7a7e-62e6-aef8-19ae2bc0caae","72f7a686-bdd7-d6a3-5781-fc2d17f71725","ea693f64-2534-190c-f2b6-fbab078bbe65","5dfd84e8-aad2-90c0-5967-2830622f183b","48442f6f-9d27-92fa-cd5a-831acbd97873","e1a93ac1-2a52-18a3-61e4-72f9d58490de","b4524d01-f46f-ff83-ed01-7c211b13edc1","cb3d9a97-5f0c-948a-5760-0c4e073c511c","ea0fa507-7c19-de31-8182-29eedc898dcb","015f0433-1a63-c2d1-b8d7-e6745148537a","bcb23c82-b2e7-8944-0e12-cd395139ec91","95244657-1618-5933-d2f3-700a5e826043","f0792b88-c280-0b7d-7b9e-a1c44d29549c","5af8276d-4381-b3cd-b430-96a3a7bde7bc","7f27a920-1084-c1cd-eaa9-9e50e97ec9b3","5561a286-29e0-f697-e285-1e77346bda81","5b9400aa-8703-a624-cf6a-9786a18892fa","33737b77-5b5b-9593-706a-54c1b6ca221d","61ea26e9-0ee5-251e-376b-33579b6fe426","983eba9e-aed0-5916-3e11-64ae473e9a47","c8746c0c-3c72-2516-4660-61fe9391a54d","e6709bc3-425a-2b2f-f3e4-e2c3b24d79f0"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":170,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":171,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":172,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"index":5,"name":"chrome/1704f00f-79e6-43a7-961b-cedd3724d5fd","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":173,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"index":15,"input":{"domain":"chrome","file":"tasks/chrome/7f52cab9-535c-4835-ac8c-391ee64dc930.json","task":{"id":"7f52cab9-535c-4835-ac8c-391ee64dc930","snapshot":"chrome","instruction":"On Google Shopping, search for drip coffee makers and filter the results to show only items that are on sale, priced between $25-60, and have a black finish. Stay on the search results page.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://shopping.google.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"conj":"and","func":["check_direct_json_object","check_url_and_content_include"],"result":[{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["q"]},{"type":"active_tab_url_and_body_text","goto_prefix":"https://www."}],"expected":[{"type":"rule","rules":{"expected":{"q":"drip coffee maker"},"expect_in_result":true}},{"type":"rule","rules":{"url_must_include":["google"],"body_must_include":["Black","On sale","$25","$60"],"body_must_exclude":[]}}]},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/7f52cab9-535c-4835-ac8c-391ee64dc930","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":174,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.587Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":175,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.task.file","value":"tasks/chrome/7f52cab9-535c-4835-ac8c-391ee64dc930.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":176,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.evaluator.func","value":"[\"check_direct_json_object\",\"check_url_and_content_include\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":177,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":178,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":179,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"15:chrome%2F7f52cab9-535c-4835-ac8c-391ee64dc930","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":180,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:43:50.588Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.pod","value":"osworld-c1fa57f3-195cbd"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":181,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.378Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/c1fa57f3-c3db-4596-8f09-020701085416/2026-10-11T19-36-09.021Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":182,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.378Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.agent.steps","value":17},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":183,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":184,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"auv.run.ids","value":["e76a6573-bb6b-76e1-f97b-03a9c20fb234","de99164c-4cc6-0e57-3542-6c6df025d065","321e0a55-1031-7b6e-1869-3931cf1c0e61","c96b9ec9-523c-cde3-8fe8-4ebf642702d2","5bb929a6-4b83-3366-8505-d80a6fe74f28","5740bf99-5897-2b0e-cdd9-648b59ec8752","69ae83a8-c2f7-249b-76b1-c2483a945cab","ab4cbee1-19c2-d201-d46d-b05ff877463c","3f8db494-7855-0281-e467-9cf0e3d4c611","1a8c4510-ca4e-5f6a-9bda-cf88bd3d549a","24ef703b-f82a-67ae-dfad-126f3896e360","8f7f131c-0095-8b1a-1223-d0491a9ec1cf","180f445e-88a8-bea9-396d-6b2e0b10b0bc","d5edecbf-0782-58ad-ee6c-0dd1445f7f8b","4778cbb6-a960-a38e-0612-2e8b879332bc","b9679b1d-5e5a-d64a-7676-dd7480cf7b7c"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":185,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":186,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":187,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"index":9,"name":"chrome/c1fa57f3-c3db-4596-8f09-020701085416","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":188,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"index":16,"input":{"domain":"chrome","file":"tasks/chrome/82279c77-8fc6-46f6-9622-3ba96f61b477.json","task":{"id":"82279c77-8fc6-46f6-9622-3ba96f61b477","snapshot":"chrome","instruction":"Find electric cars with a maximum price of $50,000 within 50 miles of 10001.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.cars.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["list_price_max","maximum_distance","zip","fuel_slugs[]"]},"expected":{"type":"rule","rules":{"expected":{"list_price_max":"50000","maximum_distance":"50","zip":"10001","fuel_slugs[]":"electric"}}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/82279c77-8fc6-46f6-9622-3ba96f61b477","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":189,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":190,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.task.file","value":"tasks/chrome/82279c77-8fc6-46f6-9622-3ba96f61b477.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":191,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":192,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":193,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":194,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"16:chrome%2F82279c77-8fc6-46f6-9622-3ba96f61b477","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":195,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:44:14.379Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.pod","value":"osworld-b7895e80-315300"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":196,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.099Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/b7895e80-f4d1-4648-bee0-4eb45a6f1fa8/2026-10-11T19-39-57.878Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":197,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.099Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.agent.steps","value":22},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":198,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.099Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":199,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.099Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"auv.run.ids","value":["a780e6d4-7751-f83b-9921-943dbabe12e2","f9394b12-1dde-eea3-e158-1405ecc8f6d1","06ed7869-1c26-b7dd-afc5-c41faaf7af81","fba25e69-f7b5-fedb-6e8d-646148cfc8be","710c806d-6372-6604-c25c-1f88bcb3f2d0","650f5bc0-0417-0557-d091-646213b18c78","0a848667-196e-d3d4-84b4-d8a576f60100","aa2e7aa9-21bd-c50a-1079-037276b3ad03","de037340-2ca8-e2d1-2670-3c88fbd16bbf","f7c20aea-0382-833e-6a73-46139fadd6f7","6f05d132-5379-5347-60e8-1fd2624db15f","67b67541-f70b-cd12-6466-1593df536398","eab81856-50ec-b4b1-7164-8d3a0b23e9a2","783f6c92-816a-a015-b9c2-a4f49684a639","f6865d7c-1f73-7bfc-ef46-84aca151852c","d0023d16-70c4-3b90-29da-80c98b449333","2360ec8f-a465-3d06-59ee-840cbdeacff1","4e7f22b2-beb7-7b2a-cd15-325bac3a4acc","81184f9e-e749-2c7c-067e-790062afde56","e2b6cbeb-86e3-6a98-7326-02372e65c173","857213c1-298b-2a26-661b-d9d1ba860d3e"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":200,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.100Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":201,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.100Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":202,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.100Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"index":13,"name":"chrome/b7895e80-f4d1-4648-bee0-4eb45a6f1fa8","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":203,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.100Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"index":17,"input":{"domain":"chrome","file":"tasks/chrome/2888b4e6-5b47-4b57-8bf5-c73827890774.json","task":{"id":"2888b4e6-5b47-4b57-8bf5-c73827890774","snapshot":"chrome","instruction":"Show me all men's large-size short-sleeve shirts with a discount of 50% or more.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.macys.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"url_path_parse","goto_prefix":"https://www.","parse_keys":["mens_clothing","shirts","Men_regular_size_t","Price_discount_range","short_sleeve"]},"expected":{"type":"rule","rules":{"expected":{"mens_clothing":true,"shirts":true,"Men_regular_size_t":"L","Price_discount_range":"50_PERCENT_ off & more","short_sleeve":true}}}},"proxy":true,"possibility_of_env_change":"medium","fixed_ip":false}},"name":"chrome/2888b4e6-5b47-4b57-8bf5-c73827890774","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":204,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.100Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":205,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.task.file","value":"tasks/chrome/2888b4e6-5b47-4b57-8bf5-c73827890774.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":206,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":207,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":208,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":209,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"17:chrome%2F2888b4e6-5b47-4b57-8bf5-c73827890774","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":210,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:46:39.101Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.pod","value":"osworld-47543840-57071c"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":211,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.104Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/47543840-672a-467d-80df-8f7c3b9788c9/2026-10-11T19-34-36.356Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":212,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.agent.steps","value":40},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":213,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":214,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"auv.run.ids","value":["e8dea926-65f9-1fba-2b47-1c086f564939","6037d52c-9356-77e3-7a31-44b8adb6b170","c127e04d-b0ab-3b77-d3b2-6a533616d4c7","7c81d50c-f890-5943-81e8-d24b4d712c1e","a332da55-4d95-9de4-c6f2-800c5978b00c","91db45c6-b587-0ea3-fbfd-7ecd7ff75c3c","071bcd09-5ef5-8729-391b-ac87667a5f98","00239632-98e5-9ccf-b76f-b8e9ca959554","2fc168c7-dac7-9046-f3f1-c9f5003b45b5","5f161b5f-4ef0-af45-0fb3-9f8d1a9d2d5d","5632a61a-9a80-f267-bd05-84e3773f775e","b217437a-ef66-1ae9-e120-777b8a146ff7","8db8fc74-4539-4852-0937-9f027c92638f","4c5b114e-f075-9e00-f961-538e495029dc","4d38f613-6928-8511-9e08-ceda9ba048cf","af89d4f6-0c84-2c56-bdf0-0c8f2a2897e2","89200d9e-3359-77f7-8203-af3dfd53c58c","f1b76b72-374f-95aa-2064-fb64854948ac","553c4e62-7525-406a-ce60-db161d61474e","3f28335e-9f44-cddd-a9fd-937c6591540c","7dca1206-0d06-78f0-f0ff-f4b44ce4a85f","6f807e51-6d18-79d6-aef0-a17ce8ec911d","d7b3c026-b9b3-6a41-964c-b62f464be8ff","03950f0b-1113-8019-c0ad-5dfa670cefe5","0a0652a1-30d7-6d22-3b7b-81a894b67c4b","02152cc2-2f06-0cc2-e344-0d94f6f65a64","77b4fb01-6de7-c009-57d9-4419b73ef361","d604b7c7-cc97-ab65-3b93-ad6b48efa13f","96ad0c50-6916-561b-a7dd-492f260bc661","27bc182b-09f6-9dbf-69cc-0fbed92136ee","f0192cd6-879e-9ee1-e52a-9cf072277e2b","0e2cefa9-bdaf-7fce-3fd8-5faea8b0ef86","f3199d5a-76c6-1aba-a5fc-b97b4fd6ad8e","06b4ee57-2eb2-4312-1488-f6d69438e965","b1fc97f9-05d0-bbb5-199e-2ca38ff116c7","b0531e70-28fb-862a-040c-44dff04f23be","872255c6-cead-0049-b016-6e43d39b9d77","45ecfe57-2ac7-9db7-88d1-a22597f28842","81f43ead-b3aa-4a71-b6e3-3cb41d8b845b"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":215,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":216,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":217,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"index":8,"name":"chrome/47543840-672a-467d-80df-8f7c3b9788c9","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":218,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.105Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"index":18,"input":{"domain":"chrome","file":"tasks/chrome/b4f95342-463e-4179-8c3f-193cd7241fb2.json","task":{"id":"b4f95342-463e-4179-8c3f-193cd7241fb2","snapshot":"chrome","instruction":"Find the Next Available dates for Diamond.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.recreation.gov/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"class","class_singleObject":{},"class_multiObject":{"camp-sortable-column-header":{"2":"camp-sortable-column-header"}}},"expected":{"type":"gotoRecreationPage_and_get_html_content","selector":"class","class":"camp-sortable-column-header","order":"2"}},"proxy":false,"possibility_of_env_change":"high","fixed_ip":false}},"name":"chrome/b4f95342-463e-4179-8c3f-193cd7241fb2","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":219,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":220,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.task.file","value":"tasks/chrome/b4f95342-463e-4179-8c3f-193cd7241fb2.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":221,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":222,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":223,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":224,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"18:chrome%2Fb4f95342-463e-4179-8c3f-193cd7241fb2","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":225,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:22.106Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.pod","value":"osworld-da46d875-0f2ebf"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":226,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/da46d875-6b82-4681-9284-653b0c7ae241/2026-10-11T19-37-52.633Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":227,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":228,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.agent.last_action","value":"none"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":229,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"auv.run.ids","value":["667ef41f-aff1-a90e-4baa-99b5c1cb7919","2d6f7e27-0bb8-e096-318f-c9372167dede","737a36b7-3e74-c5f3-d9f4-56890089bfce","30c43115-3885-1224-3b2c-94a271760e4e","0175e0c9-24c0-09fd-c11f-c14b5289ca6b","6c9862fe-f0f9-618e-469c-e0af939192a7","6da4b2e7-f8ac-e2d4-4ae3-2e9d8b1ae032","f268e396-e016-529b-7940-a106e0dfce0b","4ea7936c-2b5c-d647-3ef2-2bfb225d1358","ec58d524-7c7f-f023-a21e-d9b66cf30953","f3e5ba77-936d-4d0f-e67e-79f4500b4e4e","72bbf084-fc8e-04da-4381-cd707ee7f2e7","0cd0ba91-2598-1bf7-9c10-28ab3a183b54","96e2f246-423b-6724-e303-b0406f507684","c9c69cb6-20a0-2084-9703-bd2d3fed9591","413f2adf-8d0a-4628-0ba5-71ca6a63f3a7","febc7757-1a85-f9bf-c0ee-26b51f084b88","22e844af-1bb2-2913-8d54-6b31d99ba025","148cde6b-0fc7-b46a-fae1-ae335cecc6d8","d335e8cb-fb9b-70a7-ad7b-deace87c0a0e","86176196-fa0b-3c87-f5ed-9cd63975bdf0","87616c17-8224-6717-7205-37f3db02d27c","10466902-ddac-f6bc-e91d-05798956cbcc","e645bc21-04c3-f6d6-2ed1-cc1a461f14ef","a556eddb-da4a-4abf-0cb3-241e8b3fbe79","7b7af846-7cdc-9fc1-55f2-7deee2a1fdc8","7d888edd-fdfa-e92f-4bb9-851d0fe4abb7","eb51f97d-5c6c-2fdd-9a9c-5191469e137b","8ffe308d-3777-3b05-681b-44db8d4d7edd","2b337564-be38-1bea-e46e-efe546aece3b","4b46aa54-a3dc-4107-abe0-9edc28227441","3c3a6f4f-dc4c-a851-4dfa-6305b52efd7d","36f97f29-836d-e55e-5dec-43b22a950172","8a7f924b-ccea-8d80-9d33-819074291446","c7545e76-71ef-38ef-891b-6af3929156e2","182252ac-f9c8-0d58-c0d7-3d475d7e7e89","271eda00-a771-427e-6f94-6e82b08c8b25","fb988d72-deff-3f2c-293b-e0bd0a6102fe","326433d4-c3ae-a0f8-b850-04bfaf2cb439","6dfae1a9-bf89-84ca-e486-72f6d4f51276","73b0f574-cf0f-b2d1-195c-83cb3c148fc1","d0ee0a65-f541-3faa-5ede-985e91d6b967","f1063fc9-4897-7487-c74f-696a141a51d8","a84df944-5ccb-ff32-cfb6-b7cfbc1f5669","666f6ed7-1472-0a17-925d-8963368d27c9","5f23cfc6-6cbf-29ee-8661-4e4b7cdc3ae0","5bb6b97c-3700-f299-2a6b-7558ef731947","4b23c4b7-7b9b-99ac-f4e6-58f8323f48c3","5d0f3821-81bb-7347-c473-850caa9e0ac0","f9b315b2-4489-82c8-b860-e36022af1e9f"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":230,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":231,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.success","value":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":232,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.321Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":10,"name":"chrome/da46d875-6b82-4681-9284-653b0c7ae241","state":"failed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":233,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.322Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"index":19,"input":{"domain":"chrome","file":"tasks/chrome/f5d96daf-83a8-4c86-9686-bada31fc66ab.json","task":{"id":"f5d96daf-83a8-4c86-9686-bada31fc66ab","snapshot":"chrome","instruction":"Compare iPhone 15 Pro Max with iPhone 14 Pro Max and iPhone 13 Pro Max","source":"Mind2Web","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.apple.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["modelList"],"split_list":true},"expected":{"type":"rule","rules":{"expected":{"modelList":["iphone-15-pro-max","iphone-14-pro-max","iphone-13-pro-max"]},"ignore_list_order":true}}},"proxy":true,"possibility_of_env_change":"medium","fixed_ip":false}},"name":"chrome/f5d96daf-83a8-4c86-9686-bada31fc66ab","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":234,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":235,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.task.file","value":"tasks/chrome/f5d96daf-83a8-4c86-9686-bada31fc66ab.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":236,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":237,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":238,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":239,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":240,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:47:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.pod","value":"osworld-f5d96daf-a4fee3"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":241,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/f5d96daf-83a8-4c86-9686-bada31fc66ab/2026-10-11T19-47-40.346Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":242,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.agent.steps","value":9},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":243,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":244,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"auv.run.ids","value":["37ae7073-ca2b-3479-dbfb-178fb020c512","8668c8a7-8c23-45d0-2767-1a90c130b5a4","4d4486f6-8e08-5a8f-c107-8321427ab14b","a7ef81c1-c0b3-9be7-b077-868b6095b9eb","5406c743-1b54-4618-bf0a-faa2b8ea60d3","51b0ad72-0cbe-c770-79a0-b2047ba40a5b","dff25941-664d-2291-c7c9-8647fe04facd"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":245,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":246,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.479Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"name":"osworld.success","value":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":247,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.480Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"19:chrome%2Ff5d96daf-83a8-4c86-9686-bada31fc66ab","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":19,"name":"chrome/f5d96daf-83a8-4c86-9686-bada31fc66ab","state":"failed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":248,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.481Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"20:chrome%2F121ba48f-9e17-48ce-9bc6-a4fb17a7ebba","data":{"index":20,"input":{"domain":"chrome","file":"tasks/chrome/121ba48f-9e17-48ce-9bc6-a4fb17a7ebba.json","task":{"id":"121ba48f-9e17-48ce-9bc6-a4fb17a7ebba","snapshot":"chrome","instruction":"Find Dota 2 game and add all DLC to cart.","source":"Mind2Web","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.dota2.com/home","https://store.steampowered.com"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_added_to_steam_cart","result":{"type":"page_info","url":"https://store.steampowered.com/cart/"},"expected":{"type":"rule","rules":{"items":["The Dota 2 Official Soundtrack"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/121ba48f-9e17-48ce-9bc6-a4fb17a7ebba","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":249,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.481Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"20:chrome%2F121ba48f-9e17-48ce-9bc6-a4fb17a7ebba","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":250,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:49:23.482Z","workspaceId":"default-workspace"}
diff --git a/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/cases.jsonl b/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/cases.jsonl
index 367a560..7a4e622 100644
--- a/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/cases.jsonl
+++ b/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/cases.jsonl
@@ -1,18 +1,18 @@
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","caseName":"chrome/06fe7178-4491-4589-810f-2e2bc9502122","durationMs":146910,"endedAt":"2026-10-11T19:16:32.903Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/06fe7178-4491-4589-810f-2e2bc9502122.json","task":{"id":"06fe7178-4491-4589-810f-2e2bc9502122","snapshot":"chrome","instruction":"Can you make my computer bring back the last tab I shut down?","source":"https://www.wikihow.com/Switch-Tabs-in-Chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.lonelyplanet.com","https://www.airbnb.com","https://www.tripadvisor.com"]}},{"type":"chrome_close_tabs","parameters":{"urls_to_close":["https://www.tripadvisor.com"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_tabs","result":{"type":"open_tabs_info"},"expected":{"type":"rule","rules":{"type":"url","urls":["https://www.lonelyplanet.com","https://www.airbnb.com","https://www.tripadvisor.com"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/06fe7178-4491-4589-810f-2e2bc9502122.json","osworld.evaluator.func":"\"is_expected_tabs\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-06fe7178-624029","osworld.evidence":"__VIEVAL_DIR__/episodes/06fe7178-4491-4589-810f-2e2bc9502122/2026-10-11T19-14-06.064Z-gpt-5.5-r1","osworld.agent.steps":7,"osworld.agent.last_action":"DONE","auv.run.ids":["8ce2b33d-31cd-47b3-a869-b9cb4eab1d5d","7f8742c0-19c1-cae3-32ea-db1c53169b71","2decd904-75d7-ea71-11e7-e199b601a5d3","a505c326-bc53-cfd6-3ab2-b7b5ff4d8b07","d233deae-0dff-a34e-3e12-c1ef6e845d19","ef8e83be-bb57-3c0c-d366-3c2add975efe"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":0},"startedAt":"2026-10-11T19:14:05.993Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","caseName":"chrome/e1e75309-3ddb-4d09-92ec-de869c928143","durationMs":218639,"endedAt":"2026-10-11T19:17:44.633Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/e1e75309-3ddb-4d09-92ec-de869c928143.json","task":{"id":"e1e75309-3ddb-4d09-92ec-de869c928143","snapshot":"chrome","instruction":"Computer, can you turn the webpage I'm looking at into a PDF file, save it to my Desktop with the default filename and set the margins to none?","source":"https://in5stepstutorials.com/google-chrome/save-web-page-as-pdf-in-chrome.php","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://lilianweng.github.io/posts/2023-06-23-agent/"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"compare_pdfs","result":{"type":"vm_file","path":"/home/user/Desktop/LLM Powered Autonomous Agents _ Lil'Log.pdf","dest":"LLM Powered Autonomous Agents _ Lil'Log.pdf"},"expected":{"type":"pdf_from_url","path":"https://lilianweng.github.io/posts/2023-06-23-agent/","dest":"LLM Powered Autonomous Agents _ Lil'Log_gold.pdf"}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/e1e75309-3ddb-4d09-92ec-de869c928143.json","osworld.evaluator.func":"\"compare_pdfs\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-e1e75309-25dee7","osworld.evidence":"__VIEVAL_DIR__/episodes/e1e75309-3ddb-4d09-92ec-de869c928143/2026-10-11T19-14-06.065Z-gpt-5.5-r1","osworld.agent.steps":24,"osworld.agent.last_action":"DONE","auv.run.ids":["bbbfc80f-5c9d-3466-40dc-6ffc9fe3cec8","5efa0a92-631d-14c7-628e-1412594c3903","8c93afff-a5f8-3db9-62a1-cf6fde76a8fd","8a68cb33-b136-2bb7-5d5e-499da9c3b67c","b08b73a4-0a02-ab69-f999-6f52f47cdfbf","756c05ed-93ae-5669-7529-fa4381d834e0","bdadc082-0d52-98ac-9c4f-f36d90236668","b2dff437-c5c7-b54a-ffc9-702b714879c1","e3f53fad-c387-6078-08a2-76fab5a85c5e","bdd06d97-0078-aecb-67da-a0a52e6cd5aa","5a18dfc5-c050-04c6-2821-06a37db8440e","f6d8b193-3398-cf53-bc7f-6c21e3c3a4a0","d285b518-9ff4-dfd8-a3a7-b356697ffdb0","459b6908-d168-dd98-e2b3-b774d2f28c54","a3aa3e03-6967-4a4f-c63f-aa6f0407cddf","77cea753-b470-c14c-afd4-0fc1ee81ef03","992eafcd-9db8-0da4-c8a6-eabcaed0cf4b","fcc6b07e-1a7e-ccde-4e56-69b55dd81e73","bb542d39-1684-8e06-47b3-bb05e0535e7c","2be8f32c-3a08-31c4-8723-e2b808d3b711","9be7e44e-3013-0e5f-7201-36cb1ebd4a7b","e6b3123e-a0f3-8194-726d-6793a789bcc9","729d19e7-ed40-e269-ef3c-f658827b3e7e"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":0.9380815331090999},"startedAt":"2026-10-11T19:14:05.994Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","caseName":"chrome/35253b65-1c19-4304-8aa4-6884b8218fc0","durationMs":141074,"endedAt":"2026-10-11T19:18:53.977Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/35253b65-1c19-4304-8aa4-6884b8218fc0.json","task":{"id":"35253b65-1c19-4304-8aa4-6884b8218fc0","snapshot":"chrome","instruction":"Hey, I need a quick way back to this site. Could you whip up a shortcut on my desktop for me using Chrome's built-in feature?","source":"https://www.laptopmag.com/articles/how-to-create-desktop-shortcuts-for-web-pages-using-chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.mathsisfun.com/games/2048.html"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_shortcut_on_desktop","result":{"type":"shortcuts_on_desktop"},"expected":{"type":"rule","rules":{"type":"name","name":"Play Puzzle Game 2048"}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/35253b65-1c19-4304-8aa4-6884b8218fc0.json","osworld.evaluator.func":"\"is_shortcut_on_desktop\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-35253b65-de1eaa","osworld.evidence":"__VIEVAL_DIR__/episodes/35253b65-1c19-4304-8aa4-6884b8218fc0/2026-10-11T19-16-32.960Z-gpt-5.5-r1","osworld.agent.steps":12,"osworld.agent.last_action":"DONE","auv.run.ids":["562d89bb-7abb-c9b8-89e9-f5b7f6476fdc","c221b337-3709-b702-0d3c-38916e614403","33f22dc8-6f24-ea56-6b90-3cd8190d52a2","99797196-b5ea-b987-5276-bd0b3a19270c","a0fd65a9-32bd-b0e5-b8a3-a1c5458b387f","ef0fb452-785c-08a1-4667-78129d9c2e02","4dedcdb9-afc6-651e-f42a-06de2caefcd4","34f4119d-4df4-3ca7-cf9d-f253f90c6f61","27704877-1a99-0f93-8810-03d0f5de52f9","d30dc0a6-9abc-02ce-0a47-7396c30f8254"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:16:32.903Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","caseName":"chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca","durationMs":208165,"endedAt":"2026-10-11T19:21:12.798Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca.json","task":{"id":"2ad9387a-65d8-4e33-ad5b-7580065a27ca","snapshot":"chrome","instruction":"Can you make a new folder for me on the bookmarks bar in my internet browser? Let's call it 'Favorites.'","source":"https://www.youtube.com/watch?v=IN-Eq_UripQ","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"is_expected_bookmarks","result":{"type":"bookmarks"},"expected":{"type":"rule","rules":{"type":"bookmark_bar_folders_names","names":["Favorites"]}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca.json","osworld.evaluator.func":"\"is_expected_bookmarks\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-2ad9387a-03611b","osworld.evidence":"__VIEVAL_DIR__/episodes/2ad9387a-65d8-4e33-ad5b-7580065a27ca/2026-10-11T19-17-44.670Z-gpt-5.5-r1","osworld.agent.steps":26,"osworld.agent.last_action":"DONE","auv.run.ids":["63cc9314-526b-5d8c-c56c-6474f90ffcd3","74a785ce-182e-3dbd-63a5-29b61d79955c","a8b589ea-ec19-7b31-f717-5364234055a4","31462f1a-9636-0e8d-e59e-d3cf79817292","383a51e8-1059-33c1-318b-72b76118e6d5","9948558b-5941-2d57-fb21-40850c5f61f3","20f34f68-b134-e47b-5003-c4ece7e66b05","9e6b2088-1b0f-a47b-d18d-4792a1fda16b","e4e6c191-f988-e93e-be4d-4ca0e999849c","434aa8a4-779c-3f27-ac8e-097ce678d176","5390f775-93f1-913d-acf1-6d1b11bf790d","ffcb9b6c-576e-5849-5da1-c2cfc0f609a9","c1688640-a8db-816f-572b-1318e7d3e090","6a272e12-1ca3-125b-1929-71ad6f6b5edc","b6005878-8c1c-c18a-4e40-b22ad1e61345","e4bcdeb3-f807-de9e-a059-03a7021ae82c","7a6e804e-2635-6d32-9bb1-7d9b4caf24f1","391df434-ddc6-6483-8027-7a1b800ad797","73ff6def-2189-95a7-246f-662107a93d50","67e7f2ee-32ae-ba60-eb3a-410f16865231","7f147858-f5e9-9e5c-a165-950fdd0d78e2","f59fa940-6c43-b0c2-d6e7-bd90de15b13e","caf01e5f-724c-43dc-6bbb-e88a07f443f5","bc14f365-cc07-23c7-36ca-8cd78a711384","b4caaa6b-db0e-84a4-b707-50878f393486"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:17:44.633Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","caseName":"chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","durationMs":139064,"endedAt":"2026-10-11T19:21:13.041Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263.json","task":{"id":"7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","snapshot":"chrome","instruction":"Can you save this webpage I'm looking at to bookmarks bar so I can come back to it later?","source":"https://www.youtube.com/watch?v=ZaZ8GcTxjXA","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://blog.eleuther.ai/rotary-embeddings/","https://jalammar.github.io/illustrated-transformer/"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"is_expected_bookmarks","result":{"type":"bookmarks"},"expected":{"type":"rule","rules":{"type":"bookmark_bar_websites_urls","urls":["https://jalammar.github.io/illustrated-transformer/"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263.json","osworld.evaluator.func":"\"is_expected_bookmarks\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-7a5a7856-13547c","osworld.evidence":"__VIEVAL_DIR__/episodes/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263/2026-10-11T19-18-54.025Z-gpt-5.5-r1","osworld.agent.steps":9,"osworld.agent.last_action":"DONE","auv.run.ids":["bff34b9b-f6d2-a64a-c731-95a461421db2","c1d3b012-c69c-0543-b106-02e17fef9c70","fdddaf0d-4383-ae71-55f1-ec9759151168","a48f780c-7411-3bed-2fc6-eddc3665a1e9","3f9ca3ae-73e8-9d6d-cb89-a74f72223eab","3d3883ae-1d66-06a5-2a1a-4f4cdb9a69ea","009a7cfd-40dc-55ce-44e0-9f5210595c80","da930dff-5400-febf-35a8-0236cc13f405"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:18:53.977Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","caseName":"chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","durationMs":506796,"endedAt":"2026-10-11T19:22:32.789Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3.json","task":{"id":"7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","snapshot":"chrome","instruction":"Can you help me clean up my computer by getting rid of all the tracking things that Amazon might have saved? I want to make sure my browsing is private and those sites don't remember me.","source":"https://support.google.com/chrome/answer/95647?hl=en&ref_topic=7438325&sjid=16867045591165135686-AP#zippy=%2Cdelete-cookies-from-a-site","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.amazon.com","https://www.amazon.com/s?k=huggingface+transformers+book"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_cookie_deleted","result":{"type":"cookie_data","dest":"Cookies"},"expected":{"type":"rule","rules":{"type":"domains","domains":[".amazon.com"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3.json","osworld.evaluator.func":"\"is_cookie_deleted\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-7b6c7e24-d63d3a","osworld.evidence":"__VIEVAL_DIR__/episodes/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3/2026-10-11T19-14-06.062Z-gpt-5.5-r1","osworld.agent.steps":50,"osworld.agent.last_action":"none","auv.run.ids":["57a2c0d9-423e-24dd-f635-555b43ba3d84","e2d76e99-b9f7-2c02-6d62-1ff6613ecc61","55203674-669b-83d1-fa93-238fbf7889d2","cd308fef-d991-ea5a-6de4-c3e391006136","89dac0d5-c8d2-01ef-7025-f59ac4d43842","e6a7841d-4c1d-cb6d-4840-12945c1fe770","e97d0a1b-3fd2-d320-368d-5ac520e0d541","ad3e4591-1f0d-f5f5-1ed4-cd87d541b1f2","2e72488b-f633-67ff-e513-cb84d1107aea","441d44c4-3249-7a1f-fcf2-2487bff9d025","5bd14cde-1f00-a08c-a969-bf93cb3a1113","6af1c981-fb82-374a-0711-6aac7f46eb3f","8166a987-c12f-d7b8-b1c4-9f827249799e","119edb19-94f4-dd00-3d5e-17fe2587bb1f","8bfc851e-5738-fc7a-1a39-4c092760c9f7","c8652f37-232d-43af-92e0-0de1c0a4ab6c","5f7f1198-6ecd-f48e-dc9b-9b6dcc7e3f7c","d5d62fd1-797f-6ff3-a470-f3854018e7b0","33ca97c3-f535-8e93-35aa-badb1348c9b1","3e7bab57-5dfa-ae0d-1a2a-ea1c31788dca","9ca90c83-6ae5-22f4-26c8-857e081154f4","15617958-0d96-e799-42b8-fcb15e54f9b9","86261013-ea72-44d6-9dd0-1cb1aa763980","869164f9-862f-4c4e-901b-4b6c522be3cf","f9dcf37c-1d21-4831-fa80-2270a3e9ca0a","29037581-1849-35f1-3605-cfd7a8f5532f","a903d08b-834e-3596-7d77-e4a3eac7603d","c3482151-efe1-c0bf-519e-f0c848af4cac","5e91cdc5-ffb7-3aec-1b99-5336fb853b91","26831d79-fff7-d48a-fb6e-692b4e4c3524","326e31af-139c-2168-9091-97e7de38a2f8","25d61c14-ba73-2b5d-44b6-04a1a7361960","b1ebb94c-eb0b-4402-1fec-0aa72e72cfb0","3143d060-26ce-cda9-332d-49cc742d90e7","c102daf6-377b-e949-6429-5d7295477408","f15b186b-a04d-e66d-b63b-8795afb74db8","43fb424c-6d7a-74cd-a4cf-8fd150c8af85","96273415-a4f4-8a60-d880-5fd1ce74bf99","5655c4bf-87e8-807c-f83a-05d31af5486f","cd88d6c4-3624-1b76-d56b-30c09db9a9df","709e8263-b3e7-bc89-9553-18de97667603","39535112-9580-c5bc-4ad6-ff28af1a87d7","aa3a4b40-5715-00ea-deed-6b77245004ea","b9c81491-dcda-d5a5-70e1-7e27c54d8d9c","f4dcfb1c-8dd9-2510-870d-7fe2e806e744","b5013a28-0b93-6928-254b-80f37e64d250","7f1da95a-3d5d-8987-a5d0-ddfce3739799","122a101a-bb0a-ced4-ff1b-94cab07d0269","980a0d22-dbad-e11b-83de-4579a41fd1cc","5e9dfc2e-319a-6549-3a3a-115e51ff3acb"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:14:05.993Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","caseName":"chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","durationMs":139640,"endedAt":"2026-10-11T19:23:32.439Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3.json","task":{"id":"2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","snapshot":"chrome","instruction":"Lately I have changed my English name to Thomas. I want to update my username. Could you help me change the username in chrome profiles to Thomas?","source":"https://superuser.com/questions/1393683/how-to-change-the-username-in-google-chrome-profiles?rq=1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"chrome_inject_js","parameters":{"js":"if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }","wait_for_load":false}},{"type":"sleep","parameters":{"seconds":1}},{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"profile_name"},"expected":{"type":"rule","rules":{"expected":"Thomas"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3.json","osworld.evaluator.func":"\"exact_match\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-2ae9ba84-004323","osworld.evidence":"__VIEVAL_DIR__/episodes/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3/2026-10-11T19-21-12.909Z-gpt-5.5-r1","osworld.agent.steps":18,"osworld.agent.last_action":"DONE","auv.run.ids":["0f9a5e98-fe68-12b2-793a-423e1d0af6b4","c5ed57fe-72e7-a39d-bd11-dfd975fe9687","e54ba116-6839-8bc2-a87a-8b1e181ad407","3435fdc4-1dbf-49f3-bab1-97094a42134c","01c95781-d9db-4bd4-5f3b-162bea21a565","40adafda-c899-8bca-9a4b-fdbcc885e214","1116f716-f17f-3531-9b69-5aeabf594cea","a55fc692-d886-6171-62dc-16d4c8178ee9","5fdfb7cb-8e2d-da4c-84eb-e3f8a421d908","fb8d74aa-d814-bdb9-0c4c-77b775cf8ef2","f42fa1c1-ea4b-cb0d-a0b2-861333a51bb2","c92e144b-0dae-c45d-66e5-29c37639b403","b28cb371-6236-e9a5-3fd3-37f74c4e6035","63b1a0c5-1094-8f8a-59b0-d6570b3dec7f","960fefc4-4d1f-7a8a-0e02-bd8507e39b4b","1ab4cb89-5259-8f7a-9bb8-758fe852f8e9","838fdb09-579e-b2f4-33f9-8bf564972040"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:21:12.799Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","caseName":"chrome/af630914-714e-4a24-a7bb-f9af687d3b91","durationMs":101422,"endedAt":"2026-10-11T19:24:14.212Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/af630914-714e-4a24-a7bb-f9af687d3b91.json","task":{"id":"af630914-714e-4a24-a7bb-f9af687d3b91","snapshot":"chrome","instruction":"My grandmother has been using the Chrome lately and told me that the font size is way too small for her poor eyesight. Could you set the default font size to the largest for her?","source":"https://www.howtogeek.com/680260/how-to-change-chromes-default-text-size/","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"check_font_size","result":{"type":"chrome_font_size"},"expected":{"type":"rule","rules":{"type":"range","min":16,"max":99999}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/af630914-714e-4a24-a7bb-f9af687d3b91.json","osworld.evaluator.func":"\"check_font_size\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-af630914-1acfcc","osworld.evidence":"__VIEVAL_DIR__/episodes/af630914-714e-4a24-a7bb-f9af687d3b91/2026-10-11T19-22-32.860Z-gpt-5.5-r1","osworld.agent.steps":12,"osworld.agent.last_action":"DONE","auv.run.ids":["0fd46f09-65e2-d5ac-b323-2fa09f5bf403","330ccc1d-da9a-25e9-2f9f-ce639826c15d","559125ba-d05b-133b-205d-c25dbf2f1136","5adb7abb-0069-4d63-0a94-7efd2d4467ff","23552df7-c02d-fd80-093d-c574ecda8aa7","f6ce2758-88b5-39fe-fc71-fe5b29574a16","80d55ca2-cf68-e325-fd44-bad3b9b09894","a427ffad-066e-8c00-c9b2-cb1242c780c4","99ae98c0-f23e-69b8-1215-73980b365cb1","e4e9df49-12a3-79f3-ed70-502a311a3029","6941b4a9-36ef-1d84-f78c-5213ad9bb3c7"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:22:32.790Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","caseName":"chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d","durationMs":127883,"endedAt":"2026-10-11T19:25:40.323Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d.json","task":{"id":"3720f614-37fd-4d04-8a6b-76f54f8c222d","snapshot":"chrome","instruction":"I am from the country of Atlantis, and my mother tongue is Xenothian. Please change the Google Chrome interface language to Xenothian using only Chrome’s built-in settings.","source":"https://superuser.com/questions/984668/change-interface-language-of-chrome-to-english","config":[],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d.json","osworld.evaluator.func":"\"infeasible\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-3720f614-a3afd3","osworld.evidence":"__VIEVAL_DIR__/episodes/3720f614-37fd-4d04-8a6b-76f54f8c222d/2026-10-11T19-23-32.504Z-gpt-5.5-r1","osworld.agent.steps":21,"osworld.agent.last_action":"FAIL","auv.run.ids":["e7233e57-7f5a-a4b4-5c78-9e35691502ae","be0ede42-f98b-757a-a60d-b241457b5eed","3318b6f2-0854-039c-19f5-63afee4bc154","76466e29-2dc3-06aa-2896-2318ea2b15ae","6e47111e-a797-59fb-2d4f-aa8a0426301e","10c08eeb-38f4-1d2c-d2c3-fa32e5871ebf","505b9407-7af7-d8df-689f-2983f4abe321","9575b7a4-d4ec-66d4-bace-f9ec47084431","72f97bf4-fe88-be60-eacd-d844be41312a","d2e73cc7-885b-293d-cd6d-f1ced0cbf263","fef469ff-6a2d-c6d5-3b34-99443c89c912","cfd8c665-6316-57b8-4db6-7e1c5bc59fe2","0e2a5f36-1084-48b0-6daf-b7153e3d6cbc","8c449646-2441-5448-2ff1-5a24185a4196","5e0906d4-ace6-4ce0-f657-b6976c986dce","d654cc67-7ab3-44a0-718c-53319dc2aaf3","7e5a2393-13f7-8415-e235-942d4cf5a8e6","ae6f46ad-8ab5-cae3-14a4-14f5fe47a0d6","da810650-34a9-b977-12b5-622e97fc3285","2af4400a-f4e6-85d9-7683-fb6f8e645662"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:23:32.440Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","caseName":"chrome/99146c54-4f37-4ab8-9327-5f3291665e1e","durationMs":104925,"endedAt":"2026-10-11T19:25:59.137Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/99146c54-4f37-4ab8-9327-5f3291665e1e.json","task":{"id":"99146c54-4f37-4ab8-9327-5f3291665e1e","snapshot":"chrome","instruction":"Please help me set Chrome to delete my browsing data automatically every time I close the browser.","source":"https://www.youtube.com/watch?v=v0kxqB7Xa6I","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"data_delete_automacally"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/99146c54-4f37-4ab8-9327-5f3291665e1e.json","osworld.evaluator.func":"\"exact_match\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-99146c54-b6912e","osworld.evidence":"__VIEVAL_DIR__/episodes/99146c54-4f37-4ab8-9327-5f3291665e1e/2026-10-11T19-24-14.269Z-gpt-5.5-r1","osworld.agent.steps":15,"osworld.agent.last_action":"DONE","auv.run.ids":["d0adaa21-d213-f9cd-7161-a8302a0cf5ae","c6864c42-24ac-2e17-702f-b3946eaa0d6e","5c4bbc08-87c7-f1a9-6086-6117ce0166d8","aa3e6e0c-fc22-d519-8af7-46af279e86ea","6b3c7baf-b993-f1eb-815a-a05511c2538a","5e0e7912-97da-0ed5-92a5-c0be7fb5a3ab","dcc6481c-3a7e-6e5b-7387-70ee4ac5dcd0","dcd453f1-035f-3fcd-57c6-739cd14cb28f","5aefc831-5ced-48e1-6816-5ff3142e4c1d","3fffcefe-eb2b-d344-8fca-b38b75038f6b","a32e2490-7f44-573d-c419-b373bf59eb3a","e6e8c5c3-5d4b-de95-1460-633ecc81d135","1965eae6-bbe8-f11d-e72e-c87c58121355","3fb41d9d-285f-3035-6ae8-dab725a3af56"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:24:14.212Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","caseName":"chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381","durationMs":348077,"endedAt":"2026-10-11T19:27:01.118Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381.json","task":{"id":"480bcfea-d68f-4aaa-a0a9-2589ef319381","snapshot":"chrome","instruction":"I do not like the design of the new 2023 chrome UI. I want to keep using the original one. Can you disable the new 2023 version chrome UI for me? ","source":"https://bugartisan.medium.com/disable-the-new-chrome-ui-round-in-2023-db168271f71e","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381.json","osworld.evaluator.func":"\"infeasible\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-480bcfea-ec301a","osworld.evidence":"__VIEVAL_DIR__/episodes/480bcfea-d68f-4aaa-a0a9-2589ef319381/2026-10-11T19-21-13.134Z-gpt-5.5-r1","osworld.agent.steps":48,"osworld.agent.last_action":"DONE","auv.run.ids":["4e6d8957-8251-96cb-e992-062c46c81d6b","38fa305a-8802-a9e9-169b-47a009226e12","108d142e-ffce-3a83-e411-c5ff506ec0d1","f32796ad-9635-7719-fef3-cf7a81d39048","92bc72c4-84a8-4a77-f4ca-65ed3cf50948","a1a87341-ae9e-f3cb-1c90-47822ce105ed","872ad29c-4724-81c0-faff-e31b11449963","66aa3fa1-4558-323d-6298-61c2dc68241b","974e428d-8a56-a5ba-f192-15bd87937f34","ea313189-e736-ad7c-3ae8-7a9e852292d3","0e251863-faba-ee72-3f76-89bc2404123b","3610e612-7191-48bc-49ac-3b5efd3a6b9a","27a10af6-5306-328d-e80f-d4b13ece9460","a1736127-7b9d-cf9d-f833-aec169a8d0eb","da52a1d4-4809-f884-940e-e4c67505f814","62a5d2ad-ccb7-e5ff-0070-3f9829c6e735","e2b8c379-fc3c-4c72-a070-1c2d5f31d764","22f1e1be-cc9a-ee20-1c90-27c161529daa","1e568e0c-cca7-f993-1d9d-8bd60146c966","fb34b88b-3fb4-8472-e4a4-07d3acca8dbb","c516d635-d440-8897-f6c1-68d4e91b55f8","ce168168-9f6a-ed51-6655-3c2e3f108b97","0bb207fb-c857-61d2-ca66-37f40c96cb4b","ef16bc20-6f82-7a70-45f3-1f1cd3dec229","a8a42aa6-58ac-6b8c-c323-04eb9a81dfe1","9bed18f7-068b-fec9-dbfb-b9ae8f9ed667","94bfb787-6ba8-c951-59fc-e42672097f98","d1335548-a570-40ea-85c7-58421601170e","58e1f5b9-4bc3-98f7-c1fa-53d47a9d6d8b","8052e573-1f86-9ed8-5178-001ffb624af2","eb864f7c-e922-a1f5-82d3-19fc3a8529c3","49cf13de-6aa8-9bcd-836f-53516bbb472d","5e0bee47-1ecc-5f63-ad6a-7c7737c93acc","dd33b4ba-d774-7a5b-664d-eaf879f3fa3f","7c1aa95e-9253-f4da-9988-544345869c58","0e998bff-129a-564d-39e4-e9590d39d772","549d8216-ae88-7ebb-bc07-9370356302da","34fe8896-7301-0fac-f4d0-932f2cdc4978","8483c9e7-1df2-a2e5-445e-2fa92ded01fb","2832ae61-46af-6a38-5d69-5455b839a603","e3a4c429-f8b1-5553-54c1-9c10117ac1dc","773bbc06-953f-bcb3-aec9-5ce329a4cbd7","b1002976-3bd3-1f43-f741-fe96e42e9ab9","16832bcc-b381-584f-fa27-ec4f23725be6","cb66200e-cb9e-1ad6-c567-42532283667c","b9d98bf1-44c6-f04e-d023-94f139ad95df","b0710018-396e-e53d-9f08-a8cc98ada2ed"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":0},"startedAt":"2026-10-11T19:21:13.041Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","caseName":"chrome/12086550-11c0-466b-b367-1d9e75b3910e","durationMs":110044,"endedAt":"2026-10-11T19:27:30.367Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/12086550-11c0-466b-b367-1d9e75b3910e.json","task":{"id":"12086550-11c0-466b-b367-1d9e75b3910e","snapshot":"chrome","instruction":"Computer, please navigate to the area in my browser settings where my passwords are stored. I want to check my login information for Etsy without revealing it just yet.","source":"https://www.quora.com/What-are-the-cool-tricks-to-use-Google-Chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_active_tab_approximate","result":{"type":"active_url_from_accessTree","goto_prefix":""},"expected":{"type":"rule","rules":{"type":"url","url":"chrome://password-manager/passwords"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/12086550-11c0-466b-b367-1d9e75b3910e.json","osworld.evaluator.func":"\"is_expected_active_tab_approximate\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-12086550-2b5033","osworld.evidence":"__VIEVAL_DIR__/episodes/12086550-11c0-466b-b367-1d9e75b3910e/2026-10-11T19-25-40.365Z-gpt-5.5-r1","osworld.agent.steps":19,"osworld.agent.last_action":"DONE","auv.run.ids":["2963c2e2-1f89-a096-1ebe-7eaf92065e46","4e0372bc-b487-e3af-7231-04d2da8201b7","22fa34c3-084f-45de-5560-8a6182c54251","84e10f86-4bf1-9d06-a72b-0c78d14324e9","4cd62fa9-a1b5-7854-b81e-19b3e89150a2","eead0f83-1000-bce6-ef6f-30c7e166bf4b","22c5358f-436a-e2db-9ca2-3c422e7c72f6","6ac4a067-f900-90ff-e614-17d42c90a8ff","7788c53a-47fe-e2af-54a0-086c340ab26e","5f696e34-76b3-37bb-807a-ceec43c99ea1","ffd1dda9-3830-22f4-b2ad-de4b4cb0451b","743ab0b6-3337-e402-a743-bce3686f508f","fb895282-2852-89a5-229a-d3fff57ed378","724f8678-dd3e-1a9f-578f-f934f07ac6e5","3d883ba2-ccc5-e127-e186-d323d17db741","6fc1bbb7-a733-8c5f-6be1-82da7da34653","c745eaff-b4ea-287f-16c0-44a94bdc9a1f","e9e1dc6d-edb7-1ca1-5718-b86a58bdb4f5"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:25:40.323Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","caseName":"chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","durationMs":104339,"endedAt":"2026-10-11T19:28:45.457Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9.json","task":{"id":"93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","snapshot":"chrome","instruction":"Could you assist me in turning off the dark mode feature in Google Chrome? I've noticed that while dark mode is great for reducing glare, it actually makes it more challenging for me to read text clearly, especially with my astigmatism.","source":"https://superuser.com/questions/1417973/how-to-disable-google-chrome-dark-mode","config":[{"type":"execute","parameters":{"command":"mkdir -p /home/user/.config/google-chrome/Default && if [ ! -f /home/user/.config/google-chrome/Default/Preferences ]; then echo '{}' > /home/user/.config/google-chrome/Default/Preferences; fi && python3 -c \"import json; p='/home/user/.config/google-chrome/Default/Preferences'; d=json.load(open(p)); d.setdefault('browser', {}).setdefault('theme', {})['color_scheme'] = 2; d['browser']['theme']['color_scheme2'] = 2; json.dump(d, open(p, 'w'))\"","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"sleep","parameters":{"seconds":1}}],"func":["match_in_list","is_expected_url_pattern_match"],"result":[{"type":"chrome_appearance_mode_ui"},{"type":"active_url_from_accessTree","goto_prefix":""}],"expected":[{"type":"rule","rules":{"expected":["light","system"]}},{"type":"rule","rules":{"expected":["^chrome://settings/appearance/?$"]}}]},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9.json","osworld.evaluator.func":"[\"match_in_list\",\"is_expected_url_pattern_match\"]","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-93eabf48-e94487","osworld.evidence":"__VIEVAL_DIR__/episodes/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9/2026-10-11T19-27-01.140Z-gpt-5.5-r1","osworld.agent.steps":17,"osworld.agent.last_action":"DONE","auv.run.ids":["1f35452b-27d1-6b80-7f53-5d6891721513","2d8de726-aae4-8cf2-84e8-dd3edd47b903","e1d3f454-7f66-a132-05c7-82f3b9326713","727c8e31-b21b-85d6-7df2-035c775e8283","b472dd48-3116-5a6c-4886-4006a53d51b3","8de8d87c-959a-9ce6-77d3-7842662db847","467175cc-5c7a-613a-2b80-e73b63041ba4","89693650-e567-c4ab-a085-d6ba55f09be7","6ab39e25-6131-4688-df50-62cbb4b8f047","66f63aa8-9a02-d15e-829d-e055a98da1f5","726dbda2-3636-d07a-6356-151100372eac","70a927cd-eeb3-26a1-e56b-2c146b261045","4cdae1d1-5a80-f069-08a4-32033d574962","d4ea467e-1bbf-cb53-8f9b-5bed00b1f87e","cf1606ed-0a39-574b-b269-3f5736837607","04f5cbce-ef97-c9e5-28ed-15995841f2ac"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:27:01.118Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","caseName":"chrome/3299584d-8f11-4457-bf4c-ce98f7600250","durationMs":70860,"endedAt":"2026-10-11T19:29:56.318Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/3299584d-8f11-4457-bf4c-ce98f7600250.json","task":{"id":"3299584d-8f11-4457-bf4c-ce98f7600250","snapshot":"chrome","instruction":"On my surface pro whenever I launch Chrome it always opens \"funbrain.com.\" I don't want this. I cleared my cache but it still happens. What should I do?","source":"https://www.reddit.com/r/techsupport/comments/12zwymy/comment/jhtri65/?utm_source=share&utm_medium=web2x&context=3","config":[{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt update -y && echo {CLIENT_PASSWORD} | sudo -S apt install jq -y","shell":true}},{"type":"execute","parameters":{"command":"cd /home/user/.config/google-chrome/Default && jq '. + {\"session\":{\"restore_on_startup\":4, \"startup_urls\":[\"http://funbrain.com/\"]}}' Preferences > temp && mv temp Preferences","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"exact_match","result":{"type":"new_startup_page"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/3299584d-8f11-4457-bf4c-ce98f7600250.json","osworld.evaluator.func":"\"exact_match\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-3299584d-e2fde8","osworld.evidence":"__VIEVAL_DIR__/episodes/3299584d-8f11-4457-bf4c-ce98f7600250/2026-10-11T19-28-45.484Z-gpt-5.5-r1","osworld.agent.steps":11,"osworld.agent.last_action":"DONE","auv.run.ids":["80909ad8-f02c-f38b-0700-fd1ac6072228","47158092-f636-85dc-28e9-84f5332fe9ad","6d36679e-5b92-fdd8-3aaa-d749490482da","0a29244d-1ceb-10d4-900e-31208c13df3e","2faa9f85-7189-1239-f4db-6556fe8286c7","43c74854-caf5-1443-4c3b-986991c229b5","9400204b-25ce-5baa-33b3-e970ec3d38d2","ead1e4a4-daf0-7246-aede-e96ca8a6417e","f6fa6a21-3e4b-3858-d879-98766277151c","6ecf3c12-fdc2-ed3f-0e94-951e0e6ad249"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":1},"startedAt":"2026-10-11T19:28:45.458Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","caseName":"chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837","durationMs":241291,"endedAt":"2026-10-11T19:30:00.429Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837.json","task":{"id":"6766f2b8-8a72-417f-a9e5-56fcaa735837","snapshot":"chrome","instruction":"Could you help me unzip the downloaded extension file from /home/user/Desktop/ to /home/user/Desktop/ and configure it in Chrome's extensions?","source":"https://support.google.com/chrome/thread/205881926/it-s-possible-to-load-unpacked-extension-automatically-in-chrome?hl=en","config":[{"type":"download","parameters":{"files":[{"url":"https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache/resolve/main/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837/helloExtension.zip","path":"/home/user/Desktop/helloExtension.zip"}]}},{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt-get update -y && echo {CLIENT_PASSWORD} | sudo -S apt-get install unzip -y && unzip /home/user/Desktop/helloExtension.zip -d /home/user/Desktop/ && rm /home/user/Desktop/helloExtension.zip","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_in_list","result":{"type":"find_unpacked_extension_path"},"expected":{"type":"rule","rules":{"expected":"/home/user/Desktop/helloExtension"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837.json","osworld.evaluator.func":"\"is_in_list\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-6766f2b8-e37b73","osworld.evidence":"__VIEVAL_DIR__/episodes/6766f2b8-8a72-417f-a9e5-56fcaa735837/2026-10-11T19-25-59.183Z-gpt-5.5-r1","osworld.agent.steps":50,"osworld.agent.last_action":"none","auv.run.ids":["2114cb1d-ee83-644a-fa5c-d3d7aefc82cd","2fbf9799-eadb-cad8-2146-2e7f28fba21a","0248689b-acf0-cba5-47d3-10e6ba8a5fff","caddbc1c-8001-fea4-e957-db89ea653a9b","8dda2b9d-5fc7-ef75-a813-91a3fd04dad3","23883639-7974-f8d9-bd96-2a269054e811","22fe8694-af91-cb1f-6c7b-266179602a1d","b9a29a0f-8a64-ac79-6e61-48a8eb8b177e","dd2f09f1-b129-5dec-5a93-cf9516984d11","8c73479b-76cd-06a8-b84a-93f689a6d079","069df994-c480-d271-9bc2-35ec174f7970","0db0fc5d-574d-2e86-516e-0e5fff2e3e71","5d0d54f7-981a-5ffb-e011-8cb3de485b6a","9f3cc401-eb84-0a15-f608-793c6e6c3bc6","a2ed395b-408a-210c-e308-6ee183c3874f","a8345888-d789-baa5-1f78-2b866d20f2b0","fd92e601-364d-96a5-f762-ae44bb800d74","fd660c01-3db4-7c6e-f1c7-cdbc13703113","ab56be1b-a8f9-23de-e1f0-24aa05c931c5","e630d394-ecc8-0497-9af0-147b2255a266","1fb3d0d8-aa84-b974-c1f5-ab3818bdf1f2","58bfb27a-8c03-eeb8-61f5-da563ff337a6","6c097d75-9da0-df17-1163-1b0690ab5725","3d44b019-49bb-9b0d-5b66-3d086d66a7f4","02d4e409-88fc-2c4b-c31b-607c72b0803c","b707b42c-dc55-ecf7-1da7-d81e2f44560d","fc6655d6-40cf-69bb-94f0-0a168c08f324","c70c2156-1a66-a4bf-9bbb-02f46fbef776","7ca60128-94c4-b2f3-17fe-3deb06ca935a","2ed12090-81db-e161-4311-3cd4016a34f1","22e70f25-6f1e-12c4-2a3d-628329358291","7b58074b-8d71-22cd-bdae-09121e6778a6","f474b9a2-6b75-4649-4bc7-9a8dfa2a7baf","a5e39e0b-ed11-5330-e91e-490675ef71fd","4d722f98-c174-fe01-58ed-9847def5f5b0","ba583be9-6778-3c34-897e-8d2035aa3324","48abaa94-7815-b42a-e76a-eb7c3d57bd69","17c2fcf5-32c0-97fc-8a33-67846c6eb805","9a3d4065-bbe5-f8af-cbe8-3f5d444f66cc","b5bcfded-110c-c06f-5852-d0d97caf197e","e072df72-d4bf-70e9-c833-b27513ed770e","17e30947-fdbe-1a39-3a63-ea09cd2f8ef4","a6df051e-04cd-b402-23f6-468b9e9ba8a0","0073af34-176e-287d-0f2a-1cc099e41ecf","48722b90-33cf-ccb8-94ea-b0d9cb60e1f2","404335ea-d710-aebe-f159-fecd48db3f67","af0516e4-5fa7-795b-9920-2db84f18367a","4ecfbcd6-c1a1-6395-3901-9592c689d71e","e83badb8-d039-08c1-61f9-2a6ab08be8d0"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":0},"startedAt":"2026-10-11T19:25:59.138Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","caseName":"chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03","durationMs":232182,"endedAt":"2026-10-11T19:31:22.549Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json","task":{"id":"ae78f875-5b98-4907-bbb5-9c737fc68c03","snapshot":"chrome","instruction":"Could you please change the number of search results displayed on one page to 50? I find that having more results visible at once significantly enhances my research efficiency, as it reduces the need to constantly click through multiple pages. ","source":"https://support.google.com/chrome/thread/219988391/increase-search-results-per-page?hl=en","config":[],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json","osworld.evaluator.func":"\"infeasible\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":0},"startedAt":"2026-10-11T19:27:30.367Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","caseName":"chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626","durationMs":82433,"endedAt":"2026-10-11T19:31:22.862Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json","task":{"id":"9656a811-9b5b-4ddf-99c7-5117bcef0626","snapshot":"chrome","instruction":"I want Chrome to warn me whenever I visit a potentially harmful or unsafe website. Can you enable this safety feature?","source":"https://www.quora.com/How-do-I-set-the-security-settings-for-the-Google-Chrome-browser-for-the-best-security#:~:text=Enable%20Safe%20Browsing:%20Chrome%20has%20a%20built%2Din,Security%20%3E%20Security%20%3E%20Enable%20Safe%20Browsing.","config":[{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt update -y && echo {CLIENT_PASSWORD} | sudo -S apt install jq -y","shell":true}},{"type":"execute","parameters":{"command":"mkdir -p /home/user/.config/google-chrome/Default && if [ ! -f /home/user/.config/google-chrome/Default/Preferences ]; then echo '{}' > /home/user/.config/google-chrome/Default/Preferences; fi && cd /home/user/.config/google-chrome/Default && jq '. + {\"safebrowsing\":{\"enabled\":false,\"enhanced\":false}}' Preferences > temp && mv temp Preferences","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_safe_browsing"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json","osworld.evaluator.func":"\"exact_match\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":0},"startedAt":"2026-10-11T19:30:00.429Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","caseName":"chrome/030eeff7-b492-4218-b312-701ec99ee0cc","durationMs":88174,"endedAt":"2026-10-11T19:31:24.492Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json","task":{"id":"030eeff7-b492-4218-b312-701ec99ee0cc","snapshot":"chrome","instruction":"Can you enable the 'Do Not Track' feature in Chrome to enhance my online privacy?","source":"https://www.surreycc.gov.uk/website/cookies/do-not-track","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"chrome_inject_js","parameters":{"js":"if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }","wait_for_load":false}},{"type":"sleep","parameters":{"seconds":1}},{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_do_not_track"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json","osworld.evaluator.func":"\"exact_match\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":1,"scores":{"exact":0},"startedAt":"2026-10-11T19:29:56.318Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","caseName":"chrome/06fe7178-4491-4589-810f-2e2bc9502122","durationMs":146910,"endedAt":"2026-10-11T19:16:32.903Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/06fe7178-4491-4589-810f-2e2bc9502122.json","task":{"id":"06fe7178-4491-4589-810f-2e2bc9502122","snapshot":"chrome","instruction":"Can you make my computer bring back the last tab I shut down?","source":"https://www.wikihow.com/Switch-Tabs-in-Chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.lonelyplanet.com","https://www.airbnb.com","https://www.tripadvisor.com"]}},{"type":"chrome_close_tabs","parameters":{"urls_to_close":["https://www.tripadvisor.com"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_tabs","result":{"type":"open_tabs_info"},"expected":{"type":"rule","rules":{"type":"url","urls":["https://www.lonelyplanet.com","https://www.airbnb.com","https://www.tripadvisor.com"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/06fe7178-4491-4589-810f-2e2bc9502122.json","osworld.evaluator.func":"\"is_expected_tabs\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-06fe7178-624029","osworld.evidence":"__VIEVAL_DIR__/episodes/06fe7178-4491-4589-810f-2e2bc9502122/2026-10-11T19-14-06.064Z-gpt-5.5-r1","osworld.agent.steps":7,"osworld.agent.last_action":"DONE","auv.run.ids":["8ce2b33d-31cd-47b3-a869-b9cb4eab1d5d","7f8742c0-19c1-cae3-32ea-db1c53169b71","2decd904-75d7-ea71-11e7-e199b601a5d3","a505c326-bc53-cfd6-3ab2-b7b5ff4d8b07","d233deae-0dff-a34e-3e12-c1ef6e845d19","ef8e83be-bb57-3c0c-d366-3c2add975efe"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":0},"startedAt":"2026-10-11T19:14:05.993Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","caseName":"chrome/e1e75309-3ddb-4d09-92ec-de869c928143","durationMs":218639,"endedAt":"2026-10-11T19:17:44.633Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/e1e75309-3ddb-4d09-92ec-de869c928143.json","task":{"id":"e1e75309-3ddb-4d09-92ec-de869c928143","snapshot":"chrome","instruction":"Computer, can you turn the webpage I'm looking at into a PDF file, save it to my Desktop with the default filename and set the margins to none?","source":"https://in5stepstutorials.com/google-chrome/save-web-page-as-pdf-in-chrome.php","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://lilianweng.github.io/posts/2023-06-23-agent/"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"compare_pdfs","result":{"type":"vm_file","path":"/home/user/Desktop/LLM Powered Autonomous Agents _ Lil'Log.pdf","dest":"LLM Powered Autonomous Agents _ Lil'Log.pdf"},"expected":{"type":"pdf_from_url","path":"https://lilianweng.github.io/posts/2023-06-23-agent/","dest":"LLM Powered Autonomous Agents _ Lil'Log_gold.pdf"}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/e1e75309-3ddb-4d09-92ec-de869c928143.json","osworld.evaluator.func":"\"compare_pdfs\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-e1e75309-25dee7","osworld.evidence":"__VIEVAL_DIR__/episodes/e1e75309-3ddb-4d09-92ec-de869c928143/2026-10-11T19-14-06.065Z-gpt-5.5-r1","osworld.agent.steps":24,"osworld.agent.last_action":"DONE","auv.run.ids":["bbbfc80f-5c9d-3466-40dc-6ffc9fe3cec8","5efa0a92-631d-14c7-628e-1412594c3903","8c93afff-a5f8-3db9-62a1-cf6fde76a8fd","8a68cb33-b136-2bb7-5d5e-499da9c3b67c","b08b73a4-0a02-ab69-f999-6f52f47cdfbf","756c05ed-93ae-5669-7529-fa4381d834e0","bdadc082-0d52-98ac-9c4f-f36d90236668","b2dff437-c5c7-b54a-ffc9-702b714879c1","e3f53fad-c387-6078-08a2-76fab5a85c5e","bdd06d97-0078-aecb-67da-a0a52e6cd5aa","5a18dfc5-c050-04c6-2821-06a37db8440e","f6d8b193-3398-cf53-bc7f-6c21e3c3a4a0","d285b518-9ff4-dfd8-a3a7-b356697ffdb0","459b6908-d168-dd98-e2b3-b774d2f28c54","a3aa3e03-6967-4a4f-c63f-aa6f0407cddf","77cea753-b470-c14c-afd4-0fc1ee81ef03","992eafcd-9db8-0da4-c8a6-eabcaed0cf4b","fcc6b07e-1a7e-ccde-4e56-69b55dd81e73","bb542d39-1684-8e06-47b3-bb05e0535e7c","2be8f32c-3a08-31c4-8723-e2b808d3b711","9be7e44e-3013-0e5f-7201-36cb1ebd4a7b","e6b3123e-a0f3-8194-726d-6793a789bcc9","729d19e7-ed40-e269-ef3c-f658827b3e7e"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":0},"startedAt":"2026-10-11T19:14:05.994Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","caseName":"chrome/35253b65-1c19-4304-8aa4-6884b8218fc0","durationMs":141074,"endedAt":"2026-10-11T19:18:53.977Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/35253b65-1c19-4304-8aa4-6884b8218fc0.json","task":{"id":"35253b65-1c19-4304-8aa4-6884b8218fc0","snapshot":"chrome","instruction":"Hey, I need a quick way back to this site. Could you whip up a shortcut on my desktop for me using Chrome's built-in feature?","source":"https://www.laptopmag.com/articles/how-to-create-desktop-shortcuts-for-web-pages-using-chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.mathsisfun.com/games/2048.html"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_shortcut_on_desktop","result":{"type":"shortcuts_on_desktop"},"expected":{"type":"rule","rules":{"type":"name","name":"Play Puzzle Game 2048"}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/35253b65-1c19-4304-8aa4-6884b8218fc0.json","osworld.evaluator.func":"\"is_shortcut_on_desktop\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-35253b65-de1eaa","osworld.evidence":"__VIEVAL_DIR__/episodes/35253b65-1c19-4304-8aa4-6884b8218fc0/2026-10-11T19-16-32.960Z-gpt-5.5-r1","osworld.agent.steps":12,"osworld.agent.last_action":"DONE","auv.run.ids":["562d89bb-7abb-c9b8-89e9-f5b7f6476fdc","c221b337-3709-b702-0d3c-38916e614403","33f22dc8-6f24-ea56-6b90-3cd8190d52a2","99797196-b5ea-b987-5276-bd0b3a19270c","a0fd65a9-32bd-b0e5-b8a3-a1c5458b387f","ef0fb452-785c-08a1-4667-78129d9c2e02","4dedcdb9-afc6-651e-f42a-06de2caefcd4","34f4119d-4df4-3ca7-cf9d-f253f90c6f61","27704877-1a99-0f93-8810-03d0f5de52f9","d30dc0a6-9abc-02ce-0a47-7396c30f8254"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:16:32.903Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","caseName":"chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca","durationMs":208165,"endedAt":"2026-10-11T19:21:12.798Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca.json","task":{"id":"2ad9387a-65d8-4e33-ad5b-7580065a27ca","snapshot":"chrome","instruction":"Can you make a new folder for me on the bookmarks bar in my internet browser? Let's call it 'Favorites.'","source":"https://www.youtube.com/watch?v=IN-Eq_UripQ","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"is_expected_bookmarks","result":{"type":"bookmarks"},"expected":{"type":"rule","rules":{"type":"bookmark_bar_folders_names","names":["Favorites"]}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca.json","osworld.evaluator.func":"\"is_expected_bookmarks\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-2ad9387a-03611b","osworld.evidence":"__VIEVAL_DIR__/episodes/2ad9387a-65d8-4e33-ad5b-7580065a27ca/2026-10-11T19-17-44.670Z-gpt-5.5-r1","osworld.agent.steps":26,"osworld.agent.last_action":"DONE","auv.run.ids":["63cc9314-526b-5d8c-c56c-6474f90ffcd3","74a785ce-182e-3dbd-63a5-29b61d79955c","a8b589ea-ec19-7b31-f717-5364234055a4","31462f1a-9636-0e8d-e59e-d3cf79817292","383a51e8-1059-33c1-318b-72b76118e6d5","9948558b-5941-2d57-fb21-40850c5f61f3","20f34f68-b134-e47b-5003-c4ece7e66b05","9e6b2088-1b0f-a47b-d18d-4792a1fda16b","e4e6c191-f988-e93e-be4d-4ca0e999849c","434aa8a4-779c-3f27-ac8e-097ce678d176","5390f775-93f1-913d-acf1-6d1b11bf790d","ffcb9b6c-576e-5849-5da1-c2cfc0f609a9","c1688640-a8db-816f-572b-1318e7d3e090","6a272e12-1ca3-125b-1929-71ad6f6b5edc","b6005878-8c1c-c18a-4e40-b22ad1e61345","e4bcdeb3-f807-de9e-a059-03a7021ae82c","7a6e804e-2635-6d32-9bb1-7d9b4caf24f1","391df434-ddc6-6483-8027-7a1b800ad797","73ff6def-2189-95a7-246f-662107a93d50","67e7f2ee-32ae-ba60-eb3a-410f16865231","7f147858-f5e9-9e5c-a165-950fdd0d78e2","f59fa940-6c43-b0c2-d6e7-bd90de15b13e","caf01e5f-724c-43dc-6bbb-e88a07f443f5","bc14f365-cc07-23c7-36ca-8cd78a711384","b4caaa6b-db0e-84a4-b707-50878f393486"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:17:44.633Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","caseName":"chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","durationMs":139064,"endedAt":"2026-10-11T19:21:13.041Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263.json","task":{"id":"7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","snapshot":"chrome","instruction":"Can you save this webpage I'm looking at to bookmarks bar so I can come back to it later?","source":"https://www.youtube.com/watch?v=ZaZ8GcTxjXA","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://blog.eleuther.ai/rotary-embeddings/","https://jalammar.github.io/illustrated-transformer/"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"is_expected_bookmarks","result":{"type":"bookmarks"},"expected":{"type":"rule","rules":{"type":"bookmark_bar_websites_urls","urls":["https://jalammar.github.io/illustrated-transformer/"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263.json","osworld.evaluator.func":"\"is_expected_bookmarks\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-7a5a7856-13547c","osworld.evidence":"__VIEVAL_DIR__/episodes/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263/2026-10-11T19-18-54.025Z-gpt-5.5-r1","osworld.agent.steps":9,"osworld.agent.last_action":"DONE","auv.run.ids":["bff34b9b-f6d2-a64a-c731-95a461421db2","c1d3b012-c69c-0543-b106-02e17fef9c70","fdddaf0d-4383-ae71-55f1-ec9759151168","a48f780c-7411-3bed-2fc6-eddc3665a1e9","3f9ca3ae-73e8-9d6d-cb89-a74f72223eab","3d3883ae-1d66-06a5-2a1a-4f4cdb9a69ea","009a7cfd-40dc-55ce-44e0-9f5210595c80","da930dff-5400-febf-35a8-0236cc13f405"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:18:53.977Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","caseName":"chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","durationMs":506796,"endedAt":"2026-10-11T19:22:32.789Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3.json","task":{"id":"7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","snapshot":"chrome","instruction":"Can you help me clean up my computer by getting rid of all the tracking things that Amazon might have saved? I want to make sure my browsing is private and those sites don't remember me.","source":"https://support.google.com/chrome/answer/95647?hl=en&ref_topic=7438325&sjid=16867045591165135686-AP#zippy=%2Cdelete-cookies-from-a-site","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.amazon.com","https://www.amazon.com/s?k=huggingface+transformers+book"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_cookie_deleted","result":{"type":"cookie_data","dest":"Cookies"},"expected":{"type":"rule","rules":{"type":"domains","domains":[".amazon.com"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3.json","osworld.evaluator.func":"\"is_cookie_deleted\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-7b6c7e24-d63d3a","osworld.evidence":"__VIEVAL_DIR__/episodes/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3/2026-10-11T19-14-06.062Z-gpt-5.5-r1","osworld.agent.steps":50,"osworld.agent.last_action":"none","auv.run.ids":["57a2c0d9-423e-24dd-f635-555b43ba3d84","e2d76e99-b9f7-2c02-6d62-1ff6613ecc61","55203674-669b-83d1-fa93-238fbf7889d2","cd308fef-d991-ea5a-6de4-c3e391006136","89dac0d5-c8d2-01ef-7025-f59ac4d43842","e6a7841d-4c1d-cb6d-4840-12945c1fe770","e97d0a1b-3fd2-d320-368d-5ac520e0d541","ad3e4591-1f0d-f5f5-1ed4-cd87d541b1f2","2e72488b-f633-67ff-e513-cb84d1107aea","441d44c4-3249-7a1f-fcf2-2487bff9d025","5bd14cde-1f00-a08c-a969-bf93cb3a1113","6af1c981-fb82-374a-0711-6aac7f46eb3f","8166a987-c12f-d7b8-b1c4-9f827249799e","119edb19-94f4-dd00-3d5e-17fe2587bb1f","8bfc851e-5738-fc7a-1a39-4c092760c9f7","c8652f37-232d-43af-92e0-0de1c0a4ab6c","5f7f1198-6ecd-f48e-dc9b-9b6dcc7e3f7c","d5d62fd1-797f-6ff3-a470-f3854018e7b0","33ca97c3-f535-8e93-35aa-badb1348c9b1","3e7bab57-5dfa-ae0d-1a2a-ea1c31788dca","9ca90c83-6ae5-22f4-26c8-857e081154f4","15617958-0d96-e799-42b8-fcb15e54f9b9","86261013-ea72-44d6-9dd0-1cb1aa763980","869164f9-862f-4c4e-901b-4b6c522be3cf","f9dcf37c-1d21-4831-fa80-2270a3e9ca0a","29037581-1849-35f1-3605-cfd7a8f5532f","a903d08b-834e-3596-7d77-e4a3eac7603d","c3482151-efe1-c0bf-519e-f0c848af4cac","5e91cdc5-ffb7-3aec-1b99-5336fb853b91","26831d79-fff7-d48a-fb6e-692b4e4c3524","326e31af-139c-2168-9091-97e7de38a2f8","25d61c14-ba73-2b5d-44b6-04a1a7361960","b1ebb94c-eb0b-4402-1fec-0aa72e72cfb0","3143d060-26ce-cda9-332d-49cc742d90e7","c102daf6-377b-e949-6429-5d7295477408","f15b186b-a04d-e66d-b63b-8795afb74db8","43fb424c-6d7a-74cd-a4cf-8fd150c8af85","96273415-a4f4-8a60-d880-5fd1ce74bf99","5655c4bf-87e8-807c-f83a-05d31af5486f","cd88d6c4-3624-1b76-d56b-30c09db9a9df","709e8263-b3e7-bc89-9553-18de97667603","39535112-9580-c5bc-4ad6-ff28af1a87d7","aa3a4b40-5715-00ea-deed-6b77245004ea","b9c81491-dcda-d5a5-70e1-7e27c54d8d9c","f4dcfb1c-8dd9-2510-870d-7fe2e806e744","b5013a28-0b93-6928-254b-80f37e64d250","7f1da95a-3d5d-8987-a5d0-ddfce3739799","122a101a-bb0a-ced4-ff1b-94cab07d0269","980a0d22-dbad-e11b-83de-4579a41fd1cc","5e9dfc2e-319a-6549-3a3a-115e51ff3acb"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:14:05.993Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","caseName":"chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","durationMs":139640,"endedAt":"2026-10-11T19:23:32.439Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3.json","task":{"id":"2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","snapshot":"chrome","instruction":"Lately I have changed my English name to Thomas. I want to update my username. Could you help me change the username in chrome profiles to Thomas?","source":"https://superuser.com/questions/1393683/how-to-change-the-username-in-google-chrome-profiles?rq=1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"chrome_inject_js","parameters":{"js":"if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }","wait_for_load":false}},{"type":"sleep","parameters":{"seconds":1}},{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"profile_name"},"expected":{"type":"rule","rules":{"expected":"Thomas"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3.json","osworld.evaluator.func":"\"exact_match\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-2ae9ba84-004323","osworld.evidence":"__VIEVAL_DIR__/episodes/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3/2026-10-11T19-21-12.909Z-gpt-5.5-r1","osworld.agent.steps":18,"osworld.agent.last_action":"DONE","auv.run.ids":["0f9a5e98-fe68-12b2-793a-423e1d0af6b4","c5ed57fe-72e7-a39d-bd11-dfd975fe9687","e54ba116-6839-8bc2-a87a-8b1e181ad407","3435fdc4-1dbf-49f3-bab1-97094a42134c","01c95781-d9db-4bd4-5f3b-162bea21a565","40adafda-c899-8bca-9a4b-fdbcc885e214","1116f716-f17f-3531-9b69-5aeabf594cea","a55fc692-d886-6171-62dc-16d4c8178ee9","5fdfb7cb-8e2d-da4c-84eb-e3f8a421d908","fb8d74aa-d814-bdb9-0c4c-77b775cf8ef2","f42fa1c1-ea4b-cb0d-a0b2-861333a51bb2","c92e144b-0dae-c45d-66e5-29c37639b403","b28cb371-6236-e9a5-3fd3-37f74c4e6035","63b1a0c5-1094-8f8a-59b0-d6570b3dec7f","960fefc4-4d1f-7a8a-0e02-bd8507e39b4b","1ab4cb89-5259-8f7a-9bb8-758fe852f8e9","838fdb09-579e-b2f4-33f9-8bf564972040"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:21:12.799Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","caseName":"chrome/af630914-714e-4a24-a7bb-f9af687d3b91","durationMs":101422,"endedAt":"2026-10-11T19:24:14.212Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/af630914-714e-4a24-a7bb-f9af687d3b91.json","task":{"id":"af630914-714e-4a24-a7bb-f9af687d3b91","snapshot":"chrome","instruction":"My grandmother has been using the Chrome lately and told me that the font size is way too small for her poor eyesight. Could you set the default font size to the largest for her?","source":"https://www.howtogeek.com/680260/how-to-change-chromes-default-text-size/","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"check_font_size","result":{"type":"chrome_font_size"},"expected":{"type":"rule","rules":{"type":"range","min":16,"max":99999}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/af630914-714e-4a24-a7bb-f9af687d3b91.json","osworld.evaluator.func":"\"check_font_size\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-af630914-1acfcc","osworld.evidence":"__VIEVAL_DIR__/episodes/af630914-714e-4a24-a7bb-f9af687d3b91/2026-10-11T19-22-32.860Z-gpt-5.5-r1","osworld.agent.steps":12,"osworld.agent.last_action":"DONE","auv.run.ids":["0fd46f09-65e2-d5ac-b323-2fa09f5bf403","330ccc1d-da9a-25e9-2f9f-ce639826c15d","559125ba-d05b-133b-205d-c25dbf2f1136","5adb7abb-0069-4d63-0a94-7efd2d4467ff","23552df7-c02d-fd80-093d-c574ecda8aa7","f6ce2758-88b5-39fe-fc71-fe5b29574a16","80d55ca2-cf68-e325-fd44-bad3b9b09894","a427ffad-066e-8c00-c9b2-cb1242c780c4","99ae98c0-f23e-69b8-1215-73980b365cb1","e4e9df49-12a3-79f3-ed70-502a311a3029","6941b4a9-36ef-1d84-f78c-5213ad9bb3c7"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:22:32.790Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","caseName":"chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d","durationMs":127883,"endedAt":"2026-10-11T19:25:40.323Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d.json","task":{"id":"3720f614-37fd-4d04-8a6b-76f54f8c222d","snapshot":"chrome","instruction":"I am from the country of Atlantis, and my mother tongue is Xenothian. Please change the Google Chrome interface language to Xenothian using only Chrome’s built-in settings.","source":"https://superuser.com/questions/984668/change-interface-language-of-chrome-to-english","config":[],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d.json","osworld.evaluator.func":"\"infeasible\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-3720f614-a3afd3","osworld.evidence":"__VIEVAL_DIR__/episodes/3720f614-37fd-4d04-8a6b-76f54f8c222d/2026-10-11T19-23-32.504Z-gpt-5.5-r1","osworld.agent.steps":21,"osworld.agent.last_action":"FAIL","auv.run.ids":["e7233e57-7f5a-a4b4-5c78-9e35691502ae","be0ede42-f98b-757a-a60d-b241457b5eed","3318b6f2-0854-039c-19f5-63afee4bc154","76466e29-2dc3-06aa-2896-2318ea2b15ae","6e47111e-a797-59fb-2d4f-aa8a0426301e","10c08eeb-38f4-1d2c-d2c3-fa32e5871ebf","505b9407-7af7-d8df-689f-2983f4abe321","9575b7a4-d4ec-66d4-bace-f9ec47084431","72f97bf4-fe88-be60-eacd-d844be41312a","d2e73cc7-885b-293d-cd6d-f1ced0cbf263","fef469ff-6a2d-c6d5-3b34-99443c89c912","cfd8c665-6316-57b8-4db6-7e1c5bc59fe2","0e2a5f36-1084-48b0-6daf-b7153e3d6cbc","8c449646-2441-5448-2ff1-5a24185a4196","5e0906d4-ace6-4ce0-f657-b6976c986dce","d654cc67-7ab3-44a0-718c-53319dc2aaf3","7e5a2393-13f7-8415-e235-942d4cf5a8e6","ae6f46ad-8ab5-cae3-14a4-14f5fe47a0d6","da810650-34a9-b977-12b5-622e97fc3285","2af4400a-f4e6-85d9-7683-fb6f8e645662"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:23:32.440Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","caseName":"chrome/99146c54-4f37-4ab8-9327-5f3291665e1e","durationMs":104925,"endedAt":"2026-10-11T19:25:59.137Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/99146c54-4f37-4ab8-9327-5f3291665e1e.json","task":{"id":"99146c54-4f37-4ab8-9327-5f3291665e1e","snapshot":"chrome","instruction":"Please help me set Chrome to delete my browsing data automatically every time I close the browser.","source":"https://www.youtube.com/watch?v=v0kxqB7Xa6I","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"data_delete_automacally"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/99146c54-4f37-4ab8-9327-5f3291665e1e.json","osworld.evaluator.func":"\"exact_match\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-99146c54-b6912e","osworld.evidence":"__VIEVAL_DIR__/episodes/99146c54-4f37-4ab8-9327-5f3291665e1e/2026-10-11T19-24-14.269Z-gpt-5.5-r1","osworld.agent.steps":15,"osworld.agent.last_action":"DONE","auv.run.ids":["d0adaa21-d213-f9cd-7161-a8302a0cf5ae","c6864c42-24ac-2e17-702f-b3946eaa0d6e","5c4bbc08-87c7-f1a9-6086-6117ce0166d8","aa3e6e0c-fc22-d519-8af7-46af279e86ea","6b3c7baf-b993-f1eb-815a-a05511c2538a","5e0e7912-97da-0ed5-92a5-c0be7fb5a3ab","dcc6481c-3a7e-6e5b-7387-70ee4ac5dcd0","dcd453f1-035f-3fcd-57c6-739cd14cb28f","5aefc831-5ced-48e1-6816-5ff3142e4c1d","3fffcefe-eb2b-d344-8fca-b38b75038f6b","a32e2490-7f44-573d-c419-b373bf59eb3a","e6e8c5c3-5d4b-de95-1460-633ecc81d135","1965eae6-bbe8-f11d-e72e-c87c58121355","3fb41d9d-285f-3035-6ae8-dab725a3af56"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:24:14.212Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","caseName":"chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381","durationMs":348077,"endedAt":"2026-10-11T19:27:01.118Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381.json","task":{"id":"480bcfea-d68f-4aaa-a0a9-2589ef319381","snapshot":"chrome","instruction":"I do not like the design of the new 2023 chrome UI. I want to keep using the original one. Can you disable the new 2023 version chrome UI for me? ","source":"https://bugartisan.medium.com/disable-the-new-chrome-ui-round-in-2023-db168271f71e","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381.json","osworld.evaluator.func":"\"infeasible\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-480bcfea-ec301a","osworld.evidence":"__VIEVAL_DIR__/episodes/480bcfea-d68f-4aaa-a0a9-2589ef319381/2026-10-11T19-21-13.134Z-gpt-5.5-r1","osworld.agent.steps":48,"osworld.agent.last_action":"DONE","auv.run.ids":["4e6d8957-8251-96cb-e992-062c46c81d6b","38fa305a-8802-a9e9-169b-47a009226e12","108d142e-ffce-3a83-e411-c5ff506ec0d1","f32796ad-9635-7719-fef3-cf7a81d39048","92bc72c4-84a8-4a77-f4ca-65ed3cf50948","a1a87341-ae9e-f3cb-1c90-47822ce105ed","872ad29c-4724-81c0-faff-e31b11449963","66aa3fa1-4558-323d-6298-61c2dc68241b","974e428d-8a56-a5ba-f192-15bd87937f34","ea313189-e736-ad7c-3ae8-7a9e852292d3","0e251863-faba-ee72-3f76-89bc2404123b","3610e612-7191-48bc-49ac-3b5efd3a6b9a","27a10af6-5306-328d-e80f-d4b13ece9460","a1736127-7b9d-cf9d-f833-aec169a8d0eb","da52a1d4-4809-f884-940e-e4c67505f814","62a5d2ad-ccb7-e5ff-0070-3f9829c6e735","e2b8c379-fc3c-4c72-a070-1c2d5f31d764","22f1e1be-cc9a-ee20-1c90-27c161529daa","1e568e0c-cca7-f993-1d9d-8bd60146c966","fb34b88b-3fb4-8472-e4a4-07d3acca8dbb","c516d635-d440-8897-f6c1-68d4e91b55f8","ce168168-9f6a-ed51-6655-3c2e3f108b97","0bb207fb-c857-61d2-ca66-37f40c96cb4b","ef16bc20-6f82-7a70-45f3-1f1cd3dec229","a8a42aa6-58ac-6b8c-c323-04eb9a81dfe1","9bed18f7-068b-fec9-dbfb-b9ae8f9ed667","94bfb787-6ba8-c951-59fc-e42672097f98","d1335548-a570-40ea-85c7-58421601170e","58e1f5b9-4bc3-98f7-c1fa-53d47a9d6d8b","8052e573-1f86-9ed8-5178-001ffb624af2","eb864f7c-e922-a1f5-82d3-19fc3a8529c3","49cf13de-6aa8-9bcd-836f-53516bbb472d","5e0bee47-1ecc-5f63-ad6a-7c7737c93acc","dd33b4ba-d774-7a5b-664d-eaf879f3fa3f","7c1aa95e-9253-f4da-9988-544345869c58","0e998bff-129a-564d-39e4-e9590d39d772","549d8216-ae88-7ebb-bc07-9370356302da","34fe8896-7301-0fac-f4d0-932f2cdc4978","8483c9e7-1df2-a2e5-445e-2fa92ded01fb","2832ae61-46af-6a38-5d69-5455b839a603","e3a4c429-f8b1-5553-54c1-9c10117ac1dc","773bbc06-953f-bcb3-aec9-5ce329a4cbd7","b1002976-3bd3-1f43-f741-fe96e42e9ab9","16832bcc-b381-584f-fa27-ec4f23725be6","cb66200e-cb9e-1ad6-c567-42532283667c","b9d98bf1-44c6-f04e-d023-94f139ad95df","b0710018-396e-e53d-9f08-a8cc98ada2ed"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":0},"startedAt":"2026-10-11T19:21:13.041Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","caseName":"chrome/12086550-11c0-466b-b367-1d9e75b3910e","durationMs":110044,"endedAt":"2026-10-11T19:27:30.367Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/12086550-11c0-466b-b367-1d9e75b3910e.json","task":{"id":"12086550-11c0-466b-b367-1d9e75b3910e","snapshot":"chrome","instruction":"Computer, please navigate to the area in my browser settings where my passwords are stored. I want to check my login information for Etsy without revealing it just yet.","source":"https://www.quora.com/What-are-the-cool-tricks-to-use-Google-Chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_active_tab_approximate","result":{"type":"active_url_from_accessTree","goto_prefix":""},"expected":{"type":"rule","rules":{"type":"url","url":"chrome://password-manager/passwords"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/12086550-11c0-466b-b367-1d9e75b3910e.json","osworld.evaluator.func":"\"is_expected_active_tab_approximate\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-12086550-2b5033","osworld.evidence":"__VIEVAL_DIR__/episodes/12086550-11c0-466b-b367-1d9e75b3910e/2026-10-11T19-25-40.365Z-gpt-5.5-r1","osworld.agent.steps":19,"osworld.agent.last_action":"DONE","auv.run.ids":["2963c2e2-1f89-a096-1ebe-7eaf92065e46","4e0372bc-b487-e3af-7231-04d2da8201b7","22fa34c3-084f-45de-5560-8a6182c54251","84e10f86-4bf1-9d06-a72b-0c78d14324e9","4cd62fa9-a1b5-7854-b81e-19b3e89150a2","eead0f83-1000-bce6-ef6f-30c7e166bf4b","22c5358f-436a-e2db-9ca2-3c422e7c72f6","6ac4a067-f900-90ff-e614-17d42c90a8ff","7788c53a-47fe-e2af-54a0-086c340ab26e","5f696e34-76b3-37bb-807a-ceec43c99ea1","ffd1dda9-3830-22f4-b2ad-de4b4cb0451b","743ab0b6-3337-e402-a743-bce3686f508f","fb895282-2852-89a5-229a-d3fff57ed378","724f8678-dd3e-1a9f-578f-f934f07ac6e5","3d883ba2-ccc5-e127-e186-d323d17db741","6fc1bbb7-a733-8c5f-6be1-82da7da34653","c745eaff-b4ea-287f-16c0-44a94bdc9a1f","e9e1dc6d-edb7-1ca1-5718-b86a58bdb4f5"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:25:40.323Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","caseName":"chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","durationMs":104339,"endedAt":"2026-10-11T19:28:45.457Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9.json","task":{"id":"93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","snapshot":"chrome","instruction":"Could you assist me in turning off the dark mode feature in Google Chrome? I've noticed that while dark mode is great for reducing glare, it actually makes it more challenging for me to read text clearly, especially with my astigmatism.","source":"https://superuser.com/questions/1417973/how-to-disable-google-chrome-dark-mode","config":[{"type":"execute","parameters":{"command":"mkdir -p /home/user/.config/google-chrome/Default && if [ ! -f /home/user/.config/google-chrome/Default/Preferences ]; then echo '{}' > /home/user/.config/google-chrome/Default/Preferences; fi && python3 -c \"import json; p='/home/user/.config/google-chrome/Default/Preferences'; d=json.load(open(p)); d.setdefault('browser', {}).setdefault('theme', {})['color_scheme'] = 2; d['browser']['theme']['color_scheme2'] = 2; json.dump(d, open(p, 'w'))\"","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"sleep","parameters":{"seconds":1}}],"func":["match_in_list","is_expected_url_pattern_match"],"result":[{"type":"chrome_appearance_mode_ui"},{"type":"active_url_from_accessTree","goto_prefix":""}],"expected":[{"type":"rule","rules":{"expected":["light","system"]}},{"type":"rule","rules":{"expected":["^chrome://settings/appearance/?$"]}}]},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9.json","osworld.evaluator.func":"[\"match_in_list\",\"is_expected_url_pattern_match\"]","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-93eabf48-e94487","osworld.evidence":"__VIEVAL_DIR__/episodes/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9/2026-10-11T19-27-01.140Z-gpt-5.5-r1","osworld.agent.steps":17,"osworld.agent.last_action":"DONE","auv.run.ids":["1f35452b-27d1-6b80-7f53-5d6891721513","2d8de726-aae4-8cf2-84e8-dd3edd47b903","e1d3f454-7f66-a132-05c7-82f3b9326713","727c8e31-b21b-85d6-7df2-035c775e8283","b472dd48-3116-5a6c-4886-4006a53d51b3","8de8d87c-959a-9ce6-77d3-7842662db847","467175cc-5c7a-613a-2b80-e73b63041ba4","89693650-e567-c4ab-a085-d6ba55f09be7","6ab39e25-6131-4688-df50-62cbb4b8f047","66f63aa8-9a02-d15e-829d-e055a98da1f5","726dbda2-3636-d07a-6356-151100372eac","70a927cd-eeb3-26a1-e56b-2c146b261045","4cdae1d1-5a80-f069-08a4-32033d574962","d4ea467e-1bbf-cb53-8f9b-5bed00b1f87e","cf1606ed-0a39-574b-b269-3f5736837607","04f5cbce-ef97-c9e5-28ed-15995841f2ac"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:27:01.118Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","caseName":"chrome/3299584d-8f11-4457-bf4c-ce98f7600250","durationMs":70860,"endedAt":"2026-10-11T19:29:56.318Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/3299584d-8f11-4457-bf4c-ce98f7600250.json","task":{"id":"3299584d-8f11-4457-bf4c-ce98f7600250","snapshot":"chrome","instruction":"On my surface pro whenever I launch Chrome it always opens \"funbrain.com.\" I don't want this. I cleared my cache but it still happens. What should I do?","source":"https://www.reddit.com/r/techsupport/comments/12zwymy/comment/jhtri65/?utm_source=share&utm_medium=web2x&context=3","config":[{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt update -y && echo {CLIENT_PASSWORD} | sudo -S apt install jq -y","shell":true}},{"type":"execute","parameters":{"command":"cd /home/user/.config/google-chrome/Default && jq '. + {\"session\":{\"restore_on_startup\":4, \"startup_urls\":[\"http://funbrain.com/\"]}}' Preferences > temp && mv temp Preferences","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"exact_match","result":{"type":"new_startup_page"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/3299584d-8f11-4457-bf4c-ce98f7600250.json","osworld.evaluator.func":"\"exact_match\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-3299584d-e2fde8","osworld.evidence":"__VIEVAL_DIR__/episodes/3299584d-8f11-4457-bf4c-ce98f7600250/2026-10-11T19-28-45.484Z-gpt-5.5-r1","osworld.agent.steps":11,"osworld.agent.last_action":"DONE","auv.run.ids":["80909ad8-f02c-f38b-0700-fd1ac6072228","47158092-f636-85dc-28e9-84f5332fe9ad","6d36679e-5b92-fdd8-3aaa-d749490482da","0a29244d-1ceb-10d4-900e-31208c13df3e","2faa9f85-7189-1239-f4db-6556fe8286c7","43c74854-caf5-1443-4c3b-986991c229b5","9400204b-25ce-5baa-33b3-e970ec3d38d2","ead1e4a4-daf0-7246-aede-e96ca8a6417e","f6fa6a21-3e4b-3858-d879-98766277151c","6ecf3c12-fdc2-ed3f-0e94-951e0e6ad249"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":1},"startedAt":"2026-10-11T19:28:45.458Z","state":"passed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","caseName":"chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837","durationMs":241291,"endedAt":"2026-10-11T19:30:00.429Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837.json","task":{"id":"6766f2b8-8a72-417f-a9e5-56fcaa735837","snapshot":"chrome","instruction":"Could you help me unzip the downloaded extension file from /home/user/Desktop/ to /home/user/Desktop/ and configure it in Chrome's extensions?","source":"https://support.google.com/chrome/thread/205881926/it-s-possible-to-load-unpacked-extension-automatically-in-chrome?hl=en","config":[{"type":"download","parameters":{"files":[{"url":"https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache/resolve/main/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837/helloExtension.zip","path":"/home/user/Desktop/helloExtension.zip"}]}},{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt-get update -y && echo {CLIENT_PASSWORD} | sudo -S apt-get install unzip -y && unzip /home/user/Desktop/helloExtension.zip -d /home/user/Desktop/ && rm /home/user/Desktop/helloExtension.zip","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_in_list","result":{"type":"find_unpacked_extension_path"},"expected":{"type":"rule","rules":{"expected":"/home/user/Desktop/helloExtension"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837.json","osworld.evaluator.func":"\"is_in_list\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1,"osworld.pod":"osworld-6766f2b8-e37b73","osworld.evidence":"__VIEVAL_DIR__/episodes/6766f2b8-8a72-417f-a9e5-56fcaa735837/2026-10-11T19-25-59.183Z-gpt-5.5-r1","osworld.agent.steps":50,"osworld.agent.last_action":"none","auv.run.ids":["2114cb1d-ee83-644a-fa5c-d3d7aefc82cd","2fbf9799-eadb-cad8-2146-2e7f28fba21a","0248689b-acf0-cba5-47d3-10e6ba8a5fff","caddbc1c-8001-fea4-e957-db89ea653a9b","8dda2b9d-5fc7-ef75-a813-91a3fd04dad3","23883639-7974-f8d9-bd96-2a269054e811","22fe8694-af91-cb1f-6c7b-266179602a1d","b9a29a0f-8a64-ac79-6e61-48a8eb8b177e","dd2f09f1-b129-5dec-5a93-cf9516984d11","8c73479b-76cd-06a8-b84a-93f689a6d079","069df994-c480-d271-9bc2-35ec174f7970","0db0fc5d-574d-2e86-516e-0e5fff2e3e71","5d0d54f7-981a-5ffb-e011-8cb3de485b6a","9f3cc401-eb84-0a15-f608-793c6e6c3bc6","a2ed395b-408a-210c-e308-6ee183c3874f","a8345888-d789-baa5-1f78-2b866d20f2b0","fd92e601-364d-96a5-f762-ae44bb800d74","fd660c01-3db4-7c6e-f1c7-cdbc13703113","ab56be1b-a8f9-23de-e1f0-24aa05c931c5","e630d394-ecc8-0497-9af0-147b2255a266","1fb3d0d8-aa84-b974-c1f5-ab3818bdf1f2","58bfb27a-8c03-eeb8-61f5-da563ff337a6","6c097d75-9da0-df17-1163-1b0690ab5725","3d44b019-49bb-9b0d-5b66-3d086d66a7f4","02d4e409-88fc-2c4b-c31b-607c72b0803c","b707b42c-dc55-ecf7-1da7-d81e2f44560d","fc6655d6-40cf-69bb-94f0-0a168c08f324","c70c2156-1a66-a4bf-9bbb-02f46fbef776","7ca60128-94c4-b2f3-17fe-3deb06ca935a","2ed12090-81db-e161-4311-3cd4016a34f1","22e70f25-6f1e-12c4-2a3d-628329358291","7b58074b-8d71-22cd-bdae-09121e6778a6","f474b9a2-6b75-4649-4bc7-9a8dfa2a7baf","a5e39e0b-ed11-5330-e91e-490675ef71fd","4d722f98-c174-fe01-58ed-9847def5f5b0","ba583be9-6778-3c34-897e-8d2035aa3324","48abaa94-7815-b42a-e76a-eb7c3d57bd69","17c2fcf5-32c0-97fc-8a33-67846c6eb805","9a3d4065-bbe5-f8af-cbe8-3f5d444f66cc","b5bcfded-110c-c06f-5852-d0d97caf197e","e072df72-d4bf-70e9-c833-b27513ed770e","17e30947-fdbe-1a39-3a63-ea09cd2f8ef4","a6df051e-04cd-b402-23f6-468b9e9ba8a0","0073af34-176e-287d-0f2a-1cc099e41ecf","48722b90-33cf-ccb8-94ea-b0d9cb60e1f2","404335ea-d710-aebe-f159-fecd48db3f67","af0516e4-5fa7-795b-9920-2db84f18367a","4ecfbcd6-c1a1-6395-3901-9592c689d71e","e83badb8-d039-08c1-61f9-2a6ab08be8d0"],"auv.linux_binary.sha256":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":0},"startedAt":"2026-10-11T19:25:59.138Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","caseName":"chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03","durationMs":232182,"endedAt":"2026-10-11T19:31:22.549Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json","task":{"id":"ae78f875-5b98-4907-bbb5-9c737fc68c03","snapshot":"chrome","instruction":"Could you please change the number of search results displayed on one page to 50? I find that having more results visible at once significantly enhances my research efficiency, as it reduces the need to constantly click through multiple pages. ","source":"https://support.google.com/chrome/thread/219988391/increase-search-results-per-page?hl=en","config":[],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json","osworld.evaluator.func":"\"infeasible\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":0},"startedAt":"2026-10-11T19:27:30.367Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","caseName":"chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626","durationMs":82433,"endedAt":"2026-10-11T19:31:22.862Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json","task":{"id":"9656a811-9b5b-4ddf-99c7-5117bcef0626","snapshot":"chrome","instruction":"I want Chrome to warn me whenever I visit a potentially harmful or unsafe website. Can you enable this safety feature?","source":"https://www.quora.com/How-do-I-set-the-security-settings-for-the-Google-Chrome-browser-for-the-best-security#:~:text=Enable%20Safe%20Browsing:%20Chrome%20has%20a%20built%2Din,Security%20%3E%20Security%20%3E%20Enable%20Safe%20Browsing.","config":[{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt update -y && echo {CLIENT_PASSWORD} | sudo -S apt install jq -y","shell":true}},{"type":"execute","parameters":{"command":"mkdir -p /home/user/.config/google-chrome/Default && if [ ! -f /home/user/.config/google-chrome/Default/Preferences ]; then echo '{}' > /home/user/.config/google-chrome/Default/Preferences; fi && cd /home/user/.config/google-chrome/Default && jq '. + {\"safebrowsing\":{\"enabled\":false,\"enhanced\":false}}' Preferences > temp && mv temp Preferences","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_safe_browsing"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json","osworld.evaluator.func":"\"exact_match\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":0},"startedAt":"2026-10-11T19:30:00.429Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","caseName":"chrome/030eeff7-b492-4218-b312-701ec99ee0cc","durationMs":88174,"endedAt":"2026-10-11T19:31:24.492Z","experimentId":"matrix-run-model-gpt-5.5-repeat-1","input":{"domain":"chrome","file":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json","task":{"id":"030eeff7-b492-4218-b312-701ec99ee0cc","snapshot":"chrome","instruction":"Can you enable the 'Do Not Track' feature in Chrome to enhance my online privacy?","source":"https://www.surreycc.gov.uk/website/cookies/do-not-track","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"chrome_inject_js","parameters":{"js":"if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }","wait_for_load":false}},{"type":"sleep","parameters":{"seconds":1}},{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_do_not_track"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"metrics":{"osworld.domain":"chrome","osworld.task.file":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json","osworld.evaluator.func":"\"exact_match\"","osworld.model":"gpt-5.5","osworld.model.api":"responses","osworld.repeat":1},"projectName":"osworld-kubernetes","retryCount":0,"runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"scores":{"osworld.success":0},"startedAt":"2026-10-11T19:29:56.318Z","state":"failed","taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","workspaceId":"default-workspace","scoreDetails":{},"unscored":{}}
diff --git a/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/events.jsonl b/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/events.jsonl
index a05fac9..2a112ce 100644
--- a/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/events.jsonl
+++ b/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/events.jsonl
@@ -1,257 +1,257 @@
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"totalTasks":1},"event":"run.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":1,"time":"2026-10-11T19:14:05.992Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"displayName":"osworld [model=gpt-5.5, repeat=1]"},"event":"task.queued","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":2,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.992Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"taskCount":1},"event":"project.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":3,"time":"2026-10-11T19:14:05.992Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{},"event":"task.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":4,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"index":0,"input":{"domain":"chrome","file":"tasks/chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3.json","task":{"id":"7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","snapshot":"chrome","instruction":"Can you help me clean up my computer by getting rid of all the tracking things that Amazon might have saved? I want to make sure my browsing is private and those sites don't remember me.","source":"https://support.google.com/chrome/answer/95647?hl=en&ref_topic=7438325&sjid=16867045591165135686-AP#zippy=%2Cdelete-cookies-from-a-site","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.amazon.com","https://www.amazon.com/s?k=huggingface+transformers+book"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_cookie_deleted","result":{"type":"cookie_data","dest":"Cookies"},"expected":{"type":"rule","rules":{"type":"domains","domains":[".amazon.com"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":5,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":6,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.task.file","value":"tasks/chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":7,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.evaluator.func","value":"\"is_cookie_deleted\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":8,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":9,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":10,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":11,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"index":1,"input":{"domain":"chrome","file":"tasks/chrome/06fe7178-4491-4589-810f-2e2bc9502122.json","task":{"id":"06fe7178-4491-4589-810f-2e2bc9502122","snapshot":"chrome","instruction":"Can you make my computer bring back the last tab I shut down?","source":"https://www.wikihow.com/Switch-Tabs-in-Chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.lonelyplanet.com","https://www.airbnb.com","https://www.tripadvisor.com"]}},{"type":"chrome_close_tabs","parameters":{"urls_to_close":["https://www.tripadvisor.com"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_tabs","result":{"type":"open_tabs_info"},"expected":{"type":"rule","rules":{"type":"url","urls":["https://www.lonelyplanet.com","https://www.airbnb.com","https://www.tripadvisor.com"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/06fe7178-4491-4589-810f-2e2bc9502122","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":12,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":13,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.task.file","value":"tasks/chrome/06fe7178-4491-4589-810f-2e2bc9502122.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":14,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.evaluator.func","value":"\"is_expected_tabs\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":15,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":16,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":17,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":18,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"index":2,"input":{"domain":"chrome","file":"tasks/chrome/e1e75309-3ddb-4d09-92ec-de869c928143.json","task":{"id":"e1e75309-3ddb-4d09-92ec-de869c928143","snapshot":"chrome","instruction":"Computer, can you turn the webpage I'm looking at into a PDF file, save it to my Desktop with the default filename and set the margins to none?","source":"https://in5stepstutorials.com/google-chrome/save-web-page-as-pdf-in-chrome.php","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://lilianweng.github.io/posts/2023-06-23-agent/"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"compare_pdfs","result":{"type":"vm_file","path":"/home/user/Desktop/LLM Powered Autonomous Agents _ Lil'Log.pdf","dest":"LLM Powered Autonomous Agents _ Lil'Log.pdf"},"expected":{"type":"pdf_from_url","path":"https://lilianweng.github.io/posts/2023-06-23-agent/","dest":"LLM Powered Autonomous Agents _ Lil'Log_gold.pdf"}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/e1e75309-3ddb-4d09-92ec-de869c928143","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":19,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":20,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.task.file","value":"tasks/chrome/e1e75309-3ddb-4d09-92ec-de869c928143.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":21,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.evaluator.func","value":"\"compare_pdfs\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":22,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":23,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":24,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":25,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.pod","value":"osworld-06fe7178-624029"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":26,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.900Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/06fe7178-4491-4589-810f-2e2bc9502122/2026-10-11T19-14-06.064Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":27,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.agent.steps","value":7},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":28,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":29,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"auv.run.ids","value":["8ce2b33d-31cd-47b3-a869-b9cb4eab1d5d","7f8742c0-19c1-cae3-32ea-db1c53169b71","2decd904-75d7-ea71-11e7-e199b601a5d3","a505c326-bc53-cfd6-3ab2-b7b5ff4d8b07","d233deae-0dff-a34e-3e12-c1ef6e845d19","ef8e83be-bb57-3c0c-d366-3c2add975efe"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":30,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":31,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"kind":"exact","score":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":32,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":1,"name":"chrome/06fe7178-4491-4589-810f-2e2bc9502122","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":33,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.903Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"index":3,"input":{"domain":"chrome","file":"tasks/chrome/35253b65-1c19-4304-8aa4-6884b8218fc0.json","task":{"id":"35253b65-1c19-4304-8aa4-6884b8218fc0","snapshot":"chrome","instruction":"Hey, I need a quick way back to this site. Could you whip up a shortcut on my desktop for me using Chrome's built-in feature?","source":"https://www.laptopmag.com/articles/how-to-create-desktop-shortcuts-for-web-pages-using-chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.mathsisfun.com/games/2048.html"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_shortcut_on_desktop","result":{"type":"shortcuts_on_desktop"},"expected":{"type":"rule","rules":{"type":"name","name":"Play Puzzle Game 2048"}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/35253b65-1c19-4304-8aa4-6884b8218fc0","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":34,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.903Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":35,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.task.file","value":"tasks/chrome/35253b65-1c19-4304-8aa4-6884b8218fc0.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":36,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.evaluator.func","value":"\"is_shortcut_on_desktop\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":37,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":38,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":39,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":40,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.pod","value":"osworld-e1e75309-25dee7"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":41,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.631Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/e1e75309-3ddb-4d09-92ec-de869c928143/2026-10-11T19-14-06.065Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":42,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.agent.steps","value":24},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":43,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":44,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"auv.run.ids","value":["bbbfc80f-5c9d-3466-40dc-6ffc9fe3cec8","5efa0a92-631d-14c7-628e-1412594c3903","8c93afff-a5f8-3db9-62a1-cf6fde76a8fd","8a68cb33-b136-2bb7-5d5e-499da9c3b67c","b08b73a4-0a02-ab69-f999-6f52f47cdfbf","756c05ed-93ae-5669-7529-fa4381d834e0","bdadc082-0d52-98ac-9c4f-f36d90236668","b2dff437-c5c7-b54a-ffc9-702b714879c1","e3f53fad-c387-6078-08a2-76fab5a85c5e","bdd06d97-0078-aecb-67da-a0a52e6cd5aa","5a18dfc5-c050-04c6-2821-06a37db8440e","f6d8b193-3398-cf53-bc7f-6c21e3c3a4a0","d285b518-9ff4-dfd8-a3a7-b356697ffdb0","459b6908-d168-dd98-e2b3-b774d2f28c54","a3aa3e03-6967-4a4f-c63f-aa6f0407cddf","77cea753-b470-c14c-afd4-0fc1ee81ef03","992eafcd-9db8-0da4-c8a6-eabcaed0cf4b","fcc6b07e-1a7e-ccde-4e56-69b55dd81e73","bb542d39-1684-8e06-47b3-bb05e0535e7c","2be8f32c-3a08-31c4-8723-e2b808d3b711","9be7e44e-3013-0e5f-7201-36cb1ebd4a7b","e6b3123e-a0f3-8194-726d-6793a789bcc9","729d19e7-ed40-e269-ef3c-f658827b3e7e"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":45,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":46,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"kind":"exact","score":0.9380815331090999},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":47,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"errorMessage":"expected 0.9380815331090999 to be 1 // Object.is equality","index":2,"name":"chrome/e1e75309-3ddb-4d09-92ec-de869c928143","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":48,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.633Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"index":4,"input":{"domain":"chrome","file":"tasks/chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca.json","task":{"id":"2ad9387a-65d8-4e33-ad5b-7580065a27ca","snapshot":"chrome","instruction":"Can you make a new folder for me on the bookmarks bar in my internet browser? Let's call it 'Favorites.'","source":"https://www.youtube.com/watch?v=IN-Eq_UripQ","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"is_expected_bookmarks","result":{"type":"bookmarks"},"expected":{"type":"rule","rules":{"type":"bookmark_bar_folders_names","names":["Favorites"]}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":49,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.633Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":50,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.task.file","value":"tasks/chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":51,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.evaluator.func","value":"\"is_expected_bookmarks\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":52,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":53,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":54,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":55,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.pod","value":"osworld-35253b65-de1eaa"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":56,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.976Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/35253b65-1c19-4304-8aa4-6884b8218fc0/2026-10-11T19-16-32.960Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":57,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.976Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.agent.steps","value":12},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":58,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.976Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":59,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.976Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"auv.run.ids","value":["562d89bb-7abb-c9b8-89e9-f5b7f6476fdc","c221b337-3709-b702-0d3c-38916e614403","33f22dc8-6f24-ea56-6b90-3cd8190d52a2","99797196-b5ea-b987-5276-bd0b3a19270c","a0fd65a9-32bd-b0e5-b8a3-a1c5458b387f","ef0fb452-785c-08a1-4667-78129d9c2e02","4dedcdb9-afc6-651e-f42a-06de2caefcd4","34f4119d-4df4-3ca7-cf9d-f253f90c6f61","27704877-1a99-0f93-8810-03d0f5de52f9","d30dc0a6-9abc-02ce-0a47-7396c30f8254"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":60,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.976Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":61,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.977Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":62,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.977Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"index":3,"name":"chrome/35253b65-1c19-4304-8aa4-6884b8218fc0","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":63,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.977Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"index":5,"input":{"domain":"chrome","file":"tasks/chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263.json","task":{"id":"7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","snapshot":"chrome","instruction":"Can you save this webpage I'm looking at to bookmarks bar so I can come back to it later?","source":"https://www.youtube.com/watch?v=ZaZ8GcTxjXA","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://blog.eleuther.ai/rotary-embeddings/","https://jalammar.github.io/illustrated-transformer/"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"is_expected_bookmarks","result":{"type":"bookmarks"},"expected":{"type":"rule","rules":{"type":"bookmark_bar_websites_urls","urls":["https://jalammar.github.io/illustrated-transformer/"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":64,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.977Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":65,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.task.file","value":"tasks/chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":66,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.evaluator.func","value":"\"is_expected_bookmarks\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":67,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":68,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":69,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":70,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.pod","value":"osworld-2ad9387a-03611b"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":71,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.797Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/2ad9387a-65d8-4e33-ad5b-7580065a27ca/2026-10-11T19-17-44.670Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":72,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.797Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.agent.steps","value":26},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":73,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.797Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":74,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.797Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"auv.run.ids","value":["63cc9314-526b-5d8c-c56c-6474f90ffcd3","74a785ce-182e-3dbd-63a5-29b61d79955c","a8b589ea-ec19-7b31-f717-5364234055a4","31462f1a-9636-0e8d-e59e-d3cf79817292","383a51e8-1059-33c1-318b-72b76118e6d5","9948558b-5941-2d57-fb21-40850c5f61f3","20f34f68-b134-e47b-5003-c4ece7e66b05","9e6b2088-1b0f-a47b-d18d-4792a1fda16b","e4e6c191-f988-e93e-be4d-4ca0e999849c","434aa8a4-779c-3f27-ac8e-097ce678d176","5390f775-93f1-913d-acf1-6d1b11bf790d","ffcb9b6c-576e-5849-5da1-c2cfc0f609a9","c1688640-a8db-816f-572b-1318e7d3e090","6a272e12-1ca3-125b-1929-71ad6f6b5edc","b6005878-8c1c-c18a-4e40-b22ad1e61345","e4bcdeb3-f807-de9e-a059-03a7021ae82c","7a6e804e-2635-6d32-9bb1-7d9b4caf24f1","391df434-ddc6-6483-8027-7a1b800ad797","73ff6def-2189-95a7-246f-662107a93d50","67e7f2ee-32ae-ba60-eb3a-410f16865231","7f147858-f5e9-9e5c-a165-950fdd0d78e2","f59fa940-6c43-b0c2-d6e7-bd90de15b13e","caf01e5f-724c-43dc-6bbb-e88a07f443f5","bc14f365-cc07-23c7-36ca-8cd78a711384","b4caaa6b-db0e-84a4-b707-50878f393486"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":75,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.798Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":76,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.798Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":77,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.798Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"index":4,"name":"chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":78,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.798Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"index":6,"input":{"domain":"chrome","file":"tasks/chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3.json","task":{"id":"2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","snapshot":"chrome","instruction":"Lately I have changed my English name to Thomas. I want to update my username. Could you help me change the username in chrome profiles to Thomas?","source":"https://superuser.com/questions/1393683/how-to-change-the-username-in-google-chrome-profiles?rq=1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"chrome_inject_js","parameters":{"js":"if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }","wait_for_load":false}},{"type":"sleep","parameters":{"seconds":1}},{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"profile_name"},"expected":{"type":"rule","rules":{"expected":"Thomas"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":79,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.799Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":80,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.task.file","value":"tasks/chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":81,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":82,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":83,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":84,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":85,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.pod","value":"osworld-7a5a7856-13547c"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":86,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263/2026-10-11T19-18-54.025Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":87,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.agent.steps","value":9},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":88,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":89,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"auv.run.ids","value":["bff34b9b-f6d2-a64a-c731-95a461421db2","c1d3b012-c69c-0543-b106-02e17fef9c70","fdddaf0d-4383-ae71-55f1-ec9759151168","a48f780c-7411-3bed-2fc6-eddc3665a1e9","3f9ca3ae-73e8-9d6d-cb89-a74f72223eab","3d3883ae-1d66-06a5-2a1a-4f4cdb9a69ea","009a7cfd-40dc-55ce-44e0-9f5210595c80","da930dff-5400-febf-35a8-0236cc13f405"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":90,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":91,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":92,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"index":5,"name":"chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":93,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"index":7,"input":{"domain":"chrome","file":"tasks/chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381.json","task":{"id":"480bcfea-d68f-4aaa-a0a9-2589ef319381","snapshot":"chrome","instruction":"I do not like the design of the new 2023 chrome UI. I want to keep using the original one. Can you disable the new 2023 version chrome UI for me? ","source":"https://bugartisan.medium.com/disable-the-new-chrome-ui-round-in-2023-db168271f71e","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":94,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":95,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.task.file","value":"tasks/chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":96,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.evaluator.func","value":"\"infeasible\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":97,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":98,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":99,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":100,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.pod","value":"osworld-7b6c7e24-d63d3a"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":101,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.786Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3/2026-10-11T19-14-06.062Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":102,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":103,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.agent.last_action","value":"none"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":104,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"auv.run.ids","value":["57a2c0d9-423e-24dd-f635-555b43ba3d84","e2d76e99-b9f7-2c02-6d62-1ff6613ecc61","55203674-669b-83d1-fa93-238fbf7889d2","cd308fef-d991-ea5a-6de4-c3e391006136","89dac0d5-c8d2-01ef-7025-f59ac4d43842","e6a7841d-4c1d-cb6d-4840-12945c1fe770","e97d0a1b-3fd2-d320-368d-5ac520e0d541","ad3e4591-1f0d-f5f5-1ed4-cd87d541b1f2","2e72488b-f633-67ff-e513-cb84d1107aea","441d44c4-3249-7a1f-fcf2-2487bff9d025","5bd14cde-1f00-a08c-a969-bf93cb3a1113","6af1c981-fb82-374a-0711-6aac7f46eb3f","8166a987-c12f-d7b8-b1c4-9f827249799e","119edb19-94f4-dd00-3d5e-17fe2587bb1f","8bfc851e-5738-fc7a-1a39-4c092760c9f7","c8652f37-232d-43af-92e0-0de1c0a4ab6c","5f7f1198-6ecd-f48e-dc9b-9b6dcc7e3f7c","d5d62fd1-797f-6ff3-a470-f3854018e7b0","33ca97c3-f535-8e93-35aa-badb1348c9b1","3e7bab57-5dfa-ae0d-1a2a-ea1c31788dca","9ca90c83-6ae5-22f4-26c8-857e081154f4","15617958-0d96-e799-42b8-fcb15e54f9b9","86261013-ea72-44d6-9dd0-1cb1aa763980","869164f9-862f-4c4e-901b-4b6c522be3cf","f9dcf37c-1d21-4831-fa80-2270a3e9ca0a","29037581-1849-35f1-3605-cfd7a8f5532f","a903d08b-834e-3596-7d77-e4a3eac7603d","c3482151-efe1-c0bf-519e-f0c848af4cac","5e91cdc5-ffb7-3aec-1b99-5336fb853b91","26831d79-fff7-d48a-fb6e-692b4e4c3524","326e31af-139c-2168-9091-97e7de38a2f8","25d61c14-ba73-2b5d-44b6-04a1a7361960","b1ebb94c-eb0b-4402-1fec-0aa72e72cfb0","3143d060-26ce-cda9-332d-49cc742d90e7","c102daf6-377b-e949-6429-5d7295477408","f15b186b-a04d-e66d-b63b-8795afb74db8","43fb424c-6d7a-74cd-a4cf-8fd150c8af85","96273415-a4f4-8a60-d880-5fd1ce74bf99","5655c4bf-87e8-807c-f83a-05d31af5486f","cd88d6c4-3624-1b76-d56b-30c09db9a9df","709e8263-b3e7-bc89-9553-18de97667603","39535112-9580-c5bc-4ad6-ff28af1a87d7","aa3a4b40-5715-00ea-deed-6b77245004ea","b9c81491-dcda-d5a5-70e1-7e27c54d8d9c","f4dcfb1c-8dd9-2510-870d-7fe2e806e744","b5013a28-0b93-6928-254b-80f37e64d250","7f1da95a-3d5d-8987-a5d0-ddfce3739799","122a101a-bb0a-ced4-ff1b-94cab07d0269","980a0d22-dbad-e11b-83de-4579a41fd1cc","5e9dfc2e-319a-6549-3a3a-115e51ff3acb"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":105,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":106,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":107,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"index":0,"name":"chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":108,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.789Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"index":8,"input":{"domain":"chrome","file":"tasks/chrome/af630914-714e-4a24-a7bb-f9af687d3b91.json","task":{"id":"af630914-714e-4a24-a7bb-f9af687d3b91","snapshot":"chrome","instruction":"My grandmother has been using the Chrome lately and told me that the font size is way too small for her poor eyesight. Could you set the default font size to the largest for her?","source":"https://www.howtogeek.com/680260/how-to-change-chromes-default-text-size/","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"check_font_size","result":{"type":"chrome_font_size"},"expected":{"type":"rule","rules":{"type":"range","min":16,"max":99999}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/af630914-714e-4a24-a7bb-f9af687d3b91","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":109,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.790Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":110,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.task.file","value":"tasks/chrome/af630914-714e-4a24-a7bb-f9af687d3b91.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":111,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.evaluator.func","value":"\"check_font_size\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":112,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":113,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":114,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":115,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.pod","value":"osworld-2ae9ba84-004323"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":116,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3/2026-10-11T19-21-12.909Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":117,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.agent.steps","value":18},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":118,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":119,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"auv.run.ids","value":["0f9a5e98-fe68-12b2-793a-423e1d0af6b4","c5ed57fe-72e7-a39d-bd11-dfd975fe9687","e54ba116-6839-8bc2-a87a-8b1e181ad407","3435fdc4-1dbf-49f3-bab1-97094a42134c","01c95781-d9db-4bd4-5f3b-162bea21a565","40adafda-c899-8bca-9a4b-fdbcc885e214","1116f716-f17f-3531-9b69-5aeabf594cea","a55fc692-d886-6171-62dc-16d4c8178ee9","5fdfb7cb-8e2d-da4c-84eb-e3f8a421d908","fb8d74aa-d814-bdb9-0c4c-77b775cf8ef2","f42fa1c1-ea4b-cb0d-a0b2-861333a51bb2","c92e144b-0dae-c45d-66e5-29c37639b403","b28cb371-6236-e9a5-3fd3-37f74c4e6035","63b1a0c5-1094-8f8a-59b0-d6570b3dec7f","960fefc4-4d1f-7a8a-0e02-bd8507e39b4b","1ab4cb89-5259-8f7a-9bb8-758fe852f8e9","838fdb09-579e-b2f4-33f9-8bf564972040"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":120,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":121,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":122,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"index":6,"name":"chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":123,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.439Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"index":9,"input":{"domain":"chrome","file":"tasks/chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d.json","task":{"id":"3720f614-37fd-4d04-8a6b-76f54f8c222d","snapshot":"chrome","instruction":"I am from the country of Atlantis, and my mother tongue is Xenothian. Please change the Google Chrome interface language to Xenothian using only Chrome’s built-in settings.","source":"https://superuser.com/questions/984668/change-interface-language-of-chrome-to-english","config":[],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":124,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.440Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":125,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.task.file","value":"tasks/chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":126,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.evaluator.func","value":"\"infeasible\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":127,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":128,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":129,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":130,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.pod","value":"osworld-af630914-1acfcc"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":131,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/af630914-714e-4a24-a7bb-f9af687d3b91/2026-10-11T19-22-32.860Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":132,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.agent.steps","value":12},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":133,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":134,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"auv.run.ids","value":["0fd46f09-65e2-d5ac-b323-2fa09f5bf403","330ccc1d-da9a-25e9-2f9f-ce639826c15d","559125ba-d05b-133b-205d-c25dbf2f1136","5adb7abb-0069-4d63-0a94-7efd2d4467ff","23552df7-c02d-fd80-093d-c574ecda8aa7","f6ce2758-88b5-39fe-fc71-fe5b29574a16","80d55ca2-cf68-e325-fd44-bad3b9b09894","a427ffad-066e-8c00-c9b2-cb1242c780c4","99ae98c0-f23e-69b8-1215-73980b365cb1","e4e9df49-12a3-79f3-ed70-502a311a3029","6941b4a9-36ef-1d84-f78c-5213ad9bb3c7"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":135,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":136,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":137,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"index":8,"name":"chrome/af630914-714e-4a24-a7bb-f9af687d3b91","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":138,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"index":10,"input":{"domain":"chrome","file":"tasks/chrome/99146c54-4f37-4ab8-9327-5f3291665e1e.json","task":{"id":"99146c54-4f37-4ab8-9327-5f3291665e1e","snapshot":"chrome","instruction":"Please help me set Chrome to delete my browsing data automatically every time I close the browser.","source":"https://www.youtube.com/watch?v=v0kxqB7Xa6I","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"data_delete_automacally"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/99146c54-4f37-4ab8-9327-5f3291665e1e","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":139,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":140,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.task.file","value":"tasks/chrome/99146c54-4f37-4ab8-9327-5f3291665e1e.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":141,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":142,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.213Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":143,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.213Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":144,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.213Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":145,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.213Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.pod","value":"osworld-3720f614-a3afd3"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":146,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/3720f614-37fd-4d04-8a6b-76f54f8c222d/2026-10-11T19-23-32.504Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":147,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.agent.steps","value":21},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":148,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.agent.last_action","value":"FAIL"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":149,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"auv.run.ids","value":["e7233e57-7f5a-a4b4-5c78-9e35691502ae","be0ede42-f98b-757a-a60d-b241457b5eed","3318b6f2-0854-039c-19f5-63afee4bc154","76466e29-2dc3-06aa-2896-2318ea2b15ae","6e47111e-a797-59fb-2d4f-aa8a0426301e","10c08eeb-38f4-1d2c-d2c3-fa32e5871ebf","505b9407-7af7-d8df-689f-2983f4abe321","9575b7a4-d4ec-66d4-bace-f9ec47084431","72f97bf4-fe88-be60-eacd-d844be41312a","d2e73cc7-885b-293d-cd6d-f1ced0cbf263","fef469ff-6a2d-c6d5-3b34-99443c89c912","cfd8c665-6316-57b8-4db6-7e1c5bc59fe2","0e2a5f36-1084-48b0-6daf-b7153e3d6cbc","8c449646-2441-5448-2ff1-5a24185a4196","5e0906d4-ace6-4ce0-f657-b6976c986dce","d654cc67-7ab3-44a0-718c-53319dc2aaf3","7e5a2393-13f7-8415-e235-942d4cf5a8e6","ae6f46ad-8ab5-cae3-14a4-14f5fe47a0d6","da810650-34a9-b977-12b5-622e97fc3285","2af4400a-f4e6-85d9-7683-fb6f8e645662"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":150,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":151,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":152,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"index":9,"name":"chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":153,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"index":11,"input":{"domain":"chrome","file":"tasks/chrome/12086550-11c0-466b-b367-1d9e75b3910e.json","task":{"id":"12086550-11c0-466b-b367-1d9e75b3910e","snapshot":"chrome","instruction":"Computer, please navigate to the area in my browser settings where my passwords are stored. I want to check my login information for Etsy without revealing it just yet.","source":"https://www.quora.com/What-are-the-cool-tricks-to-use-Google-Chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_active_tab_approximate","result":{"type":"active_url_from_accessTree","goto_prefix":""},"expected":{"type":"rule","rules":{"type":"url","url":"chrome://password-manager/passwords"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/12086550-11c0-466b-b367-1d9e75b3910e","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":154,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":155,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.task.file","value":"tasks/chrome/12086550-11c0-466b-b367-1d9e75b3910e.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":156,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.evaluator.func","value":"\"is_expected_active_tab_approximate\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":157,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":158,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":159,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":160,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.pod","value":"osworld-99146c54-b6912e"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":161,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/99146c54-4f37-4ab8-9327-5f3291665e1e/2026-10-11T19-24-14.269Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":162,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.agent.steps","value":15},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":163,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":164,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"auv.run.ids","value":["d0adaa21-d213-f9cd-7161-a8302a0cf5ae","c6864c42-24ac-2e17-702f-b3946eaa0d6e","5c4bbc08-87c7-f1a9-6086-6117ce0166d8","aa3e6e0c-fc22-d519-8af7-46af279e86ea","6b3c7baf-b993-f1eb-815a-a05511c2538a","5e0e7912-97da-0ed5-92a5-c0be7fb5a3ab","dcc6481c-3a7e-6e5b-7387-70ee4ac5dcd0","dcd453f1-035f-3fcd-57c6-739cd14cb28f","5aefc831-5ced-48e1-6816-5ff3142e4c1d","3fffcefe-eb2b-d344-8fca-b38b75038f6b","a32e2490-7f44-573d-c419-b373bf59eb3a","e6e8c5c3-5d4b-de95-1460-633ecc81d135","1965eae6-bbe8-f11d-e72e-c87c58121355","3fb41d9d-285f-3035-6ae8-dab725a3af56"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":165,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":166,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":167,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"index":10,"name":"chrome/99146c54-4f37-4ab8-9327-5f3291665e1e","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":168,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"index":12,"input":{"domain":"chrome","file":"tasks/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837.json","task":{"id":"6766f2b8-8a72-417f-a9e5-56fcaa735837","snapshot":"chrome","instruction":"Could you help me unzip the downloaded extension file from /home/user/Desktop/ to /home/user/Desktop/ and configure it in Chrome's extensions?","source":"https://support.google.com/chrome/thread/205881926/it-s-possible-to-load-unpacked-extension-automatically-in-chrome?hl=en","config":[{"type":"download","parameters":{"files":[{"url":"https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache/resolve/main/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837/helloExtension.zip","path":"/home/user/Desktop/helloExtension.zip"}]}},{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt-get update -y && echo {CLIENT_PASSWORD} | sudo -S apt-get install unzip -y && unzip /home/user/Desktop/helloExtension.zip -d /home/user/Desktop/ && rm /home/user/Desktop/helloExtension.zip","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_in_list","result":{"type":"find_unpacked_extension_path"},"expected":{"type":"rule","rules":{"expected":"/home/user/Desktop/helloExtension"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":169,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":170,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.task.file","value":"tasks/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":171,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.evaluator.func","value":"\"is_in_list\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":172,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":173,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":174,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":175,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.pod","value":"osworld-480bcfea-ec301a"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":176,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/480bcfea-d68f-4aaa-a0a9-2589ef319381/2026-10-11T19-21-13.134Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":177,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.agent.steps","value":48},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":178,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":179,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"auv.run.ids","value":["4e6d8957-8251-96cb-e992-062c46c81d6b","38fa305a-8802-a9e9-169b-47a009226e12","108d142e-ffce-3a83-e411-c5ff506ec0d1","f32796ad-9635-7719-fef3-cf7a81d39048","92bc72c4-84a8-4a77-f4ca-65ed3cf50948","a1a87341-ae9e-f3cb-1c90-47822ce105ed","872ad29c-4724-81c0-faff-e31b11449963","66aa3fa1-4558-323d-6298-61c2dc68241b","974e428d-8a56-a5ba-f192-15bd87937f34","ea313189-e736-ad7c-3ae8-7a9e852292d3","0e251863-faba-ee72-3f76-89bc2404123b","3610e612-7191-48bc-49ac-3b5efd3a6b9a","27a10af6-5306-328d-e80f-d4b13ece9460","a1736127-7b9d-cf9d-f833-aec169a8d0eb","da52a1d4-4809-f884-940e-e4c67505f814","62a5d2ad-ccb7-e5ff-0070-3f9829c6e735","e2b8c379-fc3c-4c72-a070-1c2d5f31d764","22f1e1be-cc9a-ee20-1c90-27c161529daa","1e568e0c-cca7-f993-1d9d-8bd60146c966","fb34b88b-3fb4-8472-e4a4-07d3acca8dbb","c516d635-d440-8897-f6c1-68d4e91b55f8","ce168168-9f6a-ed51-6655-3c2e3f108b97","0bb207fb-c857-61d2-ca66-37f40c96cb4b","ef16bc20-6f82-7a70-45f3-1f1cd3dec229","a8a42aa6-58ac-6b8c-c323-04eb9a81dfe1","9bed18f7-068b-fec9-dbfb-b9ae8f9ed667","94bfb787-6ba8-c951-59fc-e42672097f98","d1335548-a570-40ea-85c7-58421601170e","58e1f5b9-4bc3-98f7-c1fa-53d47a9d6d8b","8052e573-1f86-9ed8-5178-001ffb624af2","eb864f7c-e922-a1f5-82d3-19fc3a8529c3","49cf13de-6aa8-9bcd-836f-53516bbb472d","5e0bee47-1ecc-5f63-ad6a-7c7737c93acc","dd33b4ba-d774-7a5b-664d-eaf879f3fa3f","7c1aa95e-9253-f4da-9988-544345869c58","0e998bff-129a-564d-39e4-e9590d39d772","549d8216-ae88-7ebb-bc07-9370356302da","34fe8896-7301-0fac-f4d0-932f2cdc4978","8483c9e7-1df2-a2e5-445e-2fa92ded01fb","2832ae61-46af-6a38-5d69-5455b839a603","e3a4c429-f8b1-5553-54c1-9c10117ac1dc","773bbc06-953f-bcb3-aec9-5ce329a4cbd7","b1002976-3bd3-1f43-f741-fe96e42e9ab9","16832bcc-b381-584f-fa27-ec4f23725be6","cb66200e-cb9e-1ad6-c567-42532283667c","b9d98bf1-44c6-f04e-d023-94f139ad95df","b0710018-396e-e53d-9f08-a8cc98ada2ed"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":180,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":181,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"kind":"exact","score":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":182,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":7,"name":"chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":183,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"index":13,"input":{"domain":"chrome","file":"tasks/chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9.json","task":{"id":"93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","snapshot":"chrome","instruction":"Could you assist me in turning off the dark mode feature in Google Chrome? I've noticed that while dark mode is great for reducing glare, it actually makes it more challenging for me to read text clearly, especially with my astigmatism.","source":"https://superuser.com/questions/1417973/how-to-disable-google-chrome-dark-mode","config":[{"type":"execute","parameters":{"command":"mkdir -p /home/user/.config/google-chrome/Default && if [ ! -f /home/user/.config/google-chrome/Default/Preferences ]; then echo '{}' > /home/user/.config/google-chrome/Default/Preferences; fi && python3 -c \"import json; p='/home/user/.config/google-chrome/Default/Preferences'; d=json.load(open(p)); d.setdefault('browser', {}).setdefault('theme', {})['color_scheme'] = 2; d['browser']['theme']['color_scheme2'] = 2; json.dump(d, open(p, 'w'))\"","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"sleep","parameters":{"seconds":1}}],"func":["match_in_list","is_expected_url_pattern_match"],"result":[{"type":"chrome_appearance_mode_ui"},{"type":"active_url_from_accessTree","goto_prefix":""}],"expected":[{"type":"rule","rules":{"expected":["light","system"]}},{"type":"rule","rules":{"expected":["^chrome://settings/appearance/?$"]}}]},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":184,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":185,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.task.file","value":"tasks/chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":186,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.evaluator.func","value":"[\"match_in_list\",\"is_expected_url_pattern_match\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":187,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":188,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":189,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":190,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.pod","value":"osworld-12086550-2b5033"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":191,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.366Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/12086550-11c0-466b-b367-1d9e75b3910e/2026-10-11T19-25-40.365Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":192,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.366Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.agent.steps","value":19},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":193,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.366Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":194,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.366Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"auv.run.ids","value":["2963c2e2-1f89-a096-1ebe-7eaf92065e46","4e0372bc-b487-e3af-7231-04d2da8201b7","22fa34c3-084f-45de-5560-8a6182c54251","84e10f86-4bf1-9d06-a72b-0c78d14324e9","4cd62fa9-a1b5-7854-b81e-19b3e89150a2","eead0f83-1000-bce6-ef6f-30c7e166bf4b","22c5358f-436a-e2db-9ca2-3c422e7c72f6","6ac4a067-f900-90ff-e614-17d42c90a8ff","7788c53a-47fe-e2af-54a0-086c340ab26e","5f696e34-76b3-37bb-807a-ceec43c99ea1","ffd1dda9-3830-22f4-b2ad-de4b4cb0451b","743ab0b6-3337-e402-a743-bce3686f508f","fb895282-2852-89a5-229a-d3fff57ed378","724f8678-dd3e-1a9f-578f-f934f07ac6e5","3d883ba2-ccc5-e127-e186-d323d17db741","6fc1bbb7-a733-8c5f-6be1-82da7da34653","c745eaff-b4ea-287f-16c0-44a94bdc9a1f","e9e1dc6d-edb7-1ca1-5718-b86a58bdb4f5"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":195,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.366Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":196,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":197,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"index":11,"name":"chrome/12086550-11c0-466b-b367-1d9e75b3910e","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":198,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"index":14,"input":{"domain":"chrome","file":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json","task":{"id":"ae78f875-5b98-4907-bbb5-9c737fc68c03","snapshot":"chrome","instruction":"Could you please change the number of search results displayed on one page to 50? I find that having more results visible at once significantly enhances my research efficiency, as it reduces the need to constantly click through multiple pages. ","source":"https://support.google.com/chrome/thread/219988391/increase-search-results-per-page?hl=en","config":[],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":199,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":200,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.task.file","value":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":201,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.evaluator.func","value":"\"infeasible\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":202,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":203,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":204,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":205,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.pod","value":"osworld-93eabf48-e94487"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":206,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9/2026-10-11T19-27-01.140Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":207,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.agent.steps","value":17},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":208,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":209,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"auv.run.ids","value":["1f35452b-27d1-6b80-7f53-5d6891721513","2d8de726-aae4-8cf2-84e8-dd3edd47b903","e1d3f454-7f66-a132-05c7-82f3b9326713","727c8e31-b21b-85d6-7df2-035c775e8283","b472dd48-3116-5a6c-4886-4006a53d51b3","8de8d87c-959a-9ce6-77d3-7842662db847","467175cc-5c7a-613a-2b80-e73b63041ba4","89693650-e567-c4ab-a085-d6ba55f09be7","6ab39e25-6131-4688-df50-62cbb4b8f047","66f63aa8-9a02-d15e-829d-e055a98da1f5","726dbda2-3636-d07a-6356-151100372eac","70a927cd-eeb3-26a1-e56b-2c146b261045","4cdae1d1-5a80-f069-08a4-32033d574962","d4ea467e-1bbf-cb53-8f9b-5bed00b1f87e","cf1606ed-0a39-574b-b269-3f5736837607","04f5cbce-ef97-c9e5-28ed-15995841f2ac"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":210,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":211,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":212,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"index":13,"name":"chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":213,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"index":15,"input":{"domain":"chrome","file":"tasks/chrome/3299584d-8f11-4457-bf4c-ce98f7600250.json","task":{"id":"3299584d-8f11-4457-bf4c-ce98f7600250","snapshot":"chrome","instruction":"On my surface pro whenever I launch Chrome it always opens \"funbrain.com.\" I don't want this. I cleared my cache but it still happens. What should I do?","source":"https://www.reddit.com/r/techsupport/comments/12zwymy/comment/jhtri65/?utm_source=share&utm_medium=web2x&context=3","config":[{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt update -y && echo {CLIENT_PASSWORD} | sudo -S apt install jq -y","shell":true}},{"type":"execute","parameters":{"command":"cd /home/user/.config/google-chrome/Default && jq '. + {\"session\":{\"restore_on_startup\":4, \"startup_urls\":[\"http://funbrain.com/\"]}}' Preferences > temp && mv temp Preferences","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"exact_match","result":{"type":"new_startup_page"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/3299584d-8f11-4457-bf4c-ce98f7600250","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":214,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":215,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.task.file","value":"tasks/chrome/3299584d-8f11-4457-bf4c-ce98f7600250.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":216,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":217,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":218,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":219,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":220,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.pod","value":"osworld-3299584d-e2fde8"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":221,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.317Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/3299584d-8f11-4457-bf4c-ce98f7600250/2026-10-11T19-28-45.484Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":222,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.agent.steps","value":11},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":223,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":224,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"auv.run.ids","value":["80909ad8-f02c-f38b-0700-fd1ac6072228","47158092-f636-85dc-28e9-84f5332fe9ad","6d36679e-5b92-fdd8-3aaa-d749490482da","0a29244d-1ceb-10d4-900e-31208c13df3e","2faa9f85-7189-1239-f4db-6556fe8286c7","43c74854-caf5-1443-4c3b-986991c229b5","9400204b-25ce-5baa-33b3-e970ec3d38d2","ead1e4a4-daf0-7246-aede-e96ca8a6417e","f6fa6a21-3e4b-3858-d879-98766277151c","6ecf3c12-fdc2-ed3f-0e94-951e0e6ad249"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":225,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":226,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":227,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"index":15,"name":"chrome/3299584d-8f11-4457-bf4c-ce98f7600250","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":228,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"index":16,"input":{"domain":"chrome","file":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json","task":{"id":"030eeff7-b492-4218-b312-701ec99ee0cc","snapshot":"chrome","instruction":"Can you enable the 'Do Not Track' feature in Chrome to enhance my online privacy?","source":"https://www.surreycc.gov.uk/website/cookies/do-not-track","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"chrome_inject_js","parameters":{"js":"if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }","wait_for_load":false}},{"type":"sleep","parameters":{"seconds":1}},{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_do_not_track"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/030eeff7-b492-4218-b312-701ec99ee0cc","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":229,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":230,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.task.file","value":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":231,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":232,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":233,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":234,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":235,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.pod","value":"osworld-6766f2b8-e37b73"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":236,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/6766f2b8-8a72-417f-a9e5-56fcaa735837/2026-10-11T19-25-59.183Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":237,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":238,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.agent.last_action","value":"none"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":239,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"auv.run.ids","value":["2114cb1d-ee83-644a-fa5c-d3d7aefc82cd","2fbf9799-eadb-cad8-2146-2e7f28fba21a","0248689b-acf0-cba5-47d3-10e6ba8a5fff","caddbc1c-8001-fea4-e957-db89ea653a9b","8dda2b9d-5fc7-ef75-a813-91a3fd04dad3","23883639-7974-f8d9-bd96-2a269054e811","22fe8694-af91-cb1f-6c7b-266179602a1d","b9a29a0f-8a64-ac79-6e61-48a8eb8b177e","dd2f09f1-b129-5dec-5a93-cf9516984d11","8c73479b-76cd-06a8-b84a-93f689a6d079","069df994-c480-d271-9bc2-35ec174f7970","0db0fc5d-574d-2e86-516e-0e5fff2e3e71","5d0d54f7-981a-5ffb-e011-8cb3de485b6a","9f3cc401-eb84-0a15-f608-793c6e6c3bc6","a2ed395b-408a-210c-e308-6ee183c3874f","a8345888-d789-baa5-1f78-2b866d20f2b0","fd92e601-364d-96a5-f762-ae44bb800d74","fd660c01-3db4-7c6e-f1c7-cdbc13703113","ab56be1b-a8f9-23de-e1f0-24aa05c931c5","e630d394-ecc8-0497-9af0-147b2255a266","1fb3d0d8-aa84-b974-c1f5-ab3818bdf1f2","58bfb27a-8c03-eeb8-61f5-da563ff337a6","6c097d75-9da0-df17-1163-1b0690ab5725","3d44b019-49bb-9b0d-5b66-3d086d66a7f4","02d4e409-88fc-2c4b-c31b-607c72b0803c","b707b42c-dc55-ecf7-1da7-d81e2f44560d","fc6655d6-40cf-69bb-94f0-0a168c08f324","c70c2156-1a66-a4bf-9bbb-02f46fbef776","7ca60128-94c4-b2f3-17fe-3deb06ca935a","2ed12090-81db-e161-4311-3cd4016a34f1","22e70f25-6f1e-12c4-2a3d-628329358291","7b58074b-8d71-22cd-bdae-09121e6778a6","f474b9a2-6b75-4649-4bc7-9a8dfa2a7baf","a5e39e0b-ed11-5330-e91e-490675ef71fd","4d722f98-c174-fe01-58ed-9847def5f5b0","ba583be9-6778-3c34-897e-8d2035aa3324","48abaa94-7815-b42a-e76a-eb7c3d57bd69","17c2fcf5-32c0-97fc-8a33-67846c6eb805","9a3d4065-bbe5-f8af-cbe8-3f5d444f66cc","b5bcfded-110c-c06f-5852-d0d97caf197e","e072df72-d4bf-70e9-c833-b27513ed770e","17e30947-fdbe-1a39-3a63-ea09cd2f8ef4","a6df051e-04cd-b402-23f6-468b9e9ba8a0","0073af34-176e-287d-0f2a-1cc099e41ecf","48722b90-33cf-ccb8-94ea-b0d9cb60e1f2","404335ea-d710-aebe-f159-fecd48db3f67","af0516e4-5fa7-795b-9920-2db84f18367a","4ecfbcd6-c1a1-6395-3901-9592c689d71e","e83badb8-d039-08c1-61f9-2a6ab08be8d0"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":240,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":241,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"kind":"exact","score":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":242,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":12,"name":"chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":243,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.429Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"index":17,"input":{"domain":"chrome","file":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json","task":{"id":"9656a811-9b5b-4ddf-99c7-5117bcef0626","snapshot":"chrome","instruction":"I want Chrome to warn me whenever I visit a potentially harmful or unsafe website. Can you enable this safety feature?","source":"https://www.quora.com/How-do-I-set-the-security-settings-for-the-Google-Chrome-browser-for-the-best-security#:~:text=Enable%20Safe%20Browsing:%20Chrome%20has%20a%20built%2Din,Security%20%3E%20Security%20%3E%20Enable%20Safe%20Browsing.","config":[{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt update -y && echo {CLIENT_PASSWORD} | sudo -S apt install jq -y","shell":true}},{"type":"execute","parameters":{"command":"mkdir -p /home/user/.config/google-chrome/Default && if [ ! -f /home/user/.config/google-chrome/Default/Preferences ]; then echo '{}' > /home/user/.config/google-chrome/Default/Preferences; fi && cd /home/user/.config/google-chrome/Default && jq '. + {\"safebrowsing\":{\"enabled\":false,\"enhanced\":false}}' Preferences > temp && mv temp Preferences","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_safe_browsing"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":244,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.429Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":245,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.task.file","value":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":246,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":247,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":248,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":249,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":250,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"signal":"SIGINT"},"event":"run.interrupted","experimentId":"matrix-run-model-gpt-5.5-repeat-1","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":251,"time":"2026-10-11T19:31:17.940Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"errorMessage":"Run interrupted by SIGINT.","index":14,"name":"chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":252,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:22.549Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"errorMessage":"Run interrupted by SIGINT.","index":17,"name":"chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":253,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:22.862Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"errorMessage":"Run interrupted by SIGINT.","index":16,"name":"chrome/030eeff7-b492-4218-b312-701ec99ee0cc","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":254,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:24.492Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"state":"passed"},"event":"task.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":255,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:24.492Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"failed":false},"event":"project.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":256,"time":"2026-10-11T19:31:24.492Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"failedTasks":0,"passedTasks":1,"skippedTasks":0,"totalTasks":1},"event":"run.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","runId":"run-1791746045991-a77ba0a4","schemaVersion":2,"seq":257,"time":"2026-10-11T19:31:24.493Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"totalTasks":1},"event":"run.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":1,"time":"2026-10-11T19:14:05.992Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"displayName":"osworld [model=gpt-5.5, repeat=1]"},"event":"task.queued","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":2,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.992Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"taskCount":1},"event":"project.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":3,"time":"2026-10-11T19:14:05.992Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{},"event":"task.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":4,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"index":0,"input":{"domain":"chrome","file":"tasks/chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3.json","task":{"id":"7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","snapshot":"chrome","instruction":"Can you help me clean up my computer by getting rid of all the tracking things that Amazon might have saved? I want to make sure my browsing is private and those sites don't remember me.","source":"https://support.google.com/chrome/answer/95647?hl=en&ref_topic=7438325&sjid=16867045591165135686-AP#zippy=%2Cdelete-cookies-from-a-site","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.amazon.com","https://www.amazon.com/s?k=huggingface+transformers+book"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_cookie_deleted","result":{"type":"cookie_data","dest":"Cookies"},"expected":{"type":"rule","rules":{"type":"domains","domains":[".amazon.com"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":5,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":6,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.task.file","value":"tasks/chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":7,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.evaluator.func","value":"\"is_cookie_deleted\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":8,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":9,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":10,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":11,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"index":1,"input":{"domain":"chrome","file":"tasks/chrome/06fe7178-4491-4589-810f-2e2bc9502122.json","task":{"id":"06fe7178-4491-4589-810f-2e2bc9502122","snapshot":"chrome","instruction":"Can you make my computer bring back the last tab I shut down?","source":"https://www.wikihow.com/Switch-Tabs-in-Chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.lonelyplanet.com","https://www.airbnb.com","https://www.tripadvisor.com"]}},{"type":"chrome_close_tabs","parameters":{"urls_to_close":["https://www.tripadvisor.com"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_tabs","result":{"type":"open_tabs_info"},"expected":{"type":"rule","rules":{"type":"url","urls":["https://www.lonelyplanet.com","https://www.airbnb.com","https://www.tripadvisor.com"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/06fe7178-4491-4589-810f-2e2bc9502122","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":12,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":13,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.task.file","value":"tasks/chrome/06fe7178-4491-4589-810f-2e2bc9502122.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":14,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.evaluator.func","value":"\"is_expected_tabs\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":15,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":16,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":17,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":18,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"index":2,"input":{"domain":"chrome","file":"tasks/chrome/e1e75309-3ddb-4d09-92ec-de869c928143.json","task":{"id":"e1e75309-3ddb-4d09-92ec-de869c928143","snapshot":"chrome","instruction":"Computer, can you turn the webpage I'm looking at into a PDF file, save it to my Desktop with the default filename and set the margins to none?","source":"https://in5stepstutorials.com/google-chrome/save-web-page-as-pdf-in-chrome.php","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://lilianweng.github.io/posts/2023-06-23-agent/"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"compare_pdfs","result":{"type":"vm_file","path":"/home/user/Desktop/LLM Powered Autonomous Agents _ Lil'Log.pdf","dest":"LLM Powered Autonomous Agents _ Lil'Log.pdf"},"expected":{"type":"pdf_from_url","path":"https://lilianweng.github.io/posts/2023-06-23-agent/","dest":"LLM Powered Autonomous Agents _ Lil'Log_gold.pdf"}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/e1e75309-3ddb-4d09-92ec-de869c928143","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":19,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":20,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.task.file","value":"tasks/chrome/e1e75309-3ddb-4d09-92ec-de869c928143.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":21,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.evaluator.func","value":"\"compare_pdfs\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":22,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":23,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":24,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":25,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:14:05.994Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.pod","value":"osworld-06fe7178-624029"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":26,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.900Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/06fe7178-4491-4589-810f-2e2bc9502122/2026-10-11T19-14-06.064Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":27,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.agent.steps","value":7},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":28,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":29,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"auv.run.ids","value":["8ce2b33d-31cd-47b3-a869-b9cb4eab1d5d","7f8742c0-19c1-cae3-32ea-db1c53169b71","2decd904-75d7-ea71-11e7-e199b601a5d3","a505c326-bc53-cfd6-3ab2-b7b5ff4d8b07","d233deae-0dff-a34e-3e12-c1ef6e845d19","ef8e83be-bb57-3c0c-d366-3c2add975efe"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":30,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":31,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"name":"osworld.success","value":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":32,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.901Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"1:chrome%2F06fe7178-4491-4589-810f-2e2bc9502122","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":1,"name":"chrome/06fe7178-4491-4589-810f-2e2bc9502122","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":33,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.903Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"index":3,"input":{"domain":"chrome","file":"tasks/chrome/35253b65-1c19-4304-8aa4-6884b8218fc0.json","task":{"id":"35253b65-1c19-4304-8aa4-6884b8218fc0","snapshot":"chrome","instruction":"Hey, I need a quick way back to this site. Could you whip up a shortcut on my desktop for me using Chrome's built-in feature?","source":"https://www.laptopmag.com/articles/how-to-create-desktop-shortcuts-for-web-pages-using-chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.mathsisfun.com/games/2048.html"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_shortcut_on_desktop","result":{"type":"shortcuts_on_desktop"},"expected":{"type":"rule","rules":{"type":"name","name":"Play Puzzle Game 2048"}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/35253b65-1c19-4304-8aa4-6884b8218fc0","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":34,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.903Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":35,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.task.file","value":"tasks/chrome/35253b65-1c19-4304-8aa4-6884b8218fc0.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":36,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.evaluator.func","value":"\"is_shortcut_on_desktop\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":37,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":38,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":39,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":40,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:16:32.904Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.pod","value":"osworld-e1e75309-25dee7"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":41,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.631Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/e1e75309-3ddb-4d09-92ec-de869c928143/2026-10-11T19-14-06.065Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":42,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.agent.steps","value":24},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":43,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":44,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"auv.run.ids","value":["bbbfc80f-5c9d-3466-40dc-6ffc9fe3cec8","5efa0a92-631d-14c7-628e-1412594c3903","8c93afff-a5f8-3db9-62a1-cf6fde76a8fd","8a68cb33-b136-2bb7-5d5e-499da9c3b67c","b08b73a4-0a02-ab69-f999-6f52f47cdfbf","756c05ed-93ae-5669-7529-fa4381d834e0","bdadc082-0d52-98ac-9c4f-f36d90236668","b2dff437-c5c7-b54a-ffc9-702b714879c1","e3f53fad-c387-6078-08a2-76fab5a85c5e","bdd06d97-0078-aecb-67da-a0a52e6cd5aa","5a18dfc5-c050-04c6-2821-06a37db8440e","f6d8b193-3398-cf53-bc7f-6c21e3c3a4a0","d285b518-9ff4-dfd8-a3a7-b356697ffdb0","459b6908-d168-dd98-e2b3-b774d2f28c54","a3aa3e03-6967-4a4f-c63f-aa6f0407cddf","77cea753-b470-c14c-afd4-0fc1ee81ef03","992eafcd-9db8-0da4-c8a6-eabcaed0cf4b","fcc6b07e-1a7e-ccde-4e56-69b55dd81e73","bb542d39-1684-8e06-47b3-bb05e0535e7c","2be8f32c-3a08-31c4-8723-e2b808d3b711","9be7e44e-3013-0e5f-7201-36cb1ebd4a7b","e6b3123e-a0f3-8194-726d-6793a789bcc9","729d19e7-ed40-e269-ef3c-f658827b3e7e"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":45,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":46,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"name":"osworld.success","value":0.9380815331090999},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":47,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.632Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"2:chrome%2Fe1e75309-3ddb-4d09-92ec-de869c928143","data":{"errorMessage":"expected 0.9380815331090999 to be 1 // Object.is equality","index":2,"name":"chrome/e1e75309-3ddb-4d09-92ec-de869c928143","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":48,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.633Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"index":4,"input":{"domain":"chrome","file":"tasks/chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca.json","task":{"id":"2ad9387a-65d8-4e33-ad5b-7580065a27ca","snapshot":"chrome","instruction":"Can you make a new folder for me on the bookmarks bar in my internet browser? Let's call it 'Favorites.'","source":"https://www.youtube.com/watch?v=IN-Eq_UripQ","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"is_expected_bookmarks","result":{"type":"bookmarks"},"expected":{"type":"rule","rules":{"type":"bookmark_bar_folders_names","names":["Favorites"]}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":49,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.633Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":50,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.task.file","value":"tasks/chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":51,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.evaluator.func","value":"\"is_expected_bookmarks\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":52,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":53,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":54,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":55,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:17:44.634Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.pod","value":"osworld-35253b65-de1eaa"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":56,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.976Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/35253b65-1c19-4304-8aa4-6884b8218fc0/2026-10-11T19-16-32.960Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":57,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.976Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.agent.steps","value":12},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":58,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.976Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":59,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.976Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"auv.run.ids","value":["562d89bb-7abb-c9b8-89e9-f5b7f6476fdc","c221b337-3709-b702-0d3c-38916e614403","33f22dc8-6f24-ea56-6b90-3cd8190d52a2","99797196-b5ea-b987-5276-bd0b3a19270c","a0fd65a9-32bd-b0e5-b8a3-a1c5458b387f","ef0fb452-785c-08a1-4667-78129d9c2e02","4dedcdb9-afc6-651e-f42a-06de2caefcd4","34f4119d-4df4-3ca7-cf9d-f253f90c6f61","27704877-1a99-0f93-8810-03d0f5de52f9","d30dc0a6-9abc-02ce-0a47-7396c30f8254"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":60,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.976Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":61,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.977Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":62,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.977Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"3:chrome%2F35253b65-1c19-4304-8aa4-6884b8218fc0","data":{"index":3,"name":"chrome/35253b65-1c19-4304-8aa4-6884b8218fc0","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":63,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.977Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"index":5,"input":{"domain":"chrome","file":"tasks/chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263.json","task":{"id":"7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","snapshot":"chrome","instruction":"Can you save this webpage I'm looking at to bookmarks bar so I can come back to it later?","source":"https://www.youtube.com/watch?v=ZaZ8GcTxjXA","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://blog.eleuther.ai/rotary-embeddings/","https://jalammar.github.io/illustrated-transformer/"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"is_expected_bookmarks","result":{"type":"bookmarks"},"expected":{"type":"rule","rules":{"type":"bookmark_bar_websites_urls","urls":["https://jalammar.github.io/illustrated-transformer/"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":64,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.977Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":65,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.task.file","value":"tasks/chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":66,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.evaluator.func","value":"\"is_expected_bookmarks\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":67,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":68,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":69,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":70,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:18:53.978Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.pod","value":"osworld-2ad9387a-03611b"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":71,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.797Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/2ad9387a-65d8-4e33-ad5b-7580065a27ca/2026-10-11T19-17-44.670Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":72,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.797Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.agent.steps","value":26},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":73,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.797Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":74,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.797Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"auv.run.ids","value":["63cc9314-526b-5d8c-c56c-6474f90ffcd3","74a785ce-182e-3dbd-63a5-29b61d79955c","a8b589ea-ec19-7b31-f717-5364234055a4","31462f1a-9636-0e8d-e59e-d3cf79817292","383a51e8-1059-33c1-318b-72b76118e6d5","9948558b-5941-2d57-fb21-40850c5f61f3","20f34f68-b134-e47b-5003-c4ece7e66b05","9e6b2088-1b0f-a47b-d18d-4792a1fda16b","e4e6c191-f988-e93e-be4d-4ca0e999849c","434aa8a4-779c-3f27-ac8e-097ce678d176","5390f775-93f1-913d-acf1-6d1b11bf790d","ffcb9b6c-576e-5849-5da1-c2cfc0f609a9","c1688640-a8db-816f-572b-1318e7d3e090","6a272e12-1ca3-125b-1929-71ad6f6b5edc","b6005878-8c1c-c18a-4e40-b22ad1e61345","e4bcdeb3-f807-de9e-a059-03a7021ae82c","7a6e804e-2635-6d32-9bb1-7d9b4caf24f1","391df434-ddc6-6483-8027-7a1b800ad797","73ff6def-2189-95a7-246f-662107a93d50","67e7f2ee-32ae-ba60-eb3a-410f16865231","7f147858-f5e9-9e5c-a165-950fdd0d78e2","f59fa940-6c43-b0c2-d6e7-bd90de15b13e","caf01e5f-724c-43dc-6bbb-e88a07f443f5","bc14f365-cc07-23c7-36ca-8cd78a711384","b4caaa6b-db0e-84a4-b707-50878f393486"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":75,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.798Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":76,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.798Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":77,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.798Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"4:chrome%2F2ad9387a-65d8-4e33-ad5b-7580065a27ca","data":{"index":4,"name":"chrome/2ad9387a-65d8-4e33-ad5b-7580065a27ca","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":78,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.798Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"index":6,"input":{"domain":"chrome","file":"tasks/chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3.json","task":{"id":"2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","snapshot":"chrome","instruction":"Lately I have changed my English name to Thomas. I want to update my username. Could you help me change the username in chrome profiles to Thomas?","source":"https://superuser.com/questions/1393683/how-to-change-the-username-in-google-chrome-profiles?rq=1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"chrome_inject_js","parameters":{"js":"if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }","wait_for_load":false}},{"type":"sleep","parameters":{"seconds":1}},{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"profile_name"},"expected":{"type":"rule","rules":{"expected":"Thomas"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":79,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.799Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":80,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.task.file","value":"tasks/chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":81,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":82,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":83,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":84,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":85,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:12.800Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.pod","value":"osworld-7a5a7856-13547c"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":86,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263/2026-10-11T19-18-54.025Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":87,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.agent.steps","value":9},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":88,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":89,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"auv.run.ids","value":["bff34b9b-f6d2-a64a-c731-95a461421db2","c1d3b012-c69c-0543-b106-02e17fef9c70","fdddaf0d-4383-ae71-55f1-ec9759151168","a48f780c-7411-3bed-2fc6-eddc3665a1e9","3f9ca3ae-73e8-9d6d-cb89-a74f72223eab","3d3883ae-1d66-06a5-2a1a-4f4cdb9a69ea","009a7cfd-40dc-55ce-44e0-9f5210595c80","da930dff-5400-febf-35a8-0236cc13f405"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":90,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":91,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":92,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.040Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"5:chrome%2F7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","data":{"index":5,"name":"chrome/7a5a7856-f1b6-42a4-ade9-1ca81ca0f263","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":93,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"index":7,"input":{"domain":"chrome","file":"tasks/chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381.json","task":{"id":"480bcfea-d68f-4aaa-a0a9-2589ef319381","snapshot":"chrome","instruction":"I do not like the design of the new 2023 chrome UI. I want to keep using the original one. Can you disable the new 2023 version chrome UI for me? ","source":"https://bugartisan.medium.com/disable-the-new-chrome-ui-round-in-2023-db168271f71e","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":94,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":95,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.task.file","value":"tasks/chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":96,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.evaluator.func","value":"\"infeasible\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":97,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":98,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":99,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":100,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:21:13.041Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.pod","value":"osworld-7b6c7e24-d63d3a"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":101,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.786Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3/2026-10-11T19-14-06.062Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":102,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":103,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.agent.last_action","value":"none"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":104,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"auv.run.ids","value":["57a2c0d9-423e-24dd-f635-555b43ba3d84","e2d76e99-b9f7-2c02-6d62-1ff6613ecc61","55203674-669b-83d1-fa93-238fbf7889d2","cd308fef-d991-ea5a-6de4-c3e391006136","89dac0d5-c8d2-01ef-7025-f59ac4d43842","e6a7841d-4c1d-cb6d-4840-12945c1fe770","e97d0a1b-3fd2-d320-368d-5ac520e0d541","ad3e4591-1f0d-f5f5-1ed4-cd87d541b1f2","2e72488b-f633-67ff-e513-cb84d1107aea","441d44c4-3249-7a1f-fcf2-2487bff9d025","5bd14cde-1f00-a08c-a969-bf93cb3a1113","6af1c981-fb82-374a-0711-6aac7f46eb3f","8166a987-c12f-d7b8-b1c4-9f827249799e","119edb19-94f4-dd00-3d5e-17fe2587bb1f","8bfc851e-5738-fc7a-1a39-4c092760c9f7","c8652f37-232d-43af-92e0-0de1c0a4ab6c","5f7f1198-6ecd-f48e-dc9b-9b6dcc7e3f7c","d5d62fd1-797f-6ff3-a470-f3854018e7b0","33ca97c3-f535-8e93-35aa-badb1348c9b1","3e7bab57-5dfa-ae0d-1a2a-ea1c31788dca","9ca90c83-6ae5-22f4-26c8-857e081154f4","15617958-0d96-e799-42b8-fcb15e54f9b9","86261013-ea72-44d6-9dd0-1cb1aa763980","869164f9-862f-4c4e-901b-4b6c522be3cf","f9dcf37c-1d21-4831-fa80-2270a3e9ca0a","29037581-1849-35f1-3605-cfd7a8f5532f","a903d08b-834e-3596-7d77-e4a3eac7603d","c3482151-efe1-c0bf-519e-f0c848af4cac","5e91cdc5-ffb7-3aec-1b99-5336fb853b91","26831d79-fff7-d48a-fb6e-692b4e4c3524","326e31af-139c-2168-9091-97e7de38a2f8","25d61c14-ba73-2b5d-44b6-04a1a7361960","b1ebb94c-eb0b-4402-1fec-0aa72e72cfb0","3143d060-26ce-cda9-332d-49cc742d90e7","c102daf6-377b-e949-6429-5d7295477408","f15b186b-a04d-e66d-b63b-8795afb74db8","43fb424c-6d7a-74cd-a4cf-8fd150c8af85","96273415-a4f4-8a60-d880-5fd1ce74bf99","5655c4bf-87e8-807c-f83a-05d31af5486f","cd88d6c4-3624-1b76-d56b-30c09db9a9df","709e8263-b3e7-bc89-9553-18de97667603","39535112-9580-c5bc-4ad6-ff28af1a87d7","aa3a4b40-5715-00ea-deed-6b77245004ea","b9c81491-dcda-d5a5-70e1-7e27c54d8d9c","f4dcfb1c-8dd9-2510-870d-7fe2e806e744","b5013a28-0b93-6928-254b-80f37e64d250","7f1da95a-3d5d-8987-a5d0-ddfce3739799","122a101a-bb0a-ced4-ff1b-94cab07d0269","980a0d22-dbad-e11b-83de-4579a41fd1cc","5e9dfc2e-319a-6549-3a3a-115e51ff3acb"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":105,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":106,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":107,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.788Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"0:chrome%2F7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","data":{"index":0,"name":"chrome/7b6c7e24-c58a-49fc-a5bb-d57b80e5b4c3","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":108,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.789Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"index":8,"input":{"domain":"chrome","file":"tasks/chrome/af630914-714e-4a24-a7bb-f9af687d3b91.json","task":{"id":"af630914-714e-4a24-a7bb-f9af687d3b91","snapshot":"chrome","instruction":"My grandmother has been using the Chrome lately and told me that the font size is way too small for her poor eyesight. Could you set the default font size to the largest for her?","source":"https://www.howtogeek.com/680260/how-to-change-chromes-default-text-size/","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"check_font_size","result":{"type":"chrome_font_size"},"expected":{"type":"rule","rules":{"type":"range","min":16,"max":99999}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/af630914-714e-4a24-a7bb-f9af687d3b91","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":109,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.790Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":110,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.task.file","value":"tasks/chrome/af630914-714e-4a24-a7bb-f9af687d3b91.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":111,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.evaluator.func","value":"\"check_font_size\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":112,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":113,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":114,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":115,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:22:32.792Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.pod","value":"osworld-2ae9ba84-004323"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":116,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3/2026-10-11T19-21-12.909Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":117,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.agent.steps","value":18},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":118,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":119,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"auv.run.ids","value":["0f9a5e98-fe68-12b2-793a-423e1d0af6b4","c5ed57fe-72e7-a39d-bd11-dfd975fe9687","e54ba116-6839-8bc2-a87a-8b1e181ad407","3435fdc4-1dbf-49f3-bab1-97094a42134c","01c95781-d9db-4bd4-5f3b-162bea21a565","40adafda-c899-8bca-9a4b-fdbcc885e214","1116f716-f17f-3531-9b69-5aeabf594cea","a55fc692-d886-6171-62dc-16d4c8178ee9","5fdfb7cb-8e2d-da4c-84eb-e3f8a421d908","fb8d74aa-d814-bdb9-0c4c-77b775cf8ef2","f42fa1c1-ea4b-cb0d-a0b2-861333a51bb2","c92e144b-0dae-c45d-66e5-29c37639b403","b28cb371-6236-e9a5-3fd3-37f74c4e6035","63b1a0c5-1094-8f8a-59b0-d6570b3dec7f","960fefc4-4d1f-7a8a-0e02-bd8507e39b4b","1ab4cb89-5259-8f7a-9bb8-758fe852f8e9","838fdb09-579e-b2f4-33f9-8bf564972040"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":120,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":121,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":122,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.438Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"6:chrome%2F2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","data":{"index":6,"name":"chrome/2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":123,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.439Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"index":9,"input":{"domain":"chrome","file":"tasks/chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d.json","task":{"id":"3720f614-37fd-4d04-8a6b-76f54f8c222d","snapshot":"chrome","instruction":"I am from the country of Atlantis, and my mother tongue is Xenothian. Please change the Google Chrome interface language to Xenothian using only Chrome’s built-in settings.","source":"https://superuser.com/questions/984668/change-interface-language-of-chrome-to-english","config":[],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":124,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.440Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":125,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.task.file","value":"tasks/chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":126,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.evaluator.func","value":"\"infeasible\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":127,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":128,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":129,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":130,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:23:32.441Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.pod","value":"osworld-af630914-1acfcc"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":131,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/af630914-714e-4a24-a7bb-f9af687d3b91/2026-10-11T19-22-32.860Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":132,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.agent.steps","value":12},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":133,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":134,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"auv.run.ids","value":["0fd46f09-65e2-d5ac-b323-2fa09f5bf403","330ccc1d-da9a-25e9-2f9f-ce639826c15d","559125ba-d05b-133b-205d-c25dbf2f1136","5adb7abb-0069-4d63-0a94-7efd2d4467ff","23552df7-c02d-fd80-093d-c574ecda8aa7","f6ce2758-88b5-39fe-fc71-fe5b29574a16","80d55ca2-cf68-e325-fd44-bad3b9b09894","a427ffad-066e-8c00-c9b2-cb1242c780c4","99ae98c0-f23e-69b8-1215-73980b365cb1","e4e9df49-12a3-79f3-ed70-502a311a3029","6941b4a9-36ef-1d84-f78c-5213ad9bb3c7"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":135,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":136,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":137,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"8:chrome%2Faf630914-714e-4a24-a7bb-f9af687d3b91","data":{"index":8,"name":"chrome/af630914-714e-4a24-a7bb-f9af687d3b91","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":138,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"index":10,"input":{"domain":"chrome","file":"tasks/chrome/99146c54-4f37-4ab8-9327-5f3291665e1e.json","task":{"id":"99146c54-4f37-4ab8-9327-5f3291665e1e","snapshot":"chrome","instruction":"Please help me set Chrome to delete my browsing data automatically every time I close the browser.","source":"https://www.youtube.com/watch?v=v0kxqB7Xa6I","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"data_delete_automacally"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/99146c54-4f37-4ab8-9327-5f3291665e1e","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":139,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":140,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.task.file","value":"tasks/chrome/99146c54-4f37-4ab8-9327-5f3291665e1e.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":141,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.212Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":142,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.213Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":143,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.213Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":144,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.213Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":145,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:24:14.213Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.pod","value":"osworld-3720f614-a3afd3"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":146,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/3720f614-37fd-4d04-8a6b-76f54f8c222d/2026-10-11T19-23-32.504Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":147,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.agent.steps","value":21},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":148,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.agent.last_action","value":"FAIL"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":149,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"auv.run.ids","value":["e7233e57-7f5a-a4b4-5c78-9e35691502ae","be0ede42-f98b-757a-a60d-b241457b5eed","3318b6f2-0854-039c-19f5-63afee4bc154","76466e29-2dc3-06aa-2896-2318ea2b15ae","6e47111e-a797-59fb-2d4f-aa8a0426301e","10c08eeb-38f4-1d2c-d2c3-fa32e5871ebf","505b9407-7af7-d8df-689f-2983f4abe321","9575b7a4-d4ec-66d4-bace-f9ec47084431","72f97bf4-fe88-be60-eacd-d844be41312a","d2e73cc7-885b-293d-cd6d-f1ced0cbf263","fef469ff-6a2d-c6d5-3b34-99443c89c912","cfd8c665-6316-57b8-4db6-7e1c5bc59fe2","0e2a5f36-1084-48b0-6daf-b7153e3d6cbc","8c449646-2441-5448-2ff1-5a24185a4196","5e0906d4-ace6-4ce0-f657-b6976c986dce","d654cc67-7ab3-44a0-718c-53319dc2aaf3","7e5a2393-13f7-8415-e235-942d4cf5a8e6","ae6f46ad-8ab5-cae3-14a4-14f5fe47a0d6","da810650-34a9-b977-12b5-622e97fc3285","2af4400a-f4e6-85d9-7683-fb6f8e645662"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":150,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":151,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":152,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"9:chrome%2F3720f614-37fd-4d04-8a6b-76f54f8c222d","data":{"index":9,"name":"chrome/3720f614-37fd-4d04-8a6b-76f54f8c222d","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":153,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"index":11,"input":{"domain":"chrome","file":"tasks/chrome/12086550-11c0-466b-b367-1d9e75b3910e.json","task":{"id":"12086550-11c0-466b-b367-1d9e75b3910e","snapshot":"chrome","instruction":"Computer, please navigate to the area in my browser settings where my passwords are stored. I want to check my login information for Etsy without revealing it just yet.","source":"https://www.quora.com/What-are-the-cool-tricks-to-use-Google-Chrome","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_active_tab_approximate","result":{"type":"active_url_from_accessTree","goto_prefix":""},"expected":{"type":"rule","rules":{"type":"url","url":"chrome://password-manager/passwords"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/12086550-11c0-466b-b367-1d9e75b3910e","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":154,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.323Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":155,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.task.file","value":"tasks/chrome/12086550-11c0-466b-b367-1d9e75b3910e.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":156,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.evaluator.func","value":"\"is_expected_active_tab_approximate\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":157,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":158,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":159,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":160,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:40.324Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.pod","value":"osworld-99146c54-b6912e"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":161,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/99146c54-4f37-4ab8-9327-5f3291665e1e/2026-10-11T19-24-14.269Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":162,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.agent.steps","value":15},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":163,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":164,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"auv.run.ids","value":["d0adaa21-d213-f9cd-7161-a8302a0cf5ae","c6864c42-24ac-2e17-702f-b3946eaa0d6e","5c4bbc08-87c7-f1a9-6086-6117ce0166d8","aa3e6e0c-fc22-d519-8af7-46af279e86ea","6b3c7baf-b993-f1eb-815a-a05511c2538a","5e0e7912-97da-0ed5-92a5-c0be7fb5a3ab","dcc6481c-3a7e-6e5b-7387-70ee4ac5dcd0","dcd453f1-035f-3fcd-57c6-739cd14cb28f","5aefc831-5ced-48e1-6816-5ff3142e4c1d","3fffcefe-eb2b-d344-8fca-b38b75038f6b","a32e2490-7f44-573d-c419-b373bf59eb3a","e6e8c5c3-5d4b-de95-1460-633ecc81d135","1965eae6-bbe8-f11d-e72e-c87c58121355","3fb41d9d-285f-3035-6ae8-dab725a3af56"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":165,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":166,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":167,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"10:chrome%2F99146c54-4f37-4ab8-9327-5f3291665e1e","data":{"index":10,"name":"chrome/99146c54-4f37-4ab8-9327-5f3291665e1e","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":168,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.137Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"index":12,"input":{"domain":"chrome","file":"tasks/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837.json","task":{"id":"6766f2b8-8a72-417f-a9e5-56fcaa735837","snapshot":"chrome","instruction":"Could you help me unzip the downloaded extension file from /home/user/Desktop/ to /home/user/Desktop/ and configure it in Chrome's extensions?","source":"https://support.google.com/chrome/thread/205881926/it-s-possible-to-load-unpacked-extension-automatically-in-chrome?hl=en","config":[{"type":"download","parameters":{"files":[{"url":"https://huggingface.co/datasets/xlangai/ubuntu_osworld_file_cache/resolve/main/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837/helloExtension.zip","path":"/home/user/Desktop/helloExtension.zip"}]}},{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt-get update -y && echo {CLIENT_PASSWORD} | sudo -S apt-get install unzip -y && unzip /home/user/Desktop/helloExtension.zip -d /home/user/Desktop/ && rm /home/user/Desktop/helloExtension.zip","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_in_list","result":{"type":"find_unpacked_extension_path"},"expected":{"type":"rule","rules":{"expected":"/home/user/Desktop/helloExtension"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":169,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":170,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.task.file","value":"tasks/chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":171,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.evaluator.func","value":"\"is_in_list\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":172,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":173,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":174,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":175,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:25:59.138Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.pod","value":"osworld-480bcfea-ec301a"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":176,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/480bcfea-d68f-4aaa-a0a9-2589ef319381/2026-10-11T19-21-13.134Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":177,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.agent.steps","value":48},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":178,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":179,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"auv.run.ids","value":["4e6d8957-8251-96cb-e992-062c46c81d6b","38fa305a-8802-a9e9-169b-47a009226e12","108d142e-ffce-3a83-e411-c5ff506ec0d1","f32796ad-9635-7719-fef3-cf7a81d39048","92bc72c4-84a8-4a77-f4ca-65ed3cf50948","a1a87341-ae9e-f3cb-1c90-47822ce105ed","872ad29c-4724-81c0-faff-e31b11449963","66aa3fa1-4558-323d-6298-61c2dc68241b","974e428d-8a56-a5ba-f192-15bd87937f34","ea313189-e736-ad7c-3ae8-7a9e852292d3","0e251863-faba-ee72-3f76-89bc2404123b","3610e612-7191-48bc-49ac-3b5efd3a6b9a","27a10af6-5306-328d-e80f-d4b13ece9460","a1736127-7b9d-cf9d-f833-aec169a8d0eb","da52a1d4-4809-f884-940e-e4c67505f814","62a5d2ad-ccb7-e5ff-0070-3f9829c6e735","e2b8c379-fc3c-4c72-a070-1c2d5f31d764","22f1e1be-cc9a-ee20-1c90-27c161529daa","1e568e0c-cca7-f993-1d9d-8bd60146c966","fb34b88b-3fb4-8472-e4a4-07d3acca8dbb","c516d635-d440-8897-f6c1-68d4e91b55f8","ce168168-9f6a-ed51-6655-3c2e3f108b97","0bb207fb-c857-61d2-ca66-37f40c96cb4b","ef16bc20-6f82-7a70-45f3-1f1cd3dec229","a8a42aa6-58ac-6b8c-c323-04eb9a81dfe1","9bed18f7-068b-fec9-dbfb-b9ae8f9ed667","94bfb787-6ba8-c951-59fc-e42672097f98","d1335548-a570-40ea-85c7-58421601170e","58e1f5b9-4bc3-98f7-c1fa-53d47a9d6d8b","8052e573-1f86-9ed8-5178-001ffb624af2","eb864f7c-e922-a1f5-82d3-19fc3a8529c3","49cf13de-6aa8-9bcd-836f-53516bbb472d","5e0bee47-1ecc-5f63-ad6a-7c7737c93acc","dd33b4ba-d774-7a5b-664d-eaf879f3fa3f","7c1aa95e-9253-f4da-9988-544345869c58","0e998bff-129a-564d-39e4-e9590d39d772","549d8216-ae88-7ebb-bc07-9370356302da","34fe8896-7301-0fac-f4d0-932f2cdc4978","8483c9e7-1df2-a2e5-445e-2fa92ded01fb","2832ae61-46af-6a38-5d69-5455b839a603","e3a4c429-f8b1-5553-54c1-9c10117ac1dc","773bbc06-953f-bcb3-aec9-5ce329a4cbd7","b1002976-3bd3-1f43-f741-fe96e42e9ab9","16832bcc-b381-584f-fa27-ec4f23725be6","cb66200e-cb9e-1ad6-c567-42532283667c","b9d98bf1-44c6-f04e-d023-94f139ad95df","b0710018-396e-e53d-9f08-a8cc98ada2ed"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":180,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":181,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"name":"osworld.success","value":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":182,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.117Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"7:chrome%2F480bcfea-d68f-4aaa-a0a9-2589ef319381","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":7,"name":"chrome/480bcfea-d68f-4aaa-a0a9-2589ef319381","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":183,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"index":13,"input":{"domain":"chrome","file":"tasks/chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9.json","task":{"id":"93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","snapshot":"chrome","instruction":"Could you assist me in turning off the dark mode feature in Google Chrome? I've noticed that while dark mode is great for reducing glare, it actually makes it more challenging for me to read text clearly, especially with my astigmatism.","source":"https://superuser.com/questions/1417973/how-to-disable-google-chrome-dark-mode","config":[{"type":"execute","parameters":{"command":"mkdir -p /home/user/.config/google-chrome/Default && if [ ! -f /home/user/.config/google-chrome/Default/Preferences ]; then echo '{}' > /home/user/.config/google-chrome/Default/Preferences; fi && python3 -c \"import json; p='/home/user/.config/google-chrome/Default/Preferences'; d=json.load(open(p)); d.setdefault('browser', {}).setdefault('theme', {})['color_scheme'] = 2; d['browser']['theme']['color_scheme2'] = 2; json.dump(d, open(p, 'w'))\"","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"sleep","parameters":{"seconds":1}}],"func":["match_in_list","is_expected_url_pattern_match"],"result":[{"type":"chrome_appearance_mode_ui"},{"type":"active_url_from_accessTree","goto_prefix":""}],"expected":[{"type":"rule","rules":{"expected":["light","system"]}},{"type":"rule","rules":{"expected":["^chrome://settings/appearance/?$"]}}]},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":184,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":185,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.task.file","value":"tasks/chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":186,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.evaluator.func","value":"[\"match_in_list\",\"is_expected_url_pattern_match\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":187,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":188,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":189,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":190,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:01.118Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.pod","value":"osworld-12086550-2b5033"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":191,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.366Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/12086550-11c0-466b-b367-1d9e75b3910e/2026-10-11T19-25-40.365Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":192,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.366Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.agent.steps","value":19},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":193,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.366Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":194,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.366Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"auv.run.ids","value":["2963c2e2-1f89-a096-1ebe-7eaf92065e46","4e0372bc-b487-e3af-7231-04d2da8201b7","22fa34c3-084f-45de-5560-8a6182c54251","84e10f86-4bf1-9d06-a72b-0c78d14324e9","4cd62fa9-a1b5-7854-b81e-19b3e89150a2","eead0f83-1000-bce6-ef6f-30c7e166bf4b","22c5358f-436a-e2db-9ca2-3c422e7c72f6","6ac4a067-f900-90ff-e614-17d42c90a8ff","7788c53a-47fe-e2af-54a0-086c340ab26e","5f696e34-76b3-37bb-807a-ceec43c99ea1","ffd1dda9-3830-22f4-b2ad-de4b4cb0451b","743ab0b6-3337-e402-a743-bce3686f508f","fb895282-2852-89a5-229a-d3fff57ed378","724f8678-dd3e-1a9f-578f-f934f07ac6e5","3d883ba2-ccc5-e127-e186-d323d17db741","6fc1bbb7-a733-8c5f-6be1-82da7da34653","c745eaff-b4ea-287f-16c0-44a94bdc9a1f","e9e1dc6d-edb7-1ca1-5718-b86a58bdb4f5"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":195,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.366Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":196,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":197,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"11:chrome%2F12086550-11c0-466b-b367-1d9e75b3910e","data":{"index":11,"name":"chrome/12086550-11c0-466b-b367-1d9e75b3910e","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":198,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"index":14,"input":{"domain":"chrome","file":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json","task":{"id":"ae78f875-5b98-4907-bbb5-9c737fc68c03","snapshot":"chrome","instruction":"Could you please change the number of search results displayed on one page to 50? I find that having more results visible at once significantly enhances my research efficiency, as it reduces the need to constantly click through multiple pages. ","source":"https://support.google.com/chrome/thread/219988391/increase-search-results-per-page?hl=en","config":[],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":199,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":200,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.task.file","value":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":201,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.evaluator.func","value":"\"infeasible\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":202,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":203,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":204,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":205,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:27:30.367Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.pod","value":"osworld-93eabf48-e94487"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":206,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9/2026-10-11T19-27-01.140Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":207,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.agent.steps","value":17},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":208,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":209,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"auv.run.ids","value":["1f35452b-27d1-6b80-7f53-5d6891721513","2d8de726-aae4-8cf2-84e8-dd3edd47b903","e1d3f454-7f66-a132-05c7-82f3b9326713","727c8e31-b21b-85d6-7df2-035c775e8283","b472dd48-3116-5a6c-4886-4006a53d51b3","8de8d87c-959a-9ce6-77d3-7842662db847","467175cc-5c7a-613a-2b80-e73b63041ba4","89693650-e567-c4ab-a085-d6ba55f09be7","6ab39e25-6131-4688-df50-62cbb4b8f047","66f63aa8-9a02-d15e-829d-e055a98da1f5","726dbda2-3636-d07a-6356-151100372eac","70a927cd-eeb3-26a1-e56b-2c146b261045","4cdae1d1-5a80-f069-08a4-32033d574962","d4ea467e-1bbf-cb53-8f9b-5bed00b1f87e","cf1606ed-0a39-574b-b269-3f5736837607","04f5cbce-ef97-c9e5-28ed-15995841f2ac"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":210,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":211,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":212,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"13:chrome%2F93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","data":{"index":13,"name":"chrome/93eabf48-6a27-4cb6-b963-7d5fe1e0d3a9","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":213,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.457Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"index":15,"input":{"domain":"chrome","file":"tasks/chrome/3299584d-8f11-4457-bf4c-ce98f7600250.json","task":{"id":"3299584d-8f11-4457-bf4c-ce98f7600250","snapshot":"chrome","instruction":"On my surface pro whenever I launch Chrome it always opens \"funbrain.com.\" I don't want this. I cleared my cache but it still happens. What should I do?","source":"https://www.reddit.com/r/techsupport/comments/12zwymy/comment/jhtri65/?utm_source=share&utm_medium=web2x&context=3","config":[{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt update -y && echo {CLIENT_PASSWORD} | sudo -S apt install jq -y","shell":true}},{"type":"execute","parameters":{"command":"cd /home/user/.config/google-chrome/Default && jq '. + {\"session\":{\"restore_on_startup\":4, \"startup_urls\":[\"http://funbrain.com/\"]}}' Preferences > temp && mv temp Preferences","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"exact_match","result":{"type":"new_startup_page"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/3299584d-8f11-4457-bf4c-ce98f7600250","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":214,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":215,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.task.file","value":"tasks/chrome/3299584d-8f11-4457-bf4c-ce98f7600250.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":216,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":217,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":218,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":219,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":220,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:28:45.458Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.pod","value":"osworld-3299584d-e2fde8"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":221,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.317Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/3299584d-8f11-4457-bf4c-ce98f7600250/2026-10-11T19-28-45.484Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":222,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.agent.steps","value":11},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":223,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":224,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"auv.run.ids","value":["80909ad8-f02c-f38b-0700-fd1ac6072228","47158092-f636-85dc-28e9-84f5332fe9ad","6d36679e-5b92-fdd8-3aaa-d749490482da","0a29244d-1ceb-10d4-900e-31208c13df3e","2faa9f85-7189-1239-f4db-6556fe8286c7","43c74854-caf5-1443-4c3b-986991c229b5","9400204b-25ce-5baa-33b3-e970ec3d38d2","ead1e4a4-daf0-7246-aede-e96ca8a6417e","f6fa6a21-3e4b-3858-d879-98766277151c","6ecf3c12-fdc2-ed3f-0e94-951e0e6ad249"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":225,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":226,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":227,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"15:chrome%2F3299584d-8f11-4457-bf4c-ce98f7600250","data":{"index":15,"name":"chrome/3299584d-8f11-4457-bf4c-ce98f7600250","state":"passed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":228,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"index":16,"input":{"domain":"chrome","file":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json","task":{"id":"030eeff7-b492-4218-b312-701ec99ee0cc","snapshot":"chrome","instruction":"Can you enable the 'Do Not Track' feature in Chrome to enhance my online privacy?","source":"https://www.surreycc.gov.uk/website/cookies/do-not-track","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"chrome_inject_js","parameters":{"js":"if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }","wait_for_load":false}},{"type":"sleep","parameters":{"seconds":1}},{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_do_not_track"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/030eeff7-b492-4218-b312-701ec99ee0cc","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":229,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":230,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.task.file","value":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":231,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":232,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":233,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":234,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":235,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:29:56.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.pod","value":"osworld-6766f2b8-e37b73"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":236,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/6766f2b8-8a72-417f-a9e5-56fcaa735837/2026-10-11T19-25-59.183Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":237,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":238,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.agent.last_action","value":"none"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":239,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"auv.run.ids","value":["2114cb1d-ee83-644a-fa5c-d3d7aefc82cd","2fbf9799-eadb-cad8-2146-2e7f28fba21a","0248689b-acf0-cba5-47d3-10e6ba8a5fff","caddbc1c-8001-fea4-e957-db89ea653a9b","8dda2b9d-5fc7-ef75-a813-91a3fd04dad3","23883639-7974-f8d9-bd96-2a269054e811","22fe8694-af91-cb1f-6c7b-266179602a1d","b9a29a0f-8a64-ac79-6e61-48a8eb8b177e","dd2f09f1-b129-5dec-5a93-cf9516984d11","8c73479b-76cd-06a8-b84a-93f689a6d079","069df994-c480-d271-9bc2-35ec174f7970","0db0fc5d-574d-2e86-516e-0e5fff2e3e71","5d0d54f7-981a-5ffb-e011-8cb3de485b6a","9f3cc401-eb84-0a15-f608-793c6e6c3bc6","a2ed395b-408a-210c-e308-6ee183c3874f","a8345888-d789-baa5-1f78-2b866d20f2b0","fd92e601-364d-96a5-f762-ae44bb800d74","fd660c01-3db4-7c6e-f1c7-cdbc13703113","ab56be1b-a8f9-23de-e1f0-24aa05c931c5","e630d394-ecc8-0497-9af0-147b2255a266","1fb3d0d8-aa84-b974-c1f5-ab3818bdf1f2","58bfb27a-8c03-eeb8-61f5-da563ff337a6","6c097d75-9da0-df17-1163-1b0690ab5725","3d44b019-49bb-9b0d-5b66-3d086d66a7f4","02d4e409-88fc-2c4b-c31b-607c72b0803c","b707b42c-dc55-ecf7-1da7-d81e2f44560d","fc6655d6-40cf-69bb-94f0-0a168c08f324","c70c2156-1a66-a4bf-9bbb-02f46fbef776","7ca60128-94c4-b2f3-17fe-3deb06ca935a","2ed12090-81db-e161-4311-3cd4016a34f1","22e70f25-6f1e-12c4-2a3d-628329358291","7b58074b-8d71-22cd-bdae-09121e6778a6","f474b9a2-6b75-4649-4bc7-9a8dfa2a7baf","a5e39e0b-ed11-5330-e91e-490675ef71fd","4d722f98-c174-fe01-58ed-9847def5f5b0","ba583be9-6778-3c34-897e-8d2035aa3324","48abaa94-7815-b42a-e76a-eb7c3d57bd69","17c2fcf5-32c0-97fc-8a33-67846c6eb805","9a3d4065-bbe5-f8af-cbe8-3f5d444f66cc","b5bcfded-110c-c06f-5852-d0d97caf197e","e072df72-d4bf-70e9-c833-b27513ed770e","17e30947-fdbe-1a39-3a63-ea09cd2f8ef4","a6df051e-04cd-b402-23f6-468b9e9ba8a0","0073af34-176e-287d-0f2a-1cc099e41ecf","48722b90-33cf-ccb8-94ea-b0d9cb60e1f2","404335ea-d710-aebe-f159-fecd48db3f67","af0516e4-5fa7-795b-9920-2db84f18367a","4ecfbcd6-c1a1-6395-3901-9592c689d71e","e83badb8-d039-08c1-61f9-2a6ab08be8d0"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":240,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":241,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"name":"osworld.success","value":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":242,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.428Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"12:chrome%2F6766f2b8-8a72-417f-a9e5-56fcaa735837","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":12,"name":"chrome/6766f2b8-8a72-417f-a9e5-56fcaa735837","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":243,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.429Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"index":17,"input":{"domain":"chrome","file":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json","task":{"id":"9656a811-9b5b-4ddf-99c7-5117bcef0626","snapshot":"chrome","instruction":"I want Chrome to warn me whenever I visit a potentially harmful or unsafe website. Can you enable this safety feature?","source":"https://www.quora.com/How-do-I-set-the-security-settings-for-the-Google-Chrome-browser-for-the-best-security#:~:text=Enable%20Safe%20Browsing:%20Chrome%20has%20a%20built%2Din,Security%20%3E%20Security%20%3E%20Enable%20Safe%20Browsing.","config":[{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt update -y && echo {CLIENT_PASSWORD} | sudo -S apt install jq -y","shell":true}},{"type":"execute","parameters":{"command":"mkdir -p /home/user/.config/google-chrome/Default && if [ ! -f /home/user/.config/google-chrome/Default/Preferences ]; then echo '{}' > /home/user/.config/google-chrome/Default/Preferences; fi && cd /home/user/.config/google-chrome/Default && jq '. + {\"safebrowsing\":{\"enabled\":false,\"enhanced\":false}}' Preferences > temp && mv temp Preferences","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_safe_browsing"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626","total":360},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":244,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.429Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":245,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.task.file","value":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":246,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":247,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":248,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":249,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":250,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:30:00.430Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"signal":"SIGINT"},"event":"run.interrupted","experimentId":"matrix-run-model-gpt-5.5-repeat-1","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":251,"time":"2026-10-11T19:31:17.940Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"14:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"errorMessage":"Run interrupted by SIGINT.","index":14,"name":"chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":252,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:22.549Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"17:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"errorMessage":"Run interrupted by SIGINT.","index":17,"name":"chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":253,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:22.862Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","caseId":"16:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"errorMessage":"Run interrupted by SIGINT.","index":16,"name":"chrome/030eeff7-b492-4218-b312-701ec99ee0cc","state":"failed","total":360},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":254,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:24.492Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"state":"passed"},"event":"task.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":255,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:24.492Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"failed":false},"event":"project.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":256,"time":"2026-10-11T19:31:24.492Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-14-05-991Z","data":{"failedTasks":0,"passedTasks":1,"skippedTasks":0,"totalTasks":1},"event":"run.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","runId":"run-1791746045991-a77ba0a4","schemaVersion":3,"seq":257,"time":"2026-10-11T19:31:24.493Z","workspaceId":"default-workspace"}
diff --git a/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/otlp/metrics.json b/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/otlp/metrics.json
index f91eab4..de13491 100644
--- a/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/otlp/metrics.json
+++ b/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/otlp/metrics.json
@@ -2538,7 +2538,7 @@
                   }
                 ]
               },
-              "name": "vieval.score.exact"
+              "name": "vieval.score.osworld.success"
             }
           ],
           "scope": {
diff --git a/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/run-summary.json b/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/run-summary.json
index 9462b9c..dab5927 100644
--- a/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/run-summary.json
+++ b/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4/run-summary.json
@@ -2,6 +2,27 @@
   "attemptId": "attempt-2026-10-11T19-14-05-991Z",
   "configFilePath": "/home/user/Git/github.com/moeru-ai/auv-pr-258/evals/osworld/vieval.config.ts",
   "experimentId": "matrix-run-model-gpt-5.5-repeat-1",
+  "headline": {
+    "completion": {
+      "mean": 1,
+      "n": 18,
+      "name": "completion",
+      "sum": 18,
+      "total": 18,
+      "unscored": {}
+    },
+    "headline": {
+      "kind": "score",
+      "score": {
+        "mean": 0.6111111111111112,
+        "n": 18,
+        "name": "osworld.success",
+        "sum": 11,
+        "total": 18,
+        "unscored": {}
+      }
+    }
+  },
   "projects": [
     {
       "caseFailures": [
@@ -73,27 +94,104 @@
       "result": {
         "inferenceExecutors": [
           {
-            "exactAverage": 0.6111111111111112,
-            "hybridAverage": 0.6111111111111112,
-            "inferenceExecutorId": "default",
-            "judgeAverage": null,
-            "runCount": 1
+            "completion": {
+              "mean": 1,
+              "n": 18,
+              "name": "completion",
+              "sum": 18,
+              "total": 18,
+              "unscored": {}
+            },
+            "headline": {
+              "completion": {
+                "mean": 1,
+                "n": 18,
+                "name": "completion",
+                "sum": 18,
+                "total": 18,
+                "unscored": {}
+              },
+              "headline": {
+                "kind": "score",
+                "score": {
+                  "mean": 0.6111111111111112,
+                  "n": 18,
+                  "name": "osworld.success",
+                  "sum": 11,
+                  "total": 18,
+                  "unscored": {}
+                }
+              }
+            },
+            "runCount": 1,
+            "scores": [
+              {
+                "mean": 0.6111111111111112,
+                "n": 18,
+                "name": "osworld.success",
+                "sum": 11,
+                "total": 18,
+                "unscored": {}
+              }
+            ],
+            "inferenceExecutorId": "default"
           }
         ],
         "overall": {
-          "exactAverage": 0.6111111111111112,
-          "hybridAverage": 0.6111111111111112,
-          "judgeAverage": null,
-          "runCount": 1
+          "completion": {
+            "mean": 1,
+            "n": 18,
+            "name": "completion",
+            "sum": 18,
+            "total": 18,
+            "unscored": {}
+          },
+          "headline": {
+            "completion": {
+              "mean": 1,
+              "n": 18,
+              "name": "completion",
+              "sum": 18,
+              "total": 18,
+              "unscored": {}
+            },
+            "headline": {
+              "kind": "score",
+              "score": {
+                "mean": 0.6111111111111112,
+                "n": 18,
+                "name": "osworld.success",
+                "sum": 11,
+                "total": 18,
+                "unscored": {}
+              }
+            }
+          },
+          "runCount": 1,
+          "scores": [
+            {
+              "mean": 0.6111111111111112,
+              "n": 18,
+              "name": "osworld.success",
+              "sum": 11,
+              "total": 18,
+              "unscored": {}
+            }
+          ]
         },
         "runs": [
           {
+            "completion": {
+              "mean": 1,
+              "n": 18,
+              "name": "completion",
+              "sum": 18,
+              "total": 18,
+              "unscored": {}
+            },
             "entryId": "evals/osworld/eval/osworld",
-            "exactAverage": 0.6111111111111112,
-            "hybridAverage": 0.6111111111111112,
             "id": "evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default",
             "inferenceExecutorId": "default",
-            "judgeAverage": null,
             "matrix": {
               "eval": {},
               "meta": {
@@ -104,7 +202,26 @@
                 "model": "gpt-5.5",
                 "repeat": "1"
               }
-            }
+            },
+            "passed": {
+              "mean": 0.6111111111111112,
+              "n": 18,
+              "name": "passed",
+              "sum": 11,
+              "total": 18,
+              "unscored": {}
+            },
+            "primary": "osworld.success",
+            "scores": [
+              {
+                "mean": 0.6111111111111112,
+                "n": 18,
+                "name": "osworld.success",
+                "sum": 11,
+                "total": 18,
+                "unscored": {}
+              }
+            ]
           }
         ]
       },
@@ -114,5 +231,7 @@
   "interruptSignal": "SIGINT",
   "reportDirectory": "__VIEVAL_DIR__/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4",
   "runId": "run-1791746045991-a77ba0a4",
-  "workspaceId": "default-workspace"
+  "workspaceId": "default-workspace",
+  "scoreDefinitions": [],
+  "schemaVersion": 2
 }
diff --git a/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-31-43-580Z/run-1791747103580-dd3a8517/events.jsonl b/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-31-43-580Z/run-1791747103580-dd3a8517/events.jsonl
index 66a7bdf..d95ffc9 100644
--- a/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-31-43-580Z/run-1791747103580-dd3a8517/events.jsonl
+++ b/packages/ui/test/fixtures/osworld/.vieval/reports/default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-31-43-580Z/run-1791747103580-dd3a8517/events.jsonl
@@ -1,150 +1,150 @@
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","data":{"totalTasks":1},"event":"run.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":1,"time":"2026-10-11T19:31:43.581Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","data":{"displayName":"osworld [model=gpt-5.5, repeat=1]"},"event":"task.queued","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":2,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.582Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","data":{"taskCount":1},"event":"project.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":3,"time":"2026-10-11T19:31:43.582Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","data":{},"event":"task.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":4,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.582Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"index":0,"input":{"domain":"chrome","file":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json","task":{"id":"ae78f875-5b98-4907-bbb5-9c737fc68c03","snapshot":"chrome","instruction":"Could you please change the number of search results displayed on one page to 50? I find that having more results visible at once significantly enhances my research efficiency, as it reduces the need to constantly click through multiple pages. ","source":"https://support.google.com/chrome/thread/219988391/increase-search-results-per-page?hl=en","config":[],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":5,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.582Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":6,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.task.file","value":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":7,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.evaluator.func","value":"\"infeasible\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":8,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":9,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":10,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":11,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"index":1,"input":{"domain":"chrome","file":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json","task":{"id":"030eeff7-b492-4218-b312-701ec99ee0cc","snapshot":"chrome","instruction":"Can you enable the 'Do Not Track' feature in Chrome to enhance my online privacy?","source":"https://www.surreycc.gov.uk/website/cookies/do-not-track","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"chrome_inject_js","parameters":{"js":"if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }","wait_for_load":false}},{"type":"sleep","parameters":{"seconds":1}},{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_do_not_track"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/030eeff7-b492-4218-b312-701ec99ee0cc","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":12,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":13,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.task.file","value":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":14,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":15,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":16,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":17,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":18,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"index":2,"input":{"domain":"chrome","file":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json","task":{"id":"9656a811-9b5b-4ddf-99c7-5117bcef0626","snapshot":"chrome","instruction":"I want Chrome to warn me whenever I visit a potentially harmful or unsafe website. Can you enable this safety feature?","source":"https://www.quora.com/How-do-I-set-the-security-settings-for-the-Google-Chrome-browser-for-the-best-security#:~:text=Enable%20Safe%20Browsing:%20Chrome%20has%20a%20built%2Din,Security%20%3E%20Security%20%3E%20Enable%20Safe%20Browsing.","config":[{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt update -y && echo {CLIENT_PASSWORD} | sudo -S apt install jq -y","shell":true}},{"type":"execute","parameters":{"command":"mkdir -p /home/user/.config/google-chrome/Default && if [ ! -f /home/user/.config/google-chrome/Default/Preferences ]; then echo '{}' > /home/user/.config/google-chrome/Default/Preferences; fi && cd /home/user/.config/google-chrome/Default && jq '. + {\"safebrowsing\":{\"enabled\":false,\"enhanced\":false}}' Preferences > temp && mv temp Preferences","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_safe_browsing"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":19,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":20,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.task.file","value":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":21,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":22,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":23,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":24,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":25,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"index":3,"input":{"domain":"chrome","file":"tasks/chrome/fc6d8143-9452-4171-9459-7f515143419a.json","task":{"id":"fc6d8143-9452-4171-9459-7f515143419a","snapshot":"chrome","instruction":"Find flights from New York–Kennedy Airport to Chicago O'Hare Airport for tomorrow.","source":"test_task_0","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.delta.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"class","class_singleObject":{"mach-flight-context-info__wrapper--date":"time"},"class_multiObject_child":{"mach-flight-context-info__wrapper__info--separator":{"0":"start","1":"end"}}},"expected":{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"tomorrow"},"expected":{"start":"JFK","end":"ORD","time":"{DoW}, {Month} {Day0D}, {Year}"}}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/fc6d8143-9452-4171-9459-7f515143419a","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":26,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":27,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.task.file","value":"tasks/chrome/fc6d8143-9452-4171-9459-7f515143419a.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":28,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":29,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":30,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":31,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":32,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"index":4,"input":{"domain":"chrome","file":"tasks/chrome/a96b564e-dbe9-42c3-9ccf-b4498073438a.json","task":{"id":"a96b564e-dbe9-42c3-9ccf-b4498073438a","snapshot":"chrome","instruction":"In the FlightAware Discussions forum, navigate to the FlightAware > General category and open the topic with the most posts or replies.","source":"test_task_0","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.flightaware.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_active_tab","result":{"type":"active_tab_info","goto_prefix":"https://"},"expected":{"type":"rule","rules":{"type":"url","url":"https://discussions.flightaware.com/t/the-banter-thread/4412"}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/a96b564e-dbe9-42c3-9ccf-b4498073438a","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":33,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":34,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.task.file","value":"tasks/chrome/a96b564e-dbe9-42c3-9ccf-b4498073438a.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":35,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.evaluator.func","value":"\"is_expected_active_tab\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":36,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":37,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":38,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":39,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"index":5,"input":{"domain":"chrome","file":"tasks/chrome/1704f00f-79e6-43a7-961b-cedd3724d5fd.json","task":{"id":"1704f00f-79e6-43a7-961b-cedd3724d5fd","snapshot":"chrome","instruction":"Find a large car from next Monday to Friday in Zurich, sorted by price.","source":"test_task_0","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.rentalcars.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":["check_direct_json_object","check_direct_json_object"],"conj":"or","result":[{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["locationName","dropLocationName","filterCriteria_carCategory","filterCriteria_sortBy","puDay","puMonth","puYear","doDay","doMonth","doYear"]},{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["locationName","dropLocationName","filterCriteria_carCategory","filterCriteria_sortBy","puDay","puMonth","puYear","doDay","doMonth","doYear"]}],"expected":[{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"next Monday split","to":"next Friday split"},"timezone":"Europe/Zurich","expected":{"locationName":"Zürich","dropLocationName":"Zürich","filterCriteria_carCategory":"large","filterCriteria_sortBy":"PRICE","puDay":"{DayD}","puMonth":"{MonthD}","puYear":"{Year}","doDay":"{DayD}","doMonth":"{MonthD}","doYear":"{Year}"}}},{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"week after next Monday split","to":"week after next Friday split"},"timezone":"Europe/Zurich","expected":{"locationName":"Zürich","dropLocationName":"Zürich","filterCriteria_carCategory":"large","filterCriteria_sortBy":"PRICE","puDay":"{DayD}","puMonth":"{MonthD}","puYear":"{Year}","doDay":"{DayD}","doMonth":"{MonthD}","doYear":"{Year}"}}}]},"proxy":true,"possibility_of_env_change":"medium","fixed_ip":false}},"name":"chrome/1704f00f-79e6-43a7-961b-cedd3724d5fd","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":40,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":41,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.task.file","value":"tasks/chrome/1704f00f-79e6-43a7-961b-cedd3724d5fd.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":42,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.evaluator.func","value":"[\"check_direct_json_object\",\"check_direct_json_object\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":43,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":44,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":45,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":46,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"index":6,"input":{"domain":"chrome","file":"tasks/chrome/f3b19d1e-2d48-44e9-b4e1-defcae1a0197.json","task":{"id":"f3b19d1e-2d48-44e9-b4e1-defcae1a0197","snapshot":"chrome","instruction":"Find the FAQ page about ticket delivery.","source":"test_task_0","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://premier.ticketek.com.au/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_url_pattern_match","result":{"type":"active_tab_info"},"expected":{"type":"rule","rules":{"expected":["(?i)ticket-delivery(?:-faqs)?"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/f3b19d1e-2d48-44e9-b4e1-defcae1a0197","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":47,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":48,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.task.file","value":"tasks/chrome/f3b19d1e-2d48-44e9-b4e1-defcae1a0197.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":49,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.evaluator.func","value":"\"is_expected_url_pattern_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":50,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":51,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":52,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":53,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"index":7,"input":{"domain":"chrome","file":"tasks/chrome/82bc8d6a-36eb-4d2d-8801-ef714fb1e55a.json","task":{"id":"82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","snapshot":"chrome","instruction":"On next Monday, look up a flight from Mumbai to Stockholm.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.qatarairways.com/en-hk/homepage.html"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["fromStation","toStation","departing"],"replace":{"departing":"time"}},"expected":{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"next Monday"},"expected":{"fromStation":["BOM"],"toStation":["STO","ARN"],"time":["{Year}-{Month0D}-{Day0D}"]},"expect_in_result":true}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":54,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":55,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.task.file","value":"tasks/chrome/82bc8d6a-36eb-4d2d-8801-ef714fb1e55a.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":56,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":57,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":58,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":59,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":60,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.pod","value":"osworld-9656a811-379d46"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":61,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.317Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/9656a811-9b5b-4ddf-99c7-5117bcef0626/2026-10-11T19-31-43.730Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":62,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.317Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.agent.steps","value":15},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":63,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":64,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"auv.run.ids","value":["09356360-7e20-fd51-e536-951effdfc464","3ab585e0-2ac6-5125-3623-438b41782395","d106b6a2-ec7b-6d8e-9640-24e290c68b50","cfcd13e5-4286-b177-1d8a-8c3c1568c134","6cced7df-a019-cef5-dfec-eda665e627bb","e8cdfce9-9be7-2a90-0e94-e60323b10ae4","e6d3b66c-f9ef-363e-8d97-3860d430c27e","93e50900-0470-2766-8b06-5003073faf44","35b95abc-8d64-40cf-1440-1cea539001a4","22d8325f-1695-4df2-90bb-2d9f667d1b7a","3fbf4db4-85e7-441a-691e-d3985ea05abe","5885e48f-8188-28b0-56a0-202b93ab60b6","7b3bfe2e-16f4-f0c1-e047-b22802bd0643","9936ebb1-9c01-236f-8899-9ee8f90df086"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":65,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":66,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":67,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"index":2,"name":"chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":68,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"index":8,"input":{"domain":"chrome","file":"tasks/chrome/47543840-672a-467d-80df-8f7c3b9788c9.json","task":{"id":"47543840-672a-467d-80df-8f7c3b9788c9","snapshot":"chrome","instruction":"On the current website, show me the cars available for pickup at Boston Logan Intl Airport from the 10th to the 11th of next month, sorted by the number of seats to find the largest capacity.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.budget.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":["is_expected_url_pattern_match","check_direct_json_object","check_direct_json_object"],"conj":"and","result":[{"type":"active_url_from_accessTree","goto_prefix":"https://www."},{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"class","class_singleObject":{},"class_multiObject":{"location-info":{"0":"start_location","1":"end_location"},"day-time-info":{"0":"from","1":"to"}}},{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"xpath","xpathObject":{"/html/body/div[6]/div[2]/div[1]/div/div/div[2]/section[1]/div[1]/form/div[1]/div[1]/div[2]/div/a":"rank"}}],"expected":[{"type":"rule","rules":{"expected":["reservation#/vehicles"]}},{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"10th next month","to":"11th next month"},"expected":{"start_location":"Boston Logan Intl Airport,\n\t\t\t\t\t\t\t\tBOS \n\t\t\t\t\t\t\t  \n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t \n\t\t\t\t\t\t\t\t \n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t      \n\t\t\t\t\t\t\t        \n\t\t\t\t\t\t\t        \n\t\t\t\t\t\t\t          \n\t\t\t\t\t\t\t            \n\t\t\t\t\t\t\t            Pick-Up","end_location":"Boston Logan Intl Airport,\n\t\t\t\t\t\t\t\tBOS","from":"{DoW}, {Month} {Day0D}, 12:00 PM","to":"{DoW}, {Month} {Day0D}, 12:00 PM"}}},{"type":"rule","rules":{"expected":{"rank":"Number of Seats (High to Low)"}}}]},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/47543840-672a-467d-80df-8f7c3b9788c9","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":69,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":70,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.task.file","value":"tasks/chrome/47543840-672a-467d-80df-8f7c3b9788c9.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":71,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.evaluator.func","value":"[\"is_expected_url_pattern_match\",\"check_direct_json_object\",\"check_direct_json_object\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":72,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":73,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":74,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":75,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.pod","value":"osworld-f3b19d1e-56a5dc"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":76,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/f3b19d1e-2d48-44e9-b4e1-defcae1a0197/2026-10-11T19-31-43.727Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":77,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.agent.steps","value":12},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":78,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":79,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"auv.run.ids","value":["321881c7-a091-7583-d23d-e6ad203828de","96b3de90-73b0-9d01-a606-eee203f2fdd4","cdef10b7-3a0c-150a-6b08-4f5f34d3557a","b55741cc-a5aa-a0bf-5630-90dc6ed12a46","c89cb02f-f44c-2512-9b69-5c24a7a4375d","b985df6b-797f-0d89-0bd7-0e3cdd71ee84","74c6af62-7dcf-722b-8da6-1e6bd70a383a","1840ab4c-8b19-9713-7e1f-e8945803a02d","77a94f3a-c89d-bd0c-030a-dd30ba2ec845","67c7266f-b8ba-f8ec-9d66-79d08bce5c87","7fbfe49d-67d1-03dd-68ec-d64a65324b2a"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":80,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":81,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":82,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"index":6,"name":"chrome/f3b19d1e-2d48-44e9-b4e1-defcae1a0197","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":83,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.992Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"index":9,"input":{"domain":"chrome","file":"tasks/chrome/c1fa57f3-c3db-4596-8f09-020701085416.json","task":{"id":"c1fa57f3-c3db-4596-8f09-020701085416","snapshot":"chrome","instruction":"Open the baggage fee calculator in United Airlines website.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.united.com/en/us"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_url_pattern_match","result":{"type":"active_tab_info","goto_prefix":"https://www."},"expected":{"type":"rule","rules":{"expected":["united\\.com/en/us/checked-bag-fee-calculator(/.*)?"]}}},"proxy":true,"possibility_of_env_change":"medium","fixed_ip":false}},"name":"chrome/c1fa57f3-c3db-4596-8f09-020701085416","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":84,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.992Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":85,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.task.file","value":"tasks/chrome/c1fa57f3-c3db-4596-8f09-020701085416.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":86,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.evaluator.func","value":"\"is_expected_url_pattern_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":87,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":88,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":89,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":90,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.pod","value":"osworld-030eeff7-bb0813"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":91,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.593Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/030eeff7-b492-4218-b312-701ec99ee0cc/2026-10-11T19-31-43.723Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":92,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.agent.steps","value":36},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":93,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":94,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"auv.run.ids","value":["d066ac34-d263-4aa8-0c73-3169458bc22d","68b030b9-6824-220f-352d-2aaaba0426c2","58fb9c32-5d1e-8faf-d51f-169d95515173","74c5f39c-d1ec-34f9-d148-4d99265f3a67","b8737dd2-be50-8619-0efe-c2d222cae717","e21a4197-75f3-e6c6-d246-23677f058971","131eee20-d470-5c71-94d1-c44a38084e59","7d0584cd-bc14-4880-3097-a2d0ec23faad","f4c69802-a0ce-0c65-2afd-d34548ca8692","a40070db-bc3b-1091-39b8-6158fc4072c0","7a3c8073-ba18-1f6e-1ffa-444be7aa2012","463cbbdd-5faf-eebe-a6db-20f884d18e12","3dcbe336-c718-93ca-6494-796eefe5f974","d8037475-6dc9-1554-32bc-a2a14d88ac65","662afd64-ec9a-ec26-7679-b78578606dca","f6fe55dc-104d-8ece-6259-29eabae863f8","30afc19e-7f90-d601-1ea6-a37660af65d2","b645caf6-7396-1a21-9605-b9bd78ed55ce","fe3a333f-083c-c159-8b96-0abb57fde82a","8195093f-22c0-a9a4-6308-8ec494b719bc","1557e2be-7b2b-ab95-52e6-4e32249f9029","ee08d293-af07-6bdc-8cfe-80f0fa61d39e","5dd67a34-10f0-31a7-20be-a3934b3e63a6","2df42faf-a179-d363-1113-febe8d2bcf5b","c68769b2-1cf8-bbdf-44cc-5c25a1aa4c95","3b38ab74-2310-9454-75ba-ca3ca0006b43","1fc27498-1f9d-f9ff-cc33-26b26380fa6d","2eaf1139-0d37-cc1b-6d45-3f45b16856f5","ae525f10-1fd7-f36e-f42b-208c278f41ef","5732b2f8-8766-cbab-2b6b-fd2c7b3110ca","62d0e113-1a6c-7aea-84c1-0185f993089b","11e3a8a4-463d-0776-ea13-35d49434287f","67f1746c-5a88-f554-4ba4-466735f7acd1","10774639-fda1-2a82-8db3-a13bbf21d054","224a3456-6255-57bb-6bec-ac901f881d77"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":95,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":96,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":97,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"index":1,"name":"chrome/030eeff7-b492-4218-b312-701ec99ee0cc","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":98,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"index":10,"input":{"domain":"chrome","file":"tasks/chrome/da46d875-6b82-4681-9284-653b0c7ae241.json","task":{"id":"da46d875-6b82-4681-9284-653b0c7ae241","snapshot":"chrome","instruction":"Book an appointment to apply for a transportation access pass at the Charlie Card store on the first Monday eight months later at any available time from 9:00 am to 12:00 pm, fill in my details (James Smith, james.smith@gmail.com). And do not click \"book\" directly. Let me review it.","source":"test_task_2","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.mbta.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"conj":"and","func":["is_expected_url_pattern_match","check_direct_json_object","check_direct_json_object","check_direct_json_object"],"result":[{"type":"active_url_from_accessTree"},{"type":"active_tab_html_parse","category":"class","class_multiObject_only_child":{"HAZ16":{"0":"content"}}},{"type":"active_tab_html_parse","category":"class","class_multiObject_only_child":{"HAZ16":{"1":"time"}}},{"type":"active_tab_html_parse","category":"input","inputObject":{"/html/body/div[2]/div/form/div[7]/div/div/div[1]/input[1]":"name","/html/body/div[2]/div/form/div[7]/div/div/div[1]/input[2]":"mail"}}],"expected":[{"type":"rule","rules":{"expected":["book/CharlieCardStoreAppointments@mbta.com/"]}},{"type":"rule","rules":{"expected":{"content":"Apply for Transportation Access Pass (TAP) CharlieCard non-auto approval"}}},{"type":"rule_relativeTime","rules":{"expected":{"time":["{MonthFull} {Day0D}, 9:00 AM","{MonthFull} {Day0D}, 9:15 AM","{MonthFull} {Day0D}, 9:30 AM","{MonthFull} {Day0D}, 9:45 AM","{MonthFull} {Day0D}, 10:00 AM","{MonthFull} {Day0D}, 10:15 AM","{MonthFull} {Day0D}, 10:30 AM","{MonthFull} {Day0D}, 10:45 AM","{MonthFull} {Day0D}, 11:00 AM","{MonthFull} {Day0D}, 11:15 AM","{MonthFull} {Day0D}, 11:30 AM","{MonthFull} {Day0D}, 11:45 AM","{MonthFull} {Day0D}, 12:00 PM"]},"relativeTime":{"from":"first monday eight months later"},"expect_in_result":true}},{"type":"rule","rules":{"expected":{"name":"James Smith","mail":"james.smith@gmail.com"}}}]},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/da46d875-6b82-4681-9284-653b0c7ae241","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":99,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":100,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.task.file","value":"tasks/chrome/da46d875-6b82-4681-9284-653b0c7ae241.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":101,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.evaluator.func","value":"[\"is_expected_url_pattern_match\",\"check_direct_json_object\",\"check_direct_json_object\",\"check_direct_json_object\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":102,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":103,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":104,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":105,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.pod","value":"osworld-a96b564e-99fa48"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":106,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/a96b564e-dbe9-42c3-9ccf-b4498073438a/2026-10-11T19-31-43.726Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":107,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.agent.steps","value":30},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":108,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":109,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"auv.run.ids","value":["20a2c74b-40f5-fc97-c4de-3cafb4fe92f3","37f95add-7363-9408-151d-fc55a5e1ed2a","5a50cd53-2ec5-f978-7d3c-ea3894f8a0d7","1dee0662-853c-8b74-81f9-0dab91395465","79079097-f326-053e-8717-b1468f489e33","4357905f-f218-fb89-0b5b-7bb16defbdf3","745c0d3d-9a65-e421-d177-41ac5a11ce00","cb3bc12f-4a97-73fd-f658-590917216c03","82df1c52-627e-6b0f-a5ad-eab72a95dffe","622a5337-c816-88ca-e5be-f6f0c3dd8d53","443fa610-bc06-d710-3e50-962232b230b1","819d7849-f018-a72e-5fba-a857a54fa672","c731a497-e6e3-eff6-346c-721efac9eb22","c0b25702-5708-eb81-6ca8-a2c6ab42328c","2a060dc4-3048-af16-8985-9353b4cee657","5fffa7e9-2070-ef21-2243-07d79b0d9e99","bf4d8e85-27b0-f2ed-82c2-0f284dc0c629","b2576050-cc86-1bdc-f9fc-26b065f42a7b","d226e082-1ac1-2ce5-986c-33053fc93cbe","71d8c2fc-bede-f911-6cb5-bcefbdbbb7ef","8a79dfc5-3889-c3bb-d001-34c10b26d026","39c7bd5d-28ba-738c-8448-a8a65e3cea85","547e95a9-dbc5-e893-4d96-2cbe16d0a655","12dd4087-f644-12c2-f749-0fcaba531cbb","ad0fa48c-ec22-52f0-1560-b321b4ba8997","d3d86a6c-7066-4cad-74be-8461b0f5ae28","d0a780b5-d721-2acd-bcd4-c9355477a2e6","6f989cb8-76c8-e6d5-7a68-10aacad1334f","babe4fa1-7a2c-680b-a6af-31ec70b5fab6"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":110,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":111,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"kind":"exact","score":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":112,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"index":4,"name":"chrome/a96b564e-dbe9-42c3-9ccf-b4498073438a","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":113,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"index":11,"input":{"domain":"chrome","file":"tasks/chrome/6c4c23a1-42a4-43cc-9db1-2f86ff3738cc.json","task":{"id":"6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","snapshot":"chrome","instruction":"Find flights from Seattle to New York on 5th next month and only show those that can be purchased with miles.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.delta.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"class","class_singleObject":{"mach-flight-context-info__wrapper--date":"time","mach-global-tabs-small__wrapper__tab--active":"category"},"class_multiObject_child":{"mach-flight-context-info__wrapper__info--separator":{"0":"start","1":"end"}}},"expected":{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"5th next month"},"expected":{"start":"SEA","end":"NYC","time":"{DoW}, {Month} {Day0D}, {Year}","category":"Miles"}}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":114,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":115,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.task.file","value":"tasks/chrome/6c4c23a1-42a4-43cc-9db1-2f86ff3738cc.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":116,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":117,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":118,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":119,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":120,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.pod","value":"osworld-82bc8d6a-1f5bdf"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":121,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/82bc8d6a-36eb-4d2d-8801-ef714fb1e55a/2026-10-11T19-31-43.729Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":122,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.agent.steps","value":28},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":123,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":124,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"auv.run.ids","value":["b14a99d4-e004-77a5-9fda-a8c775e54e4c","7085eda9-09e0-ca6c-eb91-34ba330808bd","3c926368-ee2d-1706-97b4-d9d8546e55c9","3a11d6f3-2a96-f78a-a455-2a8d61ae2d88","e5657894-c00f-f714-f19b-5d62dfda0881","9abbc5ea-be61-fbf7-8154-b0b3bb644b58","c0a1ec26-3e00-2299-2a65-38b6e3cae820","d63d4174-6fe0-6a99-fd41-5983df26dc2f","6ce3d0a9-92d9-7da6-6d82-e95f2cb77c43","fff88bde-bde2-cbc3-28bc-169e5c6ee303","aecc9873-2c80-4178-f9d1-ba77cf24dfd4","2189ac00-06bf-501b-4819-638cf60f0b21","28e5f77d-95ed-2f2d-95ce-32a42750730c","42bd4b72-674d-a586-f1fc-b262ab7f1573","a2322838-0caf-d4a2-ce4d-676517a48eca","bd5cf952-d7a9-1e62-fba0-ff3d9293f3bd","31058be6-b1d2-2547-f6af-d2b9aceb50f5","35406297-0e0f-0ead-c5c1-ff6c0c23a7de","5829084a-1549-6cb0-ec49-add93a6a400d","67ce3cfc-b0a2-cd6c-38e7-2c7bc04beb3a","749a7c39-9184-6a63-ceb0-51a340b29cc8","12b382c8-4c0d-4bed-522b-6fa76474d1b6","bb5e42c0-c71d-5f68-4460-f246d63e6217","8099d765-b8da-8e4a-4cda-c9e3111e4002","be6707f9-b69d-9fb2-6d52-2427cdbe19f3","27f9d6a8-fdd7-fffb-e1e6-3a08bfdae1b5","a7e6cd7c-5963-7add-2c9d-76e2fab0cad3"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":125,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":126,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"kind":"exact","score":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":127,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":7,"name":"chrome/82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","state":"failed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":128,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"index":12,"input":{"domain":"chrome","file":"tasks/chrome/f79439ad-3ee8-4f99-a518-0eb60e5652b0.json","task":{"id":"f79439ad-3ee8-4f99-a518-0eb60e5652b0","snapshot":"chrome","instruction":"Search for a one way flight from Dublin to Vienna on 10th next month for 2 adults.","source":"test_task_2","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.ryanair.com/gb/en"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["originIata","destinationIata","tpAdults","tpTeens","tpChildren","tpStartDate","isReturn"],"replace":{"tpStartDate":"time"}},"expected":{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"10th next month"},"expected":{"originIata":"DUB","destinationIata":"VIE","tpAdults":"2","tpTeens":"0","tpChildren":"0","time":"{Year}-{Month0D}-{DayD}","isReturn":"false"}}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/f79439ad-3ee8-4f99-a518-0eb60e5652b0","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":129,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":130,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.task.file","value":"tasks/chrome/f79439ad-3ee8-4f99-a518-0eb60e5652b0.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":131,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":132,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":133,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":134,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":135,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.pod","value":"osworld-ae78f875-6abb3a"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":136,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/ae78f875-5b98-4907-bbb5-9c737fc68c03/2026-10-11T19-31-43.722Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":137,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":138,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.agent.last_action","value":"none"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":139,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"auv.run.ids","value":["29016aee-d86c-67e4-343f-57927f6bbe7e","35027e88-5a7c-7535-9b6f-c7a4363ce41c","f8c65638-574c-eaf3-6f68-5c515270f550","d7e9e5de-8523-3b43-1245-36389ae2fb79","d51417bf-bc97-a9d4-b4f4-d40d64115167","3f1f08b9-299d-2b90-6167-c3ca2636340e","8be2aa48-d098-d4fc-fb26-6c184a360296","9c7fbf3e-0a69-128b-eb9a-21a6889c4c02","d2fe19f2-20ee-c604-8866-48e2aab77c2b","9d05e1d3-44d6-1d2f-8593-7d826989742f","37e3a9b1-4460-5c08-05ef-052c9ac3f426","5fc8e43c-5fdb-b831-be55-caee6ca86df9","ac116868-ea62-c169-4623-5b101cc67b5f","12ea7226-a238-847c-8357-d71211772651","13b1c180-4973-952d-305b-65afb9296438","f15550c6-b7c9-6805-753f-d27c64b8b1d8","22c84e2e-3910-30ed-571e-e325ab23d503","fd9e426d-a634-8514-483c-0bed04940ec4","c8b9a6f0-d66b-326b-386d-db8e79c44d36","737d552b-a933-6441-5fd4-573d74ec9998","f7d70474-0e17-4992-ddf0-83e96fea8770","4f881b40-1842-9659-7f30-ba09c59ebdb0","0fdc247e-6eb5-7be3-a976-b6e8f73e7605","42de4150-6491-57e1-363a-c581e1794bb6","039a6f68-5ead-26d7-c427-754e40184044","20114020-c011-2fa9-5027-6e230f21cd16","d293b0b7-db74-4667-ad81-79d12239f6a2","7d33033a-89cd-449f-5d89-91479319df72","a3c5f997-8740-7cc9-3b17-119fed48c035","8e8edd10-bef9-f378-01b0-f69311e2e3f9","ee483958-0f78-2f40-ffd2-12c417301bf7","b5ba6be9-eeb8-a533-95f3-a5143bfcdb57","19acad27-efbf-edec-0674-20b5e7cedc00","c3af01d2-473e-740e-b754-7f176f9ec20c","c980424c-91ae-1b24-db0a-c42cc4c6c859","a9ccddde-799f-9d9e-d58c-fb3a4fab0846","829641f5-3fbf-a4e7-7d8f-5986e1543b96","8c34075a-729e-7760-f3c2-07d6597654e3","77d7ae1d-6d77-3ba7-cec6-7dff0b887128","8adafe02-7d2f-2ecc-cbfc-720e1ad44f95","dbf9d2b2-8d74-1df5-da5f-8932d0fb3b47","b81ab5ad-8d51-6896-dc79-62f1eacd4889","9eecbd0f-7652-0fe0-7897-bea48f2124d2","b4b46475-c98f-449f-cfc6-c742befbb4b9","6be12809-860f-155a-b94a-c5e2a1419341","360bd81f-6a82-0e39-5e4b-87ddbaee8957","c2264406-c2c6-6a8f-9f47-5cdba646e7e7","df110396-bc98-8a41-7990-43586224e648","845ec44b-c44a-5d80-8e39-ef6f76950ce7","147da4a6-7e9e-efdc-181b-4ed653d92e7e"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":140,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":141,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"kind":"exact","score":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":142,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":0,"name":"chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03","state":"failed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":143,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"index":13,"input":{"domain":"chrome","file":"tasks/chrome/b7895e80-f4d1-4648-bee0-4eb45a6f1fa8.json","task":{"id":"b7895e80-f4d1-4648-bee0-4eb45a6f1fa8","snapshot":"chrome","instruction":"Find a Hotel in New York City with lowest price possible for 2 adults next weekend. Sort the search results by price and stay on the results page.","source":"test_task_0","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.tripadvisor.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":["check_direct_json_object","check_direct_json_object","check_direct_json_object","check_direct_json_object"],"conj":"or","result":[{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"xpath","xpathObject":{"//button[@data-automation='checkin']//div[contains(@class,'Wh')]//span":"from","//button[@data-automation='checkout']//div[contains(@class,'Wh')]//span":"to","//h2[@data-automation='header_geo_title']":"city","//button[@data-automation='roomsandguests']//div[contains(@class,'Wh')]":"adult","//button[contains(@aria-label,'PRICE_LOW_TO_HIGH: Price (low to high)') or contains(@aria-label,'PRICE')]//div[contains(@class,'biGQs') and contains(@class,'SewaP')]":"rank"}},{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"xpath","xpathObject":{"//button[@data-automation='checkin']//div[contains(@class,'Wh')]":"from","//button[@data-automation='checkout']//div[contains(@class,'Wh')]":"to","//h2[@data-automation='header_geo_title']":"city","//button[@data-automation='roomsandguests']//div[contains(@class,'Wh')]":"adult","//button[contains(@aria-label,'PRICE_LOW_TO_HIGH: Price (low to high)') or contains(@aria-label,'PRICE')]//div[contains(@class,'biGQs') and contains(@class,'SewaP')]":"rank"}},{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"xpath","xpathObject":{"//button[@data-automation='checkin']//div[contains(@class,'Wh')]//span":"from","//button[@data-automation='checkout']//div[contains(@class,'Wh')]//span":"to","//h2[@data-automation='header_geo_title']":"city","//button[@data-automation='roomsandguests']//div[contains(@class,'Wh')]":"adult","//button[contains(@aria-label,'PRICE_LOW_TO_HIGH: Price (low to high)') or contains(@aria-label,'PRICE')]//div[contains(@class,'biGQs') and contains(@class,'SewaP')]":"rank"}},{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"xpath","xpathObject":{"//button[@data-automation='checkin']//div[contains(@class,'Wh')]":"from","//button[@data-automation='checkout']//div[contains(@class,'Wh')]":"to","//h2[@data-automation='header_geo_title']":"city","//button[@data-automation='roomsandguests']//div[contains(@class,'Wh')]":"adult","//button[contains(@aria-label,'PRICE_LOW_TO_HIGH: Price (low to high)') or contains(@aria-label,'PRICE')]//div[contains(@class,'biGQs') and contains(@class,'SewaP')]":"rank"}}],"expected":[{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"next week Saturday","to":"next week Sunday"},"timezone":"America/New_York","expected":{"from":"{DoW}, {Month} {Day0D}","to":"{DoW}, {Month} {Day0D}","city":"New York City Hotels","adult":"Rooms/Guests1 Room, 2 Guests","rank":"Price (low to high)"}}},{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"next week Friday","to":"next week Sunday"},"timezone":"America/New_York","expected":{"from":"Check In{DoW}, {Month} {Day0D}","to":"Check Out{DoW}, {Month} {Day0D}","city":"New York City Hotels","adult":"Rooms/Guests1 Room, 2 Guests","rank":"Price (low to high)"}}},{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"week after next Saturday","to":"week after next Sunday"},"timezone":"America/New_York","expected":{"from":"{DoW}, {Month} {Day0D}","to":"{DoW}, {Month} {Day0D}","city":"New York City Hotels","adult":"Rooms/Guests1 Room, 2 Guests","rank":"Price (low to high)"}}},{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"week after next Friday","to":"week after next Sunday"},"timezone":"America/New_York","expected":{"from":"Check In{DoW}, {Month} {Day0D}","to":"Check Out{DoW}, {Month} {Day0D}","city":"New York City Hotels","adult":"Rooms/Guests1 Room, 2 Guests","rank":"Price (low to high)"}}}]},"proxy":true,"possibility_of_env_change":"high","fixed_ip":false}},"name":"chrome/b7895e80-f4d1-4648-bee0-4eb45a6f1fa8","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":144,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":145,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.task.file","value":"tasks/chrome/b7895e80-f4d1-4648-bee0-4eb45a6f1fa8.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":146,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.evaluator.func","value":"[\"check_direct_json_object\",\"check_direct_json_object\",\"check_direct_json_object\",\"check_direct_json_object\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":147,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":148,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":149,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
-{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":2,"seq":150,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","data":{"totalTasks":1},"event":"run.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":1,"time":"2026-10-11T19:31:43.581Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","data":{"displayName":"osworld [model=gpt-5.5, repeat=1]"},"event":"task.queued","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":2,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.582Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","data":{"taskCount":1},"event":"project.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":3,"time":"2026-10-11T19:31:43.582Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","data":{},"event":"task.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":4,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.582Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"index":0,"input":{"domain":"chrome","file":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json","task":{"id":"ae78f875-5b98-4907-bbb5-9c737fc68c03","snapshot":"chrome","instruction":"Could you please change the number of search results displayed on one page to 50? I find that having more results visible at once significantly enhances my research efficiency, as it reduces the need to constantly click through multiple pages. ","source":"https://support.google.com/chrome/thread/219988391/increase-search-results-per-page?hl=en","config":[],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"infeasible"},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":5,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.582Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":6,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.task.file","value":"tasks/chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":7,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.evaluator.func","value":"\"infeasible\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":8,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":9,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":10,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":11,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"index":1,"input":{"domain":"chrome","file":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json","task":{"id":"030eeff7-b492-4218-b312-701ec99ee0cc","snapshot":"chrome","instruction":"Can you enable the 'Do Not Track' feature in Chrome to enhance my online privacy?","source":"https://www.surreycc.gov.uk/website/cookies/do-not-track","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"chrome_inject_js","parameters":{"js":"if (document.activeElement && document.activeElement.blur) { document.activeElement.blur(); }","wait_for_load":false}},{"type":"sleep","parameters":{"seconds":1}},{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_do_not_track"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/030eeff7-b492-4218-b312-701ec99ee0cc","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":12,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":13,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.task.file","value":"tasks/chrome/030eeff7-b492-4218-b312-701ec99ee0cc.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":14,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":15,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":16,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":17,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":18,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"index":2,"input":{"domain":"chrome","file":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json","task":{"id":"9656a811-9b5b-4ddf-99c7-5117bcef0626","snapshot":"chrome","instruction":"I want Chrome to warn me whenever I visit a potentially harmful or unsafe website. Can you enable this safety feature?","source":"https://www.quora.com/How-do-I-set-the-security-settings-for-the-Google-Chrome-browser-for-the-best-security#:~:text=Enable%20Safe%20Browsing:%20Chrome%20has%20a%20built%2Din,Security%20%3E%20Security%20%3E%20Enable%20Safe%20Browsing.","config":[{"type":"execute","parameters":{"command":"echo {CLIENT_PASSWORD} | sudo -S apt update -y && echo {CLIENT_PASSWORD} | sudo -S apt install jq -y","shell":true}},{"type":"execute","parameters":{"command":"mkdir -p /home/user/.config/google-chrome/Default && if [ ! -f /home/user/.config/google-chrome/Default/Preferences ]; then echo '{}' > /home/user/.config/google-chrome/Default/Preferences; fi && cd /home/user/.config/google-chrome/Default && jq '. + {\"safebrowsing\":{\"enabled\":false,\"enhanced\":false}}' Preferences > temp && mv temp Preferences","shell":true}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"postconfig":[{"type":"launch","parameters":{"command":["pkill","chrome"]}},{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"sleep","parameters":{"seconds":3}}],"func":"exact_match","result":{"type":"enable_safe_browsing"},"expected":{"type":"rule","rules":{"expected":"true"}}},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":19,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":20,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.task.file","value":"tasks/chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":21,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.evaluator.func","value":"\"exact_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":22,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":23,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":24,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":25,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"index":3,"input":{"domain":"chrome","file":"tasks/chrome/fc6d8143-9452-4171-9459-7f515143419a.json","task":{"id":"fc6d8143-9452-4171-9459-7f515143419a","snapshot":"chrome","instruction":"Find flights from New York–Kennedy Airport to Chicago O'Hare Airport for tomorrow.","source":"test_task_0","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.delta.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"class","class_singleObject":{"mach-flight-context-info__wrapper--date":"time"},"class_multiObject_child":{"mach-flight-context-info__wrapper__info--separator":{"0":"start","1":"end"}}},"expected":{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"tomorrow"},"expected":{"start":"JFK","end":"ORD","time":"{DoW}, {Month} {Day0D}, {Year}"}}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/fc6d8143-9452-4171-9459-7f515143419a","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":26,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":27,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.task.file","value":"tasks/chrome/fc6d8143-9452-4171-9459-7f515143419a.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":28,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":29,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":30,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":31,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"3:chrome%2Ffc6d8143-9452-4171-9459-7f515143419a","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":32,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"index":4,"input":{"domain":"chrome","file":"tasks/chrome/a96b564e-dbe9-42c3-9ccf-b4498073438a.json","task":{"id":"a96b564e-dbe9-42c3-9ccf-b4498073438a","snapshot":"chrome","instruction":"In the FlightAware Discussions forum, navigate to the FlightAware > General category and open the topic with the most posts or replies.","source":"test_task_0","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.flightaware.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_active_tab","result":{"type":"active_tab_info","goto_prefix":"https://"},"expected":{"type":"rule","rules":{"type":"url","url":"https://discussions.flightaware.com/t/the-banter-thread/4412"}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/a96b564e-dbe9-42c3-9ccf-b4498073438a","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":33,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":34,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.task.file","value":"tasks/chrome/a96b564e-dbe9-42c3-9ccf-b4498073438a.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":35,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.evaluator.func","value":"\"is_expected_active_tab\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":36,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":37,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":38,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":39,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"index":5,"input":{"domain":"chrome","file":"tasks/chrome/1704f00f-79e6-43a7-961b-cedd3724d5fd.json","task":{"id":"1704f00f-79e6-43a7-961b-cedd3724d5fd","snapshot":"chrome","instruction":"Find a large car from next Monday to Friday in Zurich, sorted by price.","source":"test_task_0","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.rentalcars.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":["check_direct_json_object","check_direct_json_object"],"conj":"or","result":[{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["locationName","dropLocationName","filterCriteria_carCategory","filterCriteria_sortBy","puDay","puMonth","puYear","doDay","doMonth","doYear"]},{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["locationName","dropLocationName","filterCriteria_carCategory","filterCriteria_sortBy","puDay","puMonth","puYear","doDay","doMonth","doYear"]}],"expected":[{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"next Monday split","to":"next Friday split"},"timezone":"Europe/Zurich","expected":{"locationName":"Zürich","dropLocationName":"Zürich","filterCriteria_carCategory":"large","filterCriteria_sortBy":"PRICE","puDay":"{DayD}","puMonth":"{MonthD}","puYear":"{Year}","doDay":"{DayD}","doMonth":"{MonthD}","doYear":"{Year}"}}},{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"week after next Monday split","to":"week after next Friday split"},"timezone":"Europe/Zurich","expected":{"locationName":"Zürich","dropLocationName":"Zürich","filterCriteria_carCategory":"large","filterCriteria_sortBy":"PRICE","puDay":"{DayD}","puMonth":"{MonthD}","puYear":"{Year}","doDay":"{DayD}","doMonth":"{MonthD}","doYear":"{Year}"}}}]},"proxy":true,"possibility_of_env_change":"medium","fixed_ip":false}},"name":"chrome/1704f00f-79e6-43a7-961b-cedd3724d5fd","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":40,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":41,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.task.file","value":"tasks/chrome/1704f00f-79e6-43a7-961b-cedd3724d5fd.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":42,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.evaluator.func","value":"[\"check_direct_json_object\",\"check_direct_json_object\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":43,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":44,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":45,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"5:chrome%2F1704f00f-79e6-43a7-961b-cedd3724d5fd","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":46,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"index":6,"input":{"domain":"chrome","file":"tasks/chrome/f3b19d1e-2d48-44e9-b4e1-defcae1a0197.json","task":{"id":"f3b19d1e-2d48-44e9-b4e1-defcae1a0197","snapshot":"chrome","instruction":"Find the FAQ page about ticket delivery.","source":"test_task_0","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://premier.ticketek.com.au/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_url_pattern_match","result":{"type":"active_tab_info"},"expected":{"type":"rule","rules":{"expected":["(?i)ticket-delivery(?:-faqs)?"]}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/f3b19d1e-2d48-44e9-b4e1-defcae1a0197","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":47,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":48,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.task.file","value":"tasks/chrome/f3b19d1e-2d48-44e9-b4e1-defcae1a0197.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":49,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.evaluator.func","value":"\"is_expected_url_pattern_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":50,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":51,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":52,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":53,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"index":7,"input":{"domain":"chrome","file":"tasks/chrome/82bc8d6a-36eb-4d2d-8801-ef714fb1e55a.json","task":{"id":"82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","snapshot":"chrome","instruction":"On next Monday, look up a flight from Mumbai to Stockholm.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.qatarairways.com/en-hk/homepage.html"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["fromStation","toStation","departing"],"replace":{"departing":"time"}},"expected":{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"next Monday"},"expected":{"fromStation":["BOM"],"toStation":["STO","ARN"],"time":["{Year}-{Month0D}-{Day0D}"]},"expect_in_result":true}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":54,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":55,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.task.file","value":"tasks/chrome/82bc8d6a-36eb-4d2d-8801-ef714fb1e55a.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":56,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":57,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":58,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":59,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":60,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:31:43.583Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.pod","value":"osworld-9656a811-379d46"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":61,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.317Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/9656a811-9b5b-4ddf-99c7-5117bcef0626/2026-10-11T19-31-43.730Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":62,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.317Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.agent.steps","value":15},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":63,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":64,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"auv.run.ids","value":["09356360-7e20-fd51-e536-951effdfc464","3ab585e0-2ac6-5125-3623-438b41782395","d106b6a2-ec7b-6d8e-9640-24e290c68b50","cfcd13e5-4286-b177-1d8a-8c3c1568c134","6cced7df-a019-cef5-dfec-eda665e627bb","e8cdfce9-9be7-2a90-0e94-e60323b10ae4","e6d3b66c-f9ef-363e-8d97-3860d430c27e","93e50900-0470-2766-8b06-5003073faf44","35b95abc-8d64-40cf-1440-1cea539001a4","22d8325f-1695-4df2-90bb-2d9f667d1b7a","3fbf4db4-85e7-441a-691e-d3985ea05abe","5885e48f-8188-28b0-56a0-202b93ab60b6","7b3bfe2e-16f4-f0c1-e047-b22802bd0643","9936ebb1-9c01-236f-8899-9ee8f90df086"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":65,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":66,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":67,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"2:chrome%2F9656a811-9b5b-4ddf-99c7-5117bcef0626","data":{"index":2,"name":"chrome/9656a811-9b5b-4ddf-99c7-5117bcef0626","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":68,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.318Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"index":8,"input":{"domain":"chrome","file":"tasks/chrome/47543840-672a-467d-80df-8f7c3b9788c9.json","task":{"id":"47543840-672a-467d-80df-8f7c3b9788c9","snapshot":"chrome","instruction":"On the current website, show me the cars available for pickup at Boston Logan Intl Airport from the 10th to the 11th of next month, sorted by the number of seats to find the largest capacity.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.budget.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":["is_expected_url_pattern_match","check_direct_json_object","check_direct_json_object"],"conj":"and","result":[{"type":"active_url_from_accessTree","goto_prefix":"https://www."},{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"class","class_singleObject":{},"class_multiObject":{"location-info":{"0":"start_location","1":"end_location"},"day-time-info":{"0":"from","1":"to"}}},{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"xpath","xpathObject":{"/html/body/div[6]/div[2]/div[1]/div/div/div[2]/section[1]/div[1]/form/div[1]/div[1]/div[2]/div/a":"rank"}}],"expected":[{"type":"rule","rules":{"expected":["reservation#/vehicles"]}},{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"10th next month","to":"11th next month"},"expected":{"start_location":"Boston Logan Intl Airport,\n\t\t\t\t\t\t\t\tBOS \n\t\t\t\t\t\t\t  \n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t \n\t\t\t\t\t\t\t\t \n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t      \n\t\t\t\t\t\t\t        \n\t\t\t\t\t\t\t        \n\t\t\t\t\t\t\t          \n\t\t\t\t\t\t\t            \n\t\t\t\t\t\t\t            Pick-Up","end_location":"Boston Logan Intl Airport,\n\t\t\t\t\t\t\t\tBOS","from":"{DoW}, {Month} {Day0D}, 12:00 PM","to":"{DoW}, {Month} {Day0D}, 12:00 PM"}}},{"type":"rule","rules":{"expected":{"rank":"Number of Seats (High to Low)"}}}]},"proxy":false,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/47543840-672a-467d-80df-8f7c3b9788c9","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":69,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":70,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.task.file","value":"tasks/chrome/47543840-672a-467d-80df-8f7c3b9788c9.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":71,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.evaluator.func","value":"[\"is_expected_url_pattern_match\",\"check_direct_json_object\",\"check_direct_json_object\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":72,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":73,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":74,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"8:chrome%2F47543840-672a-467d-80df-8f7c3b9788c9","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":75,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:34:36.319Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.pod","value":"osworld-f3b19d1e-56a5dc"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":76,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/f3b19d1e-2d48-44e9-b4e1-defcae1a0197/2026-10-11T19-31-43.727Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":77,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.agent.steps","value":12},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":78,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":79,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"auv.run.ids","value":["321881c7-a091-7583-d23d-e6ad203828de","96b3de90-73b0-9d01-a606-eee203f2fdd4","cdef10b7-3a0c-150a-6b08-4f5f34d3557a","b55741cc-a5aa-a0bf-5630-90dc6ed12a46","c89cb02f-f44c-2512-9b69-5c24a7a4375d","b985df6b-797f-0d89-0bd7-0e3cdd71ee84","74c6af62-7dcf-722b-8da6-1e6bd70a383a","1840ab4c-8b19-9713-7e1f-e8945803a02d","77a94f3a-c89d-bd0c-030a-dd30ba2ec845","67c7266f-b8ba-f8ec-9d66-79d08bce5c87","7fbfe49d-67d1-03dd-68ec-d64a65324b2a"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":80,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":81,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":82,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.991Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"6:chrome%2Ff3b19d1e-2d48-44e9-b4e1-defcae1a0197","data":{"index":6,"name":"chrome/f3b19d1e-2d48-44e9-b4e1-defcae1a0197","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":83,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.992Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"index":9,"input":{"domain":"chrome","file":"tasks/chrome/c1fa57f3-c3db-4596-8f09-020701085416.json","task":{"id":"c1fa57f3-c3db-4596-8f09-020701085416","snapshot":"chrome","instruction":"Open the baggage fee calculator in United Airlines website.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.united.com/en/us"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"is_expected_url_pattern_match","result":{"type":"active_tab_info","goto_prefix":"https://www."},"expected":{"type":"rule","rules":{"expected":["united\\.com/en/us/checked-bag-fee-calculator(/.*)?"]}}},"proxy":true,"possibility_of_env_change":"medium","fixed_ip":false}},"name":"chrome/c1fa57f3-c3db-4596-8f09-020701085416","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":84,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.992Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":85,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.task.file","value":"tasks/chrome/c1fa57f3-c3db-4596-8f09-020701085416.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":86,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.evaluator.func","value":"\"is_expected_url_pattern_match\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":87,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":88,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":89,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"9:chrome%2Fc1fa57f3-c3db-4596-8f09-020701085416","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":90,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:36:08.993Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.pod","value":"osworld-030eeff7-bb0813"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":91,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.593Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/030eeff7-b492-4218-b312-701ec99ee0cc/2026-10-11T19-31-43.723Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":92,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.agent.steps","value":36},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":93,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":94,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"auv.run.ids","value":["d066ac34-d263-4aa8-0c73-3169458bc22d","68b030b9-6824-220f-352d-2aaaba0426c2","58fb9c32-5d1e-8faf-d51f-169d95515173","74c5f39c-d1ec-34f9-d148-4d99265f3a67","b8737dd2-be50-8619-0efe-c2d222cae717","e21a4197-75f3-e6c6-d246-23677f058971","131eee20-d470-5c71-94d1-c44a38084e59","7d0584cd-bc14-4880-3097-a2d0ec23faad","f4c69802-a0ce-0c65-2afd-d34548ca8692","a40070db-bc3b-1091-39b8-6158fc4072c0","7a3c8073-ba18-1f6e-1ffa-444be7aa2012","463cbbdd-5faf-eebe-a6db-20f884d18e12","3dcbe336-c718-93ca-6494-796eefe5f974","d8037475-6dc9-1554-32bc-a2a14d88ac65","662afd64-ec9a-ec26-7679-b78578606dca","f6fe55dc-104d-8ece-6259-29eabae863f8","30afc19e-7f90-d601-1ea6-a37660af65d2","b645caf6-7396-1a21-9605-b9bd78ed55ce","fe3a333f-083c-c159-8b96-0abb57fde82a","8195093f-22c0-a9a4-6308-8ec494b719bc","1557e2be-7b2b-ab95-52e6-4e32249f9029","ee08d293-af07-6bdc-8cfe-80f0fa61d39e","5dd67a34-10f0-31a7-20be-a3934b3e63a6","2df42faf-a179-d363-1113-febe8d2bcf5b","c68769b2-1cf8-bbdf-44cc-5c25a1aa4c95","3b38ab74-2310-9454-75ba-ca3ca0006b43","1fc27498-1f9d-f9ff-cc33-26b26380fa6d","2eaf1139-0d37-cc1b-6d45-3f45b16856f5","ae525f10-1fd7-f36e-f42b-208c278f41ef","5732b2f8-8766-cbab-2b6b-fd2c7b3110ca","62d0e113-1a6c-7aea-84c1-0185f993089b","11e3a8a4-463d-0776-ea13-35d49434287f","67f1746c-5a88-f554-4ba4-466735f7acd1","10774639-fda1-2a82-8db3-a13bbf21d054","224a3456-6255-57bb-6bec-ac901f881d77"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":95,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":96,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":97,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"1:chrome%2F030eeff7-b492-4218-b312-701ec99ee0cc","data":{"index":1,"name":"chrome/030eeff7-b492-4218-b312-701ec99ee0cc","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":98,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.594Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"index":10,"input":{"domain":"chrome","file":"tasks/chrome/da46d875-6b82-4681-9284-653b0c7ae241.json","task":{"id":"da46d875-6b82-4681-9284-653b0c7ae241","snapshot":"chrome","instruction":"Book an appointment to apply for a transportation access pass at the Charlie Card store on the first Monday eight months later at any available time from 9:00 am to 12:00 pm, fill in my details (James Smith, james.smith@gmail.com). And do not click \"book\" directly. Let me review it.","source":"test_task_2","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.mbta.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"conj":"and","func":["is_expected_url_pattern_match","check_direct_json_object","check_direct_json_object","check_direct_json_object"],"result":[{"type":"active_url_from_accessTree"},{"type":"active_tab_html_parse","category":"class","class_multiObject_only_child":{"HAZ16":{"0":"content"}}},{"type":"active_tab_html_parse","category":"class","class_multiObject_only_child":{"HAZ16":{"1":"time"}}},{"type":"active_tab_html_parse","category":"input","inputObject":{"/html/body/div[2]/div/form/div[7]/div/div/div[1]/input[1]":"name","/html/body/div[2]/div/form/div[7]/div/div/div[1]/input[2]":"mail"}}],"expected":[{"type":"rule","rules":{"expected":["book/CharlieCardStoreAppointments@mbta.com/"]}},{"type":"rule","rules":{"expected":{"content":"Apply for Transportation Access Pass (TAP) CharlieCard non-auto approval"}}},{"type":"rule_relativeTime","rules":{"expected":{"time":["{MonthFull} {Day0D}, 9:00 AM","{MonthFull} {Day0D}, 9:15 AM","{MonthFull} {Day0D}, 9:30 AM","{MonthFull} {Day0D}, 9:45 AM","{MonthFull} {Day0D}, 10:00 AM","{MonthFull} {Day0D}, 10:15 AM","{MonthFull} {Day0D}, 10:30 AM","{MonthFull} {Day0D}, 10:45 AM","{MonthFull} {Day0D}, 11:00 AM","{MonthFull} {Day0D}, 11:15 AM","{MonthFull} {Day0D}, 11:30 AM","{MonthFull} {Day0D}, 11:45 AM","{MonthFull} {Day0D}, 12:00 PM"]},"relativeTime":{"from":"first monday eight months later"},"expect_in_result":true}},{"type":"rule","rules":{"expected":{"name":"James Smith","mail":"james.smith@gmail.com"}}}]},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/da46d875-6b82-4681-9284-653b0c7ae241","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":99,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":100,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.task.file","value":"tasks/chrome/da46d875-6b82-4681-9284-653b0c7ae241.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":101,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.evaluator.func","value":"[\"is_expected_url_pattern_match\",\"check_direct_json_object\",\"check_direct_json_object\",\"check_direct_json_object\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":102,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":103,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":104,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"10:chrome%2Fda46d875-6b82-4681-9284-653b0c7ae241","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":105,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:37:52.595Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.pod","value":"osworld-a96b564e-99fa48"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":106,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/a96b564e-dbe9-42c3-9ccf-b4498073438a/2026-10-11T19-31-43.726Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":107,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.agent.steps","value":30},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":108,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":109,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"auv.run.ids","value":["20a2c74b-40f5-fc97-c4de-3cafb4fe92f3","37f95add-7363-9408-151d-fc55a5e1ed2a","5a50cd53-2ec5-f978-7d3c-ea3894f8a0d7","1dee0662-853c-8b74-81f9-0dab91395465","79079097-f326-053e-8717-b1468f489e33","4357905f-f218-fb89-0b5b-7bb16defbdf3","745c0d3d-9a65-e421-d177-41ac5a11ce00","cb3bc12f-4a97-73fd-f658-590917216c03","82df1c52-627e-6b0f-a5ad-eab72a95dffe","622a5337-c816-88ca-e5be-f6f0c3dd8d53","443fa610-bc06-d710-3e50-962232b230b1","819d7849-f018-a72e-5fba-a857a54fa672","c731a497-e6e3-eff6-346c-721efac9eb22","c0b25702-5708-eb81-6ca8-a2c6ab42328c","2a060dc4-3048-af16-8985-9353b4cee657","5fffa7e9-2070-ef21-2243-07d79b0d9e99","bf4d8e85-27b0-f2ed-82c2-0f284dc0c629","b2576050-cc86-1bdc-f9fc-26b065f42a7b","d226e082-1ac1-2ce5-986c-33053fc93cbe","71d8c2fc-bede-f911-6cb5-bcefbdbbb7ef","8a79dfc5-3889-c3bb-d001-34c10b26d026","39c7bd5d-28ba-738c-8448-a8a65e3cea85","547e95a9-dbc5-e893-4d96-2cbe16d0a655","12dd4087-f644-12c2-f749-0fcaba531cbb","ad0fa48c-ec22-52f0-1560-b321b4ba8997","d3d86a6c-7066-4cad-74be-8461b0f5ae28","d0a780b5-d721-2acd-bcd4-c9355477a2e6","6f989cb8-76c8-e6d5-7a68-10aacad1334f","babe4fa1-7a2c-680b-a6af-31ec70b5fab6"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":110,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":111,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"name":"osworld.success","value":1},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":112,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"4:chrome%2Fa96b564e-dbe9-42c3-9ccf-b4498073438a","data":{"index":4,"name":"chrome/a96b564e-dbe9-42c3-9ccf-b4498073438a","state":"passed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":113,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"index":11,"input":{"domain":"chrome","file":"tasks/chrome/6c4c23a1-42a4-43cc-9db1-2f86ff3738cc.json","task":{"id":"6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","snapshot":"chrome","instruction":"Find flights from Seattle to New York on 5th next month and only show those that can be purchased with miles.","source":"test_task_1","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.delta.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"class","class_singleObject":{"mach-flight-context-info__wrapper--date":"time","mach-global-tabs-small__wrapper__tab--active":"category"},"class_multiObject_child":{"mach-flight-context-info__wrapper__info--separator":{"0":"start","1":"end"}}},"expected":{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"5th next month"},"expected":{"start":"SEA","end":"NYC","time":"{DoW}, {Month} {Day0D}, {Year}","category":"Miles"}}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":114,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.786Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":115,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.task.file","value":"tasks/chrome/6c4c23a1-42a4-43cc-9db1-2f86ff3738cc.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":116,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":117,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":118,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":119,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"11:chrome%2F6c4c23a1-42a4-43cc-9db1-2f86ff3738cc","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":120,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:38:04.787Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.pod","value":"osworld-82bc8d6a-1f5bdf"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":121,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/82bc8d6a-36eb-4d2d-8801-ef714fb1e55a/2026-10-11T19-31-43.729Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":122,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.agent.steps","value":28},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":123,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.agent.last_action","value":"DONE"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":124,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"auv.run.ids","value":["b14a99d4-e004-77a5-9fda-a8c775e54e4c","7085eda9-09e0-ca6c-eb91-34ba330808bd","3c926368-ee2d-1706-97b4-d9d8546e55c9","3a11d6f3-2a96-f78a-a455-2a8d61ae2d88","e5657894-c00f-f714-f19b-5d62dfda0881","9abbc5ea-be61-fbf7-8154-b0b3bb644b58","c0a1ec26-3e00-2299-2a65-38b6e3cae820","d63d4174-6fe0-6a99-fd41-5983df26dc2f","6ce3d0a9-92d9-7da6-6d82-e95f2cb77c43","fff88bde-bde2-cbc3-28bc-169e5c6ee303","aecc9873-2c80-4178-f9d1-ba77cf24dfd4","2189ac00-06bf-501b-4819-638cf60f0b21","28e5f77d-95ed-2f2d-95ce-32a42750730c","42bd4b72-674d-a586-f1fc-b262ab7f1573","a2322838-0caf-d4a2-ce4d-676517a48eca","bd5cf952-d7a9-1e62-fba0-ff3d9293f3bd","31058be6-b1d2-2547-f6af-d2b9aceb50f5","35406297-0e0f-0ead-c5c1-ff6c0c23a7de","5829084a-1549-6cb0-ec49-add93a6a400d","67ce3cfc-b0a2-cd6c-38e7-2c7bc04beb3a","749a7c39-9184-6a63-ceb0-51a340b29cc8","12b382c8-4c0d-4bed-522b-6fa76474d1b6","bb5e42c0-c71d-5f68-4460-f246d63e6217","8099d765-b8da-8e4a-4cda-c9e3111e4002","be6707f9-b69d-9fb2-6d52-2427cdbe19f3","27f9d6a8-fdd7-fffb-e1e6-3a08bfdae1b5","a7e6cd7c-5963-7add-2c9d-76e2fab0cad3"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":125,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":126,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"name":"osworld.success","value":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":127,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.062Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"7:chrome%2F82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":7,"name":"chrome/82bc8d6a-36eb-4d2d-8801-ef714fb1e55a","state":"failed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":128,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"index":12,"input":{"domain":"chrome","file":"tasks/chrome/f79439ad-3ee8-4f99-a518-0eb60e5652b0.json","task":{"id":"f79439ad-3ee8-4f99-a518-0eb60e5652b0","snapshot":"chrome","instruction":"Search for a one way flight from Dublin to Vienna on 10th next month for 2 adults.","source":"test_task_2","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.ryanair.com/gb/en"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":"check_direct_json_object","result":{"type":"active_tab_url_parse","goto_prefix":"https://www.","parse_keys":["originIata","destinationIata","tpAdults","tpTeens","tpChildren","tpStartDate","isReturn"],"replace":{"tpStartDate":"time"}},"expected":{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"10th next month"},"expected":{"originIata":"DUB","destinationIata":"VIE","tpAdults":"2","tpTeens":"0","tpChildren":"0","time":"{Year}-{Month0D}-{DayD}","isReturn":"false"}}}},"proxy":true,"fixed_ip":false,"possibility_of_env_change":"low"}},"name":"chrome/f79439ad-3ee8-4f99-a518-0eb60e5652b0","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":129,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":130,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.task.file","value":"tasks/chrome/f79439ad-3ee8-4f99-a518-0eb60e5652b0.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":131,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.evaluator.func","value":"\"check_direct_json_object\""},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":132,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":133,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":134,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"12:chrome%2Ff79439ad-3ee8-4f99-a518-0eb60e5652b0","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":135,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:54.063Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.pod","value":"osworld-ae78f875-6abb3a"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":136,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.evidence","value":"__VIEVAL_DIR__/episodes/ae78f875-5b98-4907-bbb5-9c737fc68c03/2026-10-11T19-31-43.722Z-gpt-5.5-r1"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":137,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.agent.steps","value":50},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":138,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.agent.last_action","value":"none"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":139,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"auv.run.ids","value":["29016aee-d86c-67e4-343f-57927f6bbe7e","35027e88-5a7c-7535-9b6f-c7a4363ce41c","f8c65638-574c-eaf3-6f68-5c515270f550","d7e9e5de-8523-3b43-1245-36389ae2fb79","d51417bf-bc97-a9d4-b4f4-d40d64115167","3f1f08b9-299d-2b90-6167-c3ca2636340e","8be2aa48-d098-d4fc-fb26-6c184a360296","9c7fbf3e-0a69-128b-eb9a-21a6889c4c02","d2fe19f2-20ee-c604-8866-48e2aab77c2b","9d05e1d3-44d6-1d2f-8593-7d826989742f","37e3a9b1-4460-5c08-05ef-052c9ac3f426","5fc8e43c-5fdb-b831-be55-caee6ca86df9","ac116868-ea62-c169-4623-5b101cc67b5f","12ea7226-a238-847c-8357-d71211772651","13b1c180-4973-952d-305b-65afb9296438","f15550c6-b7c9-6805-753f-d27c64b8b1d8","22c84e2e-3910-30ed-571e-e325ab23d503","fd9e426d-a634-8514-483c-0bed04940ec4","c8b9a6f0-d66b-326b-386d-db8e79c44d36","737d552b-a933-6441-5fd4-573d74ec9998","f7d70474-0e17-4992-ddf0-83e96fea8770","4f881b40-1842-9659-7f30-ba09c59ebdb0","0fdc247e-6eb5-7be3-a976-b6e8f73e7605","42de4150-6491-57e1-363a-c581e1794bb6","039a6f68-5ead-26d7-c427-754e40184044","20114020-c011-2fa9-5027-6e230f21cd16","d293b0b7-db74-4667-ad81-79d12239f6a2","7d33033a-89cd-449f-5d89-91479319df72","a3c5f997-8740-7cc9-3b17-119fed48c035","8e8edd10-bef9-f378-01b0-f69311e2e3f9","ee483958-0f78-2f40-ffd2-12c417301bf7","b5ba6be9-eeb8-a533-95f3-a5143bfcdb57","19acad27-efbf-edec-0674-20b5e7cedc00","c3af01d2-473e-740e-b754-7f176f9ec20c","c980424c-91ae-1b24-db0a-c42cc4c6c859","a9ccddde-799f-9d9e-d58c-fb3a4fab0846","829641f5-3fbf-a4e7-7d8f-5986e1543b96","8c34075a-729e-7760-f3c2-07d6597654e3","77d7ae1d-6d77-3ba7-cec6-7dff0b887128","8adafe02-7d2f-2ecc-cbfc-720e1ad44f95","dbf9d2b2-8d74-1df5-da5f-8932d0fb3b47","b81ab5ad-8d51-6896-dc79-62f1eacd4889","9eecbd0f-7652-0fe0-7897-bea48f2124d2","b4b46475-c98f-449f-cfc6-c742befbb4b9","6be12809-860f-155a-b94a-c5e2a1419341","360bd81f-6a82-0e39-5e4b-87ddbaee8957","c2264406-c2c6-6a8f-9f47-5cdba646e7e7","df110396-bc98-8a41-7990-43586224e648","845ec44b-c44a-5d80-8e39-ef6f76950ce7","147da4a6-7e9e-efdc-181b-4ed653d92e7e"]},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":140,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"auv.linux_binary.sha256","value":"07d549bbb44e45df1cfd53060b0de060357fd24d59e174b341ab24a2ceb27c86"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":141,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"name":"osworld.success","value":0},"event":"score.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":142,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"0:chrome%2Fae78f875-5b98-4907-bbb5-9c737fc68c03","data":{"errorMessage":"expected +0 to be 1 // Object.is equality","index":0,"name":"chrome/ae78f875-5b98-4907-bbb5-9c737fc68c03","state":"failed","total":345},"event":"case.ended","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":143,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.836Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"index":13,"input":{"domain":"chrome","file":"tasks/chrome/b7895e80-f4d1-4648-bee0-4eb45a6f1fa8.json","task":{"id":"b7895e80-f4d1-4648-bee0-4eb45a6f1fa8","snapshot":"chrome","instruction":"Find a Hotel in New York City with lowest price possible for 2 adults next weekend. Sort the search results by price and stay on the results page.","source":"test_task_0","config":[{"type":"launch","parameters":{"command":["google-chrome","--remote-debugging-port=1337"]}},{"type":"launch","parameters":{"command":["socat","tcp-listen:9222,fork","tcp:localhost:1337"]}},{"type":"chrome_open_tabs","parameters":{"urls_to_open":["https://www.tripadvisor.com/"]}},{"type":"activate_window","parameters":{"window_name":"Google Chrome"}}],"trajectory":"trajectories/","related_apps":["chrome"],"evaluator":{"func":["check_direct_json_object","check_direct_json_object","check_direct_json_object","check_direct_json_object"],"conj":"or","result":[{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"xpath","xpathObject":{"//button[@data-automation='checkin']//div[contains(@class,'Wh')]//span":"from","//button[@data-automation='checkout']//div[contains(@class,'Wh')]//span":"to","//h2[@data-automation='header_geo_title']":"city","//button[@data-automation='roomsandguests']//div[contains(@class,'Wh')]":"adult","//button[contains(@aria-label,'PRICE_LOW_TO_HIGH: Price (low to high)') or contains(@aria-label,'PRICE')]//div[contains(@class,'biGQs') and contains(@class,'SewaP')]":"rank"}},{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"xpath","xpathObject":{"//button[@data-automation='checkin']//div[contains(@class,'Wh')]":"from","//button[@data-automation='checkout']//div[contains(@class,'Wh')]":"to","//h2[@data-automation='header_geo_title']":"city","//button[@data-automation='roomsandguests']//div[contains(@class,'Wh')]":"adult","//button[contains(@aria-label,'PRICE_LOW_TO_HIGH: Price (low to high)') or contains(@aria-label,'PRICE')]//div[contains(@class,'biGQs') and contains(@class,'SewaP')]":"rank"}},{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"xpath","xpathObject":{"//button[@data-automation='checkin']//div[contains(@class,'Wh')]//span":"from","//button[@data-automation='checkout']//div[contains(@class,'Wh')]//span":"to","//h2[@data-automation='header_geo_title']":"city","//button[@data-automation='roomsandguests']//div[contains(@class,'Wh')]":"adult","//button[contains(@aria-label,'PRICE_LOW_TO_HIGH: Price (low to high)') or contains(@aria-label,'PRICE')]//div[contains(@class,'biGQs') and contains(@class,'SewaP')]":"rank"}},{"type":"active_tab_html_parse","goto_prefix":"https://www.","category":"xpath","xpathObject":{"//button[@data-automation='checkin']//div[contains(@class,'Wh')]":"from","//button[@data-automation='checkout']//div[contains(@class,'Wh')]":"to","//h2[@data-automation='header_geo_title']":"city","//button[@data-automation='roomsandguests']//div[contains(@class,'Wh')]":"adult","//button[contains(@aria-label,'PRICE_LOW_TO_HIGH: Price (low to high)') or contains(@aria-label,'PRICE')]//div[contains(@class,'biGQs') and contains(@class,'SewaP')]":"rank"}}],"expected":[{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"next week Saturday","to":"next week Sunday"},"timezone":"America/New_York","expected":{"from":"{DoW}, {Month} {Day0D}","to":"{DoW}, {Month} {Day0D}","city":"New York City Hotels","adult":"Rooms/Guests1 Room, 2 Guests","rank":"Price (low to high)"}}},{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"next week Friday","to":"next week Sunday"},"timezone":"America/New_York","expected":{"from":"Check In{DoW}, {Month} {Day0D}","to":"Check Out{DoW}, {Month} {Day0D}","city":"New York City Hotels","adult":"Rooms/Guests1 Room, 2 Guests","rank":"Price (low to high)"}}},{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"week after next Saturday","to":"week after next Sunday"},"timezone":"America/New_York","expected":{"from":"{DoW}, {Month} {Day0D}","to":"{DoW}, {Month} {Day0D}","city":"New York City Hotels","adult":"Rooms/Guests1 Room, 2 Guests","rank":"Price (low to high)"}}},{"type":"rule_relativeTime","rules":{"relativeTime":{"from":"week after next Friday","to":"week after next Sunday"},"timezone":"America/New_York","expected":{"from":"Check In{DoW}, {Month} {Day0D}","to":"Check Out{DoW}, {Month} {Day0D}","city":"New York City Hotels","adult":"Rooms/Guests1 Room, 2 Guests","rank":"Price (low to high)"}}}]},"proxy":true,"possibility_of_env_change":"high","fixed_ip":false}},"name":"chrome/b7895e80-f4d1-4648-bee0-4eb45a6f1fa8","total":345},"event":"case.started","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":144,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.domain","value":"chrome"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":145,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.task.file","value":"tasks/chrome/b7895e80-f4d1-4648-bee0-4eb45a6f1fa8.json"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":146,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.evaluator.func","value":"[\"check_direct_json_object\",\"check_direct_json_object\",\"check_direct_json_object\",\"check_direct_json_object\"]"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":147,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.model","value":"gpt-5.5"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":148,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.model.api","value":"responses"},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":149,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
+{"attemptId":"attempt-2026-10-11T19-31-43-580Z","caseId":"13:chrome%2Fb7895e80-f4d1-4648-bee0-4eb45a6f1fa8","data":{"name":"osworld.repeat","value":1},"event":"metric.recorded","experimentId":"matrix-run-model-gpt-5.5-repeat-1","projectId":"osworld-kubernetes","runId":"run-1791747103580-dd3a8517","schemaVersion":3,"seq":150,"taskId":"evals%2Fosworld%2Feval%2Fosworld::default::run=model%3Dgpt-5.5%26repeat%3D1::eval=default","time":"2026-10-11T19:39:57.837Z","workspaceId":"default-workspace"}
diff --git a/packages/ui/test/helpers.ts b/packages/ui/test/helpers.ts
index ae48e48..a87f801 100644
--- a/packages/ui/test/helpers.ts
+++ b/packages/ui/test/helpers.ts
@@ -1,4 +1,6 @@
+import type { RunHandle, RunSummaryInfo } from '../src/server/store'
 import type { RunEvent } from '../src/shared/contract'
+import type { RunReducer } from '../src/shared/reduce'
 
 import { cp, mkdtemp, readdir, readFile, rm, writeFile } from 'node:fs/promises'
 import { tmpdir } from 'node:os'
@@ -15,7 +17,7 @@ export const fixturesDirectory = fileURLToPath(new URL('./fixtures/', import.met
 export const OSWORLD_RUNS = {
   /** Interrupted by SIGINT after 18 cases; has `run-summary.json`, `cases.jsonl`, `otlp/`. */
   archived: 'default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-14-05-991Z/run-1791746045991-a77ba0a4',
-  /** Event schema version 1. */
+  /** Event schema version 1: written by an older vieval, listed as unsupported. */
   legacy: 'default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-10T18-20-50-009Z/run-1791656450009-5299b70e',
   /** Still running when copied: `events.jsonl` only, seq 1..150 of 360 cases. */
   running: 'default-workspace/osworld-kubernetes/matrix-run-model-gpt-5.5-repeat-1/attempt-2026-10-11T19-31-43-580Z/run-1791747103580-dd3a8517',
@@ -60,7 +62,7 @@ export function createEnvelopeFactory(runId = 'run-test') {
       event,
       experimentId: 'exp-a',
       runId,
-      schemaVersion: 2,
+      schemaVersion: 3,
       seq,
       time,
       workspaceId: 'ws-a',
@@ -69,6 +71,28 @@ export function createEnvelopeFactory(runId = 'run-test') {
   }
 }
 
+/**
+ * A run handle over a reducer built in the test, as the in-process bus gives one, for view builders.
+ */
+export function createReducerHandle(reducer: RunReducer, options: { runKey?: string, summary?: RunSummaryInfo } = {}): RunHandle {
+  return {
+    artifactsRoot: () => undefined,
+    parseErrors: () => 0,
+    projectSegment: 'demo',
+    reducer: () => reducer,
+    refresh: async () => {},
+    reportDirectory: undefined,
+    reportPath: undefined,
+    reportRoot: undefined,
+    runKey: options.runKey ?? reducer.identity()?.runId ?? 'run-test',
+    source: () => 'live-bus',
+    state: () => reducer.run().terminated ? 'ended' : 'running',
+    subscribe: () => () => {},
+    summary: () => options.summary,
+    unsupportedVersion: () => undefined,
+  }
+}
+
 /**
  * Creates an empty temporary directory removed by {@link cleanupTemporaryDirectories}.
  */
diff --git a/packages/ui/test/projects/showcase/README.md b/packages/ui/test/projects/showcase/README.md
index 2a851c3..f400015 100644
--- a/packages/ui/test/projects/showcase/README.md
+++ b/packages/ui/test/projects/showcase/README.md
@@ -15,7 +15,7 @@ One run takes about 25 seconds.
 | `evals/artifacts.eval.ts` | `artifacts` | One case for each artifact preview kind: PNG `image`, a sequence of 6 WebP screenshots, WebM `video`, WAV `audio` (generated at run time, because `*.wav` is gitignored), `jsonl` with 20,500 or 21,000 rows, nested `json`, `yaml`, `toml`, plain `text` (400 lines), `html` with an inline script (the sandbox must block it: the page stays beige and says "Scripts are blocked"), and `binary` (`application/octet-stream`). The text case also writes `unannounced-notes.md` without `recordArtifact`, so the viewer lists it with `source: 'directory'`. |
 | `evals/retries.eval.ts` | `retries` | `autoAttempt: 2`: the task runs 3 rounds, because one case fails in rounds 1 and 2. `autoRetry` cases fail on their first tries and pass on a retry. One case fails for `mock:model-b` in every try. |
 | `evals/logs.eval.ts` | `logs` | `console.log/info/debug/warn/error` output, multi-line and Unicode output, one case with 3,201 lines, and a slow case (24 s for `repeat=1`, 12 s for `repeat=2`) that records `slow.step_latency_ms` every second. Use it to look at a run while it is in progress. |
-| `evals/cases.eval.ts` | `cases` | 64 short cases (a long case list). Each case records the numeric metrics `latency_ms` and `tokens` once per step, the non-numeric metrics `finish_reason`, `cached`, and `tools_used`, and `exact` and `judge` scores. `mock:model-b` gets lower scores and more failures. |
+| `evals/cases.eval.ts` | `cases` | 64 short cases (a long case list). Each case records the numeric metrics `latency_ms` and `tokens` once per step, the non-numeric metrics `finish_reason`, `cached`, and `tools_used`, and the named scores `showcase.correct` (rule grader, with a rationale) and `showcase.quality` (model grader; a few cases record it as unscored, `invalid_response`). The primary score `showcase.pass` is derived: `all` of correct ≥ 0.7 and quality ≥ 0.6. `mock:model-b` gets lower scores and more failures. |
 
 The flaky and retry outcomes are deterministic: each case counts its executions in module scope (`evals/shared.ts`).
 The `SHOWCASE_SEED` environment variable changes the scores of the `cases` task, so runs with different seeds
diff --git a/packages/ui/test/projects/showcase/evals/cases.eval.ts b/packages/ui/test/projects/showcase/evals/cases.eval.ts
index 6c3b4dd..296b334 100644
--- a/packages/ui/test/projects/showcase/evals/cases.eval.ts
+++ b/packages/ui/test/projects/showcase/evals/cases.eval.ts
@@ -2,7 +2,7 @@ import process from 'node:process'
 
 import { setTimeout as delay } from 'node:timers/promises'
 
-import { caseOf, describeTask, expect } from 'vieval'
+import { caseOf, defineScore, describeTask, expect } from 'vieval'
 
 import { modelOf, seeded } from './shared'
 
@@ -11,6 +11,24 @@ const seed = process.env.SHOWCASE_SEED ?? ''
 
 const categories = ['math', 'reading', 'coding', 'planning'] as const
 
+const correct = defineScore('showcase.correct', {
+  description: { 'en': 'How close the answer is to the reference answer.', 'zh-Hans': '答案与参考答案的接近程度。' },
+  grader: { id: 'matcher', kind: 'rule', version: '1' },
+  label: { 'en': 'Correct answer', 'zh-Hans': '答案正确' },
+})
+const quality = defineScore('showcase.quality', {
+  description: { 'en': 'The judge rates how clear and complete the answer is.', 'zh-Hans': '评委评价答案是否清楚、完整。' },
+  grader: { id: 'judge', kind: 'model', model: 'mock:judge' },
+  label: { 'en': 'Answer quality', 'zh-Hans': '回答质量' },
+})
+const pass = defineScore('showcase.pass', {
+  // A bare input passes at 1, so a partly correct answer fails unless its threshold allows it.
+  combine: { all: [{ atLeast: 0.7, score: correct }, { atLeast: 0.6, score: quality }] },
+  description: { 'en': 'The answer is at least 70% correct and the judge rates it at least 0.6.', 'zh-Hans': '答案至少 70% 正确，且评委评分不低于 0.6。' },
+  label: { 'en': 'Pass', 'zh-Hans': '通过' },
+  primary: true,
+})
+
 // 64 cheap cases: a long case list with metrics and scores that differ by model.
 const items = Array.from({ length: 64 }, (_, index) => ({
   category: categories[index % categories.length],
@@ -33,17 +51,23 @@ describeTask('cases', () => {
         context.recordMetric('tokens', Math.round(50 + seeded(`${item.id}:${step}:tokens`) * 450))
       }
 
-      const exact = Math.min(1, Math.max(0, Math.round((skill + (noise - 0.5) * 0.6) * 100) / 100))
-      const judge = Math.min(1, Math.max(0, Math.round((skill + (noise - 0.5) * 0.3 + 0.05) * 100) / 100))
+      const correctness = Math.min(1, Math.max(0, Math.round((skill + (noise - 0.5) * 0.6) * 100) / 100))
+      const rating = Math.min(1, Math.max(0, Math.round((skill + (noise - 0.5) * 0.3 + 0.05) * 100) / 100))
       context.recordMetric('finish_reason', noise > 0.9 ? 'length' : 'stop')
       context.recordMetric('cached', noise < 0.3)
       context.recordMetric('tools_used', noise > 0.5 ? ['search', 'calculator'] : ['search'])
-      context.recordScore(exact, 'exact')
-      context.recordScore(judge, 'judge')
-      console.info(`${item.id}: ${steps} steps, exact=${exact}, judge=${judge}`)
+      context.recordScore(correct, correctness, { rationale: `${Math.round(correctness * 100)}% of the reference terms appear in the answer.` })
+      // A few answers cut off by the length limit leave the judge without a verdict: unscored, not a wrong answer.
+      if (noise > 0.95) {
+        context.recordUnscored(quality, 'invalid_response', { rationale: 'The judge reply had no rating.' })
+      }
+      else {
+        context.recordScore(quality, rating, { metadata: { votes: [rating, rating] }, rationale: rating >= 0.6 ? 'Clear and complete.' : 'Misses part of the question.' })
+      }
+      console.info(`${item.id}: ${steps} steps, correct=${correctness}, quality=${rating}`)
 
-      expect(exact, `${item.id} scored below the pass threshold`).toBeGreaterThanOrEqual(0.4)
-      return { answer: `answer to ${item.question}`, exact, judge, steps }
+      expect(correctness, `${item.id} scored below the pass threshold`).toBeGreaterThanOrEqual(0.4)
+      return { answer: `answer to ${item.question}`, correct: correctness, quality: rating, steps }
     }, { input: { category: item.category, question: item.question } })
   }
-})
+}, { scores: [correct, quality, pass] })
diff --git a/packages/vieval/README.md b/packages/vieval/README.md
index c1fd627..7e7c1bc 100644
--- a/packages/vieval/README.md
+++ b/packages/vieval/README.md
@@ -141,6 +141,33 @@ describeTask('arithmetic-quality', ({ casesFromInputs }) => {
 })
 ```
 
+Declare the scores a task measures with `defineScore`. Each score is a named value in `0..1` where higher is better; one score per task is `primary` and becomes the headline, shown next to the completion rate. A derived score (`combine: { all | mean | weighted }`) is computed by the runtime when the case try settles:
+
+```ts
+import { caseOf, defineScore, describeTask } from 'vieval'
+
+const correct = defineScore('formula.correct', { label: { 'en': 'Correct', 'zh-Hans': '正确' } })
+const natural = defineScore('formula.natural', { grader: { id: 'judge', kind: 'model' } })
+const pass = defineScore('formula.pass', {
+  combine: { all: [correct, { atLeast: 0.5, score: natural }] },
+  label: 'Pass',
+  primary: true,
+})
+
+describeTask('read-formula', () => {
+  caseOf('fraction', async (context) => {
+    const { listenerScore, verdict } = await readAloud('1/2')
+    context.recordScore(correct, listenerScore)
+    if (verdict == null)
+      context.recordUnscored(natural, 'invalid_response')
+    else
+      context.recordScore(natural, verdict.score, { rationale: verdict.reason })
+  })
+}, { scores: [correct, natural, pass] })
+```
+
+A failed case counts `0` in every score. An errored or timed-out case is left out (`errors: 'exclude'`, the default) or counts `0` (`errors: 'zero'`), and lowers the completion rate either way. An unscored value never counts. Latency and cost are metrics (`defineMetric` with `better: 'lower'`), not scores.
+
 `describeEval` remains exported as an alias of `describeTask`, but new examples should prefer `describeTask` because task/case semantics are the primary runtime model.
 
 Every `describeTask` call in an eval file is its own task:
@@ -215,7 +242,8 @@ Each scheduled task receives stable matrix metadata:
 - `project`: the complete unit of work: discovery rules (`include`, `exclude`, `root`), `runMatrix` / `evalMatrix`, `models`, `inferenceExecutors`, optional `executor`, `plugins`, `reporters`, and `scheduler`.
 - `experiment`: run metadata derived from explicit `--experiment` or, when omitted, stable matrix row metadata. It does not create an extra scheduler layer.
 - `task`: one eval definition discovered from files and expanded across inference executors plus run/eval matrix rows.
-- `case`: the scoring and evidence source inside a task. `context.recordScore(...)` contributes normalized score evidence; `context.recordMetric(...)` emits benchmark metadata for reports.
+- `case`: the scoring and evidence source inside a task. `context.recordScore(score, value)` records a named score declared with `defineScore` (a value in `0..1`, higher is better; one score per task is the primary score); `context.recordUnscored(score, reason)` records that the grader could not produce one; `context.recordMetric(...)` emits benchmark data such as latency or cost for reports.
+- `completion rate`: the share of cases that produced a result (`passed` or `failed`, not `errored` or `timeout`). It is shown next to the primary score everywhere.
 - `attempt`: a full task rerun used to estimate reliability. With `autoAttempt`, each full attempt contributes evidence, so a fail-then-pass pair scores as a success rate rather than replacing the earlier failure.
 - `retry`: an in-case retry. With `autoRetry`, a case can recover inside one attempt; a retry pass still counts as that attempt passing.
 
@@ -293,6 +321,7 @@ If a project provides no `executor`, `vieval run` still discovers eval files, sc
 
 ```ts
 import { defineConfig } from 'vieval'
+import { scoreCases } from 'vieval/core/scoring'
 
 export default defineConfig({
   projects: [
@@ -315,7 +344,8 @@ export default defineConfig({
           id: task.id,
           inferenceExecutorId: task.inferenceExecutor.id,
           matrix: task.matrix,
-          scores: [{ kind: 'exact', score: success ? 1 : 0 }],
+          // One measurement as one passed outcome; `scoreCases` builds `scores`, `primary`, `completion`, and `passed`.
+          ...scoreCases([{ scores: { 'motion.success': success ? 1 : 0 }, state: 'passed', unscored: {} }]),
         }
       },
       include: ['evals/*.eval.ts'],
@@ -396,7 +426,7 @@ export default defineConfig({
 
 - One live terminal window shows `Workspaces`, `Projects`, `Tasks`, `Cases`, `Concurrency`, and `Time` rows for all workspaces.
 - A failing workspace does not stop the others.
-- At the end, a leaderboard prints one row per workspace and project, sorted by hybrid score, then the failed workspaces with the last lines of their stderr.
+- At the end, a leaderboard prints one row per workspace and project, ranked by the primary score with the completion rate in its own column, then the failed workspaces with the last lines of their stderr. If the workspaces have different primary scores, the leaderboard says so and ranks by passed cases.
 - The exit code is `1` when any workspace failed or had failing cases.
 - `--json` prints the comparison output.
 - Reports go to `.vieval/reports` next to the comparison config. The experiment defaults to the comparison id, and all workspaces share one attempt.
@@ -426,7 +456,7 @@ vieval viewer [reportRoot...] [--port <port>] [--host <host>] [--open] [--eviden
 vieval report analyze <report-directory>
 vieval report index <report-directory> [--output <path>] [--format table|json|jsonl]
 vieval report cases <report-directory> [--where <key=value>] [--group-by <key>] [--format table|json|jsonl]
-vieval report compare <left-report-directory> <right-report-directory> [--case-key <key>] [--score-kind <kind>] [--format table|json]
+vieval report compare <left-report-directory> <right-report-directory> [--case-key <key>] [--score <name>] [--format table|json]
 ```
 
 `vieval run --config <comparison config>` runs a comparison (see [Workspaces and Comparisons](#workspaces-and-comparisons)); there is no separate `vieval compare` command. `vieval report compare` diffs two existing reports.
@@ -437,9 +467,10 @@ vieval report compare <left-report-directory> <right-report-directory> [--case-k
 - Runs of one invocation share `experimentId`, `attemptId`, and `invocationId`. Runs started by a comparison also carry `comparisonId`; there, the experiment defaults to the comparison id and all workspaces share one attempt.
 - `events.jsonl` is appended while the run is in progress, so a crashed run keeps the events already flushed to disk. Each line has a `seq` that orders it within the run.
 - `run-summary.json`, `cases.jsonl`, `metrics-summary.json`, and `otlp/*.json` are derived from `events.jsonl` when the run ends.
-- `run-summary.json` has `schemaVersion: 1`, `runId`, `reportDirectory`, and exactly one project. `vieval report` commands reject other versions.
+- `run-summary.json` has `schemaVersion: 2`, `runId`, `reportDirectory`, and exactly one project. It holds the score summaries (`{ name, mean, n, total, sum, unscored }`) of every task run, the announced `scoreDefinitions`, and the resolved headline (`{ headline, completion }`) per project and for the run. `vieval report` commands reject other versions.
 - The same `workspaceId`, `experimentId`, `attemptId`, and `runId` identify the run in every file: the top-level fields of `run-summary.json`, every event envelope, every `cases.jsonl` record, and the `vieval.workspace.id` / `vieval.experiment.id` / `vieval.attempt.id` / `vieval.run.id` attributes of the OTLP resources and the `vieval.run` span.
-- For a task with auto attempts, a `cases.jsonl` record describes the last attempt and adds `attemptIndex`, `autoAttempt`, and `attempts` (state, scores, retries, and timing of each attempt). `retryCount` counts only the retries within the last attempt. `CaseRecord.schemaVersion` stays `1`, because the new fields are optional.
+- For a task with auto attempts, a `cases.jsonl` record describes the last attempt and adds `attemptIndex`, `autoAttempt`, and `attempts` (state, scores, retries, and timing of each attempt). `retryCount` counts only the retries within the last attempt.
+- `cases.jsonl` records have `schemaVersion: 2`. `scores` maps score names to the counted value (a failed case counts `0`), `unscored` maps score names to the reason the grader gave none, and `scoreDetails` keeps the `rationale`, `grader`, and `metadata` of each record.
 - `artifacts/<task>/<case>/` holds case files (see [Case ids and artifacts](#case-ids-and-artifacts)).
 
 `vieval run --json` prints `invocationId` and a `projects` list; each project has its own `runId` and `reportDirectory`. With a comparison config, `--json` prints the comparison output.
@@ -518,11 +549,12 @@ await runVievalCli({
 
 ## Public Entrypoints
 
-- `vieval`: `defineConfig`, `loadEnv`, `requiredEnvFrom`, `describeTask`, `describeEval`, `caseOf`, `casesFromInputs`, and `expect`.
+- `vieval`: `defineConfig`, `loadEnv`, `requiredEnvFrom`, `describeTask`, `describeEval`, `caseOf`, `casesFromInputs`, `defineScore`, `defineMetric`, and `expect`.
 - `vieval/config`: lower-level `defineEval`, `defineTask`, matrix types, task context types, model definitions, and plugin contracts.
 - `vieval/plugins/chat-models`: `ChatModels`, `ChatProviders`, `chatModelFrom`, `chatProviderFrom`, `chatModelMatrix`, runtime config helpers, and chat telemetry helpers.
 - `vieval/core/runner`: collection, scheduling, task context, cache runtime, scheduler, execution, and aggregation utilities.
-- `vieval/core/assertions`: assertion primitives and pipeline helpers.
+- `vieval/core/assertions`: assertion primitives and pipeline helpers. Outcomes carry a `score`; record it with `context.recordScore(definition, outcome.score)`.
+- `vieval/core/scoring`: the one scoring rule (`scoreCases`), derived scores (`deriveScores`), pooling (`poolScoreSummaries`), and headline resolution (`resolveHeadline`). Side-effect free.
 - `vieval/events`: run event definitions, payload types, and `events.jsonl` schemas.
 - `vieval/core/inference-executors`: env helpers and remote provider executors.
 - `vieval/testing/expect-extensions`: Vitest expect extensions for testing eval behavior.
@@ -620,10 +652,10 @@ See [`examples/case-output`](../../examples/case-output) for a runnable demo.
 
 Every activity of a run is one typed event on an in-process bus (`@moeru/eventa`). The terminal reporter, the report writer (`events.jsonl`), and the Vitest compat bridge are subscribers of that bus.
 
-- Event names are `<entity>.<past-tense verb>`: `run.started`, `project.started`, `task.queued`, `task.started`, `case.started`, `case.output`, `metric.defined`, `metric.recorded`, `score.recorded`, `case.ended`, `task.ended`, `project.ended`, `run.ended`, plus `inference.*` and `tool-call.*` from `vieval/plugins/chat-models`.
-- Definitions, payload types, and Valibot schemas are exported from `vieval/events`. Every line of `events.jsonl` is an envelope with `schemaVersion: 2`, a per-run `seq`, run identity, and `projectId` / `taskId` / `caseId` when they apply.
+- Event names are `<entity>.<past-tense verb>`: `run.started`, `project.started`, `task.queued`, `task.started`, `case.started`, `case.output`, `metric.defined`, `metric.recorded`, `score.defined`, `score.recorded`, `case.ended`, `task.ended`, `project.ended`, `run.ended`, plus `inference.*` and `tool-call.*` from `vieval/plugins/chat-models`.
+- Definitions, payload types, and Valibot schemas are exported from `vieval/events`. Every line of `events.jsonl` is an envelope with `schemaVersion: 3`, a per-run `seq`, run identity, and `projectId` / `taskId` / `caseId` when they apply.
 - When a task runs auto attempts (a case has `autoAttempt > 0`), `case.started` and `case.ended` of every case of the task have `attemptIndex` (0 for the first attempt) and `autoAttempt` (the attempts that the case allows). Events of other tasks do not have these fields. `retryIndex` stays the retry within one attempt.
-- Schema version policy: a new optional payload field or a new event name does not change `schemaVersion`. Readers of version 2 ignore unknown payload fields and skip unknown events, so they can read newer logs. A renamed, removed, or retyped field increments the version. `attemptIndex` and `autoAttempt` are optional fields of version 2.
+- Schema version policy: a new optional payload field or a new event name does not change `schemaVersion`. Readers of version 3 ignore unknown payload fields and skip unknown events, so they can read newer logs. A renamed, removed, or retyped field increments the version. Version 3 replaced the `exact` / `judge` score kinds with named scores (`score.defined`, and `score.recorded` with `name` and `value` or `unscored`); readers reject older envelopes with "This report was written by an older vieval. Run it again."
 - Task and case code receives `context.events`, scoped to its task and case. Emits never reject and never affect scoring.
 - A custom project `executor` forwards `context.events` into `task.entry.task.run(...)`, so case progress and output reach the terminal and the report.
 - Plugins define their own events with `defineEventa` from `vieval/events`:
@@ -638,7 +670,7 @@ Every activity of a run is one typed event on an in-process bus (`@moeru/eventa`
   })
   ```
 
-When not to use it: events are observations. Do not use them to change task results; use `recordScore` / `recordMetric` for that.
+When not to use it: events are observations. Do not use them to change task results; use `recordScore` / `recordUnscored` / `recordMetric` for that.
 
 ## Examples In This Repository
 
diff --git a/packages/vieval/package.json b/packages/vieval/package.json
index cb30533..e28e377 100644
--- a/packages/vieval/package.json
+++ b/packages/vieval/package.json
@@ -43,10 +43,6 @@
       "types": "./dist/core/assertions/index.d.mts",
       "default": "./dist/core/assertions/index.mjs"
     },
-    "./core/processors/results": {
-      "types": "./dist/core/processors/results/index.d.mts",
-      "default": "./dist/core/processors/results/index.mjs"
-    },
     "./events": {
       "types": "./dist/events/index.d.mts",
       "default": "./dist/events/index.mjs"
diff --git a/packages/vieval/src/cli/comparison.test.ts b/packages/vieval/src/cli/comparison.test.ts
index a2326eb..c8fe3b9 100644
--- a/packages/vieval/src/cli/comparison.test.ts
+++ b/packages/vieval/src/cli/comparison.test.ts
@@ -1,3 +1,5 @@
+import type { HeadlineResult, ScoreSummary } from '../core/scoring'
+import type { ComparisonOutput } from './comparison'
 import type { RunSummaryFile } from './report-artifacts'
 
 import { existsSync } from 'node:fs'
@@ -6,6 +8,7 @@ import { createRequire } from 'node:module'
 import { tmpdir } from 'node:os'
 import { join } from 'node:path'
 import { fileURLToPath } from 'node:url'
+import { stripVTControlCharacters } from 'node:util'
 
 import { glob } from 'tinyglobby'
 import { afterEach, describe, expect, it } from 'vitest'
@@ -184,3 +187,102 @@ describe('runComparison', () => {
     await expect(runComparison({ configFilePath: join(directory, 'vieval.config.ts'), reporter: quietReporter, resolveCommand: sourceCommand, workspace: ['llm'] })).rejects.toThrow('No workspace matches "llm". Workspaces: raw.')
   })
 })
+
+/**
+ * Creates a finished comparison whose workspaces each have one project with `headline` and a passed share of
+ * `passed` over 4 cases.
+ */
+function comparisonOutputOf(workspaces: Array<{ headline: HeadlineResult, id: string, passed: number }>): ComparisonOutput {
+  return {
+    attemptId: 'attempt-1',
+    comparisonId: 'comparison-1',
+    configFilePath: '/tmp/vieval.config.ts',
+    experimentId: 'experiment-1',
+    invocationId: 'invocation-1',
+    reportRoot: null,
+    workspaces: workspaces.map(workspace => ({
+      exitCode: 0,
+      id: workspace.id,
+      output: {
+        configFilePath: '/tmp/vieval.config.ts',
+        projects: [{
+          caseSummary: { errored: 0, failed: 4 - workspace.passed, passed: workspace.passed, skipped: 0, timeout: 0, total: 4 },
+          discoveredEvalFileCount: 1,
+          entryCount: 1,
+          errorMessage: null,
+          executed: true,
+          matrixSummary: null,
+          name: 'speech',
+          result: {
+            inferenceExecutors: [],
+            overall: { completion: workspace.headline.completion, headline: workspace.headline, runCount: 1, scores: [] },
+            runs: [{
+              completion: workspace.headline.completion,
+              entryId: 'speech',
+              id: 'speech',
+              inferenceExecutorId: 'default',
+              matrix: { eval: {}, meta: { evalRowId: 'default', runRowId: 'default' }, run: {} },
+              passed: summaryOf('passed', workspace.passed, 4),
+              scores: [],
+            }],
+          },
+          scoreDefinitions: [{ label: 'Pass', name: 'tts.pass', primary: true }],
+          taskCount: 1,
+        }],
+      },
+      signal: null,
+      stderrTail: [],
+    })),
+  }
+}
+
+/** Creates one score summary over `n` counted cases of `total`. */
+function summaryOf(name: string, sum: number, n: number, total = n, unscored: Record<string, number> = {}): ScoreSummary {
+  return { mean: n === 0 ? null : sum / n, n, name, sum, total, unscored }
+}
+
+/**
+ * @example
+ * describe('formatComparisonOutput', () => {})
+ */
+describe('formatComparisonOutput', () => {
+  /**
+   * @example
+   * it('ranks by the shared primary score with the completion rate in its own column', () => {})
+   */
+  it('ranks by the shared primary score with the completion rate in its own column', () => {
+    const completion = summaryOf('completion', 3, 4, 4, { errored: 1 })
+    const table = stripVTControlCharacters(formatComparisonOutput(comparisonOutputOf([
+      { headline: { completion, headline: { kind: 'score', score: summaryOf('tts.pass', 1, 4) } }, id: 'raw', passed: 4 },
+      { headline: { completion, headline: { kind: 'score', score: summaryOf('tts.pass', 3, 4) } }, id: 'rule', passed: 1 },
+    ])))
+
+    /**
+     * @example
+     * expect(table).toContain(' 1. rule/speech  Pass 75%  completed 75% (1 errored)')
+     */
+    expect(table).toContain(' 1. rule/speech  Pass 75%  completed 75% (1 errored)')
+    expect(table).toContain(' 2. raw/speech  Pass 25%  completed 75% (1 errored)')
+    expect(table).not.toContain('Primary scores differ')
+  })
+
+  /**
+   * @example
+   * it('says when primary scores differ and ranks by passed cases', () => {})
+   */
+  it('says when primary scores differ and ranks by passed cases', () => {
+    const completion = summaryOf('completion', 4, 4)
+    const table = stripVTControlCharacters(formatComparisonOutput(comparisonOutputOf([
+      { headline: { completion, headline: { kind: 'score', score: summaryOf('tts.pass', 4, 4) } }, id: 'raw', passed: 1 },
+      { headline: { completion, headline: { kind: 'score', score: summaryOf('tts.correct', 1, 4) } }, id: 'rule', passed: 3 },
+    ])))
+
+    /**
+     * @example
+     * expect(table).toContain('Primary scores differ (tts.pass, tts.correct), so the rows rank by passed cases.')
+     */
+    expect(table).toContain('Primary scores differ (tts.pass, tts.correct), so the rows rank by passed cases.')
+    expect(table).toContain(' 1. rule/speech  passed 75% (tts.correct 25%)  completed 100%')
+    expect(table).toContain(' 2. raw/speech  passed 25% (Pass 100%)  completed 100%')
+  })
+})
diff --git a/packages/vieval/src/cli/comparison.ts b/packages/vieval/src/cli/comparison.ts
index 972a2a8..8473999 100644
--- a/packages/vieval/src/cli/comparison.ts
+++ b/packages/vieval/src/cli/comparison.ts
@@ -23,12 +23,13 @@ import { glob } from 'tinyglobby'
 
 import packageJSON from '../../package.json'
 
+import { poolScoreSummaries } from '../core/scoring'
 import { runEnded, runStarted } from '../events'
 import { detectCliConfigMode, loadRawVievalConfig, supportedConfigFileNames } from './config'
 import { exitCodeFromSignal, listenForInterrupt } from './interrupt'
 import { resolveRunReportDirectory, sanitizeIdentitySegment } from './report-artifacts'
 import { resolveResumedAttemptId } from './resume'
-import { createRunReporter, forwardRunEventToReporter, hasRunFailures } from './run'
+import { createRunReporter, formatCompletion, formatHeadline, forwardRunEventToReporter, hasRunFailures } from './run'
 
 /**
  * File descriptor a workspace process sends its events and final output on.
@@ -156,24 +157,51 @@ export function comparisonExitCode(output: ComparisonOutput): number {
 }
 
 /**
- * Formats the leaderboard of a comparison: one row per project of every workspace, best hybrid score first, then
- * every workspace that failed with its reason.
+ * Formats the leaderboard of a comparison: one row per project of every workspace, then every workspace that failed
+ * with its reason.
+ *
+ * Rows rank by the mean of the primary score when every project with a result has the same primary score. Otherwise
+ * the leaderboard says that the primary scores differ and ranks by the share of passed cases, the one value every
+ * project has. The completion rate stands in its own column, because a score over few completed cases is not
+ * comparable.
  */
 export function formatComparisonOutput(output: ComparisonOutput): string {
-  const rows = output.workspaces.flatMap(workspace => (workspace.output?.projects ?? []).map(project => ({ project, workspace })))
-  rows.sort((left, right) => (right.project.result?.overall.hybridAverage ?? -1) - (left.project.result?.overall.hybridAverage ?? -1))
-  const score = (value: null | number | undefined) => value == null ? '-' : value.toFixed(3)
+  const rows = output.workspaces.flatMap(workspace => (workspace.output?.projects ?? []).map((project) => {
+    const headline = project.result?.overall.headline
+    const passed = poolScoreSummaries((project.result?.runs ?? []).flatMap(run => run.passed == null ? [] : [run.passed]))[0]
+    return {
+      headline,
+      passed,
+      primary: headline?.headline.kind === 'score' ? headline.headline.score : undefined,
+      project,
+      workspace,
+    }
+  }))
+  const scored = rows.filter(row => row.headline != null)
+  const primaryNames = [...new Set(scored.flatMap(row => row.primary == null ? [] : [row.primary.name]))]
+  const ranksByPrimary = primaryNames.length === 1 && scored.every(row => row.primary != null)
+  const rankValue = (row: typeof rows[number]) => (ranksByPrimary ? row.primary?.mean : row.passed?.mean) ?? -1
+  rows.sort((left, right) => rankValue(right) - rankValue(left))
+
   const lines = [
     ` ${c.dim('COMPARE')}  ${c.yellow(output.comparisonId)}  ${c.dim(`${output.workspaces.length} workspaces, invocation ${output.invocationId}`)}`,
     '',
-    ...rows.map(({ project, workspace }, index) => {
+    ...(ranksByPrimary || primaryNames.length === 0
+      ? []
+      : [` ${c.yellow(`Primary scores differ (${primaryNames.join(', ')}), so the rows rank by passed cases.`)}`, '']),
+    ...rows.map((row, index) => {
+      const { headline, project, workspace } = row
       const cases = project.caseSummary
       const unmeasured = (cases?.errored ?? 0) + (cases?.timeout ?? 0)
       const failed = project.errorMessage != null || (cases != null && (cases.failed > 0 || unmeasured > 0))
-      // A score over few measured cases is not comparable, so the coverage stands next to it.
-      const coverage = project.result?.overall.coverage
-      const coverageLabel = coverage == null || coverage.total === 0 ? '' : `  coverage ${score(coverage.measured / coverage.total)}`
-      return ` ${failed ? c.red('✗') : c.green('✓')} ${String(index + 1).padStart(2)}. ${workspace.id}/${project.name}  hybrid ${score(project.result?.overall.hybridAverage)}  exact ${score(project.result?.overall.exactAverage)}${coverageLabel}  cases ${cases?.passed ?? 0}/${cases?.total ?? 0}${cases != null && cases.failed > 0 ? c.red(` (${cases.failed} failed)`) : ''}${unmeasured > 0 ? c.yellow(` (${unmeasured} without result)`) : ''}${project.errorMessage == null ? '' : c.red(`  ${project.errorMessage}`)}${workspace.output?.interruptSignal == null ? '' : c.yellow(`  interrupted by ${workspace.output.interruptSignal}`)}`
+      const definitions = project.scoreDefinitions ?? []
+      const rankLabel = headline == null
+        ? 'n/a'
+        : ranksByPrimary || row.passed == null
+          ? formatHeadline(headline, definitions)
+          : `${formatHeadline({ ...headline, headline: { kind: 'passed-cases', score: row.passed } }, definitions)}${row.primary == null ? '' : c.dim(` (${formatHeadline(headline, definitions)})`)}`
+      const completionLabel = headline == null ? '' : `  ${formatCompletion(headline.completion)}`
+      return ` ${failed ? c.red('✗') : c.green('✓')} ${String(index + 1).padStart(2)}. ${workspace.id}/${project.name}  ${rankLabel}${completionLabel}  cases ${cases?.passed ?? 0}/${cases?.total ?? 0}${cases != null && cases.failed > 0 ? c.red(` (${cases.failed} failed)`) : ''}${project.errorMessage == null ? '' : c.red(`  ${project.errorMessage}`)}${workspace.output?.interruptSignal == null ? '' : c.yellow(`  interrupted by ${workspace.output.interruptSignal}`)}`
     }),
     ...output.workspaces.filter(workspace => workspace.output == null).flatMap(workspace => [
       ` ${c.red('✗')} ${workspace.id}  ${c.red(workspace.errorMessage ?? 'failed')}`,
diff --git a/packages/vieval/src/cli/report-artifacts.ts b/packages/vieval/src/cli/report-artifacts.ts
index 4f75d06..9f8d528 100644
--- a/packages/vieval/src/cli/report-artifacts.ts
+++ b/packages/vieval/src/cli/report-artifacts.ts
@@ -1,5 +1,6 @@
-import type { RunEventEnvelope } from '../events'
-import type { CliRunOutput } from './run'
+import type { HeadlineResult } from '../core/scoring'
+import type { RunEventEnvelope, ScoreDefinedData } from '../events'
+import type { CliProjectSummary, CliRunOutput } from './run'
 
 import { existsSync, readFileSync } from 'node:fs'
 import { mkdir, writeFile } from 'node:fs/promises'
@@ -9,8 +10,9 @@ import { glob } from 'tinyglobby'
 
 import * as v from 'valibot'
 
+import { emptyScoreSummary, resolveHeadline } from '../core/scoring'
 import { caseStarted, caseStartedDataSchema } from '../events'
-import { readReportEvents } from './report-log'
+import { OLDER_REPORT_MESSAGE, readReportEvents } from './report-log'
 import { buildLocalOtlpProjection } from './report-otlp'
 import { buildCaseRecords, buildMetricsSummary, encodeJsonl } from './report-records'
 
@@ -38,21 +40,43 @@ export interface ReportRunArtifact {
  * Content of one run's `run-summary.json`: the invocation output narrowed to the run's one project, plus the run's
  * own id and directory.
  */
-export interface RunSummaryFile extends CliRunOutput {
+export interface RunSummaryFile extends Omit<CliRunOutput, 'projects'> {
+  /**
+   * Headline of the run over every project, resolved with `resolveHeadline` from the projects' task runs.
+   */
+  headline: HeadlineResult
+  /** One summary per project; a run holds one project. */
+  projects: RunSummaryProject[]
   /** Directory of this run's report. */
   reportDirectory: string
   /** Id of this run (one project of the invocation). */
   runId: string
   /**
-   * Version of this file's shape; readers reject versions they do not know.
+   * Version of this file's shape; readers reject other versions.
    *
-   * 1: one project per run, `invocationId` shared by the runs of one invocation.
+   * - 1: one project per run, `invocationId` shared by the runs of one invocation.
+   * - 2: named scores. `result` holds `ScoreSummary` lists; each project has `scoreDefinitions` and `headline`, and
+   *   the run has `scoreDefinitions` and `headline`.
    */
   schemaVersion: typeof RUN_SUMMARY_SCHEMA_VERSION
+  /**
+   * Latest `score.defined` of each score name over every project, in first-announced order; labels {@link headline}.
+   */
+  scoreDefinitions: ScoreDefinedData[]
+}
+
+/**
+ * One project of {@link RunSummaryFile}.
+ */
+export interface RunSummaryProject extends CliProjectSummary {
+  /** Headline of the project (`result.overall.headline`), or `null` when the project has no result. */
+  headline: HeadlineResult | null
+  /** Latest `score.defined` of each score name the project announced. */
+  scoreDefinitions: NonNullable<CliProjectSummary['scoreDefinitions']>
 }
 
 /** Current version of {@link RunSummaryFile}. */
-export const RUN_SUMMARY_SCHEMA_VERSION = 1
+export const RUN_SUMMARY_SCHEMA_VERSION = 2
 
 /**
  * One line of `events.jsonl`.
@@ -99,10 +123,7 @@ export async function readReportArtifacts(reportPath: string): Promise<ReportRun
  */
 export function readReportRunArtifact(summaryFilePath: string): ReportRunArtifact {
   const reportDirectory = resolve(summaryFilePath, '..')
-  const summary = JSON.parse(readFileSync(summaryFilePath, 'utf-8')) as RunSummaryFile
-  if (summary.schemaVersion !== RUN_SUMMARY_SCHEMA_VERSION) {
-    throw new Error(`Unsupported run summary "${summaryFilePath}": schema version ${String(summary.schemaVersion ?? 'none')}, this vieval reads ${RUN_SUMMARY_SCHEMA_VERSION}. Use the vieval version that wrote it.`)
-  }
+  const summary = readRunSummary(summaryFilePath)
   const events = readReportEvents(resolve(reportDirectory, 'events.jsonl'))
 
   return {
@@ -114,6 +135,29 @@ export function readReportRunArtifact(summaryFilePath: string): ReportRunArtifac
   }
 }
 
+/**
+ * Reads one `run-summary.json`.
+ *
+ * Use when:
+ * - a report command needs a run's official numbers, headline, or score definitions
+ *
+ * Returns:
+ * - the parsed summary
+ * - throws {@link OLDER_REPORT_MESSAGE} for a summary of an older schema version, and an unsupported-version error
+ *   for a newer one
+ */
+export function readRunSummary(summaryFilePath: string): RunSummaryFile {
+  const summary = JSON.parse(readFileSync(summaryFilePath, 'utf-8')) as Partial<RunSummaryFile>
+  const schemaVersion = summary.schemaVersion as number | undefined
+  if (schemaVersion == null || schemaVersion < RUN_SUMMARY_SCHEMA_VERSION) {
+    throw new Error(`${OLDER_REPORT_MESSAGE} ("${summaryFilePath}", run summary schema version ${String(schemaVersion ?? 'none')})`)
+  }
+  if (schemaVersion !== RUN_SUMMARY_SCHEMA_VERSION) {
+    throw new Error(`Unsupported run summary "${summaryFilePath}": schema version ${schemaVersion}, this vieval reads ${RUN_SUMMARY_SCHEMA_VERSION}. Use the vieval version that wrote it.`)
+  }
+  return summary as RunSummaryFile
+}
+
 /**
  * Resolves the directory that holds one run's report artifacts.
  *
@@ -240,9 +284,17 @@ export async function writeRunReportArtifacts(
   const events = readReportEvents(resolve(reportDirectory, 'events.jsonl'))
   const persistedOutput: RunSummaryFile = {
     ...output,
+    headline: resolveRunHeadline(output.projects),
+    projects: output.projects.map(project => ({
+      ...project,
+      headline: project.result?.overall.headline ?? null,
+      scoreDefinitions: project.scoreDefinitions ?? [],
+    })),
     reportDirectory,
     runId: identity.runId,
     schemaVersion: RUN_SUMMARY_SCHEMA_VERSION,
+    // A later project's definition of a name replaces an earlier one, as viewers keep the latest per name.
+    scoreDefinitions: [...new Map(output.projects.flatMap(project => (project.scoreDefinitions ?? []).map(definition => [definition.name, definition] as const))).values()],
   }
 
   await mkdir(reportDirectory, { recursive: true })
@@ -274,6 +326,22 @@ export async function writeRunReportArtifacts(
   return reportDirectory
 }
 
+/**
+ * Resolves the headline of a run over the task runs of every project.
+ *
+ * Expects:
+ * - task runs of all projects; the entry id names a task, as in `aggregateRunResults`
+ */
+function resolveRunHeadline(projects: readonly CliProjectSummary[]): HeadlineResult {
+  return resolveHeadline(projects.flatMap(project => (project.result?.runs ?? []).map(run => ({
+    completion: run.completion ?? emptyScoreSummary('completion'),
+    passed: run.passed ?? emptyScoreSummary('passed'),
+    ...(run.primary == null ? {} : { primary: run.primary }),
+    scores: run.scores,
+    taskId: run.entryId,
+  }))))
+}
+
 /**
  * Collects auto attempt statistics from the `attemptIndex` of `case.started` events.
  *
diff --git a/packages/vieval/src/cli/report-attempts.test.ts b/packages/vieval/src/cli/report-attempts.test.ts
index f025d7a..45a9f33 100644
--- a/packages/vieval/src/cli/report-attempts.test.ts
+++ b/packages/vieval/src/cli/report-attempts.test.ts
@@ -30,6 +30,7 @@ async function writeRun(root: string, options: {
     await writeFile(join(runDirectory, 'run-summary.json'), JSON.stringify({
       projects: [{ caseSummary: { ...cases, skipped: 0, total: cases.errored + cases.failed + cases.passed + cases.timeout } }],
       ...(options.resumedFrom == null ? {} : { resumedFrom: options.resumedFrom }),
+      schemaVersion: 2,
     }))
   }
   await utimes(attemptDirectory, new Date(options.modifiedAt), new Date(options.modifiedAt))
@@ -54,7 +55,7 @@ describe('listAttempts', () => {
       ['attempt-1', false, ['llm', 'rule']],
     ])
     expect(rows[1]!.cases).toEqual({ errored: 2, failed: 1, passed: 9, timeout: 1, total: 13 })
-    expect(rows[1]!.coverage).toBeCloseTo(10 / 13)
+    expect(rows[1]!.completion).toBeCloseTo(10 / 13)
     expect(rows[0]!.resumedFrom).toBe('attempt-1')
     expect(formatAttemptsTable(rows)).toContain('resumed attempt-1')
   })
@@ -72,7 +73,25 @@ describe('listAttempts', () => {
     const [llm] = await listAttempts(root, { workspace: ['llm'] })
 
     expect(llm!.unfinishedRuns).toBe(1)
-    expect(llm!.coverage).toBeNull()
+    expect(llm!.completion).toBeNull()
     expect(formatAttemptsTable([llm!])).toContain('1 run without summary')
   })
+
+  /**
+   * @example
+   * a run-summary.json of schema version 1 is rejected, as every report reader rejects older reports.
+   */
+  it('rejects a run summary written by an older vieval', async () => {
+    const root = await mkdtemp(join(tmpdir(), 'vieval-attempts-'))
+    temporaryDirectories.push(root)
+    const runDirectory = join(root, 'llm', 'project', 'experiment-a', 'attempt-1', 'run-a')
+    await mkdir(runDirectory, { recursive: true })
+    await writeFile(join(runDirectory, 'run-summary.json'), JSON.stringify({ projects: [], schemaVersion: 1 }))
+
+    /**
+     * @example
+     * await expect(listAttempts(root)).rejects.toThrow('This report was written by an older vieval. Run it again.')
+     */
+    await expect(listAttempts(root)).rejects.toThrow('This report was written by an older vieval. Run it again.')
+  })
 })
diff --git a/packages/vieval/src/cli/report-attempts.ts b/packages/vieval/src/cli/report-attempts.ts
index 32fd318..e1d5b51 100644
--- a/packages/vieval/src/cli/report-attempts.ts
+++ b/packages/vieval/src/cli/report-attempts.ts
@@ -1,8 +1,6 @@
-import type { RunSummaryFile } from './report-artifacts'
-
 import process from 'node:process'
 
-import { existsSync, readFileSync } from 'node:fs'
+import { existsSync } from 'node:fs'
 import { stat } from 'node:fs/promises'
 import { dirname, join, resolve } from 'node:path'
 
@@ -10,6 +8,8 @@ import meow from 'meow'
 
 import { glob } from 'tinyglobby'
 
+import { readRunSummary } from './report-artifacts'
+
 const reportAttemptsHelpText = `
   Usage
     $ vieval report attempts [reportRoot] [options]
@@ -31,8 +31,8 @@ export interface AttemptRow {
   attemptId: string
   /** Cases summed over the runs that wrote a `run-summary.json`. */
   cases: { errored: number, failed: number, passed: number, timeout: number, total: number }
-  /** Share of settled cases with a result, or `null` when no case settled. */
-  coverage: null | number
+  /** Completion rate: the share of settled cases with a result, or `null` when no case settled. */
+  completion: null | number
   experimentId: string
   /** Signal that interrupted the attempt, when a run recorded one. */
   interruptSignal?: string
@@ -58,7 +58,7 @@ export function formatAttemptsTable(rows: readonly AttemptRow[]): string {
     return 'No attempts found.'
   }
 
-  const header = ['EXPERIMENT', 'ATTEMPT', '', 'WORKSPACES', 'CASES', 'PASSED', 'FAILED', 'NO RESULT', 'COVERAGE', 'NOTES']
+  const header = ['EXPERIMENT', 'ATTEMPT', '', 'WORKSPACES', 'CASES', 'PASSED', 'FAILED', 'NO RESULT', 'COMPLETED', 'NOTES']
   const lines = rows.map((row) => {
     const notes = [
       ...(row.resumedFrom == null ? [] : [`resumed ${row.resumedFrom}`]),
@@ -74,7 +74,7 @@ export function formatAttemptsTable(rows: readonly AttemptRow[]): string {
       String(row.cases.passed),
       String(row.cases.failed),
       String(row.cases.errored + row.cases.timeout),
-      row.coverage == null ? '-' : `${(row.coverage * 100).toFixed(1)}%`,
+      row.completion == null ? '-' : `${(row.completion * 100).toFixed(1)}%`,
       notes.join('; '),
     ]
   })
@@ -113,7 +113,7 @@ export async function listAttempts(reportRoot: string, filters: { experiment?: s
     const row = rows.get(key) ?? {
       attemptId,
       cases: { errored: 0, failed: 0, passed: 0, timeout: 0, total: 0 },
-      coverage: null,
+      completion: null,
       experimentId,
       latest: false,
       modifiedAt: 0,
@@ -132,7 +132,7 @@ export async function listAttempts(reportRoot: string, filters: { experiment?: s
         row.unfinishedRuns += 1
         continue
       }
-      const summary = JSON.parse(readFileSync(summaryPath, 'utf-8')) as RunSummaryFile
+      const summary = readRunSummary(summaryPath)
       row.interruptSignal ??= summary.interruptSignal
       row.resumedFrom ??= summary.resumedFrom
       for (const project of summary.projects) {
@@ -147,7 +147,7 @@ export async function listAttempts(reportRoot: string, filters: { experiment?: s
   const seenExperiments = new Set<string>()
   for (const row of sorted) {
     const settled = row.cases.passed + row.cases.failed + row.cases.errored + row.cases.timeout
-    row.coverage = settled === 0 ? null : (row.cases.passed + row.cases.failed) / settled
+    row.completion = settled === 0 ? null : (row.cases.passed + row.cases.failed) / settled
     row.workspaces.sort()
     // Rows are newest first, so the first row of an experiment is the one `--resume` picks.
     row.latest = !seenExperiments.has(row.experimentId)
diff --git a/packages/vieval/src/cli/report-case-compare.test.ts b/packages/vieval/src/cli/report-case-compare.test.ts
index 3a13c23..5521f64 100644
--- a/packages/vieval/src/cli/report-case-compare.test.ts
+++ b/packages/vieval/src/cli/report-case-compare.test.ts
@@ -1,32 +1,113 @@
-import { describe, expect, it } from 'vitest'
+import type { CaseRecord } from './report-records'
 
-import { buildCaseComparison, formatCaseComparisonTable } from './report-case-compare'
+import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+
+import { afterEach, describe, expect, it } from 'vitest'
+
+import { buildCaseComparison, formatCaseComparisonTable, resolveComparedScore } from './report-case-compare'
+
+const temporaryDirectories: string[] = []
+
+afterEach(async () => {
+  await Promise.all(temporaryDirectories.splice(0).map(directory => rm(directory, { force: true, recursive: true })))
+})
+
+/**
+ * Creates one current case record; `overrides` replace its fields.
+ */
+function recordOf(overrides: Partial<CaseRecord>): CaseRecord {
+  return {
+    attemptId: 'a',
+    caseId: 'case',
+    caseName: 'Case',
+    durationMs: 1,
+    endedAt: 'e',
+    experimentId: 'e',
+    metrics: {},
+    projectName: 'p',
+    retryCount: 0,
+    runId: 'r',
+    schemaVersion: 2,
+    scoreDetails: {},
+    scores: {},
+    startedAt: 's',
+    state: 'passed',
+    taskId: 't',
+    unscored: {},
+    workspaceId: 'w',
+    ...overrides,
+  }
+}
+
+/**
+ * Writes a minimal `run-summary.json` whose one project has a `score` headline over `primary`, or no headline.
+ */
+async function writeRunSummary(directory: string, primary: string | undefined): Promise<void> {
+  await mkdir(directory, { recursive: true })
+  const score = { mean: 1, n: 1, name: primary, sum: 1, total: 1, unscored: {} }
+  const completion = { mean: 1, n: 1, name: 'completion', sum: 1, total: 1, unscored: {} }
+  await writeFile(join(directory, 'run-summary.json'), JSON.stringify({
+    headline: null,
+    projects: [{ headline: primary == null ? null : { completion, headline: { kind: 'score', score } }, name: 'p' }],
+    schemaVersion: 2,
+  }), 'utf-8')
+}
 
 /**
  * @example
- * describe('report compare') verifies generic case score diffs between two runs.
+ * describe('buildCaseComparison', () => {})
  */
-describe('report compare', () => {
+describe('buildCaseComparison', () => {
   /**
    * @example
-   * it('reports matched case score deltas and changed metrics') checks the core regression analysis contract.
+   * it('reports matched case deltas of the named score and changed metrics', () => {})
    */
-  it('reports matched case score deltas and changed metrics', () => {
+  it('reports matched case deltas of the named score and changed metrics', () => {
     const comparison = buildCaseComparison({
       caseKey: 'benchmark.case.id',
-      left: [{ attemptId: 'a', caseId: 'left-1', caseName: 'Case', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.case.id': 'case-a', 'prediction': 'wrong' }, projectName: 'p', retryCount: 0, runId: 'left', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' }],
-      right: [{ attemptId: 'a', caseId: 'right-1', caseName: 'Case', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.case.id': 'case-a', 'prediction': 'right' }, projectName: 'p', retryCount: 0, runId: 'right', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' }],
-      scoreKind: 'exact',
+      left: [recordOf({ caseId: 'left-1', metrics: { 'benchmark.case.id': 'case-a', 'prediction': 'wrong' }, runId: 'left', scores: { correct: 0, natural: 1 } })],
+      right: [recordOf({ caseId: 'right-1', metrics: { 'benchmark.case.id': 'case-a', 'prediction': 'right' }, runId: 'right', scores: { correct: 1, natural: 1 } })],
+      score: 'correct',
     })
 
-    expect(comparison.overall.delta).toBe(1)
+    /**
+     * @example
+     * expect(comparison.cases[0]?.delta).toEqual({ left: 0, right: 1, score: 1 })
+     */
+    expect(comparison.score).toBe('correct')
+    expect(comparison.overall).toEqual({ delta: 1, leftAverage: 0, rightAverage: 1 })
     expect(comparison.cases[0]?.caseKey).toBe('case-a')
-    expect(comparison.cases[0]?.delta.score).toBe(1)
+    expect(comparison.cases[0]?.delta).toEqual({ left: 0, right: 1, score: 1 })
     expect(comparison.cases[0]?.metricsChanged.prediction).toEqual({ left: 'wrong', right: 'right' })
     expect(comparison.topImprovements[0]?.caseKey).toBe('case-a')
     expect(comparison.topRegressions).toHaveLength(0)
   })
 
+  /**
+   * @example
+   * it('leaves a case without a counted value on one side out of deltas instead of counting it as 0', () => {})
+   */
+  it('leaves a case without a counted value on one side out of deltas instead of counting it as 0', () => {
+    const comparison = buildCaseComparison({
+      groupBy: 'taskId',
+      left: [recordOf({ caseName: 'unscored', scores: {}, unscored: { correct: 'grader_failed' } }), recordOf({ caseName: 'counted', scores: { correct: 1 } })],
+      right: [recordOf({ caseName: 'unscored', scores: { correct: 1 } }), recordOf({ caseName: 'counted', scores: { correct: 0 } })],
+      score: 'correct',
+    })
+
+    /**
+     * @example
+     * expect(comparison.cases.find(row => row.caseKey === 't::unscored')?.delta).toEqual({ left: null, right: 1, score: null })
+     */
+    expect(comparison.cases.find(row => row.caseKey === 't::unscored')?.delta).toEqual({ left: null, right: 1, score: null })
+    expect(comparison.topImprovements).toHaveLength(0)
+    expect(comparison.topRegressions.map(row => row.caseKey)).toEqual(['t::counted'])
+    expect(comparison.overall).toEqual({ delta: -0.5, leftAverage: 1, rightAverage: 0.5 })
+    expect(comparison.groups?.['taskId=t']).toEqual({ count: 2, delta: -1, leftAverage: 1, rightAverage: 0 })
+  })
+
   // ROOT CAUSE:
   //
   // Without `--case-key` or a `benchmark.case.id` metric, cases were matched by `caseId` (`<index>:<name>`), which
@@ -34,15 +115,23 @@ describe('report compare', () => {
   // them, so the comparison threw "duplicate case key" for any multi-task run.
   //
   // We fixed this by falling back to the task id plus the case name.
+  /**
+   * @example
+   * it('matches same-named cases of different tasks separately by default', () => {})
+   */
   it('matches same-named cases of different tasks separately by default', () => {
-    const record = (runId: string, taskId: string, exact: number) => ({ attemptId: 'a', caseId: '0:case #1', caseName: 'case #1', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId, schemaVersion: 1 as const, scores: { exact }, startedAt: 's', state: 'passed' as const, taskId, workspaceId: 'w' })
+    const record = (runId: string, taskId: string, correct: number) => recordOf({ caseId: '0:case #1', caseName: 'case #1', runId, scores: { correct }, taskId })
 
     const comparison = buildCaseComparison({
       left: [record('left', 'zh::default::run=tts%3Dnone::eval=default', 0), record('left', 'en::default::run=tts%3Dnone::eval=default', 1)],
       right: [record('right', 'zh::default::run=tts%3Dnone::eval=default', 1), record('right', 'en::default::run=tts%3Dnone::eval=default', 1)],
-      scoreKind: 'exact',
+      score: 'correct',
     })
 
+    /**
+     * @example
+     * expect(comparison.cases[1]?.delta.score).toBe(1)
+     */
     expect(comparison.cases.map(row => row.caseKey)).toEqual([
       'en::default::run=tts%3Dnone::eval=default::case #1',
       'zh::default::run=tts%3Dnone::eval=default::case #1',
@@ -52,63 +141,73 @@ describe('report compare', () => {
 
   /**
    * @example
-   * it('reports added and removed cases') verifies unmatched case accounting.
+   * it('reports added and removed cases', () => {})
    */
   it('reports added and removed cases', () => {
     const comparison = buildCaseComparison({
-      left: [{ attemptId: 'a', caseId: 'removed', caseName: 'Removed', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId: 'left', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' }],
-      right: [{ attemptId: 'a', caseId: 'added', caseName: 'Added', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId: 'right', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' }],
-      scoreKind: 'exact',
+      left: [recordOf({ caseId: 'removed', caseName: 'Removed', scores: { correct: 1 } })],
+      right: [recordOf({ caseId: 'added', caseName: 'Added', scores: { correct: 0 } })],
+      score: 'correct',
     })
 
+    /**
+     * @example
+     * expect(comparison.added).toHaveLength(1)
+     */
     expect(comparison.added).toHaveLength(1)
     expect(comparison.removed).toHaveLength(1)
   })
 
   /**
    * @example
-   * it('does not silently fall back when an explicit case key is missing') keeps case matching strict.
+   * it('does not silently fall back when an explicit case key is missing', () => {})
    */
   it('does not silently fall back when an explicit case key is missing', () => {
+    /**
+     * @example
+     * expect(() => buildCaseComparison({ caseKey: 'benchmark.case.id', ... })).toThrow('Missing explicit case key')
+     */
     expect(() => buildCaseComparison({
       caseKey: 'benchmark.case.id',
-      left: [{ attemptId: 'a', caseId: 'same', caseName: 'Left', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId: 'left', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' }],
-      right: [{ attemptId: 'a', caseId: 'same', caseName: 'Right', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId: 'right', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' }],
+      left: [recordOf({ caseId: 'same', caseName: 'Left' })],
+      right: [recordOf({ caseId: 'same', caseName: 'Right' })],
+      score: 'correct',
     })).toThrow('Missing explicit case key "benchmark.case.id" for case "same".')
   })
 
   /**
    * @example
-   * it('throws for duplicate case keys on either side') prevents silent Map overwrite in comparisons.
+   * it('throws for duplicate case keys on either side', () => {})
    */
   it('throws for duplicate case keys on either side', () => {
-    expect(() => buildCaseComparison({
-      left: [
-        { attemptId: 'a', caseId: 'first', caseName: 'First', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.case.id': 'same' }, projectName: 'p', retryCount: 0, runId: 'left', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-        { attemptId: 'a', caseId: 'second', caseName: 'Second', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.case.id': 'same' }, projectName: 'p', retryCount: 0, runId: 'left', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-      ],
-      right: [],
-    })).toThrow('Duplicate case key "same" in left report.')
+    const duplicates = [
+      recordOf({ caseId: 'first', metrics: { 'benchmark.case.id': 'same' } }),
+      recordOf({ caseId: 'second', metrics: { 'benchmark.case.id': 'same' } }),
+    ]
 
-    expect(() => buildCaseComparison({
-      left: [],
-      right: [
-        { attemptId: 'a', caseId: 'first', caseName: 'First', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.case.id': 'same' }, projectName: 'p', retryCount: 0, runId: 'right', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-        { attemptId: 'a', caseId: 'second', caseName: 'Second', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.case.id': 'same' }, projectName: 'p', retryCount: 0, runId: 'right', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-      ],
-    })).toThrow('Duplicate case key "same" in right report.')
+    /**
+     * @example
+     * expect(() => buildCaseComparison({ left: duplicates, right: [], score: 'correct' })).toThrow('Duplicate case key "same" in left report.')
+     */
+    expect(() => buildCaseComparison({ left: duplicates, right: [], score: 'correct' })).toThrow('Duplicate case key "same" in left report.')
+    expect(() => buildCaseComparison({ left: [], right: duplicates, score: 'correct' })).toThrow('Duplicate case key "same" in right report.')
   })
 
   /**
    * @example
-   * it('falls back to vieval.case.id before direct caseId') checks default matching with Vieval metric ids.
+   * it('falls back to vieval.case.id before the task id and case name', () => {})
    */
-  it('falls back to vieval.case.id before direct caseId', () => {
+  it('falls back to vieval.case.id before the task id and case name', () => {
     const comparison = buildCaseComparison({
-      left: [{ attemptId: 'a', caseId: 'left-id', caseName: 'Left', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'vieval.case.id': 'canonical' }, projectName: 'p', retryCount: 0, runId: 'left', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' }],
-      right: [{ attemptId: 'a', caseId: 'right-id', caseName: 'Right', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'vieval.case.id': 'canonical' }, projectName: 'p', retryCount: 0, runId: 'right', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' }],
+      left: [recordOf({ caseId: 'left-id', caseName: 'Left', metrics: { 'vieval.case.id': 'canonical' } })],
+      right: [recordOf({ caseId: 'right-id', caseName: 'Right', metrics: { 'vieval.case.id': 'canonical' } })],
+      score: 'correct',
     })
 
+    /**
+     * @example
+     * expect(comparison.cases[0]?.caseKey).toBe('canonical')
+     */
     expect(comparison.cases[0]?.caseKey).toBe('canonical')
     expect(comparison.added).toHaveLength(0)
     expect(comparison.removed).toHaveLength(0)
@@ -116,30 +215,96 @@ describe('report compare', () => {
 
   /**
    * @example
-   * it('renders group, per-case, metric, and unmatched details in table output') keeps default CLI output useful.
+   * it('renders the score, group, per-case, metric, and unmatched details in table output', () => {})
    */
-  it('renders group, per-case, metric, and unmatched details in table output', () => {
+  it('renders the score, group, per-case, metric, and unmatched details in table output', () => {
     const comparison = buildCaseComparison({
       groupBy: 'benchmark.group',
       left: [
-        { attemptId: 'a', caseId: 'improved', caseName: 'Improved', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.group': 'date', 'prediction': 'wrong' }, projectName: 'p', retryCount: 0, runId: 'left', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-        { attemptId: 'a', caseId: 'regressed', caseName: 'Regressed', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.group': 'date' }, projectName: 'p', retryCount: 0, runId: 'left', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-        { attemptId: 'a', caseId: 'removed', caseName: 'Removed', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId: 'left', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
+        recordOf({ caseId: 'improved', caseName: 'Improved', metrics: { 'benchmark.group': 'date', 'prediction': 'wrong' }, scores: { correct: 0 } }),
+        recordOf({ caseId: 'regressed', caseName: 'Regressed', metrics: { 'benchmark.group': 'date' }, scores: { correct: 1 } }),
+        recordOf({ caseId: 'removed', caseName: 'Removed', scores: { correct: 1 } }),
       ],
       right: [
-        { attemptId: 'a', caseId: 'improved', caseName: 'Improved', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.group': 'date', 'prediction': 'right' }, projectName: 'p', retryCount: 0, runId: 'right', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-        { attemptId: 'a', caseId: 'regressed', caseName: 'Regressed', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.group': 'date' }, projectName: 'p', retryCount: 0, runId: 'right', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-        { attemptId: 'a', caseId: 'added', caseName: 'Added', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId: 'right', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
+        recordOf({ caseId: 'improved', caseName: 'Improved', metrics: { 'benchmark.group': 'date', 'prediction': 'right' }, scores: { correct: 1 } }),
+        recordOf({ caseId: 'regressed', caseName: 'Regressed', metrics: { 'benchmark.group': 'date' }, scores: {} }),
+        recordOf({ caseId: 'added', caseName: 'Added', scores: { correct: 1 } }),
       ],
+      score: 'correct',
     })
 
     const table = formatCaseComparisonTable(comparison)
 
+    /**
+     * @example
+     * expect(table).toContain('Score     correct')
+     */
+    expect(table).toContain('Score     correct')
     expect(table).toContain('benchmark.group=date')
     expect(table).toContain('Top improvements')
-    expect(table).toContain('Top regressions')
+    expect(table).toContain('t::Regressed  delta=- changedMetrics=none')
     expect(table).toContain('changedMetrics=prediction')
     expect(table).toContain('Added cases t::Added')
     expect(table).toContain('Removed cases t::Removed')
   })
 })
+
+/**
+ * @example
+ * describe('resolveComparedScore', () => {})
+ */
+describe('resolveComparedScore', () => {
+  /**
+   * @example
+   * it('uses the primary score of the run summaries', async () => {})
+   */
+  it('uses the primary score of the run summaries', async () => {
+    const root = await mkdtemp(join(tmpdir(), 'vieval-compare-score-'))
+    temporaryDirectories.push(root)
+    await writeRunSummary(join(root, 'left'), 'pass')
+    await writeRunSummary(join(root, 'right'), 'pass')
+
+    /**
+     * @example
+     * expect(await resolveComparedScore([left, right], records)).toBe('pass')
+     */
+    expect(await resolveComparedScore([join(root, 'left'), join(root, 'right')], [recordOf({ scores: { correct: 1, pass: 1 } })])).toBe('pass')
+  })
+
+  /**
+   * @example
+   * it('falls back to the only score, and asks for --score when neither decides', async () => {})
+   */
+  it('falls back to the only score, and asks for --score when neither decides', async () => {
+    const root = await mkdtemp(join(tmpdir(), 'vieval-compare-score-'))
+    temporaryDirectories.push(root)
+    await writeRunSummary(join(root, 'left'), 'pass')
+    await writeRunSummary(join(root, 'right'), 'correct')
+    await writeRunSummary(join(root, 'none'), undefined)
+
+    /**
+     * @example
+     * await expect(resolveComparedScore([left, right], records)).rejects.toThrow('Pass --score <name>')
+     */
+    expect(await resolveComparedScore([join(root, 'none')], [recordOf({ scores: { correct: 1 } })])).toBe('correct')
+    await expect(resolveComparedScore([join(root, 'left'), join(root, 'right')], [recordOf({ scores: { correct: 1, pass: 1 } })]))
+      .rejects
+      .toThrow('Pass --score <name>: the reports have different primary scores (correct, pass). Scores in the reports: correct, pass.')
+  })
+
+  /**
+   * @example
+   * it('rejects a run summary written by an older vieval', async () => {})
+   */
+  it('rejects a run summary written by an older vieval', async () => {
+    const root = await mkdtemp(join(tmpdir(), 'vieval-compare-score-'))
+    temporaryDirectories.push(root)
+    await writeFile(join(root, 'run-summary.json'), JSON.stringify({ projects: [], schemaVersion: 1 }), 'utf-8')
+
+    /**
+     * @example
+     * await expect(resolveComparedScore([root], [])).rejects.toThrow('This report was written by an older vieval. Run it again.')
+     */
+    await expect(resolveComparedScore([root], [])).rejects.toThrow('This report was written by an older vieval. Run it again.')
+  })
+})
diff --git a/packages/vieval/src/cli/report-case-compare.ts b/packages/vieval/src/cli/report-case-compare.ts
index 1981de1..df28c38 100644
--- a/packages/vieval/src/cli/report-case-compare.ts
+++ b/packages/vieval/src/cli/report-case-compare.ts
@@ -8,6 +8,7 @@ import meow from 'meow'
 
 import { errorMessageFrom } from '@moeru/std'
 
+import { readRunSummary, resolveRunSummaryPaths } from './report-artifacts'
 import { readCaseRecordsFromReport } from './report-cases'
 import { getCaseSelectorValue, stableStringify } from './report-selectors'
 
@@ -16,6 +17,8 @@ export interface BuildCaseComparisonArgs extends CaseComparisonOptions {
   left: readonly CaseRecord[]
   /** Right/candidate run case records. */
   right: readonly CaseRecord[]
+  /** Score name used for averages and deltas; see {@link resolveComparedScore} for the CLI default. */
+  score: string
 }
 
 export interface CaseComparisonOptions {
@@ -23,8 +26,6 @@ export interface CaseComparisonOptions {
   caseKey?: string
   /** Optional key used to group matched case deltas. */
   groupBy?: string
-  /** Score kind used for averages and deltas. */
-  scoreKind?: string
 }
 
 /** Full case comparison output for local report artifacts. */
@@ -39,6 +40,8 @@ export interface CaseComparisonOutput {
   overall: CaseComparisonSummary
   /** Cases present only in the left/base report. */
   removed: CaseRecord[]
+  /** Score name the averages and deltas use. */
+  score: string
   /** Matched cases sorted from largest improvement to largest regression. */
   topImprovements: CaseComparisonRow[]
   /** Matched cases sorted from largest regression to largest improvement. */
@@ -49,11 +52,14 @@ export interface CaseComparisonOutput {
 export interface CaseComparisonRow {
   /** Stable comparison key used to match the case. */
   caseKey: string
-  /** Direct left and right scores plus right-minus-left delta. */
+  /**
+   * Counted left and right values of the score plus the right-minus-left delta. A side whose case did not count the
+   * score (unscored, missing, or an excluded error) is `null`, and so is the delta.
+   */
   delta: {
-    left: number
-    right: number
-    score: number
+    left: null | number
+    right: null | number
+    score: null | number
   }
   /** Left/base case record. */
   left: CaseRecord
@@ -65,18 +71,19 @@ export interface CaseComparisonRow {
 
 /** Score summary for all compared cases or one group of compared cases. */
 export interface CaseComparisonSummary {
-  /** Difference between right and left average. */
-  delta: number
-  /** Average score in the left/base records. */
-  leftAverage: number
-  /** Average score in the right/candidate records. */
-  rightAverage: number
+  /** Difference between right and left average, or `null` when a side has no counted value. */
+  delta: null | number
+  /** Average counted score in the left/base records, or `null` when none counted. */
+  leftAverage: null | number
+  /** Average counted score in the right/candidate records, or `null` when none counted. */
+  rightAverage: null | number
 }
 
 interface ParsedReportCompareCliArguments extends CaseComparisonOptions {
   format: 'json' | 'table'
   leftReportPath: string
   rightReportPath: string
+  score?: string
 }
 
 const reportCompareHelpText = `
@@ -88,7 +95,7 @@ const reportCompareHelpText = `
   Options
     --format       Output format: table | json (default: table)
     --case-key     Case field, score name, or metric name used to match records
-    --score-kind   Score kind used for deltas (default: exact)
+    --score        Score name used for deltas (default: the primary score of the reports, or their only score)
     --group-by     Case field, score name, or metric name used for grouped deltas
 `
 
@@ -101,13 +108,14 @@ const reportCompareHelpText = `
  *
  * Expects:
  * - left and right records are normalized `cases.jsonl` rows
- * - score values are numeric and comparable by `scoreKind`
+ * - `scores` hold counted values (see `CaseRecord.scores`); a case without the score did not count it
  *
  * Returns:
- * - matched case deltas, added/removed cases, top changes, and optional group summaries
+ * - matched case deltas, added/removed cases, top changes, and optional group summaries; cases whose score did not
+ *   count on a side have a `null` delta and are left out of top changes and group averages
  */
 export function buildCaseComparison(args: BuildCaseComparisonArgs): CaseComparisonOutput {
-  const scoreKind = args.scoreKind ?? 'exact'
+  const score = args.score
   const leftByKey = indexRecordsByCaseKey(args.left, args.caseKey, 'left')
   const rightByKey = indexRecordsByCaseKey(args.right, args.caseKey, 'right')
   const cases: CaseComparisonRow[] = []
@@ -121,14 +129,14 @@ export function buildCaseComparison(args: BuildCaseComparisonArgs): CaseComparis
       continue
     }
 
-    const leftScore = getScore(leftRecord, scoreKind)
-    const rightScore = getScore(rightRecord, scoreKind)
+    const leftScore = leftRecord.scores[score] ?? null
+    const rightScore = rightRecord.scores[score] ?? null
     cases.push({
       caseKey,
       delta: {
         left: leftScore,
         right: rightScore,
-        score: rightScore - leftScore,
+        score: leftScore == null || rightScore == null ? null : rightScore - leftScore,
       },
       left: leftRecord,
       metricsChanged: diffMetrics(leftRecord.metrics, rightRecord.metrics),
@@ -142,8 +150,8 @@ export function buildCaseComparison(args: BuildCaseComparisonArgs): CaseComparis
     }
   }
 
-  const sortedCases = [...cases].sort((left, right) => {
-    const deltaOrder = right.delta.score - left.delta.score
+  const sortedCases = cases.filter(row => row.delta.score != null).sort((left, right) => {
+    const deltaOrder = right.delta.score! - left.delta.score!
     return deltaOrder === 0
       ? left.caseKey.localeCompare(right.caseKey)
       : deltaOrder
@@ -153,14 +161,14 @@ export function buildCaseComparison(args: BuildCaseComparisonArgs): CaseComparis
     added: added.sort(compareCaseRecords),
     cases: cases.sort((left, right) => left.caseKey.localeCompare(right.caseKey)),
     groups: args.groupBy == null ? undefined : buildComparisonGroups(cases, args.groupBy),
-    overall: {
-      delta: averageScore(args.right, scoreKind) - averageScore(args.left, scoreKind),
-      leftAverage: averageScore(args.left, scoreKind),
-      rightAverage: averageScore(args.right, scoreKind),
-    },
+    overall: summarizeAverages(
+      args.left.flatMap(record => record.scores[score] ?? []),
+      args.right.flatMap(record => record.scores[score] ?? []),
+    ),
     removed: removed.sort(compareCaseRecords),
-    topImprovements: sortedCases.filter(row => row.delta.score > 0).slice(0, 10),
-    topRegressions: [...sortedCases].reverse().filter(row => row.delta.score < 0).slice(0, 10),
+    score,
+    topImprovements: sortedCases.filter(row => row.delta.score! > 0).slice(0, 10),
+    topRegressions: [...sortedCases].reverse().filter(row => row.delta.score! < 0).slice(0, 10),
   }
 }
 
@@ -178,32 +186,34 @@ export function buildCaseComparison(args: BuildCaseComparisonArgs): CaseComparis
  * - multi-line text containing aggregate, group, top-change, case, and unmatched summaries
  */
 export function formatCaseComparisonTable(output: CaseComparisonOutput): string {
+  const fixed = (value: null | number) => value == null ? '-' : value.toFixed(3)
   const lines = [
     'COMPARE  vieval report cases',
+    `Score     ${output.score}`,
     `Matched   ${output.cases.length}`,
     `Added     ${output.added.length}`,
     `Removed   ${output.removed.length}`,
-    `Scores    left=${output.overall.leftAverage.toFixed(3)} right=${output.overall.rightAverage.toFixed(3)} delta=${output.overall.delta.toFixed(3)}`,
+    `Scores    left=${fixed(output.overall.leftAverage)} right=${fixed(output.overall.rightAverage)} delta=${fixed(output.overall.delta)}`,
   ]
 
   if (output.groups != null && Object.keys(output.groups).length > 0) {
     lines.push('Groups')
     for (const [groupKey, group] of Object.entries(output.groups)) {
-      lines.push(`${groupKey}  count=${group.count} left=${group.leftAverage.toFixed(3)} right=${group.rightAverage.toFixed(3)} delta=${group.delta.toFixed(3)}`)
+      lines.push(`${groupKey}  count=${group.count} left=${fixed(group.leftAverage)} right=${fixed(group.rightAverage)} delta=${fixed(group.delta)}`)
     }
   }
 
   if (output.topImprovements.length > 0) {
     lines.push('Top improvements')
     for (const row of output.topImprovements) {
-      lines.push(`${row.caseKey}  delta=${row.delta.score.toFixed(3)} left=${row.delta.left.toFixed(3)} right=${row.delta.right.toFixed(3)}`)
+      lines.push(`${row.caseKey}  delta=${fixed(row.delta.score)} left=${fixed(row.delta.left)} right=${fixed(row.delta.right)}`)
     }
   }
 
   if (output.topRegressions.length > 0) {
     lines.push('Top regressions')
     for (const row of output.topRegressions) {
-      lines.push(`${row.caseKey}  delta=${row.delta.score.toFixed(3)} left=${row.delta.left.toFixed(3)} right=${row.delta.right.toFixed(3)}`)
+      lines.push(`${row.caseKey}  delta=${fixed(row.delta.score)} left=${fixed(row.delta.left)} right=${fixed(row.delta.right)}`)
     }
   }
 
@@ -211,7 +221,7 @@ export function formatCaseComparisonTable(output: CaseComparisonOutput): string
     lines.push('Cases')
     for (const row of output.cases) {
       const changedMetricNames = Object.keys(row.metricsChanged)
-      lines.push(`${row.caseKey}  delta=${row.delta.score.toFixed(3)} changedMetrics=${changedMetricNames.length === 0 ? 'none' : changedMetricNames.join(',')}`)
+      lines.push(`${row.caseKey}  delta=${fixed(row.delta.score)} changedMetrics=${changedMetricNames.length === 0 ? 'none' : changedMetricNames.join(',')}`)
     }
   }
 
@@ -226,6 +236,41 @@ export function formatCaseComparisonTable(output: CaseComparisonOutput): string
   return lines.join('\n')
 }
 
+/**
+ * Resolves the score `vieval report compare` uses when `--score` is not given.
+ *
+ * Use when:
+ * - the user compares two reports without naming a score
+ *
+ * Expects:
+ * - `reportPaths` to be the compared report locations, whose `run-summary.json` files hold the resolved headlines
+ *
+ * Returns:
+ * - the primary score name when every report's headline that has a score names the same one
+ * - otherwise the only score name the records have
+ * - throws when neither decides, listing the score names to pick from
+ */
+export async function resolveComparedScore(reportPaths: readonly string[], records: readonly CaseRecord[]): Promise<string> {
+  const summaryPaths = (await Promise.all(reportPaths.map(reportPath => resolveRunSummaryPaths(reportPath)))).flat()
+  const primaryNames = new Set(summaryPaths.flatMap(summaryPath => readRunSummary(summaryPath).projects.flatMap((project) => {
+    const headline = project.headline?.headline
+    return headline?.kind === 'score' ? [headline.score.name] : []
+  })))
+  if (primaryNames.size === 1) {
+    return [...primaryNames][0]!
+  }
+
+  const scoreNames = [...new Set(records.flatMap(record => Object.keys(record.scores)))].sort()
+  if (scoreNames.length === 1) {
+    return scoreNames[0]!
+  }
+
+  const reason = primaryNames.size > 1
+    ? `the reports have different primary scores (${[...primaryNames].sort().join(', ')})`
+    : `the reports have no primary score`
+  throw new Error(`Pass --score <name>: ${reason}. Scores in the reports: ${scoreNames.length === 0 ? 'none' : scoreNames.join(', ')}.`)
+}
+
 /**
  * Runs the `vieval report compare` command.
  *
@@ -235,6 +280,7 @@ export function formatCaseComparisonTable(output: CaseComparisonOutput): string
  *   -> {@link import('./index').runTopLevelCli}
  *     -> {@link runReportCompareCli}
  *       -> {@link readCaseRecordsFromReport}
+ *       -> {@link resolveComparedScore} (without `--score`)
  *       -> {@link buildCaseComparison}
  *
  * Use when:
@@ -258,7 +304,7 @@ export async function runReportCompareCli(argv: readonly string[]): Promise<void
       groupBy: parsed.groupBy,
       left,
       right,
-      scoreKind: parsed.scoreKind,
+      score: parsed.score ?? await resolveComparedScore([parsed.leftReportPath, parsed.rightReportPath], [...left, ...right]),
     })
 
     if (parsed.format === 'json') {
@@ -275,16 +321,8 @@ export async function runReportCompareCli(argv: readonly string[]): Promise<void
   }
 }
 
-function averageScore(records: readonly CaseRecord[], scoreKind: string): number {
-  const values = records
-    .map(record => record.scores[scoreKind])
-    .filter((value): value is number => typeof value === 'number')
-
-  if (values.length === 0) {
-    return 0
-  }
-
-  return values.reduce((sum, value) => sum + value, 0) / values.length
+function averageOf(values: readonly number[]): null | number {
+  return values.length === 0 ? null : values.reduce((sum, value) => sum + value, 0) / values.length
 }
 
 function buildComparisonGroups(
@@ -308,14 +346,11 @@ function buildComparisonGroups(
     Object.entries(groupedRows)
       .sort(([left], [right]) => left.localeCompare(right))
       .map(([groupKey, rows]) => {
-        const leftAverage = rows.reduce((sum, row) => sum + row.delta.left, 0) / rows.length
-        const rightAverage = rows.reduce((sum, row) => sum + row.delta.right, 0) / rows.length
-
+        // Only rows that count the score on both sides compare like with like.
+        const compared = rows.filter(row => row.delta.score != null)
         return [groupKey, {
           count: rows.length,
-          delta: rightAverage - leftAverage,
-          leftAverage,
-          rightAverage,
+          ...summarizeAverages(compared.map(row => row.delta.left!), compared.map(row => row.delta.right!)),
         }]
       }),
   )
@@ -344,10 +379,6 @@ function diffMetrics(
   return changed
 }
 
-function getScore(record: CaseRecord, scoreKind: string): number {
-  return record.scores[scoreKind] ?? 0
-}
-
 function indexRecordsByCaseKey(records: readonly CaseRecord[], caseKey: string | undefined, side: 'left' | 'right'): Map<string, CaseRecord> {
   const indexed = new Map<string, CaseRecord>()
 
@@ -393,8 +424,7 @@ function parseReportCompareCliArguments(argv: readonly string[]): ParsedReportCo
       groupBy: {
         type: 'string',
       },
-      scoreKind: {
-        default: 'exact',
+      score: {
         type: 'string',
       },
     },
@@ -413,7 +443,7 @@ function parseReportCompareCliArguments(argv: readonly string[]): ParsedReportCo
     groupBy: cli.flags.groupBy,
     leftReportPath,
     rightReportPath,
-    scoreKind: cli.flags.scoreKind,
+    score: cli.flags.score,
   }
 }
 
@@ -441,3 +471,13 @@ function resolveCaseKey(record: CaseRecord, caseKey: string | undefined): string
   // cases in another order. The task id plus the case name names one case across tasks, matrix rows, and runs.
   return `${record.taskId}::${record.caseName}`
 }
+
+function summarizeAverages(left: readonly number[], right: readonly number[]): CaseComparisonSummary {
+  const leftAverage = averageOf(left)
+  const rightAverage = averageOf(right)
+  return {
+    delta: leftAverage == null || rightAverage == null ? null : rightAverage - leftAverage,
+    leftAverage,
+    rightAverage,
+  }
+}
diff --git a/packages/vieval/src/cli/report-cases.test.ts b/packages/vieval/src/cli/report-cases.test.ts
index 2e046eb..166369f 100644
--- a/packages/vieval/src/cli/report-cases.test.ts
+++ b/packages/vieval/src/cli/report-cases.test.ts
@@ -1,91 +1,162 @@
-import { mkdir, mkdtemp, writeFile } from 'node:fs/promises'
+import type { CaseRecord } from './report-records'
+
+import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'
 import { tmpdir } from 'node:os'
 import { join } from 'node:path'
 
-import { describe, expect, it } from 'vitest'
+import { afterEach, describe, expect, it } from 'vitest'
 
 import { buildReportCasesOutput, readCaseRecordsFromReport } from './report-cases'
+import { encodeJsonl } from './report-records'
+
+const temporaryDirectories: string[] = []
+
+afterEach(async () => {
+  await Promise.all(temporaryDirectories.splice(0).map(directory => rm(directory, { force: true, recursive: true })))
+})
+
+async function createTemporaryDirectory(prefix: string): Promise<string> {
+  const directory = await mkdtemp(join(tmpdir(), prefix))
+  temporaryDirectories.push(directory)
+  return directory
+}
+
+/**
+ * Creates one current case record; `overrides` replace its fields.
+ */
+function recordOf(overrides: Partial<CaseRecord>): CaseRecord {
+  return {
+    attemptId: 'a',
+    caseId: '1',
+    caseName: 'One',
+    durationMs: 1,
+    endedAt: 'e',
+    experimentId: 'e',
+    metrics: {},
+    projectName: 'p',
+    retryCount: 0,
+    runId: 'r',
+    schemaVersion: 2,
+    scoreDetails: {},
+    scores: {},
+    startedAt: 's',
+    state: 'passed',
+    taskId: 't',
+    unscored: {},
+    workspaceId: 'w',
+    ...overrides,
+  }
+}
 
 /**
  * @example
- * describe('report cases') verifies local case artifact inspection.
+ * describe('report cases', () => {})
  */
 describe('report cases', () => {
   /**
    * @example
-   * it('reads and filters cases by metric equality') checks dataframe-like filtering from CLI args.
+   * it('reads and filters cases by metric and score equality', async () => {})
    */
-  it('reads and filters cases by metric equality', async ({ task }) => {
-    const dir = task.name.replaceAll(' ', '-')
-    const reportPath = join(process.cwd(), '.tmp', dir)
-    await mkdir(reportPath, { recursive: true })
-    await writeFile(join(reportPath, 'cases.jsonl'), [
-      JSON.stringify({ attemptId: 'a', caseId: '1', caseName: 'One', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.locomo.category': 1 }, projectName: 'p', retryCount: 0, runId: 'r', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' }),
-      JSON.stringify({ attemptId: 'a', caseId: '2', caseName: 'Two', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.locomo.category': 2 }, projectName: 'p', retryCount: 0, runId: 'r', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' }),
-    ].join('\n').concat('\n'), 'utf-8')
+  it('reads and filters cases by metric and score equality', async () => {
+    const reportPath = await createTemporaryDirectory('vieval-report-cases-')
+    await writeFile(join(reportPath, 'cases.jsonl'), encodeJsonl([
+      recordOf({ caseId: '1', metrics: { 'benchmark.locomo.category': 1 }, scores: { 'locomo.correct': 1 } }),
+      recordOf({ caseId: '2', caseName: 'Two', metrics: { 'benchmark.locomo.category': 2 }, scores: { 'locomo.correct': 0 } }),
+    ]), 'utf-8')
 
     const records = await readCaseRecordsFromReport(reportPath)
-    const output = buildReportCasesOutput(records, { format: 'json', where: ['benchmark.locomo.category=2'] })
 
-    expect(output.records).toHaveLength(1)
-    expect(output.records[0]?.caseId).toBe('2')
+    /**
+     * @example
+     * expect(buildReportCasesOutput(records, { format: 'json', where: ['benchmark.locomo.category=2'] }).records.map(record => record.caseId)).toEqual(['2'])
+     */
+    expect(buildReportCasesOutput(records, { format: 'json', where: ['benchmark.locomo.category=2'] }).records.map(record => record.caseId)).toEqual(['2'])
+    expect(buildReportCasesOutput(records, { format: 'json', where: ['scores.locomo.correct=1'] }).records.map(record => record.caseId)).toEqual(['1'])
   })
 
   /**
    * @example
-   * it('groups cases by a metric key') verifies generic benchmark facet summaries.
+   * it('groups cases by a metric key into score summaries by name', () => {})
    */
-  it('groups cases by a metric key', () => {
+  it('groups cases by a metric key into score summaries by name', () => {
     const output = buildReportCasesOutput([
-      { attemptId: 'a', caseId: '1', caseName: 'One', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.locomo.category': 2 }, projectName: 'p', retryCount: 0, runId: 'r', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-      { attemptId: 'a', caseId: '2', caseName: 'Two', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.locomo.category': 2 }, projectName: 'p', retryCount: 0, runId: 'r', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
+      recordOf({ caseId: '1', metrics: { 'benchmark.locomo.category': 2 }, scores: { 'locomo.correct': 1 } }),
+      recordOf({ caseId: '2', metrics: { 'benchmark.locomo.category': 2 }, scores: {}, unscored: { 'locomo.correct': 'grader_failed' } }),
+      recordOf({ caseId: '3', metrics: { 'benchmark.locomo.category': 2 }, scores: { 'locomo.correct': 0 }, state: 'failed' }),
     ], { format: 'json', groupBy: 'benchmark.locomo.category' })
 
-    expect(output.groups?.['benchmark.locomo.category=2']?.count).toBe(2)
-    expect(output.groups?.['benchmark.locomo.category=2']?.scores.exact.average).toBe(0.5)
+    /**
+     * @example
+     * expect(output.groups?.['benchmark.locomo.category=2']?.scores['locomo.correct']).toEqual({ mean: 0.5, n: 2, ... })
+     */
+    expect(output.groups?.['benchmark.locomo.category=2']?.count).toBe(3)
+    expect(output.groups?.['benchmark.locomo.category=2']?.scores['locomo.correct']).toEqual({
+      mean: 0.5,
+      n: 2,
+      name: 'locomo.correct',
+      sum: 1,
+      total: 3,
+      unscored: { grader_failed: 1 },
+    })
   })
 
   /**
    * @example
-   * it('reads cases from direct jsonl files and multi-report roots') covers non-default report path shapes.
+   * it('reads cases from direct jsonl files and multi-report roots', async () => {})
    */
   it('reads cases from direct jsonl files and multi-report roots', async () => {
-    const reportRoot = await mkdtemp(join(tmpdir(), 'vieval-report-cases-'))
+    const reportRoot = await createTemporaryDirectory('vieval-report-cases-')
     const leftRun = join(reportRoot, 'workspace', 'project', 'experiment', 'attempt-a', 'run-a')
     const rightRun = join(reportRoot, 'workspace', 'project', 'experiment', 'attempt-b', 'run-b')
     await mkdir(leftRun, { recursive: true })
     await mkdir(rightRun, { recursive: true })
-    await writeFile(join(leftRun, 'cases.jsonl'), `${JSON.stringify({ attemptId: 'a', caseId: 'left', caseName: 'Left', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId: 'left-run', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' })}\n`, 'utf-8')
-    await writeFile(join(rightRun, 'cases.jsonl'), `${JSON.stringify({ attemptId: 'b', caseId: 'right', caseName: 'Right', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId: 'right-run', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'failed', taskId: 't', workspaceId: 'w' })}\n`, 'utf-8')
+    await writeFile(join(leftRun, 'cases.jsonl'), encodeJsonl([recordOf({ caseId: 'left', runId: 'left-run' })]), 'utf-8')
+    await writeFile(join(rightRun, 'cases.jsonl'), encodeJsonl([recordOf({ caseId: 'right', runId: 'right-run', state: 'failed' })]), 'utf-8')
 
     const directRecords = await readCaseRecordsFromReport(join(leftRun, 'cases.jsonl'))
     const rootRecords = await readCaseRecordsFromReport(reportRoot)
 
+    /**
+     * @example
+     * expect(rootRecords.map(record => record.caseId)).toEqual(['left', 'right'])
+     */
     expect(directRecords.map(record => record.caseId)).toEqual(['left'])
     expect(rootRecords.map(record => record.caseId)).toEqual(['left', 'right'])
   })
 
   /**
    * @example
-   * it('throws for malformed jsonl and missing case files') verifies bad report inputs fail loudly.
+   * it('throws for malformed jsonl, older records, and missing case files', async () => {})
    */
-  it('throws for malformed jsonl and missing case files', async () => {
-    const reportRoot = await mkdtemp(join(tmpdir(), 'vieval-report-cases-invalid-'))
+  it('throws for malformed jsonl, older records, and missing case files', async () => {
+    const reportRoot = await createTemporaryDirectory('vieval-report-cases-invalid-')
     const malformedRun = join(reportRoot, 'malformed')
+    const olderRun = join(reportRoot, 'older')
     const emptyRoot = join(reportRoot, 'empty')
     await mkdir(malformedRun, { recursive: true })
+    await mkdir(olderRun, { recursive: true })
     await mkdir(emptyRoot, { recursive: true })
     await writeFile(join(malformedRun, 'cases.jsonl'), '{"caseId":\n', 'utf-8')
+    await writeFile(join(olderRun, 'cases.jsonl'), `${JSON.stringify({ ...recordOf({}), schemaVersion: 1, scores: { exact: 1 } })}\n`, 'utf-8')
 
+    /**
+     * @example
+     * await expect(readCaseRecordsFromReport(olderRun)).rejects.toThrow('This report was written by an older vieval. Run it again.')
+     */
     await expect(readCaseRecordsFromReport(malformedRun)).rejects.toThrow('Invalid cases.jsonl line 1')
+    await expect(readCaseRecordsFromReport(olderRun)).rejects.toThrow('This report was written by an older vieval. Run it again.')
     await expect(readCaseRecordsFromReport(emptyRoot)).rejects.toThrow('No cases.jsonl files found')
   })
 
   /**
    * @example
-   * it('throws for invalid where selectors') verifies CLI selectors are not silently ignored.
+   * it('throws for invalid where selectors', () => {})
    */
   it('throws for invalid where selectors', () => {
+    /**
+     * @example
+     * expect(() => buildReportCasesOutput([], { format: 'json', where: ['benchmark.locomo.category'] })).toThrow('Invalid selector')
+     */
     expect(() => buildReportCasesOutput([], { format: 'json', where: ['benchmark.locomo.category'] })).toThrow('Invalid selector "benchmark.locomo.category". Expected "key=value".')
   })
 })
diff --git a/packages/vieval/src/cli/report-cases.ts b/packages/vieval/src/cli/report-cases.ts
index 332f5c4..588526f 100644
--- a/packages/vieval/src/cli/report-cases.ts
+++ b/packages/vieval/src/cli/report-cases.ts
@@ -1,10 +1,11 @@
 #!/usr/bin/env node
 
-import type { CaseRecord, ScoreSummary } from './report-records'
+import type { ScoreSummary } from '../core/scoring'
+import type { CaseRecord } from './report-records'
 
 import process from 'node:process'
 
-import { existsSync, readFileSync } from 'node:fs'
+import { existsSync } from 'node:fs'
 import { resolve } from 'node:path'
 
 import meow from 'meow'
@@ -12,7 +13,7 @@ import meow from 'meow'
 import { errorMessageFrom } from '@moeru/std'
 import { glob } from 'tinyglobby'
 
-import { encodeJsonl } from './report-records'
+import { encodeJsonl, readCaseRecordFile, summarizeRecordScores } from './report-records'
 import { getCaseSelectorValue } from './report-selectors'
 
 export type ReportCasesFormat = 'json' | 'jsonl' | 'table'
@@ -21,8 +22,8 @@ export type ReportCasesFormat = 'json' | 'jsonl' | 'table'
 export interface ReportCasesGroupSummary {
   /** Number of case records in this group. */
   count: number
-  /** Score averages/sums/counts for this group. */
-  scores: ScoreSummary
+  /** Score summaries of this group by score name, counted as in `metrics-summary.json`. */
+  scores: Record<string, ScoreSummary>
 }
 
 /** Options for inspecting normalized `cases.jsonl` report artifacts. */
@@ -97,10 +98,11 @@ export function buildReportCasesOutput(
  * - callers may pass a run directory, a `cases.jsonl` file, or a report root
  *
  * Expects:
- * - discovered `cases.jsonl` files contain one `CaseRecord` JSON object per line
+ * - discovered `cases.jsonl` files contain one current `CaseRecord` JSON object per line
  *
  * Returns:
  * - all parsed case records sorted by discovered file path order
+ * - throws for records written by an older vieval (see {@link readCaseRecordFile})
  */
 export async function readCaseRecordsFromReport(reportPath: string): Promise<CaseRecord[]> {
   const caseFilePaths = await resolveCaseRecordPaths(reportPath)
@@ -108,27 +110,7 @@ export async function readCaseRecordsFromReport(reportPath: string): Promise<Cas
     throw new Error(`No cases.jsonl files found under "${resolve(reportPath)}".`)
   }
 
-  const records: CaseRecord[] = []
-
-  for (const caseFilePath of caseFilePaths) {
-    const contents = readFileSync(caseFilePath, 'utf-8')
-    const lines = contents.split('\n')
-    for (const [index, line] of lines.entries()) {
-      const trimmed = line.trim()
-      if (trimmed.length === 0) {
-        continue
-      }
-
-      try {
-        records.push(JSON.parse(trimmed) as CaseRecord)
-      }
-      catch (error) {
-        throw new Error(`Invalid cases.jsonl line ${index + 1} in "${caseFilePath}": ${errorMessageFrom(error) ?? 'Unknown JSON parse failure.'}`)
-      }
-    }
-  }
-
-  return records
+  return caseFilePaths.flatMap(caseFilePath => readCaseRecordFile(caseFilePath))
 }
 
 /**
@@ -175,16 +157,8 @@ export async function runReportCasesCli(argv: readonly string[]): Promise<void>
   }
 }
 
-function addScores(summary: ScoreSummary, scores: Record<string, number>): void {
-  for (const [scoreName, value] of Object.entries(scores)) {
-    summary[scoreName] ??= { average: 0, count: 0, sum: 0 }
-    summary[scoreName].count += 1
-    summary[scoreName].sum += value
-  }
-}
-
 function buildCaseGroups(records: readonly CaseRecord[], groupBy: string): Record<string, ReportCasesGroupSummary> {
-  const groups: Record<string, ReportCasesGroupSummary> = {}
+  const groups = new Map<string, CaseRecord[]>()
 
   for (const record of records) {
     const resolved = getCaseSelectorValue(record, groupBy)
@@ -193,33 +167,19 @@ function buildCaseGroups(records: readonly CaseRecord[], groupBy: string): Recor
     }
 
     const groupKey = `${groupBy}=${String(resolved.value)}`
-    groups[groupKey] ??= { count: 0, scores: {} }
-    groups[groupKey].count += 1
-    addScores(groups[groupKey].scores, record.scores)
+    groups.set(groupKey, [...(groups.get(groupKey) ?? []), record])
   }
 
   return Object.fromEntries(
-    Object.entries(groups)
+    [...groups]
       .sort(([left], [right]) => left.localeCompare(right))
-      .map(([groupKey, group]) => [groupKey, {
-        count: group.count,
-        scores: finalizeScores(group.scores),
+      .map(([groupKey, groupRecords]) => [groupKey, {
+        count: groupRecords.length,
+        scores: summarizeRecordScores(groupRecords),
       } satisfies ReportCasesGroupSummary]),
   )
 }
 
-function finalizeScores(summary: ScoreSummary): ScoreSummary {
-  return Object.fromEntries(
-    Object.entries(summary)
-      .sort(([left], [right]) => left.localeCompare(right))
-      .map(([scoreName, bucket]) => [scoreName, {
-        average: bucket.count === 0 ? 0 : bucket.sum / bucket.count,
-        count: bucket.count,
-        sum: bucket.sum,
-      }]),
-  )
-}
-
 function formatCasesTable(output: ReportCasesOutput): string {
   const lines = [
     'CASES  vieval report',
@@ -229,8 +189,8 @@ function formatCasesTable(output: ReportCasesOutput): string {
   if (output.groups != null) {
     lines.push('Groups')
     for (const [groupKey, group] of Object.entries(output.groups)) {
-      const scoreText = Object.entries(group.scores)
-        .map(([scoreName, bucket]) => `${scoreName}=${bucket.average.toFixed(3)}`)
+      const scoreText = Object.values(group.scores)
+        .map(summary => `${summary.name}=${summary.mean == null ? '-' : summary.mean.toFixed(3)} (${summary.n}/${summary.total})`)
         .join(' ')
       lines.push(`${groupKey}  count=${group.count}${scoreText.length > 0 ? ` ${scoreText}` : ''}`)
     }
diff --git a/packages/vieval/src/cli/report-log.test.ts b/packages/vieval/src/cli/report-log.test.ts
index 468b246..fcfec61 100644
--- a/packages/vieval/src/cli/report-log.test.ts
+++ b/packages/vieval/src/cli/report-log.test.ts
@@ -102,17 +102,35 @@ describe('readReportEvents', () => {
 
   /**
    * @example
-   * it('rejects a log written by an older report schema with its line number', async () => {})
+   * it('rejects a log of event schema version 2 as written by an older vieval, with its line number', async () => {})
    */
-  it('rejects a log written by an older report schema with its line number', async () => {
+  it('rejects a log of event schema version 2 as written by an older vieval, with its line number', async () => {
     const filePath = join(await mkdtemp(join(tmpdir(), 'vieval-report-log-')), 'events.jsonl')
     temporaryDirectories.push(join(filePath, '..'))
-    await writeFile(filePath, `${JSON.stringify({ event: 'RunStarted', schemaVersion: 1, version: 1 })}\n`, 'utf-8')
+    const olderEnvelope = { attemptId: 'a', caseId: 'c', data: { kind: 'exact', score: 1 }, event: 'score.recorded', experimentId: 'e', runId: 'r', schemaVersion: 2, seq: 1, taskId: 't', time: '2026-10-20T00:00:00.000Z', workspaceId: 'w' }
+    await writeFile(filePath, `${JSON.stringify(olderEnvelope)}\n`, 'utf-8')
 
     /**
      * @example
-     * expect(() => readReportEvents(filePath)).toThrow(`${filePath}:1`)
+     * expect(() => readReportEvents(filePath)).toThrow('This report was written by an older vieval. Run it again.')
      */
+    expect(() => readReportEvents(filePath)).toThrow('This report was written by an older vieval. Run it again.')
     expect(() => readReportEvents(filePath)).toThrow(`${filePath}:1`)
   })
+
+  /**
+   * @example
+   * it('rejects a line that is not an event envelope with its line number', async () => {})
+   */
+  it('rejects a line that is not an event envelope with its line number', async () => {
+    const filePath = join(await mkdtemp(join(tmpdir(), 'vieval-report-log-')), 'events.jsonl')
+    temporaryDirectories.push(join(filePath, '..'))
+    await writeFile(filePath, `${JSON.stringify({ event: 'run.started' })}\n`, 'utf-8')
+
+    /**
+     * @example
+     * expect(() => readReportEvents(filePath)).toThrow(`Unsupported event record at ${filePath}:1`)
+     */
+    expect(() => readReportEvents(filePath)).toThrow(`Unsupported event record at ${filePath}:1`)
+  })
 })
diff --git a/packages/vieval/src/cli/report-log.ts b/packages/vieval/src/cli/report-log.ts
index 20344ac..2c82bc2 100644
--- a/packages/vieval/src/cli/report-log.ts
+++ b/packages/vieval/src/cli/report-log.ts
@@ -9,6 +9,13 @@ import * as v from 'valibot'
 
 import { RUN_EVENT_SCHEMA_VERSION, runEventEnvelopeSchema } from '../events'
 
+/**
+ * Error message of every report reader that meets a file written before the current report schema.
+ *
+ * Old reports are not migrated: a reader rejects them, and the eval runs again to write a current one.
+ */
+export const OLDER_REPORT_MESSAGE = 'This report was written by an older vieval. Run it again.'
+
 /**
  * Append-only `events.jsonl` writer for one run.
  */
@@ -86,7 +93,8 @@ export function createReportEventLog(filePath: string): ReportEventLog {
  *
  * Returns:
  * - envelopes in file order, or an empty list when the file does not exist
- * - throws when a line belongs to another report schema version
+ * - throws {@link OLDER_REPORT_MESSAGE} with the line when an envelope has an older `schemaVersion`, and a malformed
+ *   record error for any other line that is not a current envelope
  */
 export function readReportEvents(filePath: string): RunEventEnvelope[] {
   if (!existsSync(filePath)) {
@@ -98,9 +106,14 @@ export function readReportEvents(filePath: string): RunEventEnvelope[] {
     .map((line, index) => ({ line, lineNumber: index + 1 }))
     .filter(({ line }) => line.trim().length > 0)
     .map(({ line, lineNumber }) => {
-      const result = v.safeParse(runEventEnvelopeSchema, JSON.parse(line))
+      const value: unknown = JSON.parse(line)
+      const schemaVersion = (value as null | { schemaVersion?: unknown })?.schemaVersion
+      if (typeof schemaVersion === 'number' && schemaVersion < RUN_EVENT_SCHEMA_VERSION) {
+        throw new Error(`${OLDER_REPORT_MESSAGE} (${filePath}:${lineNumber}, event schema version ${schemaVersion})`)
+      }
+      const result = v.safeParse(runEventEnvelopeSchema, value)
       if (!result.success) {
-        throw new Error(`Unsupported event record at ${filePath}:${lineNumber}: expected report schema version ${RUN_EVENT_SCHEMA_VERSION}. Reports written by older vieval versions cannot be read; re-run the eval to regenerate them.`)
+        throw new Error(`Unsupported event record at ${filePath}:${lineNumber}: expected an event envelope of report schema version ${RUN_EVENT_SCHEMA_VERSION}.`)
       }
       return result.output
     })
diff --git a/packages/vieval/src/cli/report-otlp.test.ts b/packages/vieval/src/cli/report-otlp.test.ts
index 05ac9eb..71a535f 100644
--- a/packages/vieval/src/cli/report-otlp.test.ts
+++ b/packages/vieval/src/cli/report-otlp.test.ts
@@ -37,11 +37,33 @@ describe('local OTLP projection', () => {
           projectName: 'p',
           retryCount: 0,
           runId: 'run-1',
-          schemaVersion: 1,
-          scores: { exact: 0.5 },
+          schemaVersion: 2,
+          scoreDetails: {},
+          scores: { 'locomo.correct': 0.5 },
           startedAt: '2026-05-08T00:00:00.000Z',
           state: 'passed',
           taskId: 'task-1',
+          unscored: { 'locomo.natural': 'grader_failed' },
+          workspaceId: 'w',
+        },
+        {
+          attemptId: 'a',
+          caseId: 'case-2',
+          caseName: 'Case 2',
+          durationMs: 10,
+          endedAt: '2026-05-08T00:00:00.010Z',
+          experimentId: 'e',
+          metrics: {},
+          projectName: 'p',
+          retryCount: 0,
+          runId: 'run-1',
+          schemaVersion: 2,
+          scoreDetails: {},
+          scores: { 'locomo.correct': 1, 'locomo.natural': 1 },
+          startedAt: '2026-05-08T00:00:00.000Z',
+          state: 'passed',
+          taskId: 'task-1',
+          unscored: {},
           workspaceId: 'w',
         },
       ],
@@ -67,6 +89,7 @@ describe('local OTLP projection', () => {
       'vieval.project',
       'vieval.task',
       'vieval.case',
+      'vieval.case',
     ])
     expect(spans?.[1]?.attributes).toContainEqual({
       key: 'vieval.project.name',
@@ -100,13 +123,19 @@ describe('local OTLP projection', () => {
     })
     expect(logRecords?.[0]?.eventName).toBe('vieval.case')
     expect(logRecords?.[0]?.body).toEqual({
-      stringValue: JSON.stringify({ caseId: 'case-1', scores: { exact: 0.5 }, state: 'passed' }),
+      stringValue: JSON.stringify({ caseId: 'case-1', scores: { 'locomo.correct': 0.5 }, state: 'passed', unscored: { 'locomo.natural': 'grader_failed' } }),
     })
     expect(logRecords?.[0]?.attributes).toContainEqual({
       key: 'benchmark.id',
       value: { stringValue: 'locomo' },
     })
-    expect(metrics?.[0]?.name).toBe('vieval.score.exact')
+    /**
+     * @example
+     * expect(metrics?.map(metric => metric.name)).toEqual(['vieval.score.locomo.correct', 'vieval.score.locomo.natural'])
+     */
+    expect(metrics?.map(metric => metric.name)).toEqual(['vieval.score.locomo.correct', 'vieval.score.locomo.natural'])
+    // An unscored value is not a data point: it says nothing about the case.
+    expect(metrics?.[1]?.gauge.dataPoints.map(point => point.asDouble)).toEqual([1])
     expect(metrics?.[0]?.gauge.dataPoints[0]).toEqual({
       asDouble: 0.5,
       attributes: expect.arrayContaining([
@@ -144,11 +173,13 @@ describe('local OTLP projection', () => {
           projectName: 'p',
           retryCount: 0,
           runId: 'run-1',
-          schemaVersion: 1,
-          scores: { exact: 1 },
+          schemaVersion: 2,
+          scoreDetails: {},
+          scores: { 'suite.correct': 1 },
           startedAt: '2026-05-08T00:00:00.000Z',
           state: 'passed',
           taskId: 'task-1',
+          unscored: {},
           workspaceId: 'w',
         },
       ],
@@ -181,11 +212,13 @@ describe('local OTLP projection', () => {
           projectName: 'p',
           retryCount: 0,
           runId: 'run-1',
-          schemaVersion: 1,
-          scores: { exact: 1 },
+          schemaVersion: 2,
+          scoreDetails: {},
+          scores: { 'suite.correct': 1 },
           startedAt: '2026-05-08T00:00:00.123456Z',
           state: 'passed',
           taskId: 'task-1',
+          unscored: {},
           workspaceId: 'w',
         },
       ],
@@ -220,11 +253,13 @@ describe('local OTLP projection', () => {
           projectName: 'p',
           retryCount: 0,
           runId: 'run-1',
-          schemaVersion: 1,
-          scores: { exact: 1 },
+          schemaVersion: 2,
+          scoreDetails: {},
+          scores: { 'suite.correct': 1 },
           startedAt: '2026-05-08T00:00:00.000Z',
           state: 'passed',
           taskId: 'task-1',
+          unscored: {},
           workspaceId: 'w',
         },
       ],
diff --git a/packages/vieval/src/cli/report-otlp.ts b/packages/vieval/src/cli/report-otlp.ts
index e030f2a..acd5891 100644
--- a/packages/vieval/src/cli/report-otlp.ts
+++ b/packages/vieval/src/cli/report-otlp.ts
@@ -150,6 +150,7 @@ export function buildLocalOtlpProjection(args: { identity: RunEventIdentity, rec
                 caseId: record.caseId,
                 scores: record.scores,
                 state: record.state,
+                unscored: record.unscored,
               }),
             },
             eventName: 'vieval.case',
@@ -163,12 +164,12 @@ export function buildLocalOtlpProjection(args: { identity: RunEventIdentity, rec
       resourceMetrics: [{
         resource,
         scopeMetrics: [{
-          metrics: collectScoreKinds(args.records).map(kind => ({
+          metrics: collectScoreNames(args.records).map(name => ({
             gauge: {
               dataPoints: args.records
-                .filter(record => typeof record.scores[kind] === 'number')
+                .filter(record => typeof record.scores[name] === 'number')
                 .map(record => ({
-                  asDouble: record.scores[kind]!,
+                  asDouble: record.scores[name]!,
                   attributes: toAttributes({
                     ...record.metrics,
                     'vieval.case.id': record.caseId,
@@ -177,7 +178,7 @@ export function buildLocalOtlpProjection(args: { identity: RunEventIdentity, rec
                   timeUnixNano: isoToUnixNano(record.endedAt),
                 })),
             },
-            name: `vieval.score.${kind}`,
+            name: `vieval.score.${name}`,
           })),
           scope: { name: 'vieval' },
         }],
@@ -208,7 +209,7 @@ function collectProjectNames(records: readonly CaseRecord[]): string[] {
     .sort((left, right) => left.localeCompare(right))
 }
 
-function collectScoreKinds(records: readonly CaseRecord[]): string[] {
+function collectScoreNames(records: readonly CaseRecord[]): string[] {
   return [...new Set(records.flatMap(record => Object.keys(record.scores)))]
     .sort((left, right) => left.localeCompare(right))
 }
diff --git a/packages/vieval/src/cli/report-records.test.ts b/packages/vieval/src/cli/report-records.test.ts
index 77a7168..9c16dcb 100644
--- a/packages/vieval/src/cli/report-records.test.ts
+++ b/packages/vieval/src/cli/report-records.test.ts
@@ -1,13 +1,43 @@
-import type { RunEventEnvelope } from '../events'
+import type { ScoredCaseOutcome } from '../core/scoring'
+import type { RunEventEnvelope, ScoreDefinedData } from '../events'
+import type { CaseRecord } from './report-records'
 
-import { describe, expect, it } from 'vitest'
+import { mkdtemp, rm, writeFile } from 'node:fs/promises'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
 
-import { buildCaseRecords, buildMetricsSummary, encodeJsonl } from './report-records'
+import { afterEach, describe, expect, it } from 'vitest'
+
+import { scoreCases } from '../core/scoring'
+import { buildCaseRecords, buildMetricsSummary, encodeJsonl, readCaseRecordFile, summarizeRecordScores } from './report-records'
 
 let nextSeq = 0
+const temporaryDirectories: string[] = []
+
+afterEach(async () => {
+  await Promise.all(temporaryDirectories.splice(0).map(directory => rm(directory, { force: true, recursive: true })))
+})
+
+/**
+ * Creates the events of one settled case `caseId` of `task-1`: a start, the given score events, and an end.
+ */
+function caseEventsOf(caseId: string, state: string, scoreEvents: Array<Record<string, unknown>> = []): RunEventEnvelope[] {
+  return [
+    envelopeOf('case.started', { index: 0, name: caseId, total: 1 }, { caseId }),
+    ...scoreEvents.map(data => envelopeOf('score.recorded', data, { caseId })),
+    envelopeOf('case.ended', { index: 0, name: caseId, state, total: 1 }, { caseId }),
+  ]
+}
+
+/**
+ * Creates one task-scoped `score.defined` envelope of `task-1`.
+ */
+function definitionOf(data: ScoreDefinedData): RunEventEnvelope {
+  return envelopeOf('score.defined', data, { caseId: undefined })
+}
 
 /**
- * Creates one schema 2 envelope for `case-1` of `task-1`; `overrides` replace envelope fields.
+ * Creates one schema 3 envelope for `case-1` of `task-1`; `overrides` replace envelope fields.
  */
 function envelopeOf(event: string, data: unknown, overrides: Partial<Omit<RunEventEnvelope, 'data' | 'event'>> = {}): RunEventEnvelope {
   nextSeq += 1
@@ -19,7 +49,7 @@ function envelopeOf(event: string, data: unknown, overrides: Partial<Omit<RunEve
     experimentId: 'default',
     projectId: 'project-1',
     runId: 'run-1',
-    schemaVersion: 2,
+    schemaVersion: 3,
     seq: nextSeq,
     taskId: 'task-1',
     time: '2026-05-08T00:00:00.000Z',
@@ -28,21 +58,48 @@ function envelopeOf(event: string, data: unknown, overrides: Partial<Omit<RunEve
   }
 }
 
+/**
+ * Creates one current case record; `overrides` replace its fields.
+ */
+function recordOf(overrides: Partial<CaseRecord>): CaseRecord {
+  return {
+    attemptId: 'a',
+    caseId: '1',
+    caseName: '1',
+    durationMs: 1,
+    endedAt: 'e',
+    experimentId: 'e',
+    metrics: {},
+    projectName: 'p',
+    retryCount: 0,
+    runId: 'r',
+    schemaVersion: 2,
+    scoreDetails: {},
+    scores: {},
+    startedAt: 's',
+    state: 'passed',
+    taskId: 't',
+    unscored: {},
+    workspaceId: 'w',
+    ...overrides,
+  }
+}
+
 /**
  * @example
- * describe('report records') verifies lifecycle events become inspectable case artifacts.
+ * describe('buildCaseRecords', () => {})
  */
-describe('report records', () => {
+describe('buildCaseRecords', () => {
   /**
    * @example
-   * it('builds a case record from case lifecycle, score, and metric events') checks the minimum report inspect contract.
+   * it('builds a case record with named scores, their details, and metrics', () => {})
    */
-  it('builds a case record from case lifecycle, score, and metric events', () => {
+  it('builds a case record with named scores, their details, and metrics', () => {
     const records = buildCaseRecords({
       events: [
         envelopeOf('case.started', { index: 0, input: { prompt: 'When?' }, name: 'Case 1', retryIndex: 1, total: 1 }, { projectId: 'project-from-event', time: '2026-05-08T00:00:00.000Z' }),
         envelopeOf('metric.recorded', { name: 'benchmark.id', value: 'locomo' }),
-        envelopeOf('score.recorded', { kind: 'exact', score: 0.75 }),
+        envelopeOf('score.recorded', { grader: { id: 'judge', kind: 'model', model: 'gpt-5.5' }, name: 'locomo.correct', rationale: 'The date matches.', value: 0.75 }),
         envelopeOf('case.ended', { index: 0, name: 'Case 1', output: { answer: 'May 8' }, state: 'passed', total: 1 }, { time: '2026-05-08T00:00:01.250Z' }),
       ],
       projectName: 'locomo-lobehub',
@@ -50,7 +107,7 @@ describe('report records', () => {
 
     /**
      * @example
-     * expect(records[0]).toMatchObject({ caseName: 'Case 1', durationMs: 1250, ... })
+     * expect(records[0]).toMatchObject({ schemaVersion: 2, scores: { 'locomo.correct': 0.75 }, ... })
      */
     expect(records).toHaveLength(1)
     expect(records[0]).toMatchObject({
@@ -66,39 +123,119 @@ describe('report records', () => {
       projectName: 'project-from-event',
       retryCount: 1,
       runId: 'run-1',
-      schemaVersion: 1,
-      scores: { exact: 0.75 },
+      schemaVersion: 2,
+      scores: { 'locomo.correct': 0.75 },
       startedAt: '2026-05-08T00:00:00.000Z',
       state: 'passed',
       taskId: 'task-1',
+      unscored: {},
       workspaceId: 'workspace-1',
     })
+    /**
+     * @example
+     * expect(records[0]?.scoreDetails).toEqual({ 'locomo.correct': { grader: {...}, rationale: 'The date matches.' } })
+     */
+    expect(records[0]?.scoreDetails).toEqual({
+      'locomo.correct': { grader: { id: 'judge', kind: 'model', model: 'gpt-5.5' }, rationale: 'The date matches.' },
+    })
   })
 
   /**
    * @example
-   * an errored case record has no exact score; a failed one gets exact 0.
+   * it('counts every case state with the scoring rule of scoreCases', () => {})
    */
-  it('gives a default exact score only to cases that produced a result', () => {
+  it('counts every case state with the scoring rule of scoreCases', () => {
+    // ROOT CAUSE:
+    //
+    // `cases.jsonl` used its own rule, `scores.exact ??= passed ? 1 : 0`, so a failed case kept a value it recorded
+    // before the assertion failed, while `run-summary.json` counted it as 0. The two files disagreed.
+    //
+    // We fixed this by counting records with the state rule of `scoreCases` over the task's score names.
     const records = buildCaseRecords({
       events: [
-        envelopeOf('case.started', { index: 0, name: 'errored', total: 2 }, { caseId: 'case-errored' }),
-        envelopeOf('case.ended', { errorMessage: 'fetch failed', errorName: 'TypeError', index: 0, name: 'errored', state: 'errored', total: 2 }, { caseId: 'case-errored' }),
-        envelopeOf('case.started', { index: 1, name: 'failed', total: 2 }, { caseId: 'case-failed' }),
-        envelopeOf('case.ended', { errorMessage: 'expected 1 to be 2', errorName: 'AssertionError', index: 1, name: 'failed', state: 'failed', total: 2 }, { caseId: 'case-failed' }),
+        definitionOf({ name: 'correct' }),
+        definitionOf({ errors: 'zero', name: 'answered' }),
+        ...caseEventsOf('passed', 'passed', [{ name: 'correct', value: 1 }, { name: 'answered', value: 1 }]),
+        ...caseEventsOf('passed-missing', 'passed', [{ name: 'answered', value: 1 }]),
+        ...caseEventsOf('passed-unscored', 'passed', [{ name: 'correct', unscored: { reason: 'grader_failed' } }, { name: 'answered', value: 1 }]),
+        ...caseEventsOf('failed', 'failed', [{ name: 'correct', value: 1 }]),
+        ...caseEventsOf('errored', 'errored'),
+        ...caseEventsOf('timeout', 'timeout'),
+        ...caseEventsOf('skipped', 'skipped'),
       ],
       projectName: 'project-1',
     })
+    const failed = records.find(record => record.caseName === 'failed')
 
+    /**
+     * @example
+     * expect(failed?.scores).toEqual({ answered: 0, correct: 0 })
+     */
+    // The per-state rule itself is `countCase`, tested in `core/scoring`; here the records must use it.
+    expect(failed?.scores).toEqual({ answered: 0, correct: 0 })
+
+    const outcomes: ScoredCaseOutcome[] = [
+      { scores: { answered: 1, correct: 1 }, state: 'passed', unscored: {} },
+      { scores: { answered: 1 }, state: 'passed', unscored: {} },
+      { scores: { answered: 1 }, state: 'passed', unscored: { correct: 'grader_failed' } },
+      { scores: { correct: 1 }, state: 'failed', unscored: {} },
+      { scores: {}, state: 'errored', unscored: {} },
+      { scores: {}, state: 'timeout', unscored: {} },
+      { scores: {}, state: 'skipped', unscored: {} },
+    ]
+    const expected = scoreCases(outcomes, [{ name: 'correct' }, { errors: 'zero', name: 'answered' }]).scores
+    /**
+     * @example
+     * expect(summarizeRecordScores(records)).toEqual({ answered: <scoreCases summary>, correct: <scoreCases summary> })
+     */
+    expect(summarizeRecordScores(records)).toEqual(Object.fromEntries(expected.map(summary => [summary.name, summary])))
+  })
+
+  /**
+   * @example
+   * it('keeps the last of a value and an unscored record of one name', () => {})
+   */
+  it('keeps the last of a value and an unscored record of one name', () => {
+    const records = buildCaseRecords({
+      events: [
+        ...caseEventsOf('rescored', 'passed', [{ name: 'natural', unscored: { reason: 'invalid_response' } }, { name: 'natural', value: 0.5 }]),
+        ...caseEventsOf('unscored', 'passed', [{ name: 'natural', rationale: 'first try', value: 1 }, { name: 'natural', unscored: { reason: 'refusal' } }]),
+      ],
+      projectName: 'project-1',
+    })
     const byName = Object.fromEntries(records.map(record => [record.caseName, record]))
-    expect(byName.errored?.state).toBe('errored')
-    expect(byName.errored?.scores.exact).toBeUndefined()
-    expect(byName.failed?.scores.exact).toBe(0)
+
+    /**
+     * @example
+     * expect(byName.unscored?.unscored).toEqual({ natural: 'refusal' })
+     */
+    expect(byName.rescored?.scores).toEqual({ natural: 0.5 })
+    expect(byName.rescored?.unscored).toEqual({})
+    expect(byName.unscored?.scores).toEqual({})
+    expect(byName.unscored?.unscored).toEqual({ natural: 'refusal' })
+    expect(byName.unscored?.scoreDetails).toEqual({})
+  })
+
+  /**
+   * @example
+   * it('marks derived scores in their details', () => {})
+   */
+  it('marks derived scores in their details', () => {
+    const records = buildCaseRecords({
+      events: caseEventsOf('case', 'passed', [{ name: 'correct', value: 1 }, { derived: true, name: 'pass', value: 1 }]),
+      projectName: 'project-1',
+    })
+
+    /**
+     * @example
+     * expect(records[0]?.scoreDetails).toEqual({ pass: { derived: true } })
+     */
+    expect(records[0]?.scoreDetails).toEqual({ pass: { derived: true } })
   })
 
   /**
    * @example
-   * it('uses the fallback project name and ignores invalid durations') covers events from custom project executors.
+   * it('uses the fallback project name and ignores invalid durations', () => {})
    */
   it('uses the fallback project name and ignores invalid durations', () => {
     const records = buildCaseRecords({
@@ -122,7 +259,7 @@ describe('report records', () => {
 
   /**
    * @example
-   * it('skips events whose payload does not match the event schema') keeps one malformed line from breaking a report.
+   * it('skips events whose payload does not match the event schema', () => {})
    */
   it('skips events whose payload does not match the event schema', () => {
     const records = buildCaseRecords({
@@ -142,13 +279,13 @@ describe('report records', () => {
 
   /**
    * @example
-   * it('keeps only the final retry attempt payload while preserving full case duration') verifies retry event ordering.
+   * it('keeps only the final retry payload while preserving full case duration', () => {})
    */
-  it('keeps only the final retry attempt payload while preserving full case duration', () => {
+  it('keeps only the final retry payload while preserving full case duration', () => {
     const records = buildCaseRecords({
       events: [
         envelopeOf('case.started', { autoRetry: 1, index: 0, name: 'case', retryIndex: 0, total: 1 }, { time: '2026-05-08T00:00:00.000Z' }),
-        envelopeOf('score.recorded', { kind: 'exact', score: 0.2 }),
+        envelopeOf('score.recorded', { name: 'correct', value: 0.2 }),
         envelopeOf('metric.recorded', { name: 'attempt.metric', value: 'failed-attempt' }),
         envelopeOf('case.started', { autoRetry: 1, index: 0, name: 'case', retryIndex: 1, total: 1 }, { time: '2026-05-08T00:00:03.000Z' }),
         envelopeOf('metric.recorded', { name: 'attempt.metric', value: 'final-attempt' }),
@@ -159,34 +296,41 @@ describe('report records', () => {
 
     /**
      * @example
-     * expect(records[0]?.metrics).toEqual({ 'attempt.metric': 'final-attempt' })
+     * expect(records[0]?.scores).toEqual({})
      */
     expect(records).toHaveLength(1)
     expect(records[0]?.startedAt).toBe('2026-05-08T00:00:00.000Z')
     expect(records[0]?.endedAt).toBe('2026-05-08T00:00:05.000Z')
     expect(records[0]?.durationMs).toBe(5000)
     expect(records[0]?.retryCount).toBe(1)
-    expect(records[0]?.scores).toEqual({ exact: 1 })
+    // The final try recorded no score, and a passed case without a value has no score.
+    expect(records[0]?.scores).toEqual({})
     expect(records[0]?.metrics).toEqual({ 'attempt.metric': 'final-attempt' })
   })
 
   /**
    * @example
-   * it('keeps every auto attempt of a case and counts only in-attempt retries') verifies `attemptIndex` projection.
+   * it('keeps every auto attempt of a case with counted scores and counts only in-attempt retries', () => {})
    */
-  it('keeps every auto attempt of a case and counts only in-attempt retries', () => {
+  it('keeps every auto attempt of a case with counted scores and counts only in-attempt retries', () => {
     const records = buildCaseRecords({
       events: [
         envelopeOf('case.started', { attemptIndex: 0, autoAttempt: 1, autoRetry: 1, index: 0, name: 'Case 1', retryIndex: 0, total: 1 }, { time: '2026-05-08T00:00:00.000Z' }),
         envelopeOf('case.started', { attemptIndex: 0, autoAttempt: 1, autoRetry: 1, index: 0, name: 'Case 1', retryIndex: 1, total: 1 }, { time: '2026-05-08T00:00:01.000Z' }),
+        envelopeOf('score.recorded', { name: 'correct', value: 1 }),
         envelopeOf('case.ended', { attemptIndex: 0, autoAttempt: 1, errorMessage: 'boom', index: 0, name: 'Case 1', state: 'failed', total: 1 }, { time: '2026-05-08T00:00:02.000Z' }),
         envelopeOf('case.started', { attemptIndex: 1, autoAttempt: 1, autoRetry: 1, index: 0, name: 'Case 1', retryIndex: 0, total: 1 }, { time: '2026-05-08T00:00:03.000Z' }),
-        envelopeOf('score.recorded', { kind: 'exact', score: 0.5 }),
+        envelopeOf('score.recorded', { name: 'correct', value: 0.5 }),
+        envelopeOf('score.recorded', { name: 'natural', unscored: { reason: 'no_response' } }),
         envelopeOf('case.ended', { attemptIndex: 1, autoAttempt: 1, index: 0, name: 'Case 1', state: 'passed', total: 1 }, { time: '2026-05-08T00:00:03.500Z' }),
       ],
       projectName: 'project-1',
     })
 
+    /**
+     * @example
+     * expect(records[0]?.attempts?.[0]?.scores).toEqual({ correct: 0, natural: 0 })
+     */
     expect(records).toHaveLength(1)
     expect(records[0]).toMatchObject({
       attemptId: 'attempt-1',
@@ -194,8 +338,9 @@ describe('report records', () => {
       autoAttempt: 1,
       durationMs: 3500,
       retryCount: 0,
-      scores: { exact: 0.5 },
+      scores: { correct: 0.5 },
       state: 'passed',
+      unscored: { natural: 'no_response' },
     })
     expect(records[0]?.attempts).toEqual([
       {
@@ -204,25 +349,28 @@ describe('report records', () => {
         endedAt: '2026-05-08T00:00:02.000Z',
         errorMessage: 'boom',
         retryCount: 1,
-        scores: { exact: 0 },
+        // The failed attempt scores 0 on every score of the task, including `natural` from the later attempt.
+        scores: { correct: 0, natural: 0 },
         startedAt: '2026-05-08T00:00:00.000Z',
         state: 'failed',
+        unscored: {},
       },
       {
         attemptIndex: 1,
         durationMs: 500,
         endedAt: '2026-05-08T00:00:03.500Z',
         retryCount: 0,
-        scores: { exact: 0.5 },
+        scores: { correct: 0.5 },
         startedAt: '2026-05-08T00:00:03.000Z',
         state: 'passed',
+        unscored: { natural: 'no_response' },
       },
     ])
   })
 
   /**
    * @example
-   * it('omits auto attempt fields for logs without attemptIndex') keeps records of other tasks and older logs unchanged.
+   * it('omits auto attempt fields for logs without attemptIndex', () => {})
    */
   it('omits auto attempt fields for logs without attemptIndex', () => {
     const records = buildCaseRecords({
@@ -233,6 +381,10 @@ describe('report records', () => {
       projectName: 'project-1',
     })
 
+    /**
+     * @example
+     * expect(Object.keys(records[0]!).sort()).toEqual(['attemptId', ..., 'workspaceId'])
+     */
     expect(Object.keys(records[0]!).sort()).toEqual([
       'attemptId',
       'caseId',
@@ -245,72 +397,95 @@ describe('report records', () => {
       'retryCount',
       'runId',
       'schemaVersion',
+      'scoreDetails',
       'scores',
       'startedAt',
       'state',
       'taskId',
+      'unscored',
       'workspaceId',
     ])
   })
+})
 
+/**
+ * @example
+ * describe('buildMetricsSummary', () => {})
+ */
+describe('buildMetricsSummary', () => {
   /**
    * @example
-   * it('adds default exact scores from terminal case state') keeps ordinary pass/fail cases visible in metric projections.
+   * it('summarizes scores by name overall and per metric group', () => {})
    */
-  it('adds default exact scores from terminal case state', () => {
-    const records = buildCaseRecords({
-      events: [
-        envelopeOf('case.started', { index: 0, name: 'passed-case', total: 2 }, { caseId: 'passed-case' }),
-        envelopeOf('case.ended', { index: 0, name: 'passed-case', state: 'passed', total: 2 }, { caseId: 'passed-case' }),
-        envelopeOf('case.started', { index: 1, name: 'failed-case', total: 2 }, { caseId: 'failed-case' }),
-        envelopeOf('case.ended', { index: 1, name: 'failed-case', state: 'failed', total: 2 }, { caseId: 'failed-case' }),
-      ],
-      projectName: 'project-1',
-    })
+  it('summarizes scores by name overall and per metric group', () => {
+    const summary = buildMetricsSummary([
+      recordOf({ caseId: '1', metrics: { 'benchmark.category': 2 }, scores: { correct: 1, natural: 0.5 } }),
+      recordOf({ caseId: '2', metrics: { 'benchmark.category': 2 }, scores: { correct: 0 }, unscored: { natural: 'grader_failed' } }),
+      recordOf({ caseId: '3', metrics: { 'benchmark.category': 3 }, scores: { correct: 0.5, natural: 1 } }),
+      recordOf({ caseId: '4', metrics: { nullable: null }, scores: {}, state: 'errored' }),
+      recordOf({ caseId: '5', scores: { other: 1 }, taskId: 'other-task' }),
+    ], ['benchmark.category', 'taskId', 'nullable'])
 
     /**
      * @example
-     * expect(records.map(record => [record.caseId, record.scores.exact])).toEqual([['passed-case', 1], ['failed-case', 0]])
+     * expect(summary.overall.correct).toEqual({ mean: 0.5, n: 3, name: 'correct', sum: 1.5, total: 4, unscored: {} })
      */
-    expect(records.map(record => [record.caseId, record.scores.exact])).toEqual([
-      ['passed-case', 1],
-      ['failed-case', 0],
-    ])
+    expect(summary.overall.correct).toEqual({ mean: 0.5, n: 3, name: 'correct', sum: 1.5, total: 4, unscored: {} })
+    expect(summary.overall.natural).toEqual({ mean: 0.75, n: 2, name: 'natural', sum: 1.5, total: 4, unscored: { grader_failed: 1 } })
+    // A record of another task does not count toward `total` of this task's scores, and the reverse.
+    expect(summary.overall.other).toEqual({ mean: 1, n: 1, name: 'other', sum: 1, total: 1, unscored: {} })
+    expect(summary.groups['benchmark.category=2']?.correct?.mean).toBe(0.5)
+    expect(summary.groups['benchmark.category=3']?.correct?.mean).toBe(0.5)
+    expect(summary.groups['taskId=t']?.correct?.mean).toBe(0.5)
+    expect(summary.groups['nullable=null']?.correct).toBeUndefined()
   })
+})
 
+/**
+ * @example
+ * describe('readCaseRecordFile', () => {})
+ */
+describe('readCaseRecordFile', () => {
   /**
    * @example
-   * it('summarizes score averages by score kind and metric group') verifies generic group-by analysis input.
+   * it('reads current records and rejects records of an older vieval', async () => {})
    */
-  it('summarizes score averages by score kind and metric group', () => {
-    const summary = buildMetricsSummary([
-      { attemptId: 'a', caseId: '1', caseName: '1', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.category': 2 }, projectName: 'p', retryCount: 0, runId: 'r', schemaVersion: 1, scores: { exact: 1, judge: 0.5 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-      { attemptId: 'a', caseId: '2', caseName: '2', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.category': 2 }, projectName: 'p', retryCount: 0, runId: 'r', schemaVersion: 1, scores: { exact: 0 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-      { attemptId: 'a', caseId: '3', caseName: '3', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { 'benchmark.category': 3 }, projectName: 'p', retryCount: 0, runId: 'r', schemaVersion: 1, scores: { exact: 0.5, judge: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-      { attemptId: 'a', caseId: '4', caseName: '4', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: { nullable: null }, projectName: 'p', retryCount: 0, runId: 'r', schemaVersion: 1, scores: { exact: 1 }, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-    ], ['benchmark.category', 'taskId', 'nullable'])
+  it('reads current records and rejects records of an older vieval', async () => {
+    const directory = await mkdtemp(join(tmpdir(), 'vieval-case-records-'))
+    temporaryDirectories.push(directory)
+    const currentPath = join(directory, 'current.jsonl')
+    const olderPath = join(directory, 'older.jsonl')
+    await writeFile(currentPath, encodeJsonl([recordOf({ caseId: '1' })]), 'utf-8')
+    await writeFile(olderPath, `${JSON.stringify({ ...recordOf({ caseId: '1' }), schemaVersion: 1, scores: { exact: 1 } })}\n`, 'utf-8')
 
-    expect(summary.overall.exact.count).toBe(4)
-    expect(summary.overall.exact.sum).toBe(2.5)
-    expect(summary.overall.exact.average).toBe(0.625)
-    expect(summary.overall.judge.count).toBe(2)
-    expect(summary.overall.judge.average).toBe(0.75)
-    expect(summary.groups['benchmark.category=2']?.exact.average).toBe(0.5)
-    expect(summary.groups['benchmark.category=3']?.exact.average).toBe(0.5)
-    expect(summary.groups['taskId=t']?.exact.average).toBe(0.625)
-    expect(summary.groups['nullable=null']?.exact.average).toBe(1)
+    /**
+     * @example
+     * expect(() => readCaseRecordFile(olderPath)).toThrow('This report was written by an older vieval. Run it again.')
+     */
+    expect(readCaseRecordFile(currentPath).map(record => record.caseId)).toEqual(['1'])
+    expect(() => readCaseRecordFile(olderPath)).toThrow('This report was written by an older vieval. Run it again.')
   })
+})
 
+/**
+ * @example
+ * describe('encodeJsonl', () => {})
+ */
+describe('encodeJsonl', () => {
   /**
    * @example
-   * it('encodes case records as newline-delimited JSON') documents the local dataframe-friendly format.
+   * it('encodes case records as newline-delimited JSON', () => {})
    */
   it('encodes case records as newline-delimited JSON', () => {
     const jsonl = encodeJsonl([
-      { attemptId: 'a', caseId: '1', caseName: '1', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId: 'r', schemaVersion: 1, scores: {}, startedAt: 's', state: 'passed', taskId: 't', workspaceId: 'w' },
-      { attemptId: 'a', caseId: '2', caseName: '2', durationMs: 1, endedAt: 'e', experimentId: 'e', metrics: {}, projectName: 'p', retryCount: 0, runId: 'r', schemaVersion: 1, scores: {}, startedAt: 's', state: 'failed', taskId: 't', workspaceId: 'w' },
+      recordOf({ caseId: '1' }),
+      recordOf({ caseId: '2', state: 'failed' }),
     ])
 
+    /**
+     * @example
+     * expect(jsonl.split('\n')).toHaveLength(3)
+     */
     expect(jsonl.endsWith('\n')).toBe(true)
     expect(jsonl.split('\n')).toHaveLength(3)
     expect(jsonl.trim()).toContain('"caseId":"1"')
@@ -319,9 +494,13 @@ describe('report records', () => {
 
   /**
    * @example
-   * it('encodes empty case records as an empty string') fixes the no-record artifact behavior.
+   * it('encodes empty case records as an empty string', () => {})
    */
   it('encodes empty case records as an empty string', () => {
+    /**
+     * @example
+     * expect(encodeJsonl([])).toBe('')
+     */
     expect(encodeJsonl([])).toBe('')
   })
 })
diff --git a/packages/vieval/src/cli/report-records.ts b/packages/vieval/src/cli/report-records.ts
index 8d43ad9..230ba5a 100644
--- a/packages/vieval/src/cli/report-records.ts
+++ b/packages/vieval/src/cli/report-records.ts
@@ -1,10 +1,18 @@
-import type { CaseOutcomeState } from '../core/scoring'
-import type { CaseEndedData, CaseStartedData, RunEventEnvelope } from '../events'
+import type { CaseOutcomeState, ScoreSummary } from '../core/scoring'
+import type { CaseEndedData, CaseStartedData, Grader, RunEventEnvelope, ScoreDefinedData } from '../events'
+
+import { readFileSync } from 'node:fs'
+
+import { errorMessageFrom } from '@moeru/std'
 
 import * as v from 'valibot'
 
-import { isMeasuredState } from '../core/scoring'
-import { caseEnded, caseEndedDataSchema, caseStarted, caseStartedDataSchema, metricRecorded, metricRecordedDataSchema, scoreRecorded, scoreRecordedDataSchema } from '../events'
+import { countCase } from '../core/scoring'
+import { caseEnded, caseEndedDataSchema, caseStarted, caseStartedDataSchema, metricRecorded, metricRecordedDataSchema, scoreDefined, scoreDefinedDataSchema, scoreRecorded, scoreRecordedDataSchema } from '../events'
+import { OLDER_REPORT_MESSAGE } from './report-log'
+
+/** Current version of {@link CaseRecord}; `cases.jsonl` readers reject other versions. */
+export const CASE_RECORD_SCHEMA_VERSION = 2
 
 /** Arguments for building normalized case records from report events. */
 export interface BuildCaseRecordsArgs {
@@ -28,12 +36,14 @@ export interface CaseAttemptRecord {
   errorMessage?: string
   /** Retries within this attempt. */
   retryCount: number
-  /** Score values of this attempt keyed by score kind. */
+  /** Counted score values of this attempt by score name, with the same rule as {@link CaseRecord.scores}. */
   scores: Record<string, number>
   /** Timestamp of the attempt's first `case.started`. */
   startedAt: string
   /** Terminal state of this attempt. */
   state: CaseRecordState
+  /** Scores this attempt recorded as unscored, by score name, with the reason. */
+  unscored: Record<string, string>
 }
 
 /** Scalar and array metric values persisted in case records. */
@@ -95,8 +105,17 @@ export interface CaseRecord {
   /** Run identity attached to the run. */
   runId: string
   /** Case record schema version. */
-  schemaVersion: 1
-  /** Score values keyed by score kind. */
+  schemaVersion: typeof CASE_RECORD_SCHEMA_VERSION
+  /** What each score record of the final outcome carried (rationale, grader, metadata), by score name. */
+  scoreDetails: Record<string, CaseScoreDetail>
+  /**
+   * Counted score values by score name, with the rule of `scoreCases`, so `cases.jsonl` and `run-summary.json`
+   * agree:
+   * - `passed`: the recorded value; a score the case did not record, or recorded as unscored, is absent
+   * - `failed`: 0 for every score of the task, whatever the case recorded
+   * - `errored` and `timeout`: 0 for scores declared `errors: 'zero'`, otherwise absent
+   * - `skipped`: no score
+   */
   scores: Record<string, number>
   /** ISO-ish case start timestamp, or the best event timestamp available. */
   startedAt: string
@@ -104,6 +123,8 @@ export interface CaseRecord {
   state: CaseRecordState
   /** Scheduled task id that owns the case. */
   taskId: string
+  /** Scores the final outcome recorded as unscored, by score name, with the reason. They never count. */
+  unscored: Record<string, string>
   /** Workspace identity attached to the run. */
   workspaceId: string
 }
@@ -111,31 +132,37 @@ export interface CaseRecord {
 /** Terminal state stored for one normalized case record. */
 export type CaseRecordState = CaseOutcomeState
 
-/** Generic metrics summary for overall scores and requested group facets. */
+/**
+ * What one `score.recorded` of a case carried besides its value.
+ */
+export interface CaseScoreDetail {
+  /** Whether the runtime derived the value from other scores. */
+  derived?: boolean
+  /** What produced the value, when the record named it. */
+  grader?: Grader
+  /** Other details of the value. */
+  metadata?: Record<string, unknown>
+  /** Why the score has its value. */
+  rationale?: string
+}
+
+/** Content of `metrics-summary.json`: score summaries overall and per requested group facet. */
 export interface MetricsSummary {
-  /** Score aggregates keyed by `<groupKey>=<groupValue>`. */
-  groups: Record<string, ScoreSummary>
-  /** Overall score aggregates keyed by score kind. */
-  overall: ScoreSummary
+  /** Score summaries by score name, per group keyed by `<groupKey>=<groupValue>`. */
+  groups: Record<string, Record<string, ScoreSummary>>
+  /** Score summaries by score name over every record. */
+  overall: Record<string, ScoreSummary>
 }
 
-/** Generic score summary keyed by score kind. */
-export type ScoreSummary = Record<string, ScoreSummaryBucket>
-
-/** Score aggregate for one score kind. */
-export interface ScoreSummaryBucket {
-  /** Average score for this kind, or `0` when count is zero. */
-  average: number
-  /** Number of records that provided this score kind. */
-  count: number
-  /** Sum of all score values for this kind. */
-  sum: number
+/** One auto attempt as recorded, before the task's score names are known. */
+interface AttemptDraft extends Omit<CaseAttemptRecord, 'scores'> {
+  recorded: Record<string, number>
 }
 
 interface CaseDraft {
   attemptId: string
   attemptIndex?: number
-  attempts?: CaseAttemptRecord[]
+  attempts?: AttemptDraft[]
   attemptStartedAt?: string
   autoAttempt?: number
   caseId: string
@@ -149,11 +176,14 @@ interface CaseDraft {
   requeueCount: number
   retryCount: number
   runId: string
+  scoreDetails: Record<string, CaseScoreDetail>
+  /** Values the final outcome recorded (or the runtime derived), before the state rule counts them. */
   scores: Record<string, number>
   startCount: number
   startedAt?: string
   state?: CaseRecordState
   taskId: string
+  unscored: Record<string, string>
   workspaceId: string
 }
 
@@ -165,18 +195,30 @@ interface CaseDraft {
  * - report commands need one final record per observed case outcome
  *
  * Expects:
- * - events are schema 2 envelopes ordered by `seq`
- * - case events carry `taskId` and `caseId`; payloads that fail their event schema are skipped
+ * - events are schema 3 envelopes ordered by `seq`
+ * - case events carry `taskId` and `caseId`, `score.defined` carries `taskId`; payloads that fail their event schema
+ *   are skipped
  *
  * Returns:
- * - records for cases that emitted an end lifecycle event
+ * - records for cases that emitted an end lifecycle event, with scores counted by the state rule of `scoreCases`
+ *   over the task's scores: its `score.defined` names plus every name a case of the task recorded
  */
 export function buildCaseRecords(args: BuildCaseRecordsArgs): CaseRecord[] {
   const drafts = new Map<string, CaseDraft>()
   const completedKeys: string[] = []
+  const definitionsByTask = new Map<string, Map<string, ScoreDefinedData>>()
 
   for (const event of args.events) {
     const { caseId, taskId } = event
+    if (taskId != null && event.event === scoreDefined.id) {
+      const data = parseData(scoreDefinedDataSchema, event)
+      if (data != null) {
+        const definitions = definitionsByTask.get(taskId) ?? new Map<string, ScoreDefinedData>()
+        definitions.set(data.name, data)
+        definitionsByTask.set(taskId, definitions)
+      }
+      continue
+    }
     if (caseId == null || taskId == null) {
       continue
     }
@@ -196,7 +238,7 @@ export function buildCaseRecords(args: BuildCaseRecordsArgs): CaseRecord[] {
     else if (event.event === scoreRecorded.id) {
       const data = parseData(scoreRecordedDataSchema, event)
       if (data != null) {
-        getOrCreateDraft(drafts, event, args, caseId).scores[data.kind] = data.score
+        applyScoreRecord(getOrCreateDraft(drafts, event, args, caseId), data)
       }
     }
     else if (event.event === caseEnded.id) {
@@ -211,33 +253,41 @@ export function buildCaseRecords(args: BuildCaseRecordsArgs): CaseRecord[] {
     }
   }
 
-  return completedKeys
+  const completed = completedKeys
     .map(key => drafts.get(key))
     .filter((draft): draft is CaseDraft => draft != null && draft.endedAt != null)
-    .map(toCaseRecord)
+
+  // A failed case scores 0 on every score of its task, so the task's names are collected before any case is counted.
+  const taskScoreNames = new Map<string, Set<string>>()
+  for (const draft of completed) {
+    const names = taskScoreNames.get(draft.taskId) ?? new Set(definitionsByTask.get(draft.taskId)?.keys())
+    for (const outcome of [draft, ...(draft.attempts ?? []).map(attempt => ({ scores: attempt.recorded, unscored: attempt.unscored }))]) {
+      Object.keys(outcome.scores).forEach(name => names.add(name))
+      Object.keys(outcome.unscored).forEach(name => names.add(name))
+    }
+    taskScoreNames.set(draft.taskId, names)
+  }
+
+  return completed.map(draft => toCaseRecord(draft, [...taskScoreNames.get(draft.taskId) ?? []].sort().map(name => definitionsByTask.get(draft.taskId)?.get(name) ?? { name })))
 }
 
 /**
- * Builds generic score summaries overall and grouped by arbitrary keys.
+ * Builds score summaries by score name, overall and grouped by arbitrary keys.
  *
  * Use when:
- * - report artifacts need benchmark-neutral aggregate score views
+ * - report artifacts need benchmark-neutral aggregate score views (`metrics-summary.json`)
  * - callers want to group by metrics such as `benchmark.category` or direct record fields such as `taskId`
  *
  * Expects:
  * - `groupByKeys` are stable metric names or direct `CaseRecord` field names
- * - record score values are normalized numeric scores
+ * - records built by {@link buildCaseRecords}
  *
  * Returns:
- * - overall score buckets and group buckets keyed by `<key>=<value>`
+ * - overall summaries and group summaries keyed by `<key>=<value>`, each from {@link summarizeRecordScores}
  */
 export function buildMetricsSummary(records: readonly CaseRecord[], groupByKeys: readonly string[]): MetricsSummary {
-  const overall: ScoreSummary = {}
-  const groups: Record<string, ScoreSummary> = {}
-
+  const groups = new Map<string, CaseRecord[]>()
   for (const record of records) {
-    addRecordScores(overall, record)
-
     for (const groupByKey of groupByKeys) {
       const groupValue = getGroupValue(record, groupByKey)
       if (!groupValue.exists) {
@@ -245,14 +295,13 @@ export function buildMetricsSummary(records: readonly CaseRecord[], groupByKeys:
       }
 
       const groupKey = `${groupByKey}=${String(groupValue.value)}`
-      groups[groupKey] ??= {}
-      addRecordScores(groups[groupKey], record)
+      groups.set(groupKey, [...(groups.get(groupKey) ?? []), record])
     }
   }
 
   return {
-    groups: finalizeSummaryGroups(groups),
-    overall: finalizeScoreSummary(overall),
+    groups: Object.fromEntries([...groups].map(([groupKey, groupRecords]) => [groupKey, summarizeRecordScores(groupRecords)])),
+    overall: summarizeRecordScores(records),
   }
 }
 
@@ -277,16 +326,89 @@ export function encodeJsonl(records: readonly CaseRecord[]): string {
   return `${records.map(record => JSON.stringify(record)).join('\n')}\n`
 }
 
-function addRecordScores(summary: ScoreSummary, record: CaseRecord): void {
-  for (const [kind, score] of Object.entries(record.scores)) {
-    if (!Number.isFinite(score)) {
-      continue
+/**
+ * Reads the case records of one `cases.jsonl` file.
+ *
+ * Use when:
+ * - a report command or `--resume` reads the cases of a finished run
+ *
+ * Expects:
+ * - one {@link CaseRecord} JSON object per non-empty line
+ *
+ * Returns:
+ * - records in file order
+ * - throws {@link OLDER_REPORT_MESSAGE} for a record of an older schema version, and a parse error with the line
+ *   number for a line that is not JSON
+ */
+export function readCaseRecordFile(filePath: string): CaseRecord[] {
+  return readFileSync(filePath, 'utf-8').split('\n').flatMap((line, index) => {
+    if (line.trim() === '') {
+      return []
+    }
+
+    let record: CaseRecord
+    try {
+      record = JSON.parse(line) as CaseRecord
     }
+    catch (error) {
+      throw new Error(`Invalid cases.jsonl line ${index + 1} in "${filePath}": ${errorMessageFrom(error) ?? 'Unknown JSON parse failure.'}`)
+    }
+    if (record.schemaVersion !== CASE_RECORD_SCHEMA_VERSION) {
+      throw new Error(`${OLDER_REPORT_MESSAGE} ("${filePath}" line ${index + 1}, case record schema version ${String(record.schemaVersion ?? 'none')})`)
+    }
+    return [record]
+  })
+}
+
+/**
+ * Pools the counted scores of case records into one summary per score name.
+ *
+ * Use when:
+ * - `metrics-summary.json` or `vieval report cases --group-by` summarizes a group of records
+ *
+ * Expects:
+ * - records built by {@link buildCaseRecords}, so `scores` already holds counted values
+ *
+ * Returns:
+ * - per score name, sorted by name: `sum` and `n` over the records that count it, `total` over the records of
+ *   tasks that have the score and were not skipped, and the unscored reasons of passed records; the same numbers
+ *   `scoreCases` gives for the cases of the records
+ */
+export function summarizeRecordScores(records: readonly CaseRecord[]): Record<string, ScoreSummary> {
+  // A record of a task without the score must not add to its `total`, so names are resolved per task.
+  const namesByTask = new Map<string, Set<string>>()
+  for (const record of records) {
+    const names = namesByTask.get(record.taskId) ?? new Set<string>()
+    Object.keys(record.scores).forEach(name => names.add(name))
+    Object.keys(record.unscored).forEach(name => names.add(name))
+    namesByTask.set(record.taskId, names)
+  }
 
-    summary[kind] ??= { average: 0, count: 0, sum: 0 }
-    summary[kind].count += 1
-    summary[kind].sum += score
+  const summaries = new Map<string, ScoreSummary>()
+  for (const record of records) {
+    if (record.state === 'skipped') {
+      continue
+    }
+    for (const name of namesByTask.get(record.taskId) ?? []) {
+      const summary = summaries.get(name) ?? { mean: null, n: 0, name, sum: 0, total: 0, unscored: {} }
+      summaries.set(name, summary)
+      summary.total += 1
+      const value = record.scores[name]
+      if (value != null && Number.isFinite(value)) {
+        summary.n += 1
+        summary.sum += value
+        continue
+      }
+      const reason = record.state === 'passed' ? record.unscored[name] : undefined
+      if (reason != null) {
+        summary.unscored[reason] = (summary.unscored[reason] ?? 0) + 1
+      }
+    }
   }
+
+  return Object.fromEntries([...summaries.values()]
+    .sort((left, right) => left.name.localeCompare(right.name))
+    .map(summary => [summary.name, { ...summary, mean: summary.n === 0 ? null : summary.sum / summary.n }]))
 }
 
 function applyCaseEnd(draft: CaseDraft, event: RunEventEnvelope, data: CaseEndedData): void {
@@ -294,11 +416,9 @@ function applyCaseEnd(draft: CaseDraft, event: RunEventEnvelope, data: CaseEnded
   draft.endedAt = event.time
   draft.output = data.output ?? draft.output
   draft.state = data.state
-  // Only a case with a result gets a default exact score. An errored or timed-out case measured nothing, so it has
-  // no exact score and report means leave it out, as the task score does.
-  if (isMeasuredState(draft.state)) {
-    draft.scores.exact ??= draft.state === 'passed' ? 1 : 0
-  }
+  // The comment here previously described the default `exact` score of a case with a result. It was removed with the
+  // `exact` score itself: a case that records no score has no score, and the state rule counts values in
+  // `toCaseRecord`, once the task's score names are known.
 
   if (data.attemptIndex != null) {
     const startedAt = draft.attemptStartedAt ?? event.time
@@ -308,10 +428,11 @@ function applyCaseEnd(draft: CaseDraft, event: RunEventEnvelope, data: CaseEnded
       durationMs: calculateDurationMs(startedAt, event.time),
       endedAt: event.time,
       ...(data.errorMessage == null ? {} : { errorMessage: data.errorMessage }),
+      recorded: { ...draft.scores },
       retryCount: draft.retryCount,
-      scores: { ...draft.scores },
       startedAt,
       state: data.state,
+      unscored: { ...draft.unscored },
     })
   }
 }
@@ -323,8 +444,10 @@ function applyCaseStart(draft: CaseDraft, event: RunEventEnvelope, data: CaseSta
   draft.endedAt = undefined
   draft.metrics = {}
   draft.output = undefined
+  draft.scoreDetails = {}
   draft.scores = {}
   draft.state = undefined
+  draft.unscored = {}
   draft.input = data.input
   draft.requeueCount = Math.max(draft.requeueCount, data.requeueIndex ?? 0)
 
@@ -355,6 +478,30 @@ function applyCaseStart(draft: CaseDraft, event: RunEventEnvelope, data: CaseSta
  * - `1250`
  * - `0`
  */
+function applyScoreRecord(draft: CaseDraft, data: v.InferOutput<typeof scoreRecordedDataSchema>): void {
+  // A value and an unscored record of one name replace each other; the last one of the try wins.
+  if ('value' in data) {
+    draft.scores[data.name] = data.value
+    delete draft.unscored[data.name]
+  }
+  else {
+    draft.unscored[data.name] = data.unscored.reason
+    delete draft.scores[data.name]
+  }
+
+  const detail: CaseScoreDetail = {
+    ...(data.derived == null ? {} : { derived: data.derived }),
+    ...(data.grader == null ? {} : { grader: data.grader }),
+    ...(data.metadata == null ? {} : { metadata: data.metadata }),
+    ...(data.rationale == null ? {} : { rationale: data.rationale }),
+  }
+  if (Object.keys(detail).length === 0) {
+    delete draft.scoreDetails[data.name]
+    return
+  }
+  draft.scoreDetails[data.name] = detail
+}
+
 function calculateDurationMs(startedAt: string, endedAt: string): number {
   const started = Date.parse(startedAt)
   const ended = Date.parse(endedAt)
@@ -369,25 +516,6 @@ function createCaseKey(taskId: string, caseId: string): string {
   return `${taskId}\u0000${caseId}`
 }
 
-function finalizeScoreSummary(summary: ScoreSummary): ScoreSummary {
-  return Object.fromEntries(
-    Object.entries(summary).map(([kind, bucket]) => [
-      kind,
-      {
-        average: bucket.count === 0 ? 0 : bucket.sum / bucket.count,
-        count: bucket.count,
-        sum: bucket.sum,
-      },
-    ]),
-  )
-}
-
-function finalizeSummaryGroups(groups: Record<string, ScoreSummary>): Record<string, ScoreSummary> {
-  return Object.fromEntries(
-    Object.entries(groups).map(([key, summary]) => [key, finalizeScoreSummary(summary)]),
-  )
-}
-
 function getGroupValue(record: CaseRecord, key: string): { exists: false } | { exists: true, value: CaseMetricValue } {
   if (Object.hasOwn(record.metrics, key)) {
     return {
@@ -431,9 +559,11 @@ function getOrCreateDraft(
     requeueCount: 0,
     retryCount: 0,
     runId: event.runId,
+    scoreDetails: {},
     scores: {},
     startCount: 0,
     taskId,
+    unscored: {},
     workspaceId: event.workspaceId,
   }
   drafts.set(key, draft)
@@ -456,14 +586,28 @@ function parseData<TSchema extends v.GenericSchema>(schema: TSchema, event: RunE
   return result.success ? result.output : undefined
 }
 
-function toCaseRecord(draft: CaseDraft): CaseRecord {
+/**
+ * Builds the final record of one case, counting its values with `countCase`.
+ *
+ * Expects:
+ * - `taskScores` to list every score of the case's task, so a failed case counts 0 in each
+ */
+function toCaseRecord(draft: CaseDraft, taskScores: readonly ScoreDefinedData[]): CaseRecord {
   const startedAt = draft.startedAt ?? draft.endedAt ?? ''
   const endedAt = draft.endedAt ?? startedAt
+  const state = draft.state ?? 'failed'
 
   return {
     attemptId: draft.attemptId,
     ...(draft.attemptIndex == null ? {} : { attemptIndex: draft.attemptIndex }),
-    ...(draft.attempts == null ? {} : { attempts: draft.attempts }),
+    ...(draft.attempts == null
+      ? {}
+      : {
+          attempts: draft.attempts.map(({ recorded, ...attempt }) => ({
+            ...attempt,
+            scores: countCase({ scores: recorded, state: attempt.state, unscored: attempt.unscored }, taskScores),
+          })),
+        }),
     ...(draft.autoAttempt == null ? {} : { autoAttempt: draft.autoAttempt }),
     caseId: draft.caseId,
     caseName: draft.caseName,
@@ -477,11 +621,13 @@ function toCaseRecord(draft: CaseDraft): CaseRecord {
     ...(draft.requeueCount === 0 ? {} : { requeueCount: draft.requeueCount }),
     retryCount: draft.retryCount,
     runId: draft.runId,
-    schemaVersion: 1,
-    scores: draft.scores,
+    schemaVersion: CASE_RECORD_SCHEMA_VERSION,
+    scoreDetails: draft.scoreDetails,
+    scores: countCase({ scores: draft.scores, state, unscored: draft.unscored }, taskScores),
     startedAt,
-    state: draft.state ?? 'failed',
+    state,
     taskId: draft.taskId,
+    unscored: draft.unscored,
     workspaceId: draft.workspaceId,
   }
 }
diff --git a/packages/vieval/src/cli/resume.test.ts b/packages/vieval/src/cli/resume.test.ts
index 71b280f..220a9cd 100644
--- a/packages/vieval/src/cli/resume.test.ts
+++ b/packages/vieval/src/cli/resume.test.ts
@@ -16,7 +16,7 @@ afterEach(async () => {
 async function writeRun(root: string, path: { attempt: string, run: string }, records: Array<{ attemptIndex?: number, caseName: string, state: string, taskId: string }>): Promise<string> {
   const runDirectory = join(root, 'workspace-a', 'project-a', 'experiment-a', path.attempt, path.run)
   await mkdir(runDirectory, { recursive: true })
-  await writeFile(join(runDirectory, 'cases.jsonl'), `${records.map(record => JSON.stringify({ caseId: `0:${record.caseName}`, ...record })).join('\n')}\n`)
+  await writeFile(join(runDirectory, 'cases.jsonl'), `${records.map(record => JSON.stringify({ caseId: `0:${record.caseName}`, schemaVersion: 2, ...record })).join('\n')}\n`)
   return join(root, 'workspace-a', 'project-a', 'experiment-a', path.attempt)
 }
 
diff --git a/packages/vieval/src/cli/resume.ts b/packages/vieval/src/cli/resume.ts
index 76f1cda..66a3f03 100644
--- a/packages/vieval/src/cli/resume.ts
+++ b/packages/vieval/src/cli/resume.ts
@@ -1,14 +1,14 @@
 import type { CaseRecord } from './report-records'
 
-import { existsSync, readFileSync } from 'node:fs'
+import { existsSync } from 'node:fs'
 import { stat } from 'node:fs/promises'
 import { isAbsolute, join, relative, resolve } from 'node:path'
 
-import { errorMessageFrom } from '@moeru/std'
 import { glob } from 'tinyglobby'
 
 import { isMeasuredState } from '../core/scoring'
 import { sanitizeIdentitySegment } from './report-artifacts'
+import { readCaseRecordFile } from './report-records'
 
 /** `--resume` value that picks the most recent attempt of the experiment. */
 export const LATEST_ATTEMPT = 'latest'
@@ -118,7 +118,7 @@ export function formatResumeHint(input: ResumeHintInput): string | undefined {
  * - case names with a result in their last record (the highest auto attempt), by task id
  *
  * Throws:
- * - when the workspace has no such attempt, or a `cases.jsonl` line is not JSON
+ * - when the workspace has no such attempt, or a `cases.jsonl` line is not JSON or was written by an older vieval
  */
 export async function readMeasuredCases(options: {
   attemptId: string
@@ -141,7 +141,7 @@ export async function readMeasuredCases(options: {
   // Keyed by task and case id: the last record of an auto-attempted case decides whether it has a result.
   const lastRecords = new Map<string, CaseRecord>()
   for (const caseFile of caseFiles) {
-    for (const record of readCaseRecords(caseFile)) {
+    for (const record of readCaseRecordFile(caseFile)) {
       const key = `${record.taskId}\u0000${record.caseId}`
       const previous = lastRecords.get(key)
       if (previous == null || (record.attemptIndex ?? 0) >= (previous.attemptIndex ?? 0)) {
@@ -228,20 +228,6 @@ function quoteShellArgument(value: string): string {
   return /^[\w@%+=:,./-]+$/.test(value) ? value : `'${value.replaceAll('\'', '\'\\\'\'')}'`
 }
 
-function readCaseRecords(filePath: string): CaseRecord[] {
-  return readFileSync(filePath, 'utf-8').split('\n').flatMap((line, index) => {
-    if (line.trim() === '') {
-      return []
-    }
-    try {
-      return [JSON.parse(line) as CaseRecord]
-    }
-    catch (error) {
-      throw new Error(`Invalid cases.jsonl line ${index + 1} in "${filePath}": ${errorMessageFrom(error) ?? 'unknown parse error'}`)
-    }
-  })
-}
-
 /**
  * Makes a config path relative to where the command is pasted, keeping a leading `./` so it reads as a path.
  *
diff --git a/packages/vieval/src/cli/run-hooks.test.ts b/packages/vieval/src/cli/run-hooks.test.ts
index de2de99..b13355a 100644
--- a/packages/vieval/src/cli/run-hooks.test.ts
+++ b/packages/vieval/src/cli/run-hooks.test.ts
@@ -191,7 +191,7 @@ describe('runVievalCli auto attempt reports', () => {
       ['case.ended', '0:chrome%20%231', 1, 1],
       ['case.ended', '1:chrome%20%232', 1, 1],
     ])
-    expect(events.every(event => event.schemaVersion === 2)).toBe(true)
+    expect(events.every(event => event.schemaVersion === 3)).toBe(true)
 
     const flaky = cases.find(record => record.caseName === 'chrome #2')
     expect(flaky).toMatchObject({ attemptIndex: 1, autoAttempt: 1, retryCount: 0, state: 'passed' })
diff --git a/packages/vieval/src/cli/run.test.ts b/packages/vieval/src/cli/run.test.ts
index de64caa..fcfa468 100644
--- a/packages/vieval/src/cli/run.test.ts
+++ b/packages/vieval/src/cli/run.test.ts
@@ -1,4 +1,9 @@
+import type { AggregatedRunResults } from '../core/runner'
+import type { HeadlineResult, ScoreSummary } from '../core/scoring'
+import type { ScoreDefinedData } from '../events'
 import type { RunSummaryFile } from './report-artifacts'
+import type { CaseRecord, MetricsSummary } from './report-records'
+import type { CliProjectSummary } from './run'
 
 import { Buffer } from 'node:buffer'
 import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises'
@@ -23,6 +28,11 @@ const resumeProjectDirectory = join(packageDirectory, 'tests', 'projects', 'resu
 const fixtureProjectDirectory = join(packageDirectory, 'tests', 'projects', 'example-pattern-byoa-bring-your-own-agent')
 const taskProjectDirectory = join(packageDirectory, 'tests', 'projects', 'example-api-defining-new-task')
 const temporaryDirectories: string[] = []
+/**
+ * Source of one score summary that fixture executors and custom tasks return, as `TaskRunOutput.scores` expects; the
+ * built-in task DSL builds the same shape with `scoreCases`.
+ */
+const fixtureScoreSource = `{ mean: 1, n: 1, name: 'fixture.score', sum: 1, total: 1, unscored: {} }`
 const blockingRunStateKey = '__VIEVAL_BLOCKING_RUN_STATE__'
 const caseConcurrencyRunStateKey = '__VIEVAL_CASE_CONCURRENCY_RUN_STATE__'
 const telemetryRunEndStateKey = '__VIEVAL_TELEMETRY_RUN_END_STATE__'
@@ -142,12 +152,7 @@ export default defineEval({
     id: '${evalName}',
     run() {
       return {
-        scores: [
-          {
-            kind: 'exact',
-            score: 1,
-          },
-        ],
+        scores: [${fixtureScoreSource}],
       }
     },
   }),
@@ -193,12 +198,7 @@ export default defineEval({
           id: task.id,
           inferenceExecutorId: task.inferenceExecutor.id,
           matrix: task.matrix,
-          scores: [
-            {
-              kind: 'exact',
-              score: 1,
-            },
-          ],
+          scores: [${fixtureScoreSource}],
         }
       },
     }
@@ -458,10 +458,53 @@ function setCaseConcurrencyRunState(state: CaseConcurrencyRunState): void {
   })[caseConcurrencyRunStateKey] = state
 }
 
+/** A headline of a project whose cases all passed and recorded no score. */
+const passedHeadline: HeadlineResult = {
+  completion: { mean: 1, n: 1, name: 'completion', sum: 1, total: 1, unscored: {} },
+  headline: { kind: 'passed-cases', score: { mean: 1, n: 1, name: 'passed', sum: 1, total: 1, unscored: {} } },
+}
+
+/** Creates one executed project summary for formatting tests, with its result built from `headline`. */
+function projectSummaryOf(options: { headline: HeadlineResult, name: string, scoreDefinitions?: ScoreDefinedData[] }): CliProjectSummary {
+  return {
+    discoveredEvalFileCount: 1,
+    entryCount: 1,
+    errorMessage: null,
+    executed: true,
+    matrixSummary: null,
+    name: options.name,
+    result: resultOf(options.headline, 1),
+    ...(options.scoreDefinitions == null ? {} : { scoreDefinitions: options.scoreDefinitions }),
+    taskCount: 1,
+  }
+}
+
+/** Creates aggregated results whose overall summary carries `headline`. */
+function resultOf(headline: HeadlineResult, runCount: number): AggregatedRunResults {
+  return {
+    inferenceExecutors: [],
+    overall: { completion: headline.completion, headline, runCount, scores: [] },
+    runs: [],
+  }
+}
+
 function stripAnsi(value: string): string {
   return stripVTControlCharacters(value)
 }
 
+/** Creates one score summary over `n` counted cases; `total` defaults to `n` plus the unscored counts. */
+function summaryOf(name: string, options: { n: number, sum: number, unscored?: Record<string, number> }): ScoreSummary {
+  const unscored = options.unscored ?? {}
+  return {
+    mean: options.n === 0 ? null : options.sum / options.n,
+    n: options.n,
+    name,
+    sum: options.sum,
+    total: options.n + Object.values(unscored).reduce((sum, count) => sum + count, 0),
+    unscored,
+  }
+}
+
 /**
  * How long a test waits for the run to reach a state (cases started, released, or ended).
  *
@@ -518,7 +561,9 @@ describe('runVievalCli', () => {
     expect(output.projects).toHaveLength(1)
     expect(output.projects[0].executed).toBe(true)
     expect(output.projects[0].result?.overall.runCount).toBe(2)
-    expect(output.projects[0].result?.overall.hybridAverage).toBe(1)
+    // The fixture executor returns its score summaries directly; they pool over both runs.
+    expect(output.projects[0].result?.overall.scores.length).toBeGreaterThan(0)
+    expect(output.projects[0].result?.overall.scores.every(score => score.mean === 1)).toBe(true)
   })
 
   it('injects cache runtime into task execution and persists cache artifacts', async () => {
@@ -610,8 +655,10 @@ describeTask('cache-runtime-task', () => {
     const eventsText = await readFile(join(output.projects[0]!.reportDirectory!, 'events.jsonl'), 'utf-8')
     const summary = JSON.parse(summaryText) as RunSummaryFile
 
-    expect(summary.schemaVersion).toBe(1)
+    expect(summary.schemaVersion).toBe(2)
     expect(summary.projects.map(project => project.name)).toEqual([output.projects[0]!.name])
+    expect(summary.projects[0]?.headline).toEqual(output.projects[0]!.result?.overall.headline)
+    expect(summary.headline).toEqual(output.projects[0]!.result?.overall.headline)
     expect(summaryText).toContain('"workspaceId": "packages-vieval"')
     expect(summaryText).toContain('"experimentId": "baseline"')
     expect(summaryText).toContain('"attemptId": "attempt-1"')
@@ -637,14 +684,19 @@ describeTask('cache-runtime-task', () => {
     const projectDirectory = await createDslProject({
       evalFiles: {
         'semantic-artifact.eval.ts': `
-import { caseOf, describeTask } from '${vievalImportPath}'
+import { caseOf, defineScore, describeTask } from '${vievalImportPath}'
+
+const correct = defineScore('locomo.correct', {
+  label: { 'en': 'Correct answer', 'zh-Hans': '回答正确' },
+  primary: true,
+})
 
 describeTask('semantic-artifact-task', () => {
   caseOf('semantic-artifact-case', (context) => {
     context.recordMetric('benchmark.id', 'locomo')
     context.recordMetric('benchmark.case.id', 'case-semantic')
     context.recordMetric('benchmark.locomo.category', 2)
-    context.recordScore(0.75)
+    context.recordScore(correct, 0.75, { rationale: 'The date matches.' })
     return {
       answer: '7 May 2023',
     }
@@ -672,15 +724,9 @@ describeTask('semantic-artifact-task', () => {
     const casesText = await readFile(join(output.projects[0]!.reportDirectory!, 'cases.jsonl'), 'utf-8')
     const metricsText = await readFile(join(output.projects[0]!.reportDirectory!, 'metrics-summary.json'), 'utf-8')
     const tracesText = await readFile(join(output.projects[0]!.reportDirectory!, 'otlp', 'traces.json'), 'utf-8')
-    const caseRecord = JSON.parse(casesText.trim()) as {
-      input?: { question?: string, sampleId?: string }
-      metrics?: Record<string, unknown>
-      output?: { answer?: string }
-      scores?: Record<string, number>
-    }
-    const metricsSummary = JSON.parse(metricsText) as {
-      overall?: Record<string, { average?: number }>
-    }
+    const caseRecord = JSON.parse(casesText.trim()) as CaseRecord & { input?: { question?: string }, output?: { answer?: string } }
+    const metricsSummary = JSON.parse(metricsText) as MetricsSummary
+    const summary = JSON.parse(await readFile(join(output.projects[0]!.reportDirectory!, 'run-summary.json'), 'utf-8')) as RunSummaryFile
     const traces = JSON.parse(tracesText) as {
       resourceSpans?: Array<{
         scopeSpans?: Array<{
@@ -697,8 +743,14 @@ describeTask('semantic-artifact-task', () => {
     expect(caseRecord.output?.answer).toBe('7 May 2023')
     expect(caseRecord.metrics?.['benchmark.id']).toBe('locomo')
     expect(caseRecord.metrics?.['benchmark.case.id']).toBe('case-semantic')
-    expect(caseRecord.scores?.exact).toBe(0.75)
-    expect(metricsSummary.overall?.exact.average).toBe(0.75)
+    expect(caseRecord.scores).toEqual({ 'locomo.correct': 0.75 })
+    expect(caseRecord.scoreDetails).toEqual({ 'locomo.correct': { rationale: 'The date matches.' } })
+    expect(metricsSummary.overall['locomo.correct']?.mean).toBe(0.75)
+    expect(summary.projects[0]?.scoreDefinitions).toEqual([{ label: { 'en': 'Correct answer', 'zh-Hans': '回答正确' }, name: 'locomo.correct', primary: true }])
+    expect(summary.scoreDefinitions).toEqual(summary.projects[0]?.scoreDefinitions)
+    expect(summary.headline.headline).toMatchObject({ kind: 'score', score: { mean: 0.75, name: 'locomo.correct' } })
+    expect(summary.projects[0]?.headline?.headline).toMatchObject({ kind: 'score', score: { mean: 0.75, name: 'locomo.correct' } })
+    expect(stripAnsi(formatVievalCliRunOutput(output))).toContain('Correct answer 75% · completed 100%')
     expect(spans.some(span => span.name === 'vieval.case' && span.attributes?.some(attribute => attribute.key === 'benchmark.locomo.category'))).toBe(true)
   })
 
@@ -805,6 +857,7 @@ describeTask('otel-task', () => {
     context.telemetry?.addEvent('case-context-observed', {
       'vieval.case.fixture': 'otel-case',
     })
+    context.recordScore('otel.correct', 1)
   })
 })
 `,
@@ -905,11 +958,11 @@ export default defineConfig({
         }>
       }>
     }
-    const exactMetric = metricsArtifact.resourceMetrics?.[0]?.scopeMetrics?.[0]?.metrics?.find(metric => metric.name === 'vieval.score.exact')
+    const scoreMetric = metricsArtifact.resourceMetrics?.[0]?.scopeMetrics?.[0]?.metrics?.find(metric => metric.name === 'vieval.score.otel.correct')
 
     expect(output.projects[0]!.reportDirectory).toBeDefined()
     expect(output.projects[0]?.caseSummary?.passed).toBe(1)
-    expect(exactMetric?.gauge?.dataPoints?.length).toBeGreaterThan(0)
+    expect(scoreMetric?.gauge?.dataPoints?.length).toBeGreaterThan(0)
     expect(openTelemetrySpanCalls).toEqual(expect.arrayContaining([
       {
         attributes: expect.objectContaining({
@@ -978,7 +1031,7 @@ export default defineEval({
     id: 'executor-task',
     run() {
       return {
-        scores: [{ kind: 'exact', score: 1 }],
+        scores: [${fixtureScoreSource}],
       }
     },
   }),
@@ -1016,7 +1069,7 @@ export default defineConfig({
           id: task.id,
           inferenceExecutorId: task.inferenceExecutor.id,
           matrix: task.matrix,
-          scores: [{ kind: 'exact', score: 1 }],
+          scores: [${fixtureScoreSource}],
         }
       },
     },
@@ -1146,47 +1199,89 @@ export default defineConfig({
 
   /**
    * @example
-   * it('formats final project hybrid score for terminal readability') verifies score precision stays concise.
+   * it('prints the primary score with its label and the completion rate next to it', () => {})
    */
-  it('formats final project hybrid score for terminal readability', () => {
+  it('prints the primary score with its label and the completion rate next to it', () => {
     const summary = stripAnsi(formatVievalCliRunOutput({
       configFilePath: '/tmp/vieval.config.ts',
       projects: [
-        {
-          caseSummary: {
-            errored: 0,
-            failed: 0,
-            passed: 1986,
-            skipped: 0,
-            timeout: 0,
-            total: 1986,
+        projectSummaryOf({
+          headline: {
+            completion: summaryOf('completion', { n: 155, sum: 150, unscored: { errored: 4, timeout: 1 } }),
+            headline: { kind: 'score', score: summaryOf('tts.pass', { n: 150, sum: 131.25 }) },
           },
-          discoveredEvalFileCount: 1,
-          durationMs: 9_986_250,
-          entryCount: 1,
-          errorMessage: null,
-          executed: true,
-          matrixSummary: null,
-          name: 'locomo-lobehub',
-          result: {
-            inferenceExecutors: [],
-            overall: {
-              coverage: null,
-              exactAverage: 0.3851948392189298,
-              hybridAverage: 0.3851948392189298,
-              judgeAverage: null,
-              runCount: 1,
-              strictExactAverage: 0.3851948392189298,
+          name: 'tts-speech-formula',
+          scoreDefinitions: [{ label: { 'en': 'Pass', 'zh-Hans': '通过' }, name: 'tts.pass', primary: true }],
+        }),
+        projectSummaryOf({
+          headline: {
+            completion: summaryOf('completion', { n: 4, sum: 4 }),
+            headline: { kind: 'score', score: summaryOf('locomo.correct', { n: 3, sum: 1.155584517656789 }) },
+          },
+          name: 'locomo',
+          scoreDefinitions: [{ label: { 'zh-Hans': '回答正确' }, name: 'locomo.correct' }],
+        }),
+        projectSummaryOf({
+          headline: {
+            completion: summaryOf('completion', { n: 2, sum: 2 }),
+            headline: { kind: 'score', score: summaryOf('unlabeled.score', { n: 2, sum: 2 }) },
+          },
+          name: 'unlabeled',
+        }),
+      ],
+    }))
+
+    /**
+     * @example
+     * expect(summary).toContain('Pass 87.5% · completed 96.8% (4 errored, 1 timed out)')
+     */
+    expect(summary).toContain('Pass 87.5% · completed 96.8% (4 errored, 1 timed out)')
+    // Without an `en` label, the first label is shown.
+    expect(summary).toContain('回答正确 38.5% · completed 100%')
+    expect(summary).toContain('unlabeled.score 100% · completed 100%')
+    expect(summary).not.toContain('0.385')
+  })
+
+  /**
+   * @example
+   * it('prints the share of passed cases when no score was recorded, and each task when primary scores differ', () => {})
+   */
+  it('prints the share of passed cases when no score was recorded, and each task when primary scores differ', () => {
+    const summary = stripAnsi(formatVievalCliRunOutput({
+      configFilePath: '/tmp/vieval.config.ts',
+      projects: [
+        projectSummaryOf({
+          headline: {
+            completion: summaryOf('completion', { n: 4, sum: 4 }),
+            headline: { kind: 'passed-cases', score: summaryOf('passed', { n: 4, sum: 2 }) },
+          },
+          name: 'unscored-project',
+        }),
+        projectSummaryOf({
+          headline: {
+            completion: summaryOf('completion', { n: 3, sum: 2, unscored: { timeout: 1 } }),
+            headline: {
+              kind: 'per-task',
+              tasks: [
+                { score: summaryOf('a.correct', { n: 1, sum: 1 }), taskId: 'evals/a.eval.ts' },
+                { score: summaryOf('b.natural', { n: 1, sum: 0.5 }), taskId: 'evals/b.eval.ts' },
+              ],
             },
-            runs: [],
           },
-          taskCount: 1,
-        },
+          name: 'mixed-project',
+          scoreDefinitions: [{ label: 'Natural reading', name: 'b.natural', primary: true }],
+        }),
       ],
     }))
 
-    expect(summary).toContain('hybrid 0.385')
-    expect(summary).not.toContain('0.3851948392189298')
+    /**
+     * @example
+     * expect(summary).toContain('passed 50% · completed 100%')
+     */
+    expect(summary).toContain('passed 50% · completed 100%')
+    expect(summary).toContain('primary scores differ per task · completed 66.7% (1 timed out)')
+    expect(summary).toContain('evals/a.eval.ts  a.correct 100%')
+    expect(summary).toContain('evals/b.eval.ts  Natural reading 50%')
   })
 
   it('reports skipped/passed/failed counts together when mixed', () => {
@@ -1200,18 +1295,7 @@ export default defineConfig({
           executed: true,
           matrixSummary: null,
           name: 'passed-project',
-          result: {
-            inferenceExecutors: [],
-            overall: {
-              coverage: null,
-              exactAverage: 1,
-              hybridAverage: 1,
-              judgeAverage: null,
-              runCount: 1,
-              strictExactAverage: 1,
-            },
-            runs: [],
-          },
+          result: resultOf(passedHeadline, 1),
           taskCount: 1,
         },
         {
@@ -1258,7 +1342,8 @@ export default defineConfig({
     })
     expect(output.projects[0]?.matrixSummary).not.toBeNull()
     expect(output.projects[0].result?.overall.runCount).toBe(2)
-    expect(output.projects[0].result?.overall.hybridAverage).toBe(1)
+    // One task declares a primary score; the other records none, so it does not change the headline.
+    expect(output.projects[0].result?.overall.headline.headline).toMatchObject({ kind: 'score', score: { mean: 1, name: 'task_default.models_resolved' } })
   })
 
   it('runs projects concurrently up to the run-wide project pool while each project keeps its task pool', async () => {
@@ -1589,6 +1674,52 @@ export default defineConfig({
     }
   })
 
+  // ROOT CAUSE:
+  //
+  // If a case errored in its first auto attempt and passed in a later one, the project summary counted it as errored,
+  // listed its first error as a failure, and suggested resuming it.
+  // This happened because `subscribeProjectCaseSummary` counted only the first `case.ended` of each case.
+  //
+  // if (!seenCaseIds.has(projectCaseId)) { seenCaseIds.add(projectCaseId); counters[data.state] += 1 }
+  //
+  // We fixed this by keeping the last `case.ended` of each case, and only its failure.
+  // finalStates.set(key, data.state)
+  /**
+   * @example
+   * expect(output.projects[0]!.caseSummary).toMatchObject({ errored: 0, passed: 1, total: 1 })
+   */
+  it('counts a case that errored in one auto attempt and passed in a later one as passed', async () => {
+    const vievalImportPath = join(packageDirectory, 'src', 'index.ts').replaceAll('\\', '/')
+    const directory = await mkdtemp(join(tmpdir(), 'vieval-final-state-'))
+    temporaryDirectories.push(directory)
+    await writeFile(join(directory, 'flaky.eval.ts'), `
+import { describeTask } from '${vievalImportPath}'
+
+let runs = 0
+describeTask('flaky', ({ caseOf }) => {
+  caseOf('first-attempt-errors', async () => {
+    runs += 1
+    if (runs === 1)
+      throw new Error('provider unavailable')
+  }, { autoAttempt: 1, input: undefined })
+})
+`, 'utf-8')
+    await writeFile(join(directory, 'vieval.config.ts'), `
+import { defineConfig } from '${vievalImportPath}'
+
+export default defineConfig({
+  models: [{ aliases: [], id: 'openai:gpt-4.1-mini', model: 'gpt-4.1-mini', inferenceExecutor: 'openai', inferenceExecutorId: 'openai:gpt-4.1-mini' }],
+  projects: [{ name: 'flaky', include: ['*.eval.ts'] }],
+})
+`, 'utf-8')
+
+    const output = await runVievalCli({ configFilePath: join(directory, 'vieval.config.ts'), cwd: directory, report: false })
+
+    expect(output.projects[0]!.caseSummary).toMatchObject({ errored: 0, passed: 1, total: 1 })
+    expect(output.projects[0]!.caseFailures).toEqual([])
+    expect(output.projects[0]!.result?.overall.completion).toMatchObject({ mean: 1, n: 1 })
+  })
+
   it('runs only the tasks and cases selected by name or glob', async () => {
     const directory = await createLanguageProjectsFixture()
 
@@ -1822,7 +1953,8 @@ describeTask('cli-case-concurrency', (task) => {
       // Only `flaky › provider` ran: the wrong answer is a result, and `stable` had nothing left, so it was skipped.
       expect(resumed.projects[0]?.caseSummary).toEqual({ errored: 0, failed: 0, passed: 1, skipped: 0, timeout: 0, total: 1 })
       expect(resumed.projects[0]?.taskCount).toBe(1)
-      expect(resumed.projects[0]?.result?.overall.exactAverage).toBe(1)
+      // The resume fixture records no score, so its headline is the share of passed cases.
+      expect(resumed.projects[0]?.result?.overall.headline.headline).toMatchObject({ kind: 'passed-cases', score: { mean: 1 } })
 
       const summary = JSON.parse(await readFile(join(resumed.projects[0]!.reportDirectory!, 'run-summary.json'), 'utf-8')) as RunSummaryFile
       expect(summary.resumedFrom).toBe('attempt-first')
@@ -1906,11 +2038,12 @@ describeTask('cli-case-concurrency', (task) => {
       })
 
       expect(output.projects[0]?.executed).toBe(true)
-      // The fixture's first attempt throws a plain error, so that case counts as `errored`.
+      // The fixture's first attempt of case-b throws a plain error, and the second attempt passes: a case counts by its
+      // final state, so both cases count as passed.
       expect(output.projects[0]?.caseSummary).toEqual({
-        errored: 1,
+        errored: 0,
         failed: 0,
-        passed: 1,
+        passed: 2,
         skipped: 0,
         timeout: 0,
         total: 2,
@@ -1997,18 +2130,7 @@ describeTask('failed-case-task', () => {
             runRows: 8,
           },
           name: 'example-api-config-matrix',
-          result: {
-            inferenceExecutors: [],
-            overall: {
-              coverage: null,
-              exactAverage: 1,
-              hybridAverage: 1,
-              judgeAverage: null,
-              runCount: 288,
-              strictExactAverage: 1,
-            },
-            runs: [],
-          },
+          result: resultOf(passedHeadline, 288),
           taskCount: 288,
         },
       ],
@@ -2039,12 +2161,8 @@ export default defineEval({
     run() {
       const score = process.env.VIEVAL_TEST_ENV_KEY === 'from-config' ? 1 : 0
       return {
-        scores: [
-          {
-            kind: 'exact',
-            score,
-          },
-        ],
+        primary: 'env.applied',
+        scores: [{ mean: score, n: 1, name: 'env.applied', sum: score, total: 1, unscored: {} }],
       }
     },
   }),
@@ -2094,7 +2212,7 @@ export default defineConfig({
     expect(output.projects).toHaveLength(1)
     expect(output.projects[0].errorMessage).toBeNull()
     expect(output.projects[0].executed).toBe(true)
-    expect(output.projects[0].result?.overall.hybridAverage).toBe(1)
+    expect(output.projects[0].result?.overall.headline.headline).toMatchObject({ kind: 'score', score: { mean: 1, name: 'env.applied' } })
     expect(process.env.VIEVAL_TEST_ENV_KEY).toBe('outside-value')
 
     if (previousEnv == null) {
@@ -2510,7 +2628,7 @@ export default defineEval({
     id: 'exec-task',
     run() {
       return {
-        scores: [{ kind: 'exact', score: 1 }],
+        scores: [${fixtureScoreSource}],
       }
     },
   }),
@@ -2552,7 +2670,7 @@ export default defineConfig({
         id: 'exec-task',
         matrix: {},
         inferenceExecutorId: 'openai:gpt-4.1-mini',
-        scores: [{ kind: 'exact', score: 1 }],
+        scores: [${fixtureScoreSource}],
       }),
     },
     {
diff --git a/packages/vieval/src/cli/run.ts b/packages/vieval/src/cli/run.ts
index b20ef42..db7b6f3 100644
--- a/packages/vieval/src/cli/run.ts
+++ b/packages/vieval/src/cli/run.ts
@@ -2,8 +2,9 @@ import type { TaskRunStorage } from '../config'
 import type { AggregatedRunResults, ScheduledTask, ScheduledTaskExecutor, TaskExecutionContext } from '../core/runner'
 import type { RunEventBus } from '../core/runner/event-bus'
 import type { Scheduler, SchedulerConcurrencyConfig } from '../core/scheduler/types'
+import type { HeadlineResult, ScoreSummary } from '../core/scoring'
 import type { TelemetryRuntime } from '../core/telemetry'
-import type { RunEventEmitter, RunEventEnvelope } from '../events'
+import type { CaseEndedData, RunEventEmitter, RunEventEnvelope, ScoreDefinedData } from '../events'
 import type { CliProjectConcurrencyConfig, CliProjectExecutorContext, LoadVievalCliConfigOptions, NormalizedCliProjectConfig } from './config'
 import type { InterruptSignal } from './interrupt'
 import type { ReportEventLog } from './report-log'
@@ -29,7 +30,7 @@ import { collectEvalEntries, createFilesystemTaskCacheRuntime, createRunnerSched
 import { createProcessOutputInterceptor } from '../core/runner/case-output'
 import { createRunEventBus } from '../core/runner/event-bus'
 import { createNoopTelemetryRuntime, createOpenTelemetryRuntime } from '../core/telemetry'
-import { caseEnded, caseEndedDataSchema, caseOutput, caseOutputDataSchema, caseRequeued, caseRequeuedDataSchema, caseStarted, caseStartedDataSchema, projectEnded, projectStarted, runEnded, runInterrupted, runStarted, taskEnded, taskEndedDataSchema, taskQueued, taskQueuedDataSchema, taskStarted } from '../events'
+import { caseEnded, caseEndedDataSchema, caseOutput, caseOutputDataSchema, caseRequeued, caseRequeuedDataSchema, caseStarted, caseStartedDataSchema, projectEnded, projectStarted, runEnded, runInterrupted, runStarted, scoreDefined, taskEnded, taskEndedDataSchema, taskQueued, taskQueuedDataSchema, taskStarted } from '../events'
 import { loadVievalCliConfig } from './config'
 import { discoverEvalFiles } from './discovery'
 import { exitCodeFromSignal, listenForInterrupt } from './interrupt'
@@ -93,6 +94,11 @@ export interface CliProjectSummary {
   result: AggregatedRunResults | null
   /** Run id of this project's run; every project of an invocation runs as its own run. */
   runId?: string
+  /**
+   * Latest `score.defined` of each score name the project's tasks announced, in first-announced order. Labels the
+   * headline; empty for a custom executor that does not emit score events.
+   */
+  scoreDefinitions?: ScoreDefinedData[]
   taskCount: number
 }
 
@@ -423,6 +429,49 @@ export function createRunReporter(options: RunVievalCliReporterOptions | undefin
   }
 }
 
+/**
+ * Formats the completion rate shown next to every headline.
+ *
+ * Before:
+ * - completion 30/31 with `{ errored: 1 }`; 4/4 with no unscored counts
+ *
+ * After:
+ * - "completed 96.8% (1 errored)"; "completed 100%"
+ */
+export function formatCompletion(completion: ScoreSummary): string {
+  const errored = completion.unscored.errored ?? 0
+  const timedOut = completion.unscored.timeout ?? 0
+  const details = [
+    ...(errored === 0 ? [] : [`${errored} errored`]),
+    ...(timedOut === 0 ? [] : [`${timedOut} timed out`]),
+  ]
+  return `completed ${formatPercent(completion.mean)}${details.length === 0 ? '' : ` (${details.join(', ')})`}`
+}
+
+/**
+ * Formats a resolved headline with the label of its score.
+ *
+ * Use when:
+ * - the run summary line or the comparison leaderboard shows what a project scored
+ *
+ * Expects:
+ * - `definitions` to hold the project's `score.defined` payloads; a score without one is shown by its name
+ *
+ * Returns:
+ * - `score`: `<label> <mean%>`, for example `Pass 87.5%`
+ * - `passed-cases`: `passed <rate%>`, because no score was recorded
+ * - `per-task`: `primary scores differ per task`; callers list each task's primary score below it
+ */
+export function formatHeadline(result: HeadlineResult, definitions: readonly ScoreDefinedData[]): string {
+  if (result.headline.kind === 'score') {
+    return formatScore(result.headline.score, definitions)
+  }
+  if (result.headline.kind === 'passed-cases') {
+    return `passed ${formatPercent(result.headline.score.mean)}`
+  }
+  return 'primary scores differ per task'
+}
+
 /**
  * Formats CLI run output as human-readable lines.
  */
@@ -499,7 +548,7 @@ export function formatVievalCliRunOutput(output: CliRunOutput): string {
     if (!project.executed) {
       skippedProjects += 1
       const countLabel = colors.dim(`(${project.taskCount} tasks)`)
-      const detailsLabel = colors.dim(` ${project.discoveredEvalFileCount} files, ${project.entryCount} entries, 0 runs, hybrid n/a`)
+      const detailsLabel = colors.dim(` ${project.discoveredEvalFileCount} files, ${project.entryCount} entries, 0 runs`)
       const matrixSummary = formatMatrixSummary(project.matrixSummary)
       lines.push(` ${colors.dim('○')} ${badge}${countLabel}${detailsLabel}${formatDuration(project.durationMs, colors)}`)
       if (matrixSummary != null) {
@@ -518,17 +567,23 @@ export function formatVievalCliRunOutput(output: CliRunOutput): string {
     else {
       passedProjects += 1
     }
-    const hybridAverageLabel = formatHybridAverage(project.result?.overall.hybridAverage)
     const runCount = project.result?.overall.runCount ?? 0
     const countLabel = colors.dim(`(${project.taskCount} tasks)`)
     const caseSummaryLabel = project.caseSummary == null
       ? ''
       : `, cases ${project.caseSummary.passed} passed | ${project.caseSummary.failed} failed | ${project.caseSummary.errored} errored | ${project.caseSummary.timeout} timeout`
-    const coverage = project.result?.overall.coverage
-    const coverageLabel = coverage == null || coverage.total === 0 ? '' : `, coverage ${formatHybridAverage(coverage.measured / coverage.total)}`
-    const detailsLabel = colors.dim(` ${project.discoveredEvalFileCount} files, ${project.entryCount} entries, ${runCount} runs${caseSummaryLabel}, hybrid ${hybridAverageLabel}${coverageLabel}`)
+    const detailsLabel = colors.dim(` ${project.discoveredEvalFileCount} files, ${project.entryCount} entries, ${runCount} runs${caseSummaryLabel}`)
     const matrixSummary = formatMatrixSummary(project.matrixSummary)
     lines.push(` ${hasFailedCases ? colors.red('❯') : colors.green('✓')} ${badge}${countLabel}${detailsLabel}${formatDuration(project.durationMs, colors)}`)
+    const headline = project.result?.overall.headline
+    if (headline != null) {
+      lines.push(`   ${formatHeadline(headline, project.scoreDefinitions ?? [])} · ${formatCompletion(headline.completion)}`)
+      if (headline.headline.kind === 'per-task') {
+        for (const task of headline.headline.tasks) {
+          lines.push(`     ${colors.dim(task.taskId)}  ${formatScore(task.score, project.scoreDefinitions ?? [])}`)
+        }
+      }
+    }
     if (matrixSummary != null) {
       lines.push(`   ${colors.dim(matrixSummary)}`)
     }
@@ -1046,11 +1101,13 @@ function createAutoTaskExecutor(): ScheduledTaskExecutor {
     })
 
     return {
-      ...(output.coverage == null ? {} : { coverage: { ...output.coverage } }),
+      ...(output.completion == null ? {} : { completion: output.completion }),
       entryId: task.entry.id,
       id: task.id,
       inferenceExecutorId: task.inferenceExecutor.id,
       matrix: task.matrix,
+      ...(output.passed == null ? {} : { passed: output.passed }),
+      ...(output.primary == null ? {} : { primary: output.primary }),
       scores: [...output.scores],
     }
   }
@@ -1293,6 +1350,7 @@ async function executePreparedProject(
   const settledTaskIds = new Set<string>()
   const projectEvents = bus.scope({ projectId: prepared.name })
   const caseSummary = subscribeProjectCaseSummary(bus, prepared.name)
+  const scoreDefinitions = subscribeProjectScoreDefinitions(bus, prepared.name)
   const rawTaskExecutor = prepared.project.executor ?? createAutoTaskExecutor()
   const taskExecutor: ScheduledTaskExecutor = async (task, context) => {
     const result = await telemetry.withSpan('vieval.task', {
@@ -1356,6 +1414,7 @@ async function executePreparedProject(
       matrixSummary: createProjectMatrixSummary(prepared.tasks),
       name: prepared.name,
       result: aggregated,
+      scoreDefinitions: [...scoreDefinitions.values()],
       taskCount: prepared.tasks.length,
     }
   }
@@ -1391,6 +1450,7 @@ async function executePreparedProject(
       matrixSummary: createProjectMatrixSummary(prepared.tasks),
       name: prepared.name,
       result: null,
+      scoreDefinitions: [...scoreDefinitions.values()],
       taskCount: prepared.tasks.length,
     }
   }
@@ -1449,12 +1509,33 @@ function formatDuration(durationMs: number | undefined, colors: CliColorPalette)
   return color(` ${rounded}${colors.dim('ms')}`)
 }
 
-function formatHybridAverage(hybridAverage: null | number | undefined): string {
-  if (hybridAverage == null) {
+/**
+ * Normalizes a `0..1` value into a percentage with at most one decimal.
+ *
+ * Before:
+ * - 0.875, 1, null
+ *
+ * After:
+ * - "87.5%", "100%", "n/a"
+ */
+function formatPercent(value: null | number): string {
+  if (value == null) {
     return 'n/a'
   }
 
-  return hybridAverage.toFixed(3).replace(/\.?0+$/, '')
+  return `${(value * 100).toFixed(1).replace(/\.0$/, '')}%`
+}
+
+/**
+ * Formats one score summary as `<label> <mean%>`; the label is the definition's `en` text, else its first text,
+ * else the score name.
+ */
+function formatScore(score: ScoreSummary, definitions: readonly ScoreDefinedData[]): string {
+  const label = definitions.find(definition => definition.name === score.name)?.label
+  const text = label == null
+    ? score.name
+    : typeof label === 'string' ? label : label.en ?? Object.values(label)[0] ?? score.name
+  return `${text} ${formatPercent(score.mean)}`
 }
 
 /**
@@ -1787,29 +1868,29 @@ function smallestDeclared(configured: number | undefined, cli: number | undefine
  * Counts the case results of one project from `case.ended` events.
  *
  * Returns:
- * - the failures list, filled as cases fail, and a snapshot function for the project summary
+ * - the final failure of each case that did not pass, and a snapshot function for the project summary; both count each
+ *   case once, by its last `case.ended`
  */
 function subscribeProjectCaseSummary(bus: RunEventBus, projectName: string): {
   failures: CliProjectCaseFailure[]
   toSummary: () => CliProjectCaseSummary
 } {
-  const counters = { errored: 0, failed: 0, passed: 0, skipped: 0, timeout: 0 }
-  const seenCaseIds = new Set<string>()
-  const failures: CliProjectCaseFailure[] = []
+  // The last `case.ended` of a case is its final state: an auto attempt ends the same case again, and a case that
+  // errored in its first attempt but passed in a later one has a result.
+  const finalStates = new Map<string, CaseEndedData['state']>()
+  // Only the final failure of a case is listed, so a case that passed in a later attempt is not reported as failed.
+  const finalFailures = new Map<string, CliProjectCaseFailure>()
 
   bus.on(caseEnded, ({ caseId, data, projectId, taskId }) => {
     if (projectId !== projectName || taskId == null || caseId == null) {
       return
     }
 
-    const projectCaseId = `${taskId}:${caseId}`
-    if (!seenCaseIds.has(projectCaseId)) {
-      seenCaseIds.add(projectCaseId)
-      counters[data.state] += 1
-    }
-
+    const key = `${taskId}:${caseId}`
+    finalStates.set(key, data.state)
+    finalFailures.delete(key)
     if ((data.state === 'failed' || data.state === 'errored' || data.state === 'timeout') && data.errorMessage != null) {
-      failures.push({
+      finalFailures.set(key, {
         caseId,
         caseName: data.name,
         errorMessage: data.errorMessage,
@@ -1819,11 +1900,35 @@ function subscribeProjectCaseSummary(bus: RunEventBus, projectName: string): {
   })
 
   return {
-    failures,
-    toSummary: () => ({ ...counters, total: seenCaseIds.size }),
+    get failures() {
+      return [...finalFailures.values()]
+    },
+    toSummary: () => {
+      const counters = { errored: 0, failed: 0, passed: 0, skipped: 0, timeout: 0 }
+      for (const state of finalStates.values()) {
+        counters[state] += 1
+      }
+      return { ...counters, total: finalStates.size }
+    },
   }
 }
 
+/**
+ * Keeps the latest `score.defined` of each score name of one project.
+ *
+ * Returns:
+ * - a map filled as tasks announce scores; a later announcement of a name replaces the earlier one, as in viewers
+ */
+function subscribeProjectScoreDefinitions(bus: RunEventBus, projectName: string): Map<string, ScoreDefinedData> {
+  const definitions = new Map<string, ScoreDefinedData>()
+  bus.on(scoreDefined, ({ data, projectId }) => {
+    if (projectId === projectName) {
+      definitions.set(data.name, data)
+    }
+  })
+  return definitions
+}
+
 /**
  * Forwards the events of one in-process run to the terminal reporter.
  *
diff --git a/packages/vieval/src/config/types.ts b/packages/vieval/src/config/types.ts
index ab8cbde..73a71c3 100644
--- a/packages/vieval/src/config/types.ts
+++ b/packages/vieval/src/config/types.ts
@@ -1,8 +1,7 @@
-import type { RunScore } from '../core/runner'
 import type { ScheduledTask } from '../core/runner/schedule'
 import type { TaskExecutionContext } from '../core/runner/task-context'
 import type { Scheduler } from '../core/scheduler/types'
-import type { CaseCoverage } from '../core/scoring'
+import type { ScoreSummary } from '../core/scoring'
 import type { TelemetryRuntime } from '../core/telemetry'
 import type { RunEventEmitter } from '../events'
 
@@ -628,13 +627,21 @@ export interface TaskRunContext {
  */
 export interface TaskRunOutput {
   /**
-   * Measured and settled case counts of this task run. A task without cases leaves it out.
+   * Completion rate of this task run (`scoreCases`). A task without cases leaves it out.
    */
-  coverage?: CaseCoverage
+  completion?: ScoreSummary
   /**
-   * Scores emitted by this task run.
+   * Share of passed cases of this task run. A task without cases leaves it out.
    */
-  scores: readonly RunScore[]
+  passed?: ScoreSummary
+  /**
+   * Name of the headline score of this task run, when it has one.
+   */
+  primary?: string
+  /**
+   * One summary per score of this task run.
+   */
+  scores: readonly ScoreSummary[]
 }
 
 /**
diff --git a/packages/vieval/src/core/assertions/index.test.ts b/packages/vieval/src/core/assertions/index.test.ts
index 1cef1ca..32fc2af 100644
--- a/packages/vieval/src/core/assertions/index.test.ts
+++ b/packages/vieval/src/core/assertions/index.test.ts
@@ -11,7 +11,6 @@ import {
   expectRubric,
   expectStructuredOutput,
   expectToolCallArgs,
-  toRunScores,
 } from './index'
 
 describe('assertions', () => {
@@ -32,11 +31,7 @@ describe('assertions', () => {
     expect(outcomes.map(outcome => outcome.pass)).toEqual([true, true])
     expect(collectFailedAssertions(outcomes)).toEqual([])
 
-    const runScores = toRunScores(outcomes)
-    expect(runScores).toEqual([
-      { kind: 'exact', score: 1 },
-      { kind: 'exact', score: 1 },
-    ])
+    expect(outcomes.map(outcome => outcome.score)).toEqual([1, 1])
   })
 
   it('supports regex, structured output, and tool-call argument checks', async () => {
@@ -109,7 +104,7 @@ describe('assertions', () => {
     })
 
     expect(outcomes.map(outcome => outcome.pass)).toEqual([true, true])
-    expect(outcomes[1]?.scoreKind).toBe('judge')
+    expect(outcomes[1]?.score).toBeGreaterThanOrEqual(0.8)
   })
 
   it('supports stateful custom assertions for dynamic measurements', async () => {
@@ -127,7 +122,6 @@ describe('assertions', () => {
           }
         },
         id: 'stateful-threshold',
-        scoreKind: 'exact',
       }),
     ], {
       state: new Map([['calls', 1]]),
diff --git a/packages/vieval/src/core/assertions/index.ts b/packages/vieval/src/core/assertions/index.ts
index 012ff0d..d6b75ce 100644
--- a/packages/vieval/src/core/assertions/index.ts
+++ b/packages/vieval/src/core/assertions/index.ts
@@ -1,5 +1,3 @@
-import type { RunScore, RunScoreKind } from '../runner/aggregate'
-
 /**
  * Async assertion function used by eval scenarios.
  */
@@ -44,13 +42,9 @@ export interface AssertionOutcome {
    */
   reason: string
   /**
-   * Normalized score in the `0..1` range.
+   * Normalized score in the `0..1` range. Record it with `context.recordScore(definition, outcome.score)`.
    */
   score: number
-  /**
-   * Assertion family emitted as run score kind.
-   */
-  scoreKind: RunScoreKind
 }
 
 /**
@@ -74,10 +68,6 @@ export interface CustomAssertionOptions {
    * Stable assertion id.
    */
   id: string
-  /**
-   * Score family emitted by this custom assertion.
-   */
-  scoreKind: RunScoreKind
 }
 
 /**
@@ -269,13 +259,13 @@ export async function evaluateAssertions(
  * Creates a custom assertion with fully user-defined logic.
  *
  * Example:
- * `expectCustom({ id: 'stateful-window', scoreKind: 'exact', evaluate: (ctx) => ... })`
+ * `expectCustom({ id: 'stateful-window', evaluate: (ctx) => ... })`
  */
 export function expectCustom(options: CustomAssertionOptions): Assertion {
   return async (context) => {
     const result = await options.evaluate(context)
 
-    return createOutcome(options.id, options.scoreKind, result.pass, result.score, result.reason)
+    return createOutcome(options.id, result.pass, result.score, result.reason)
   }
 }
 
@@ -288,7 +278,7 @@ export function expectCustom(options: CustomAssertionOptions): Assertion {
 export function expectMustExclude(options: MustExcludeAssertionOptions): Assertion {
   return async (context) => {
     if (options.keywords.length === 0) {
-      return createOutcome(options.id, 'exact', true, 1, 'No excluded keywords configured.')
+      return createOutcome(options.id, true, 1, 'No excluded keywords configured.')
     }
 
     const caseSensitive = options.caseSensitive ?? false
@@ -303,7 +293,6 @@ export function expectMustExclude(options: MustExcludeAssertionOptions): Asserti
 
     return createOutcome(
       options.id,
-      'exact',
       pass,
       score,
       pass
@@ -322,7 +311,7 @@ export function expectMustExclude(options: MustExcludeAssertionOptions): Asserti
 export function expectMustInclude(options: MustIncludeAssertionOptions): Assertion {
   return async (context) => {
     if (options.keywords.length === 0) {
-      return createOutcome(options.id, 'exact', true, 1, 'No required keywords configured.')
+      return createOutcome(options.id, true, 1, 'No required keywords configured.')
     }
 
     const caseSensitive = options.caseSensitive ?? false
@@ -341,7 +330,6 @@ export function expectMustInclude(options: MustIncludeAssertionOptions): Asserti
 
     return createOutcome(
       options.id,
-      'exact',
       pass,
       score,
       pass
@@ -363,7 +351,6 @@ export function expectNot(assertion: Assertion, options: { id: string }): Assert
 
     return createOutcome(
       options.id,
-      baseOutcome.scoreKind,
       !baseOutcome.pass,
       1 - baseOutcome.score,
       `NOT(${baseOutcome.id}): ${baseOutcome.reason}`,
@@ -383,7 +370,6 @@ export function expectRegex(options: RegexAssertionOptions): Assertion {
 
     return createOutcome(
       options.id,
-      'exact',
       pass,
       pass ? 1 : 0,
       pass ? 'Regex matched response text.' : `Regex did not match: ${options.pattern}`,
@@ -406,7 +392,6 @@ export function expectRubric(options: RubricAssertionOptions): Assertion {
 
     return createOutcome(
       options.id,
-      'judge',
       pass,
       normalizedScore,
       `${result.reason}${result.judgeModel ? ` (judge: ${result.judgeModel})` : ''}`,
@@ -426,7 +411,6 @@ export function expectStructuredOutput<TValue>(options: StructuredOutputAssertio
 
     return createOutcome(
       options.id,
-      'exact',
       pass,
       pass ? 1 : 0,
       pass ? 'Structured output matched validator.' : (options.failureReason ?? 'Structured output validation failed.'),
@@ -445,14 +429,13 @@ export function expectToolCallArgs(options: ToolCallArgsAssertionOptions): Asser
     const targetCall = (context.toolCalls ?? []).find(call => call.name === options.toolName)
 
     if (targetCall == null) {
-      return createOutcome(options.id, 'exact', false, 0, `Missing tool call: ${options.toolName}`)
+      return createOutcome(options.id, false, 0, `Missing tool call: ${options.toolName}`)
     }
 
     const pass = options.validate(targetCall.args)
 
     return createOutcome(
       options.id,
-      'exact',
       pass,
       pass ? 1 : 0,
       pass ? `Tool call args validated for ${options.toolName}.` : `Tool call args validation failed for ${options.toolName}.`,
@@ -476,16 +459,6 @@ export function normalizeMatchText(value: string, caseSensitive: boolean): strin
   return compactedWhitespace.toLowerCase()
 }
 
-/**
- * Converts assertion outcomes to run-score tuples consumed by aggregation.
- */
-export function toRunScores(outcomes: readonly AssertionOutcome[]): RunScore[] {
-  return outcomes.map(outcome => ({
-    kind: outcome.scoreKind,
-    score: outcome.score,
-  }))
-}
-
 function clampScore(score: number): number {
   if (Number.isNaN(score)) {
     return 0
@@ -504,7 +477,6 @@ function clampScore(score: number): number {
 
 function createOutcome(
   id: string,
-  scoreKind: RunScoreKind,
   pass: boolean,
   score: number,
   reason: string,
@@ -514,6 +486,5 @@ function createOutcome(
     pass,
     reason,
     score: clampScore(score),
-    scoreKind,
   }
 }
diff --git a/packages/vieval/src/core/processors/results/index.test.ts b/packages/vieval/src/core/processors/results/index.test.ts
deleted file mode 100644
index 584f4d2..0000000
--- a/packages/vieval/src/core/processors/results/index.test.ts
+++ /dev/null
@@ -1,145 +0,0 @@
-import type { AggregatedRunResults } from '../../runner/aggregate'
-
-import { describe, expect, it } from 'vitest'
-
-import { processRunResults } from './index'
-
-function createAggregatedResults(overrides: Partial<AggregatedRunResults> = {}): AggregatedRunResults {
-  return {
-    inferenceExecutors: [{
-      coverage: null,
-      exactAverage: 0.8,
-      hybridAverage: 0.8,
-      inferenceExecutorId: 'openai:gpt-4.1-mini',
-      judgeAverage: 0.8,
-      runCount: 2,
-      strictExactAverage: 0.8,
-    }],
-    overall: {
-      coverage: null,
-      exactAverage: 0.8,
-      hybridAverage: 0.8,
-      judgeAverage: 0.8,
-      runCount: 2,
-      strictExactAverage: 0.8,
-    },
-    runs: [{
-      coverage: null,
-      entryId: 'plugins/airi-plugin-game-chess/src/agent/evals/chess-commentary',
-      exactAverage: 0.8,
-      hybridAverage: 0.8,
-      id: 'run-1',
-      inferenceExecutorId: 'openai:gpt-4.1-mini',
-      judgeAverage: 0.8,
-      matrix: createScheduledTaskMatrix(),
-      strictExactAverage: 0.8,
-    }, {
-      coverage: null,
-      entryId: 'plugins/airi-plugin-game-chess/src/agent/evals/chess-commentary',
-      exactAverage: 0.7,
-      hybridAverage: 0.7,
-      id: 'run-2',
-      inferenceExecutorId: 'openai:gpt-4.1-mini',
-      judgeAverage: 0.7,
-      matrix: createScheduledTaskMatrix(),
-      strictExactAverage: 0.7,
-    }],
-    ...overrides,
-  }
-}
-
-function createScheduledTaskMatrix() {
-  return {
-    eval: {},
-    meta: {
-      evalRowId: 'default',
-      runRowId: 'default',
-    },
-    run: {},
-  }
-}
-
-describe('processRunResults', () => {
-  it('passes when threshold and hard-limit policies are satisfied', () => {
-    const decision = processRunResults(createAggregatedResults(), {
-      maxFailedRuns: {
-        maxFailedRuns: 0,
-        minRunHybridScore: 0.7,
-      },
-      threshold: {
-        minOverallHybridScore: 0.75,
-        minProviderHybridScore: 0.75,
-      },
-    })
-
-    expect(decision.pass).toBe(true)
-    expect(decision.violations).toEqual([])
-  })
-
-  it('fails threshold policy when overall or inferenceExecutor hybrid scores are below limits', () => {
-    const decision = processRunResults(createAggregatedResults({
-      inferenceExecutors: [{
-        coverage: null,
-        exactAverage: 0.55,
-        hybridAverage: 0.55,
-        inferenceExecutorId: 'openai:gpt-4.1-mini',
-        judgeAverage: 0.55,
-        runCount: 2,
-        strictExactAverage: 0.55,
-      }],
-      overall: {
-        coverage: null,
-        exactAverage: 0.5,
-        hybridAverage: 0.5,
-        judgeAverage: 0.5,
-        runCount: 2,
-        strictExactAverage: 0.5,
-      },
-    }), {
-      threshold: {
-        minOverallHybridScore: 0.7,
-        minProviderHybridScore: 0.6,
-      },
-    })
-
-    expect(decision.pass).toBe(false)
-    expect(decision.violations.map(violation => violation.policyId)).toEqual([
-      'threshold:overall-hybrid',
-      'threshold:inferenceExecutor-hybrid',
-    ])
-  })
-
-  it('fails hard-limit policy when failed runs exceed maxFailedRuns', () => {
-    const decision = processRunResults(createAggregatedResults({
-      runs: [{
-        coverage: null,
-        entryId: 'plugins/airi-plugin-game-chess/src/agent/evals/chess-commentary',
-        exactAverage: 0.3,
-        hybridAverage: 0.3,
-        id: 'run-1',
-        inferenceExecutorId: 'openai:gpt-4.1-mini',
-        judgeAverage: 0.3,
-        matrix: createScheduledTaskMatrix(),
-        strictExactAverage: 0.3,
-      }, {
-        coverage: null,
-        entryId: 'plugins/airi-plugin-game-chess/src/agent/evals/chess-commentary',
-        exactAverage: 0.2,
-        hybridAverage: 0.2,
-        id: 'run-2',
-        inferenceExecutorId: 'openai:gpt-4.1-mini',
-        judgeAverage: 0.2,
-        matrix: createScheduledTaskMatrix(),
-        strictExactAverage: 0.2,
-      }],
-    }), {
-      maxFailedRuns: {
-        maxFailedRuns: 1,
-        minRunHybridScore: 0.6,
-      },
-    })
-
-    expect(decision.pass).toBe(false)
-    expect(decision.violations.map(violation => violation.policyId)).toContain('hard-limit:max-failed-runs')
-  })
-})
diff --git a/packages/vieval/src/core/processors/results/index.ts b/packages/vieval/src/core/processors/results/index.ts
deleted file mode 100644
index d252ce1..0000000
--- a/packages/vieval/src/core/processors/results/index.ts
+++ /dev/null
@@ -1,70 +0,0 @@
-import type { AggregatedRunResults } from '../../runner/aggregate'
-import type { HybridThresholdPolicyOptions, ResultPolicyViolation } from './policies/hybrid-threshold'
-import type { MaxFailedRunsPolicyOptions } from './policies/max-failed-runs'
-
-import { evaluateHybridThresholdPolicy } from './policies/hybrid-threshold'
-import { evaluateMaxFailedRunsPolicy } from './policies/max-failed-runs'
-
-/**
- * Configures result-processing policies for eval gating.
- */
-export interface ProcessRunResultsOptions {
-  /**
-   * Hard-limit failed-run policy options.
-   */
-  maxFailedRuns?: MaxFailedRunsPolicyOptions
-  /**
-   * Threshold policy options.
-   */
-  threshold?: HybridThresholdPolicyOptions
-}
-
-/**
- * Final gate decision returned by result processors.
- */
-export interface ResultGateDecision {
-  /**
-   * Whether the result batch passes all policies.
-   */
-  pass: boolean
-  /**
-   * Collected policy violations.
-   */
-  violations: ResultPolicyViolation[]
-}
-
-/**
- * Processes aggregated run results through built-in gating policies.
- *
- * Call stack:
- *
- * {@link runScheduledTasks}
- *   -> {@link aggregateRunResults}
- *     -> {@link processRunResults}
- *       -> {@link evaluateHybridThresholdPolicy}
- *       -> {@link evaluateMaxFailedRunsPolicy}
- *         -> {@link ResultGateDecision}
- */
-export function processRunResults(
-  results: AggregatedRunResults,
-  options: ProcessRunResultsOptions = {},
-): ResultGateDecision {
-  const thresholdViolations = evaluateHybridThresholdPolicy(results, options.threshold)
-  const maxFailedRunsViolations = evaluateMaxFailedRunsPolicy(results, options.maxFailedRuns)
-
-  const violations = [
-    ...thresholdViolations,
-    ...maxFailedRunsViolations,
-  ]
-
-  return {
-    pass: violations.length === 0,
-    violations,
-  }
-}
-
-export type {
-  HybridThresholdPolicyOptions,
-  MaxFailedRunsPolicyOptions,
-  ResultPolicyViolation,
-}
diff --git a/packages/vieval/src/core/processors/results/policies/hybrid-threshold.ts b/packages/vieval/src/core/processors/results/policies/hybrid-threshold.ts
deleted file mode 100644
index 06904fb..0000000
--- a/packages/vieval/src/core/processors/results/policies/hybrid-threshold.ts
+++ /dev/null
@@ -1,65 +0,0 @@
-import type { AggregatedRunResults } from '../../../runner/aggregate'
-
-/**
- * Configures hybrid-threshold policy behavior.
- */
-export interface HybridThresholdPolicyOptions {
-  /**
-   * Minimum required overall hybrid score.
-   *
-   * @default 0.7
-   */
-  minOverallHybridScore?: number
-  /**
-   * Minimum required inferenceExecutor hybrid score.
-   *
-   * @default 0.6
-   */
-  minProviderHybridScore?: number
-}
-
-/**
- * Violation emitted when result policies fail.
- */
-export interface ResultPolicyViolation {
-  /**
-   * Stable policy id.
-   */
-  policyId: string
-  /**
-   * Human-readable violation reason.
-   */
-  reason: string
-}
-
-/**
- * Evaluates threshold policy against aggregated results.
- */
-export function evaluateHybridThresholdPolicy(
-  results: AggregatedRunResults,
-  options: HybridThresholdPolicyOptions = {},
-): ResultPolicyViolation[] {
-  const minOverallHybridScore = options.minOverallHybridScore ?? 0.7
-  const minProviderHybridScore = options.minProviderHybridScore ?? 0.6
-
-  const violations: ResultPolicyViolation[] = []
-
-  const overallHybridAverage = results.overall.hybridAverage
-  if (overallHybridAverage == null || overallHybridAverage < minOverallHybridScore) {
-    violations.push({
-      policyId: 'threshold:overall-hybrid',
-      reason: `Overall hybrid average ${overallHybridAverage ?? 'null'} is below ${minOverallHybridScore}.`,
-    })
-  }
-
-  for (const inferenceExecutor of results.inferenceExecutors) {
-    if (inferenceExecutor.hybridAverage == null || inferenceExecutor.hybridAverage < minProviderHybridScore) {
-      violations.push({
-        policyId: 'threshold:inferenceExecutor-hybrid',
-        reason: `Provider ${inferenceExecutor.inferenceExecutorId} hybrid average ${inferenceExecutor.hybridAverage ?? 'null'} is below ${minProviderHybridScore}.`,
-      })
-    }
-  }
-
-  return violations
-}
diff --git a/packages/vieval/src/core/processors/results/policies/max-failed-runs.ts b/packages/vieval/src/core/processors/results/policies/max-failed-runs.ts
deleted file mode 100644
index 374f8e9..0000000
--- a/packages/vieval/src/core/processors/results/policies/max-failed-runs.ts
+++ /dev/null
@@ -1,48 +0,0 @@
-import type { AggregatedRunResults } from '../../../runner/aggregate'
-import type { ResultPolicyViolation } from './hybrid-threshold'
-
-/**
- * Configures hard-limit policy for failed runs.
- */
-export interface MaxFailedRunsPolicyOptions {
-  /**
-   * Maximum allowed failed run count.
-   *
-   * @default 0
-   */
-  maxFailedRuns?: number
-  /**
-   * Hybrid score threshold below which a run counts as failed.
-   *
-   * @default 0.6
-   */
-  minRunHybridScore?: number
-}
-
-/**
- * Evaluates hard-limit policy for failed runs.
- */
-export function evaluateMaxFailedRunsPolicy(
-  results: AggregatedRunResults,
-  options: MaxFailedRunsPolicyOptions = {},
-): ResultPolicyViolation[] {
-  const maxFailedRuns = options.maxFailedRuns ?? 0
-  const minRunHybridScore = options.minRunHybridScore ?? 0.6
-
-  const failedRuns = results.runs.filter((run) => {
-    if (run.hybridAverage == null) {
-      return true
-    }
-
-    return run.hybridAverage < minRunHybridScore
-  })
-
-  if (failedRuns.length <= maxFailedRuns) {
-    return []
-  }
-
-  return [{
-    policyId: 'hard-limit:max-failed-runs',
-    reason: `Failed runs ${failedRuns.length} exceed maxFailedRuns ${maxFailedRuns} with minRunHybridScore ${minRunHybridScore}.`,
-  }]
-}
diff --git a/packages/vieval/src/core/runner/aggregate.test.ts b/packages/vieval/src/core/runner/aggregate.test.ts
index bdb35f1..efbf41d 100644
--- a/packages/vieval/src/core/runner/aggregate.test.ts
+++ b/packages/vieval/src/core/runner/aggregate.test.ts
@@ -1,3 +1,5 @@
+import type { ScoreSummary } from '../scoring'
+import type { RunResult } from './aggregate'
 import type { ScheduledTaskMatrix } from './schedule'
 
 import { describe, expect, it } from 'vitest'
@@ -26,6 +28,23 @@ function createScheduledTaskMatrix(
   }
 }
 
+function run(id: string, inferenceExecutorId: string, entryId: string, primary: string | undefined, scores: ScoreSummary[], completion: ScoreSummary): RunResult {
+  return {
+    completion,
+    entryId,
+    id,
+    inferenceExecutorId,
+    matrix: createScheduledTaskMatrix(),
+    passed: summary('passed', completion.sum, completion.n),
+    ...(primary == null ? {} : { primary }),
+    scores,
+  }
+}
+
+function summary(name: string, sum: number, n: number, total = n): ScoreSummary {
+  return { mean: n === 0 ? null : sum / n, n, name, sum, total, unscored: {} }
+}
+
 describe('aggregateRunResults', () => {
   it('preserves scoped matrix dimensions as stable run artifacts', () => {
     const resultMatrix = createScheduledTaskMatrix({
@@ -35,15 +54,13 @@ describe('aggregateRunResults', () => {
       rubric: 'strict',
     })
 
-    const summary = aggregateRunResults([
+    const result = aggregateRunResults([
       {
         entryId: 'agent/chess-commentary/chess-commentary',
         id: 'run-1',
         inferenceExecutorId: 'openai:gpt-4.1-mini',
         matrix: resultMatrix,
-        scores: [
-          { kind: 'exact', score: 1 },
-        ],
+        scores: [summary('pass', 1, 1)],
       },
     ])
 
@@ -52,7 +69,7 @@ describe('aggregateRunResults', () => {
     resultMatrix.meta.runRowId = 'mutated-run-row-id'
     resultMatrix.meta.evalRowId = 'mutated-eval-row-id'
 
-    expect(summary.runs[0]?.matrix).toEqual({
+    expect(result.runs[0]?.matrix).toEqual({
       eval: {
         rubric: 'strict',
       },
@@ -67,143 +84,48 @@ describe('aggregateRunResults', () => {
     })
   })
 
-  it('produces a hybrid score by averaging exact and judge category means', () => {
-    const summary = aggregateRunResults([
-      {
-        entryId: 'agent/chess-commentary/chess-commentary',
-        id: 'run-1',
-        inferenceExecutorId: 'openai:gpt-4.1-mini',
-        matrix: createScheduledTaskMatrix({
-          difficulty: 'rapid',
-        }),
-        scores: [
-          { kind: 'exact', score: 1 },
-          { kind: 'exact', score: 0.5 },
-          { kind: 'judge', score: 0.75 },
-          { kind: 'judge', score: 0.25 },
-        ],
-      },
-      {
-        entryId: 'agent/chess-commentary/chess-commentary',
-        id: 'run-2',
-        inferenceExecutorId: 'openai:gpt-4.1-mini',
-        matrix: createScheduledTaskMatrix({
-          difficulty: 'blitz',
-        }),
-        scores: [
-          { kind: 'exact', score: 0.25 },
-          { kind: 'judge', score: 0.5 },
-        ],
-      },
+  /**
+   * @example
+   * pass 1/1 on executor a and 0/3 on executor b -> overall 1/4 = 0.25 (cases pooled), not the mean of means 0.5.
+   */
+  it('pools each score by cases per executor and overall', () => {
+    const result = aggregateRunResults([
+      run('run-1', 'a', 'entry', 'pass', [summary('pass', 1, 1)], summary('completion', 1, 1)),
+      run('run-2', 'b', 'entry', 'pass', [summary('pass', 0, 3, 4)], summary('completion', 3, 4)),
     ])
 
-    expect(summary.runs).toHaveLength(2)
-    expect(summary.inferenceExecutors).toEqual([
-      {
-        coverage: null,
-        exactAverage: 7 / 12,
-        hybridAverage: expect.any(Number),
-        inferenceExecutorId: 'openai:gpt-4.1-mini',
-        judgeAverage: 0.5,
-        runCount: 2,
-        // Runs without coverage have no unmeasured cases, so the strict score equals the exact score.
-        strictExactAverage: 7 / 12,
-      },
-    ])
-    expect(summary.inferenceExecutors[0]?.hybridAverage).toBeCloseTo(13 / 24)
-    expect(summary.overall).toEqual({
-      coverage: null,
-      exactAverage: 7 / 12,
-      hybridAverage: expect.any(Number),
-      judgeAverage: 0.5,
-      runCount: 2,
-      strictExactAverage: 7 / 12,
-    })
-    expect(summary.overall.hybridAverage).toBeCloseTo(13 / 24)
+    expect(result.inferenceExecutors.map(executor => [executor.inferenceExecutorId, executor.scores[0]?.mean, executor.runCount])).toEqual([['a', 1, 1], ['b', 0, 1]])
+    expect(result.overall.scores).toEqual([summary('pass', 1, 4, 5)])
+    expect(result.overall.completion).toEqual(summary('completion', 4, 5))
+    expect(result.overall.headline.headline).toEqual({ kind: 'score', score: summary('pass', 1, 4, 5) })
   })
 
-  it('falls back to the available score family when only one side exists', () => {
-    const summary = aggregateRunResults([
-      {
-        entryId: 'agent/chess-commentary/chess-commentary',
-        id: 'run-1',
-        inferenceExecutorId: 'openai:gpt-4.1',
-        matrix: createScheduledTaskMatrix(),
-        scores: [
-          { kind: 'judge', score: 0.8 },
-        ],
-      },
+  /**
+   * @example
+   * two entries with primary `pass` and `accuracy` -> the overall headline lists each entry.
+   */
+  it('lists each task in the headline when the primary names differ', () => {
+    const result = aggregateRunResults([
+      run('run-1', 'a', 'entry-1', 'pass', [summary('pass', 1, 1)], summary('completion', 1, 1)),
+      run('run-2', 'a', 'entry-2', 'accuracy', [summary('accuracy', 1, 2)], summary('completion', 2, 2)),
     ])
 
-    expect(summary.inferenceExecutors).toEqual([
-      {
-        coverage: null,
-        exactAverage: null,
-        hybridAverage: 0.8,
-        inferenceExecutorId: 'openai:gpt-4.1',
-        judgeAverage: 0.8,
-        runCount: 1,
-        strictExactAverage: null,
-      },
-    ])
+    expect(result.overall.headline.headline).toEqual({
+      kind: 'per-task',
+      tasks: [{ score: summary('pass', 1, 1), taskId: 'entry-1' }, { score: summary('accuracy', 1, 2), taskId: 'entry-2' }],
+    })
+    expect(result.runs.map(item => item.primary)).toEqual(['pass', 'accuracy'])
   })
 
-  // ROOT CAUSE:
-  //
-  // In the `tts-speech-formula` comparison of 2026-10-19, the `llm` workspace graded 87.6% of its measured cases right,
-  // but 93 of its cases errored on provider 429s. Errored cases scored exact 0, so its exact score sank to the level
-  // of a workspace that measured every case and got far fewer right.
-  //
-  // collectCaseOutcomeScores: if (outcome.state !== 'passed') exact.push(0)
-  //
-  // We fixed this by scoring only measured cases in `exactAverage`, and reporting `coverage` and the old
-  // every-case score as `strictExactAverage` next to it.
   /**
    * @example
-   * one run measured 3 of 4 cases with exact 1 -> exact 1, coverage 3/4, strict exact 0.75.
+   * a custom executor run without completion -> completion null for the run and the group.
    */
-  it('scores measured cases in exactAverage and reports coverage and strict exact next to it', () => {
-    const summary = aggregateRunResults([
-      {
-        coverage: { measured: 3, total: 4 },
-        entryId: 'tasks/read-aloud/ko',
-        id: 'run-1',
-        inferenceExecutorId: 'default',
-        matrix: createScheduledTaskMatrix(),
-        scores: [{ kind: 'exact', score: 1 }],
-      },
-      {
-        coverage: { measured: 0, total: 2 },
-        entryId: 'tasks/read-aloud/zh',
-        id: 'run-2',
-        inferenceExecutorId: 'default',
-        matrix: createScheduledTaskMatrix(),
-        scores: [],
-      },
-    ])
+  it('keeps completion null for runs that do not report their cases', () => {
+    const result = aggregateRunResults([{ entryId: 'custom', id: 'run-1', inferenceExecutorId: 'a', matrix: createScheduledTaskMatrix(), scores: [summary('quality', 1, 2)] }])
 
-    expect(summary.runs[0]?.exactAverage).toBe(1)
-    expect(summary.runs[0]?.coverage).toEqual({ measured: 3, total: 4 })
-    expect(summary.runs[0]?.strictExactAverage).toBe(0.75)
-    expect(summary.runs[1]?.exactAverage).toBeNull()
-    expect(summary.runs[1]?.strictExactAverage).toBe(0)
-    expect(summary.overall.exactAverage).toBe(1)
-    expect(summary.overall.coverage).toEqual({ measured: 3, total: 6 })
-    expect(summary.overall.strictExactAverage).toBe(0.375)
-  })
-
-  it('throws a deterministic error for unknown score kinds', () => {
-    expect(() => aggregateRunResults([
-      {
-        entryId: 'agent/chess-commentary/chess-commentary',
-        id: 'run-1',
-        inferenceExecutorId: 'openai:gpt-4.1',
-        matrix: createScheduledTaskMatrix(),
-        scores: [
-          { kind: 'judge', score: 0.8 },
-          { kind: 'rubric' as 'exact', score: 0.2 },
-        ],
-      },
-    ])).toThrowError('Unknown eval score kind "rubric".')
+    expect(result.runs[0]?.completion).toBeNull()
+    expect(result.overall.completion).toBeNull()
+    expect(result.overall.headline.headline).toEqual({ kind: 'passed-cases', score: summary('passed', 0, 0) })
   })
 })
diff --git a/packages/vieval/src/core/runner/aggregate.ts b/packages/vieval/src/core/runner/aggregate.ts
index 509e7aa..364be25 100644
--- a/packages/vieval/src/core/runner/aggregate.ts
+++ b/packages/vieval/src/core/runner/aggregate.ts
@@ -1,40 +1,16 @@
-import type { CaseCoverage } from '../scoring'
+import type { HeadlineResult, ScoreSummary } from '../scoring'
 import type { ScheduledTaskMatrix } from './schedule'
 
-import { hybridAverage, strictExactOf } from '../scoring'
+import { emptyScoreSummary, poolScoreSummaries, resolveHeadline } from '../scoring'
 
 /**
- * Stores inferenceExecutor-level score aggregates across multiple runs.
+ * Score summaries of the runs of one inference executor.
  */
-export interface AggregatedProviderSummary {
-  /**
-   * Measured and settled cases summed over the runs, or `null` when no run reported its cases.
-   */
-  coverage: CaseCoverage | null
-  /**
-   * Mean of all exact-match scores over measured cases, or `null` when absent.
-   */
-  exactAverage: null | number
-  /**
-   * Hybrid average derived from the inferenceExecutor exact and judge means.
-   */
-  hybridAverage: null | number
+export interface AggregatedProviderSummary extends AggregatedScoreSummary {
   /**
    * Stable inferenceExecutor id.
    */
   inferenceExecutorId: string
-  /**
-   * Mean of all judge-based scores or `null` when absent.
-   */
-  judgeAverage: null | number
-  /**
-   * Number of runs included in this inferenceExecutor bucket.
-   */
-  runCount: number
-  /**
-   * Mean of the per-run strict exact scores, where a case without a result scores `0`, or `null` when absent.
-   */
-  strictExactAverage: null | number
 }
 
 /**
@@ -48,34 +24,25 @@ export interface AggregatedRunResults {
   /**
    * Overall summary across every run.
    */
-  overall: Omit<AggregatedProviderSummary, 'inferenceExecutorId'>
+  overall: AggregatedScoreSummary
   /**
-   * Per-run normalized score summaries.
+   * Per-run summaries.
    */
   runs: AggregatedRunSummary[]
 }
 
 /**
- * Stores the per-run score averages after normalization.
+ * Score summaries of one run.
  */
 export interface AggregatedRunSummary {
   /**
-   * Measured and settled cases of this run, or `null` when the task did not report its cases.
+   * Completion rate of the run, or `null` when the task did not report its cases.
    */
-  coverage: CaseCoverage | null
+  completion: null | ScoreSummary
   /**
    * Collected eval entry id.
    */
   entryId: string
-  /**
-   * Mean of exact-match scores over measured cases, or `null` when absent.
-   */
-  exactAverage: null | number
-  /**
-   * Hybrid average. Uses both families when present, otherwise falls back to the
-   * single available family.
-   */
-  hybridAverage: null | number
   /**
    * Stable run id.
    */
@@ -84,18 +51,44 @@ export interface AggregatedRunSummary {
    * Stable inferenceExecutor id.
    */
   inferenceExecutorId: string
-  /**
-   * Mean of judge-based scores or `null` when absent.
-   */
-  judgeAverage: null | number
   /**
    * Concrete matrix selection used by the run.
    */
   matrix: ScheduledTaskMatrix
   /**
-   * Mean of the per-run strict exact scores, where a case without a result scores `0`, or `null` when absent.
+   * Share of passed cases, or `null` when the task did not report its cases.
+   */
+  passed: null | ScoreSummary
+  /**
+   * Name of the run's headline score, when it has one.
+   */
+  primary?: string
+  /**
+   * One summary per score of the run.
+   */
+  scores: ScoreSummary[]
+}
+
+/**
+ * Score summaries pooled over a group of runs: one inference executor, or every run.
+ */
+export interface AggregatedScoreSummary {
+  /**
+   * Completion rate pooled over the runs that reported their cases, or `null` when none did.
+   */
+  completion: null | ScoreSummary
+  /**
+   * Headline of the group, resolved with {@link resolveHeadline}.
+   */
+  headline: HeadlineResult
+  /**
+   * Number of runs in the group.
    */
-  strictExactAverage: null | number
+  runCount: number
+  /**
+   * Each score pooled by name over the runs, in first-seen order.
+   */
+  scores: ScoreSummary[]
 }
 
 /**
@@ -103,9 +96,9 @@ export interface AggregatedRunSummary {
  */
 export interface RunResult {
   /**
-   * Measured and settled cases of the run, when the task reports them (the task DSL always does).
+   * Completion rate of the run, when the task reports its cases (the task DSL always does).
    */
-  coverage?: CaseCoverage
+  completion?: ScoreSummary
   /**
    * Collected eval entry id.
    */
@@ -123,94 +116,59 @@ export interface RunResult {
    */
   matrix: ScheduledTaskMatrix
   /**
-   * Raw scores emitted by the eval.
+   * Share of passed cases, when the task reports its cases.
    */
-  scores: readonly RunScore[]
-}
-
-/**
- * Represents one normalized score emitted by a completed eval run.
- */
-export interface RunScore {
+  passed?: ScoreSummary
   /**
-   * Score family used for aggregation.
+   * Name of the run's headline score, when it has one.
    */
-  kind: RunScoreKind
+  primary?: string
   /**
-   * Normalized score in the `0..1` range.
+   * One summary per score emitted by the eval.
    */
-  score: number
+  scores: readonly ScoreSummary[]
 }
 
 /**
- * Identifies the scoring family for a single eval score.
- */
-export type RunScoreKind = 'exact' | 'judge'
-
-interface ScoreBuckets {
-  exact: number[]
-  judge: number[]
-}
-
-/**
- * Aggregates exact-match and judge-based scores into hybrid runner summaries.
+ * Pools the score summaries of runner results per run, per inference executor, and overall.
  *
  * Call stack:
  *
  * {@link runScheduledTasks}
  *   -> {@link aggregateRunResults}
- *     -> {@link createRunSummary}
- *     -> {@link createProviderSummary}
- *       -> `report output`
+ *     -> {@link poolScoreSummaries}
+ *     -> {@link resolveHeadline}
+ *       -> `run-summary.json`
  *
  * Use when:
- * - a runner batch mixes deterministic exact checks with judge-based grading
- * - inferenceExecutor comparison should preserve both score families and one hybrid view
+ * - a runner batch ends and its runs are summarized for reports, the CLI, and comparisons
  *
  * Expects:
- * - each score to be normalized to the `0..1` range before aggregation
- * - `scores.kind` to use only `'exact'` or `'judge'`
+ * - score values in `0..1`; summaries of one name pool their cases, so a large run weighs more than a small one
+ *
+ * Returns:
+ * - per-run summaries, per-executor summaries sorted by id, and the overall summary, each group with its headline
  */
 export function aggregateRunResults(results: readonly RunResult[]): AggregatedRunResults {
-  const runs = results.map(createRunSummary)
-
-  const inferenceExecutorIds = Array.from(new Set(results.map(result => result.inferenceExecutorId)))
-  const inferenceExecutors = inferenceExecutorIds
-    .map((inferenceExecutorId) => {
-      const providerResults = results.filter(result => result.inferenceExecutorId === inferenceExecutorId)
-      return createProviderSummary(inferenceExecutorId, providerResults)
-    })
-    .sort((left, right) => left.inferenceExecutorId.localeCompare(right.inferenceExecutorId))
-
-  const overall = createProviderSummary(
-    'overall',
-    results,
-  )
-
-  const { inferenceExecutorId: _overallId, ...overallSummary } = overall
+  const inferenceExecutorIds = Array.from(new Set(results.map(result => result.inferenceExecutorId))).sort()
 
   return {
-    inferenceExecutors,
-    overall: overallSummary,
-    runs,
-  }
-}
-
-function assertKnownScoreKind(kind: string): RunScoreKind {
-  if (kind === 'exact' || kind === 'judge') {
-    return kind
-  }
-
-  throw new TypeError(`Unknown eval score kind "${kind}".`)
-}
-
-function average(scores: readonly number[]): null | number {
-  if (scores.length === 0) {
-    return null
+    inferenceExecutors: inferenceExecutorIds.map(inferenceExecutorId => ({
+      inferenceExecutorId,
+      ...summarizeGroup(results.filter(result => result.inferenceExecutorId === inferenceExecutorId)),
+    })),
+    overall: summarizeGroup(results),
+    runs: results.map(result => ({
+      completion: result.completion ?? null,
+      entryId: result.entryId,
+      id: result.id,
+      inferenceExecutorId: result.inferenceExecutorId,
+      matrix: cloneScheduledTaskMatrix(result.matrix),
+      passed: result.passed ?? null,
+      ...(result.primary == null ? {} : { primary: result.primary }),
+      scores: [...result.scores],
+    })),
   }
-
-  const total = scores.reduce((sum, score) => sum + score, 0)
-  return total / scores.length
 }
 
 function cloneScheduledTaskMatrix(matrix: ScheduledTaskMatrix): ScheduledTaskMatrix {
@@ -227,100 +185,19 @@ function cloneScheduledTaskMatrix(matrix: ScheduledTaskMatrix): ScheduledTaskMat
   }
 }
 
-function collectScoreBuckets(scores: readonly RunScore[]): ScoreBuckets {
-  const buckets: ScoreBuckets = {
-    exact: [],
-    judge: [],
-  }
-
-  for (const score of scores) {
-    const kind = assertKnownScoreKind(score.kind)
-
-    if (kind === 'exact') {
-      buckets.exact.push(score.score)
-      continue
-    }
-
-    buckets.judge.push(score.score)
-  }
-
-  return buckets
-}
-
-function createProviderSummary(inferenceExecutorId: string, results: readonly RunResult[]): AggregatedProviderSummary {
-  const exactScores: number[] = []
-  const judgeScores: number[] = []
-  const strictExactScores: number[] = []
-  const coverages = results.flatMap(result => result.coverage == null ? [] : [result.coverage])
-
-  for (const result of results) {
-    const buckets = collectScoreBuckets(result.scores)
-    exactScores.push(...buckets.exact)
-    judgeScores.push(...buckets.judge)
-
-    strictExactScores.push(...strictExactScoresOf(buckets.exact, result.coverage))
-  }
-
-  const exactAverage = average(exactScores)
-  const judgeAverage = average(judgeScores)
-
+function summarizeGroup(results: readonly RunResult[]): AggregatedScoreSummary {
+  const completions = results.flatMap(result => result.completion == null ? [] : [result.completion])
   return {
-    coverage: coverages.length === 0
-      ? null
-      : {
-          measured: coverages.reduce((sum, coverage) => sum + coverage.measured, 0),
-          total: coverages.reduce((sum, coverage) => sum + coverage.total, 0),
-        },
-    exactAverage,
-    hybridAverage: hybridAverage(exactAverage, judgeAverage),
-    inferenceExecutorId,
-    judgeAverage,
+    completion: completions.length === 0 ? null : poolScoreSummaries(completions)[0]!,
+    // A run is one task under one matrix row, so its eval entry identifies the task for per-task headlines.
+    headline: resolveHeadline(results.map(result => ({
+      completion: result.completion ?? emptyScoreSummary('completion'),
+      passed: result.passed ?? emptyScoreSummary('passed'),
+      ...(result.primary == null ? {} : { primary: result.primary }),
+      scores: [...result.scores],
+      taskId: result.entryId,
+    }))),
     runCount: results.length,
-    strictExactAverage: average(strictExactScores),
+    scores: poolScoreSummaries(results.flatMap(result => result.scores)),
   }
 }
-
-function createRunSummary(result: RunResult): AggregatedRunSummary {
-  const buckets = collectScoreBuckets(result.scores)
-  const exactAverage = average(buckets.exact)
-  const judgeAverage = average(buckets.judge)
-
-  return {
-    coverage: result.coverage == null ? null : { ...result.coverage },
-    entryId: result.entryId,
-    exactAverage,
-    hybridAverage: hybridAverage(exactAverage, judgeAverage),
-    id: result.id,
-    inferenceExecutorId: result.inferenceExecutorId,
-    judgeAverage,
-    matrix: cloneScheduledTaskMatrix(result.matrix),
-    strictExactAverage: runStrictExact(exactAverage, result.coverage),
-  }
-}
-
-/**
- * Strict exact score of one run. A run without coverage (a custom executor) has no unmeasured cases to count,
- * so its strict score equals its exact score.
- */
-function runStrictExact(exactAverage: null | number, coverage: CaseCoverage | undefined): null | number {
-  return coverage == null ? exactAverage : strictExactOf(exactAverage, coverage)
-}
-
-/**
- * Exact scores of one run scaled to strict scores, so pooling them across runs mirrors how `exactAverage` pools the
- * exact scores: each score shrinks by the run's measured share, and a run that settled cases without measuring any
- * adds one `0`.
- */
-function strictExactScoresOf(exactScores: readonly number[], coverage: CaseCoverage | undefined): number[] {
-  if (coverage == null) {
-    return [...exactScores]
-  }
-  if (coverage.total === 0) {
-    return []
-  }
-  if (exactScores.length === 0) {
-    return [0]
-  }
-
-  return exactScores.map(score => score * coverage.measured / coverage.total)
-}
diff --git a/packages/vieval/src/core/runner/event-bus.test.ts b/packages/vieval/src/core/runner/event-bus.test.ts
index 4184ddc..0d94455 100644
--- a/packages/vieval/src/core/runner/event-bus.test.ts
+++ b/packages/vieval/src/core/runner/event-bus.test.ts
@@ -30,8 +30,8 @@ describe('createRunEventBus', () => {
      * expect(envelopes.map(({ time: _time, ...envelope }) => envelope)).toEqual([...])
      */
     expect(envelopes.map(({ time: _time, ...envelope }) => envelope)).toEqual([
-      { ...identity, caseId: undefined, data: { totalTasks: 1 }, event: 'run.started', projectId: undefined, schemaVersion: 2, seq: 1, taskId: undefined },
-      { ...identity, caseId: '0:case', data: { index: 0, name: 'case', total: 1 }, event: 'case.started', projectId: 'project-1', schemaVersion: 2, seq: 2, taskId: 'task-1' },
+      { ...identity, caseId: undefined, data: { totalTasks: 1 }, event: 'run.started', projectId: undefined, schemaVersion: 3, seq: 1, taskId: undefined },
+      { ...identity, caseId: '0:case', data: { index: 0, name: 'case', total: 1 }, event: 'case.started', projectId: 'project-1', schemaVersion: 3, seq: 2, taskId: 'task-1' },
     ])
   })
 
diff --git a/packages/vieval/src/core/runner/run.test.ts b/packages/vieval/src/core/runner/run.test.ts
index fa08004..7284485 100644
--- a/packages/vieval/src/core/runner/run.test.ts
+++ b/packages/vieval/src/core/runner/run.test.ts
@@ -50,7 +50,7 @@ function createSuccessfulRunResult(task: ScheduledTask): RunResult {
     id: task.id,
     inferenceExecutorId: task.inferenceExecutor.id,
     matrix: task.matrix,
-    scores: [{ kind: 'exact', score: 1 }],
+    scores: [{ mean: 1, n: 1, name: 'correct', sum: 1, total: 1, unscored: {} }],
   }
 }
 
@@ -67,7 +67,7 @@ describe('runScheduledTasks', () => {
         inferenceExecutorId: task.inferenceExecutor.id,
         matrix: task.matrix,
         scores: [
-          { kind: 'exact', score: 1 },
+          { mean: 1, n: 1, name: 'correct', sum: 1, total: 1, unscored: {} },
         ],
       }
     }
diff --git a/packages/vieval/src/core/scoring/index.test.ts b/packages/vieval/src/core/scoring/index.test.ts
index be266b7..f6c654f 100644
--- a/packages/vieval/src/core/scoring/index.test.ts
+++ b/packages/vieval/src/core/scoring/index.test.ts
@@ -1,77 +1,270 @@
+import type { ScoreDefinedData } from '../../events'
+import type { HeadlineInput, ScoredCaseOutcome } from './index'
+
 import { describe, expect, it } from 'vitest'
 
-import { isMeasuredState, scoreCaseOutcomes, strictExactOf } from './index'
+import { countCase, deriveScores, emptyScoreSummary, isMeasuredState, poolScoreSummaries, resolveHeadline, scoreCases } from './index'
+
+function outcome(state: ScoredCaseOutcome['state'], scores: Record<string, number> = {}, unscored: Record<string, string> = {}): ScoredCaseOutcome {
+  return { scores, state, unscored }
+}
 
-describe('scoreCaseOutcomes', () => {
+describe('scoreCases', () => {
   /**
    * @example
-   * passed 1, passed 0.5, failed, errored, timeout -> exact 0.5 over 3 measured, strict exact 0.3 over 5.
+   * passed 1, passed 0.5, failed, errored, timeout, skipped -> correct mean 0.5 over n 3 of total 5.
    */
-  it('scores measured cases in exact and every settled case in strict exact', () => {
-    const scores = scoreCaseOutcomes([
-      { scores: {}, state: 'passed' },
-      { scores: { exact: 0.5 }, state: 'passed' },
-      { scores: {}, state: 'failed' },
-      { scores: {}, state: 'errored' },
-      { scores: {}, state: 'timeout' },
-    ])
+  it('counts passed values and failed zeros, and leaves errored, timed-out, and skipped cases out by default', () => {
+    const scoring = scoreCases([
+      outcome('passed', { correct: 1 }),
+      outcome('passed', { correct: 0.5 }),
+      outcome('failed', { correct: 1 }),
+      outcome('errored'),
+      outcome('timeout'),
+      outcome('skipped'),
+    ], [{ name: 'correct' }])
+
+    expect(scoring.scores).toEqual([{ mean: 0.5, n: 3, name: 'correct', sum: 1.5, total: 5, unscored: {} }])
+  })
+
+  /**
+   * @example
+   * errors: 'zero' -> errored and timed-out cases count as 0: mean 1/3 over 3.
+   */
+  it('counts errored and timed-out cases as 0 when the score sets errors: zero', () => {
+    const scoring = scoreCases([
+      outcome('passed', { pass: 1 }),
+      outcome('errored'),
+      outcome('timeout'),
+    ], [{ errors: 'zero', name: 'pass' }])
 
-    expect(scores.exact).toBe(0.5)
-    expect(scores.coverage).toEqual({ measured: 3, total: 5 })
-    expect(scores.strictExact).toBeCloseTo(0.3)
+    expect(scoring.scores[0]).toEqual({ mean: 1 / 3, n: 3, name: 'pass', sum: 1, total: 3, unscored: {} })
   })
 
   /**
    * @example
-   * skipped outcomes stay out of both the score and the coverage.
+   * an unscored passed case never counts, whatever `errors` says, and is counted per reason.
    */
-  it('leaves skipped cases out of the score and the coverage', () => {
-    const scores = scoreCaseOutcomes([
-      { scores: {}, state: 'passed' },
-      { scores: {}, state: 'skipped' },
+  it('never counts unscored values and sums their reasons', () => {
+    const scoring = scoreCases([
+      outcome('passed', { natural: 1 }),
+      outcome('passed', {}, { natural: 'grader_failed' }),
+      outcome('passed', {}, { natural: 'grader_failed' }),
+      outcome('passed', {}, { natural: 'refusal' }),
+    ], [{ errors: 'zero', name: 'natural' }])
+
+    expect(scoring.scores[0]).toEqual({ mean: 1, n: 1, name: 'natural', sum: 1, total: 4, unscored: { grader_failed: 2, refusal: 1 } })
+  })
+
+  /**
+   * @example
+   * a declared score that a passed case never recorded is missing: in total, not in n.
+   */
+  it('leaves a declared score that a passed case did not record out of the mean', () => {
+    const scoring = scoreCases([outcome('passed', { correct: 1 }), outcome('passed')], [{ name: 'correct' }])
+
+    expect(scoring.scores[0]).toMatchObject({ mean: 1, n: 1, total: 2 })
+  })
+
+  /**
+   * @example
+   * no case recorded a score -> no scores; completion and passed still count every settled case.
+   */
+  it('reports completion and passed shares, with errored and timed-out counts', () => {
+    const scoring = scoreCases([
+      outcome('passed'),
+      outcome('failed'),
+      outcome('errored'),
+      outcome('errored'),
+      outcome('timeout'),
+      outcome('skipped'),
     ])
 
-    expect(scores.exact).toBe(1)
-    expect(scores.coverage).toEqual({ measured: 1, total: 1 })
+    expect(scoring.scores).toEqual([])
+    expect(scoring.completion).toEqual({ mean: 0.4, n: 5, name: 'completion', sum: 2, total: 5, unscored: { errored: 2, timeout: 1 } })
+    expect(scoring.passed).toMatchObject({ mean: 0.2, n: 5, sum: 1 })
+    expect(scoring.primary).toBeUndefined()
+  })
+
+  /**
+   * @example
+   * case a errored, then passed in a later auto attempt; case b errored twice -> scores count both rounds of a,
+   * completion counts each case once by its final outcome: 1 of 2.
+   */
+  it('counts each case once by its final outcome in completion and passed, and every outcome in the scores', () => {
+    const scoring = scoreCases([
+      { ...outcome('errored'), caseId: 'a' },
+      { ...outcome('errored'), caseId: 'b' },
+      { ...outcome('passed', { pass: 1 }), caseId: 'a' },
+      { ...outcome('errored'), caseId: 'b' },
+    ], [{ errors: 'zero', name: 'pass' }])
+
+    expect(scoring.scores[0]).toMatchObject({ n: 4, sum: 1, total: 4 })
+    expect(scoring.completion).toEqual({ mean: 0.5, n: 2, name: 'completion', sum: 1, total: 2, unscored: { errored: 1 } })
+    expect(scoring.passed).toMatchObject({ mean: 0.5, n: 2, sum: 1 })
   })
 
   /**
    * @example
-   * every case errored -> exact null, strict exact 0.
+   * declared names keep declaration order; names recorded without a definition follow in name order.
    */
-  it('has no exact score but a strict exact score of 0 when nothing was measured', () => {
-    const scores = scoreCaseOutcomes([{ scores: {}, state: 'errored' }])
+  it('orders declared scores first, then recorded-only names', () => {
+    const scoring = scoreCases([outcome('passed', { a: 1, natural: 1, z: 1 })], [{ name: 'natural' }])
+
+    expect(scoring.scores.map(summary => summary.name)).toEqual(['natural', 'a', 'z'])
+  })
 
-    expect(scores.exact).toBeNull()
-    expect(scores.hybrid).toBeNull()
-    expect(scores.strictExact).toBe(0)
+  /**
+   * @example
+   * primary: the declared primary score, else the only score, else none.
+   */
+  it('resolves the primary score of the group', () => {
+    expect(scoreCases([outcome('passed', { correct: 1, pass: 1 })], [{ name: 'correct' }, { name: 'pass', primary: true }]).primary).toBe('pass')
+    expect(scoreCases([outcome('passed', { only: 1 })]).primary).toBe('only')
+    expect(scoreCases([outcome('passed', { a: 1, b: 1 })]).primary).toBeUndefined()
   })
+})
+
+describe('countCase', () => {
+  const definitions = [{ name: 'correct' }, { errors: 'zero' as const, name: 'answered' }]
 
   /**
    * @example
-   * judge scores come only from passed cases; hybrid averages exact and judge.
+   * failed -> 0 in every score of the task; errored -> 0 only in `errors: 'zero'` scores; unscored -> absent.
    */
-  it('averages judge scores of passed cases into the hybrid score', () => {
-    const scores = scoreCaseOutcomes([
-      { scores: { judge: 0.5 }, state: 'passed' },
-      { scores: {}, state: 'failed' },
+  it('counts one outcome by its state', () => {
+    expect(countCase(outcome('passed', { answered: 1, correct: 1 }), definitions)).toEqual({ answered: 1, correct: 1 })
+    expect(countCase(outcome('passed', { answered: 1 }, { correct: 'grader_failed' }), definitions)).toEqual({ answered: 1 })
+    expect(countCase(outcome('failed', { correct: 1 }), definitions)).toEqual({ answered: 0, correct: 0 })
+    expect(countCase(outcome('errored'), definitions)).toEqual({ answered: 0 })
+    expect(countCase(outcome('timeout'), definitions)).toEqual({ answered: 0 })
+    expect(countCase(outcome('skipped'), definitions)).toEqual({})
+  })
+})
+
+describe('deriveScores', () => {
+  const correct: ScoreDefinedData = { name: 'correct' }
+  const natural: ScoreDefinedData = { name: 'natural' }
+
+  /**
+   * @example
+   * all [correct, natural] -> 1 only when both are 1; a mixed reading (0.5) fails without a threshold.
+   */
+  it('derives all as binary with a default threshold of 1', () => {
+    const pass: ScoreDefinedData = { combine: { inputs: [{ name: 'correct' }, { name: 'natural' }], kind: 'all' }, name: 'pass' }
+
+    expect(deriveScores([correct, natural, pass], { correct: 1, natural: 1 }, {})).toEqual([{ name: 'pass', value: 1 }])
+    expect(deriveScores([correct, natural, pass], { correct: 1, natural: 0.5 }, {})).toEqual([{ name: 'pass', value: 0 }])
+  })
+
+  /**
+   * @example
+   * all [correct, { natural, atLeast: 0.5 }] -> a mixed reading passes.
+   */
+  it('passes an all input that reaches its own threshold', () => {
+    const pass: ScoreDefinedData = { combine: { inputs: [{ name: 'correct' }, { atLeast: 0.5, name: 'natural' }], kind: 'all' }, name: 'pass' }
+
+    expect(deriveScores([pass], { correct: 1, natural: 0.5 }, {})).toEqual([{ name: 'pass', value: 1 }])
+    expect(deriveScores([pass], { correct: 1, natural: 0 }, {})).toEqual([{ name: 'pass', value: 0 }])
+  })
+
+  /**
+   * @example
+   * mean [0.5, 1] -> 0.75; weighted [0 x1, 1 x3] -> 0.75.
+   */
+  it('derives mean and weighted means', () => {
+    expect(deriveScores([{ combine: { inputs: [{ name: 'a' }, { name: 'b' }], kind: 'mean' }, name: 'm' }], { a: 0.5, b: 1 }, {})).toEqual([{ name: 'm', value: 0.75 }])
+    expect(deriveScores([{ combine: { inputs: [{ name: 'a', weight: 1 }, { name: 'b', weight: 3 }], kind: 'weighted' }, name: 'w' }], { a: 0, b: 1 }, {})).toEqual([{ name: 'w', value: 0.75 }])
+  })
+
+  /**
+   * @example
+   * a missing input -> no derived value; an unscored input -> unscored 'input_unscored'.
+   */
+  it('leaves a derived score out when an input is missing, and marks it unscored when an input is unscored', () => {
+    const pass: ScoreDefinedData = { combine: { inputs: [{ name: 'correct' }, { name: 'natural' }], kind: 'all' }, name: 'pass' }
+
+    expect(deriveScores([pass], { correct: 1 }, {})).toEqual([])
+    expect(deriveScores([pass], { correct: 1 }, { natural: 'grader_failed' })).toEqual([{ name: 'pass', unscored: 'input_unscored' }])
+  })
+
+  /**
+   * @example
+   * a derived score may use a derived score declared before it.
+   */
+  it('chains derived scores in declaration order', () => {
+    const both: ScoreDefinedData = { combine: { inputs: [{ name: 'a' }, { name: 'b' }], kind: 'all' }, name: 'both' }
+    const final: ScoreDefinedData = { combine: { inputs: [{ name: 'both' }, { name: 'c' }], kind: 'mean' }, name: 'final' }
+
+    expect(deriveScores([both, final], { a: 1, b: 1, c: 0 }, {})).toEqual([{ name: 'both', value: 1 }, { name: 'final', value: 0.5 }])
+  })
+})
+
+describe('poolScoreSummaries', () => {
+  /**
+   * @example
+   * runs of 1/1 and 0/3 pool to 1/4 = 0.25, not to the mean of means 0.5.
+   */
+  it('pools cases instead of averaging means', () => {
+    const pooled = poolScoreSummaries([
+      { mean: 1, n: 1, name: 'pass', sum: 1, total: 1, unscored: {} },
+      { mean: 0, n: 3, name: 'pass', sum: 0, total: 4, unscored: { grader_failed: 1 } },
     ])
 
-    expect(scores.exact).toBe(0.5)
-    expect(scores.judge).toBe(0.5)
-    expect(scores.hybrid).toBe(0.5)
+    expect(pooled).toEqual([{ mean: 0.25, n: 4, name: 'pass', sum: 1, total: 5, unscored: { grader_failed: 1 } }])
   })
 })
 
-describe('strictExactOf', () => {
+describe('resolveHeadline', () => {
+  const summary = (name: string, mean: number, n = 1) => ({ mean, n, name, sum: mean * n, total: n, unscored: {} })
+  const task = (taskId: string, primary: string | undefined, scores: ReturnType<typeof summary>[]): HeadlineInput => ({
+    completion: summary('completion', 1),
+    passed: summary('passed', 1),
+    ...(primary == null ? {} : { primary }),
+    scores,
+    taskId,
+  })
+
+  /**
+   * @example
+   * two tasks with primary `pass` -> one pooled `score` headline.
+   */
+  it('pools a primary score shared by every task', () => {
+    const result = resolveHeadline([task('a', 'pass', [summary('pass', 1)]), task('b', 'pass', [summary('pass', 0)])])
+
+    expect(result.headline).toEqual({ kind: 'score', score: { mean: 0.5, n: 2, name: 'pass', sum: 1, total: 2, unscored: {} } })
+    expect(result.completion).toMatchObject({ mean: 1, n: 2 })
+  })
+
+  /**
+   * @example
+   * primary `pass` and primary `accuracy` -> `per-task`, one entry per task.
+   */
+  it('lists each task when the primary names differ', () => {
+    const result = resolveHeadline([task('a', 'pass', [summary('pass', 1)]), task('b', 'accuracy', [summary('accuracy', 0.5)])])
+
+    expect(result.headline).toEqual({
+      kind: 'per-task',
+      tasks: [{ score: summary('pass', 1), taskId: 'a' }, { score: summary('accuracy', 0.5), taskId: 'b' }],
+    })
+  })
+
+  /**
+   * @example
+   * no task has a primary score -> `passed-cases`.
+   */
+  it('falls back to the share of passed cases', () => {
+    const result = resolveHeadline([task('a', undefined, [])])
+
+    expect(result.headline).toEqual({ kind: 'passed-cases', score: summary('passed', 1) })
+  })
+
   /**
    * @example
-   * strictExactOf(0.8, { measured: 3, total: 4 }) -> 0.6
+   * no tasks -> passed-cases over no cases, completion over no cases.
    */
-  it('scales the measured mean by the measured share', () => {
-    expect(strictExactOf(0.8, { measured: 3, total: 4 })).toBeCloseTo(0.6)
-    expect(strictExactOf(null, { measured: 0, total: 2 })).toBe(0)
-    expect(strictExactOf(null, { measured: 0, total: 0 })).toBeNull()
+  it('resolves an empty group', () => {
+    expect(resolveHeadline([])).toEqual({ completion: emptyScoreSummary('completion'), headline: { kind: 'passed-cases', score: emptyScoreSummary('passed') } })
   })
 })
 
@@ -80,7 +273,7 @@ describe('isMeasuredState', () => {
    * @example
    * passed and failed produced a result; errored, timeout, and skipped did not.
    */
-  it('counts only passed and failed as measured', () => {
+  it('tells results from no results', () => {
     expect(isMeasuredState('passed')).toBe(true)
     expect(isMeasuredState('failed')).toBe(true)
     expect(isMeasuredState('errored')).toBe(false)
diff --git a/packages/vieval/src/core/scoring/index.ts b/packages/vieval/src/core/scoring/index.ts
index 295e58f..f7287e9 100644
--- a/packages/vieval/src/core/scoring/index.ts
+++ b/packages/vieval/src/core/scoring/index.ts
@@ -1,168 +1,397 @@
+import type { ScoreDefinedData } from '../../events'
+
 /**
- * How many settled cases produced a result.
+ * Terminal state of one case try, as `case.ended` reports it.
+ *
+ * - `passed`: the case ran to the end without a failed assertion
+ * - `failed`: an assertion failed, so the case measured a wrong result
+ * - `errored`: the case threw something other than an assertion (a provider error, a bug, an interrupt), so it
+ *   measured nothing
+ * - `timeout`: the case ran out of time, so it measured nothing
+ * - `skipped`: the case did not run
  */
-export interface CaseCoverage {
-  /**
-   * Cases that ended `passed` or `failed`: they produced a result, right or wrong.
-   */
-  measured: number
-  /**
-   * Cases that settled: `passed`, `failed`, `errored`, or `timeout`. Skipped and running cases are not counted.
-   */
-  total: number
-}
+export type CaseOutcomeState = 'errored' | 'failed' | 'passed' | 'skipped' | 'timeout'
 
 /**
- * Scores of a group of case outcomes, for example one task.
+ * Scores of a group of case outcomes, for example one task run.
  */
-export interface CaseOutcomeScores {
+export interface CaseScoring {
   /**
-   * Measured and settled case counts. `strictExact` equals `exact * measured / total`.
+   * Share of settled cases whose final outcome produced a result (`passed` or `failed`).
    */
-  coverage: CaseCoverage
+  completion: ScoreSummary
   /**
-   * Mean exact score of the measured cases, or `null` when no case produced a result.
+   * Share of settled cases whose final outcome passed.
    */
-  exact: null | number
+  passed: ScoreSummary
   /**
-   * Mean of `exact` and `judge` when both exist, otherwise the one that exists.
+   * Name of the headline score: the score declared `primary`, or the only score of the group.
    */
-  hybrid: null | number
+  primary?: string
   /**
-   * Mean judge score of the passed cases that recorded one, or `null`.
+   * One summary per score: declared scores in declaration order, then recorded-only names in name order.
    */
-  judge: null | number
+  scores: ScoreSummary[]
+}
+
+/**
+ * One value the runtime derived from other scores of a case try.
+ */
+export type DerivedScore = { name: string, unscored: string } | { name: string, value: number }
+
+/**
+ * The resolved headline of a task, project, or run.
+ *
+ * - `score`: the primary score
+ * - `passed-cases`: no score was recorded, so the share of passed cases
+ * - `per-task`: the tasks have different primary scores, so each one is listed
+ */
+export type Headline
+  = | { kind: 'passed-cases', score: ScoreSummary }
+    | { kind: 'per-task', tasks: Array<{ score: ScoreSummary, taskId: string }> }
+    | { kind: 'score', score: ScoreSummary }
+
+/**
+ * One task's scoring, as {@link resolveHeadline} reads it.
+ */
+export interface HeadlineInput extends CaseScoring {
   /**
-   * Mean exact score of all settled cases, where a case without a result scores `0`, or `null` when nothing settled.
-   * It mixes the quality of the results with the reliability of the run; read it next to `coverage`.
+   * Task the scoring belongs to.
    */
-  strictExact: null | number
+  taskId: string
 }
 
 /**
- * Terminal state of one case try, as `case.ended` reports it.
- *
- * - `passed`: the case ran to the end without a failed assertion
- * - `failed`: an assertion failed, so the case measured a wrong result
- * - `errored`: the case threw something other than an assertion (a provider error, a bug, an interrupt), so it
- *   measured nothing
- * - `timeout`: the case ran out of time, so it measured nothing
- * - `skipped`: the case did not run
+ * Headline and completion rate of a task, project, or run, written into results once so every reader shows the same.
  */
-export type CaseOutcomeState = 'errored' | 'failed' | 'passed' | 'skipped' | 'timeout'
+export interface HeadlineResult {
+  /**
+   * Share of settled cases that produced a result, shown next to the headline.
+   */
+  completion: ScoreSummary
+  /**
+   * The headline value.
+   */
+  headline: Headline
+}
 
 /**
- * One settled case outcome with the scores that it recorded.
+ * One settled case outcome with the score values it recorded.
  */
 export interface ScoredCaseOutcome {
   /**
-   * Scores recorded by `recordScore`, keyed by kind. Only a `passed` outcome reads them.
+   * Case the outcome belongs to. Outcomes of one case (one per auto attempt) share it, and only the last one counts
+   * toward `completion` and `passed`. An outcome without it counts on its own.
+   */
+  caseId?: string
+  /**
+   * Score values of the outcome by name, recorded by the case or derived when its try settled.
    */
-  scores: { exact?: number, judge?: number }
+  scores: Readonly<Record<string, number>>
   /**
    * Terminal state of the outcome.
    */
   state: CaseOutcomeState
+  /**
+   * Scores the grader could not produce, by name, with the reason.
+   */
+  unscored: Readonly<Record<string, string>>
 }
 
 /**
- * Combines an exact and a judge mean into one hybrid score.
+ * Aggregate of one named value over a group of case outcomes, for example one score of one task.
  *
- * Returns:
- * - their mean when both exist, otherwise the one that exists, otherwise `null`
+ * The same shape carries a declared score, the completion rate (`completion`), and the share of passed cases
+ * (`passed`), so summaries of tasks, runs, and executors pool the same way.
  */
-export function hybridAverage(exact: null | number, judge: null | number): null | number {
-  if (exact != null && judge != null) {
-    return (exact + judge) / 2
-  }
-
-  return exact ?? judge
+export interface ScoreSummary {
+  /**
+   * Mean over the counted cases (`sum / n`), or `null` when no case counted.
+   */
+  mean: null | number
+  /**
+   * Cases that counted toward `mean`.
+   */
+  n: number
+  /**
+   * Score name; `completion` and `passed` for the built-in summaries.
+   */
+  name: string
+  /**
+   * Sum of the counted values.
+   */
+  sum: number
+  /**
+   * Settled cases that were not skipped. `total - n` cases did not count: they were unscored, missing, or left out
+   * by `errors: 'exclude'`.
+   */
+  total: number
+  /**
+   * Cases that did not count, per reason. For a score: the unscored reasons (`grader_failed`, ...). For
+   * `completion`: `errored` and `timeout`, the states that produced no result.
+   */
+  unscored: Record<string, number>
 }
 
+/** Reason of a derived score whose input was unscored. */
+const INPUT_UNSCORED = 'input_unscored'
+
 /**
- * Returns whether a case outcome produced a result.
+ * Counts the values of one case outcome with the scoring rule of vieval.
  *
  * Use when:
- * - counting coverage or deciding whether an outcome enters the main score
+ * - one case's values must agree with the summaries of its task, for example a line of `cases.jsonl`, the spread
+ *   under a matrix cell, or a point of a metric panel
+ *
+ * Expects:
+ * - `definitions` to list every score of the case's task, so a failed case counts 0 even in a score only other cases
+ *   recorded; a score without `score.defined` is a bare `{ name }`
  *
  * Returns:
- * - `true` for `passed` and `failed`; `false` for `errored`, `timeout`, and `skipped`
+ * - by score name: the recorded value of a `passed` outcome, 0 for a `failed` outcome, 0 for an `errored` or
+ *   `timeout` outcome in a score with `errors: 'zero'`; a score the outcome does not count in (unscored, missing,
+ *   left out, skipped) is absent
  */
-export function isMeasuredState(state: CaseOutcomeState): boolean {
-  return state === 'passed' || state === 'failed'
+export function countCase(outcome: ScoredCaseOutcome, definitions: readonly Pick<ScoreDefinedData, 'errors' | 'name'>[]): Record<string, number> {
+  if (outcome.state === 'passed') {
+    return Object.fromEntries(definitions.flatMap(({ name }) => outcome.unscored[name] == null && outcome.scores[name] != null ? [[name, outcome.scores[name]]] : []))
+  }
+  if (outcome.state === 'skipped') {
+    return {}
+  }
+  return Object.fromEntries(definitions.flatMap(({ errors, name }) => outcome.state === 'failed' || errors === 'zero' ? [[name, 0]] : []))
 }
 
 /**
- * Scores a group of case outcomes with the scoring rule of vieval.
+ * Computes the derived scores of one case try from the values it recorded.
  *
  * Use when:
- * - a task, the CLI, or the viewer turns case outcomes into task scores; all of them use this one rule
+ * - a case try settles and its derived scores (`score.defined` with `combine`) must be emitted
  *
  * Expects:
- * - only settled outcomes; running outcomes are left out by the caller
- * - every outcome of every attempt, so repeated attempts each add evidence
+ * - `definitions` in declaration order; a derived score may use derived scores declared before it
  *
  * Returns:
- * - `exact` over measured outcomes: a passed outcome scores its recorded exact score, or `1` without one; a failed
- *   outcome scores `0`; errored, timed-out, and skipped outcomes do not enter it
- * - `strictExact` over all settled outcomes, where errored and timed-out outcomes score `0`
- * - `coverage` with the measured and settled counts
+ * - one entry per derived score whose inputs all exist: a value, or `unscored: 'input_unscored'` when an input was
+ *   unscored. A derived score with a missing input is left out.
+ *   - `all`: 1 when every input reaches its `atLeast` (default 1), else 0
+ *   - `mean`: the mean of the inputs
+ *   - `weighted`: the mean weighted by `weight` (default 1)
  */
-export function scoreCaseOutcomes(outcomes: readonly ScoredCaseOutcome[]): CaseOutcomeScores {
-  const exactScores: number[] = []
-  const judgeScores: number[] = []
-  let total = 0
-
-  for (const outcome of outcomes) {
-    if (outcome.state === 'skipped') {
+export function deriveScores(
+  definitions: readonly ScoreDefinedData[],
+  values: Readonly<Record<string, number>>,
+  unscored: Readonly<Record<string, string>>,
+): DerivedScore[] {
+  const knownValues: Record<string, number> = { ...values }
+  const knownUnscored: Record<string, string> = { ...unscored }
+  const derived: DerivedScore[] = []
+  for (const definition of definitions) {
+    const combine = definition.combine
+    if (combine == null) {
       continue
     }
-
-    total += 1
-    if (outcome.state === 'failed') {
-      exactScores.push(0)
+    const inputs = combine.inputs
+    if (inputs.some(input => knownUnscored[input.name] != null)) {
+      knownUnscored[definition.name] = INPUT_UNSCORED
+      derived.push({ name: definition.name, unscored: INPUT_UNSCORED })
       continue
     }
-    if (outcome.state !== 'passed') {
+    if (inputs.length === 0 || inputs.some(input => knownValues[input.name] == null)) {
       continue
     }
 
-    exactScores.push(outcome.scores.exact ?? 1)
-    if (outcome.scores.judge != null) {
-      judgeScores.push(outcome.scores.judge)
+    let value: number
+    if (combine.kind === 'all') {
+      value = inputs.every(input => knownValues[input.name]! >= (input.atLeast ?? 1)) ? 1 : 0
+    }
+    else if (combine.kind === 'mean') {
+      value = inputs.reduce((sum, input) => sum + knownValues[input.name]!, 0) / inputs.length
+    }
+    else {
+      const totalWeight = inputs.reduce((sum, input) => sum + (input.weight ?? 1), 0)
+      value = totalWeight === 0 ? 0 : inputs.reduce((sum, input) => sum + knownValues[input.name]! * (input.weight ?? 1), 0) / totalWeight
     }
+    knownValues[definition.name] = value
+    derived.push({ name: definition.name, value })
   }
+  return derived
+}
 
-  const exact = average(exactScores)
-  const judge = average(judgeScores)
-  const coverage = { measured: exactScores.length, total }
+/**
+ * Returns a summary of `name` over no cases.
+ *
+ * Use when:
+ * - a group has no case outcomes, for example a custom task executor that reports scores but no completion
+ */
+export function emptyScoreSummary(name: string): ScoreSummary {
+  return emptySummary(name, 0)
+}
 
+/**
+ * Returns whether a case outcome produced a result.
+ *
+ * Use when:
+ * - counting completion or deciding whether an outcome is a result
+ *
+ * Returns:
+ * - `true` for `passed` and `failed`; `false` for `errored`, `timeout`, and `skipped`
+ */
+export function isMeasuredState(state: CaseOutcomeState): boolean {
+  return state === 'passed' || state === 'failed'
+}
+
+/**
+ * Pools summaries of the same name, for example one score over several task runs.
+ *
+ * Use when:
+ * - combining task runs into a run, executor, or overall summary; cases are pooled, not task means averaged
+ *
+ * Returns:
+ * - one summary per name, in first-seen order, with `sum`, `n`, `total`, and unscored counts added up
+ */
+export function poolScoreSummaries(summaries: readonly ScoreSummary[]): ScoreSummary[] {
+  const pooled = new Map<string, ScoreSummary>()
+  for (const summary of summaries) {
+    const target = pooled.get(summary.name) ?? emptySummary(summary.name, 0)
+    target.n += summary.n
+    target.sum += summary.sum
+    target.total += summary.total
+    for (const [reason, count] of Object.entries(summary.unscored)) {
+      target.unscored[reason] = (target.unscored[reason] ?? 0) + count
+    }
+    pooled.set(summary.name, target)
+  }
+  return [...pooled.values()].map(finalize)
+}
+
+/**
+ * Resolves the headline of a group of task scorings.
+ *
+ * Use when:
+ * - a task, project, or run result is written, or a viewer shows a live run; every reader uses this one rule
+ *
+ * Expects:
+ * - every task run of the group; several runs of one task share a `taskId`
+ *
+ * Returns:
+ * - `score` when every task that has scores shares one primary score name (its pooled summary)
+ * - `per-task` when the primary names differ, with one pooled summary per task
+ * - `passed-cases` when no task has a primary score
+ * - the pooled completion rate in every case
+ */
+export function resolveHeadline(tasks: readonly HeadlineInput[]): HeadlineResult {
+  const completion = poolScoreSummaries(tasks.map(task => task.completion))[0] ?? emptySummary('completion', 0)
+  const primaries = tasks.flatMap((task) => {
+    const score = task.primary == null ? undefined : task.scores.find(summary => summary.name === task.primary)
+    return score == null ? [] : [{ score, taskId: task.taskId }]
+  })
+
+  if (primaries.length === 0) {
+    const passed = poolScoreSummaries(tasks.map(task => task.passed))[0] ?? emptySummary('passed', 0)
+    return { completion, headline: { kind: 'passed-cases', score: passed } }
+  }
+  if (new Set(primaries.map(({ score }) => score.name)).size === 1) {
+    return { completion, headline: { kind: 'score', score: poolScoreSummaries(primaries.map(({ score }) => score))[0]! } }
+  }
+
+  const byTask = new Map<string, ScoreSummary[]>()
+  for (const { score, taskId } of primaries) {
+    byTask.set(taskId, [...(byTask.get(taskId) ?? []), score])
+  }
   return {
-    coverage,
-    exact,
-    hybrid: hybridAverage(exact, judge),
-    judge,
-    strictExact: strictExactOf(exact, coverage),
+    completion,
+    headline: {
+      kind: 'per-task',
+      tasks: [...byTask].map(([taskId, scores]) => ({ score: poolScoreSummaries(scores)[0]!, taskId })),
+    },
   }
 }
 
 /**
- * Scores a group with every settled case in the denominator.
+ * Scores a group of case outcomes with the scoring rule of vieval.
  *
  * Use when:
- * - only the measured mean and the coverage of a group are known, for example when combining task results
+ * - a task, the CLI, or the viewer turns case outcomes into score summaries; all of them use this one rule
+ *
+ * Expects:
+ * - only settled outcomes; running outcomes are left out by the caller
+ * - every outcome of every attempt, so repeated attempts each add evidence
+ * - `definitions` to be the scores the group announced; names recorded without a definition count with defaults
  *
  * Returns:
- * - `exact * measured / total`; `0` when cases settled but none produced a result; `null` when nothing settled
+ * - per score, over every settled outcome that was not skipped:
+ *   - `passed`: its recorded value counts; an unscored value is counted per reason; a missing value does not count
+ *   - `failed`: 0, because an assertion failed and the case measured a wrong result
+ *   - `errored` and `timeout`: 0 with `errors: 'zero'`, otherwise not counted
+ * - `completion` (passed or failed over settled) and `passed` (passed over settled), over the final outcome of each
+ *   case: an auto attempt adds evidence to the scores, but a case is one case in the completion rate, the same count
+ *   as the case states a reader sees
+ * - `primary`: the score declared primary, or the only score of the group
  */
-export function strictExactOf(exact: null | number, coverage: CaseCoverage): null | number {
-  if (coverage.total === 0) {
-    return null
+export function scoreCases(outcomes: readonly ScoredCaseOutcome[], definitions: readonly ScoreDefinedData[] = []): CaseScoring {
+  const declared = new Set(definitions.map(definition => definition.name))
+  const recordedNames = new Set<string>()
+  for (const outcome of outcomes) {
+    Object.keys(outcome.scores).forEach(name => recordedNames.add(name))
+    Object.keys(outcome.unscored).forEach(name => recordedNames.add(name))
+  }
+  // Every score of the group: a name recorded without a definition counts with the defaults.
+  const scoreDefinitions = [
+    ...definitions,
+    ...[...recordedNames].filter(name => !declared.has(name)).sort().map(name => ({ name })),
+  ]
+
+  const settled = outcomes.filter(outcome => outcome.state !== 'skipped')
+  const summaries = new Map(scoreDefinitions.map(definition => [definition.name, emptySummary(definition.name, settled.length)]))
+  for (const outcome of settled) {
+    const counted = countCase(outcome, scoreDefinitions)
+    for (const [name, summary] of summaries) {
+      const value = counted[name]
+      const reason = outcome.state === 'passed' ? outcome.unscored[name] : undefined
+      if (value != null) {
+        addValue(summary, value)
+      }
+      else if (reason != null) {
+        summary.unscored[reason] = (summary.unscored[reason] ?? 0) + 1
+      }
+    }
+  }
+  const scores = [...summaries.values()].map(finalize)
+
+  // The last outcome of each case is its final state; outcomes arrive in settle order.
+  const finals = new Map<object | string, ScoredCaseOutcome>()
+  for (const outcome of settled) {
+    finals.set(outcome.caseId ?? outcome, outcome)
   }
+  const completion = emptySummary('completion', finals.size)
+  const passed = emptySummary('passed', finals.size)
+  for (const outcome of finals.values()) {
+    addValue(completion, isMeasuredState(outcome.state) ? 1 : 0)
+    addValue(passed, outcome.state === 'passed' ? 1 : 0)
+    if (!isMeasuredState(outcome.state)) {
+      completion.unscored[outcome.state] = (completion.unscored[outcome.state] ?? 0) + 1
+    }
+  }
+
+  const primary = definitions.find(definition => definition.primary === true)?.name ?? (scores.length === 1 ? scores[0]!.name : undefined)
+  return {
+    completion: finalize(completion),
+    passed: finalize(passed),
+    ...(primary == null ? {} : { primary }),
+    scores,
+  }
+}
+
+function addValue(summary: ScoreSummary, value: number): void {
+  summary.n += 1
+  summary.sum += value
+}
 
-  return (exact ?? 0) * coverage.measured / coverage.total
+function emptySummary(name: string, total: number): ScoreSummary {
+  return { mean: null, n: 0, name, sum: 0, total, unscored: {} }
 }
 
-function average(values: readonly number[]): null | number {
-  return values.length === 0 ? null : values.reduce((sum, value) => sum + value, 0) / values.length
+function finalize(summary: ScoreSummary): ScoreSummary {
+  return { ...summary, mean: summary.n === 0 ? null : summary.sum / summary.n }
 }
diff --git a/packages/vieval/src/dsl/announcements.ts b/packages/vieval/src/dsl/announcements.ts
new file mode 100644
index 0000000..695aaa7
--- /dev/null
+++ b/packages/vieval/src/dsl/announcements.ts
@@ -0,0 +1,43 @@
+/**
+ * Tracks which definitions (`metric.defined`, `score.defined`) one task already announced.
+ *
+ * @param TDefined - Payload of the definition event; `name` identifies the definition.
+ */
+export interface Announcements<TDefined extends { name: string }> {
+  /**
+   * Records one definition.
+   *
+   * Returns:
+   * - `true` when the definition is new for the task or differs from its last announcement, so it must be emitted
+   */
+  announce: (defined: TDefined) => boolean
+  /**
+   * Latest announced definition of each name, in first-announced order.
+   */
+  latest: () => TDefined[]
+}
+
+/**
+ * Creates the announcement tracker of one task.
+ *
+ * Use when:
+ * - a task starts running its cases; every case of the task shares the tracker, so a definition is announced once,
+ *   and again only when it changes
+ *
+ * Returns:
+ * - a tracker comparing definitions by content, independent of field order
+ */
+export function createAnnouncements<TDefined extends { name: string }>(): Announcements<TDefined> {
+  const announced = new Map<string, { defined: TDefined, key: string }>()
+  return {
+    announce(defined) {
+      // Field order differs between a definition and inline metadata, so the comparison key sorts the fields.
+      const key = JSON.stringify(Object.entries(defined).sort(([a], [b]) => a.localeCompare(b)))
+      if (announced.get(defined.name)?.key === key)
+        return false
+      announced.set(defined.name, { defined, key })
+      return true
+    },
+    latest: () => [...announced.values()].map(({ defined }) => defined),
+  }
+}
diff --git a/packages/vieval/src/dsl/metrics.ts b/packages/vieval/src/dsl/metrics.ts
index 30e9c76..5855522 100644
--- a/packages/vieval/src/dsl/metrics.ts
+++ b/packages/vieval/src/dsl/metrics.ts
@@ -1,21 +1,5 @@
 import type { MetricDefinedData } from '../events'
-
-/**
- * Tracks which metric metadata a task already announced.
- *
- * Use when:
- * - `recordMetric` must decide whether to emit `metric.defined` before a value
- */
-export interface MetricAnnouncements {
-  /**
-   * Resolves the name and metadata of one `recordMetric` call.
-   *
-   * Returns:
-   * - `name` of the metric, and `defined` when its metadata is new or changed for this task (to emit before the
-   *   value); a definition's fields are overridden by inline `metadata` field by field
-   */
-  resolve: (metric: MetricDefinition | string, metadata: MetricMetadata | undefined) => { defined?: MetricDefinedData, name: string }
-}
+import type { Announcements } from './announcements'
 
 /**
  * One metric declared once with {@link defineMetric}; pass it to `context.recordMetric` instead of its name.
@@ -33,33 +17,6 @@ export interface MetricDefinition extends MetricMetadata {
  */
 export type MetricMetadata = Omit<MetricDefinedData, 'name'>
 
-/**
- * Creates the announcement tracker of one task.
- *
- * Use when:
- * - a task starts running its cases; every case of the task shares the tracker, so metadata is announced once
- *
- * Returns:
- * - a tracker that announces metadata on first use and again when it differs from the last announcement
- */
-export function createMetricAnnouncements(): MetricAnnouncements {
-  const announced = new Map<string, string>()
-  return {
-    resolve(metric, metadata) {
-      const name = typeof metric === 'string' ? metric : metric.name
-      const defined: MetricDefinedData = { ...(typeof metric === 'string' ? {} : metric), ...metadata, name }
-      if (Object.keys(defined).length === 1)
-        return { name }
-      // Field order differs between a definition and inline metadata, so the comparison key sorts the fields.
-      const key = JSON.stringify(Object.entries(defined).sort(([a], [b]) => a.localeCompare(b)))
-      if (announced.get(name) === key)
-        return { name }
-      announced.set(name, key)
-      return { defined, name }
-    },
-  }
-}
-
 /**
  * Declares one metric with the text and unit viewers show for it.
  *
@@ -86,3 +43,25 @@ export function createMetricAnnouncements(): MetricAnnouncements {
 export function defineMetric(name: string, metadata: MetricMetadata = {}): MetricDefinition {
   return Object.freeze({ ...metadata, name })
 }
+
+/**
+ * Resolves the name and the `metric.defined` payload of one `recordMetric` call.
+ *
+ * Use when:
+ * - `recordMetric` must decide whether to emit `metric.defined` before a value
+ *
+ * Returns:
+ * - `name` of the metric, and `defined` when its metadata is new or changed for the task (to emit before the value);
+ *   a definition's fields are overridden by inline `metadata` field by field
+ */
+export function resolveMetricAnnouncement(
+  announcements: Announcements<MetricDefinedData>,
+  metric: MetricDefinition | string,
+  metadata: MetricMetadata | undefined,
+): { defined?: MetricDefinedData, name: string } {
+  const name = typeof metric === 'string' ? metric : metric.name
+  const defined: MetricDefinedData = { ...(typeof metric === 'string' ? {} : metric), ...metadata, name }
+  if (Object.keys(defined).length === 1 || !announcements.announce(defined))
+    return { name }
+  return { defined, name }
+}
diff --git a/packages/vieval/src/dsl/scores.ts b/packages/vieval/src/dsl/scores.ts
new file mode 100644
index 0000000..a979063
--- /dev/null
+++ b/packages/vieval/src/dsl/scores.ts
@@ -0,0 +1,217 @@
+import type { Grader, LocalizedText, ScoreDefinedData } from '../events'
+import type { Announcements } from './announcements'
+
+/**
+ * How a derived score combines other scores of the same case try. The runtime computes it when the try settles.
+ */
+export type ScoreCombination
+  = | {
+    /**
+     * 1 when every input reaches its threshold, else 0. A bare input passes at 1; `{ score, atLeast }` sets a lower
+     * threshold, for example `{ score: natural, atLeast: 0.5 }`.
+     */
+    all: ReadonlyArray<ScoreRef | { atLeast: number, score: ScoreRef }>
+  }
+  | {
+    /** The mean of the inputs. */
+    mean: readonly ScoreRef[]
+  }
+  | {
+    /** The mean weighted by `weight`; weights are normalized. */
+    weighted: ReadonlyArray<{ score: ScoreRef, weight: number }>
+  }
+
+/**
+ * One score declared once with {@link defineScore}; pass it to `context.recordScore` instead of its name.
+ */
+export interface ScoreDefinition extends ScoreMetadata {
+  /** Score name written to `score.recorded`. */
+  readonly name: string
+}
+
+/**
+ * Label, description, and scoring rules of one score, as shown by viewers.
+ */
+export interface ScoreMetadata {
+  /**
+   * Derives this score from other scores of the same case try. A case cannot record a derived score.
+   */
+  combine?: ScoreCombination
+  /**
+   * What the score measures and how to read it. One string, or one per BCP 47 locale.
+   */
+  description?: LocalizedText
+  /**
+   * How an errored or timed-out case counts. `exclude` leaves it out, and the completion rate shows how many were
+   * left out. `zero` counts it as 0.
+   *
+   * @default 'exclude'
+   */
+  errors?: 'exclude' | 'zero'
+  /**
+   * What usually produces the value. A record can override it, for example when the judge model is a matrix axis.
+   */
+  grader?: Grader
+  /**
+   * Name shown instead of the score name. One string, or one per BCP 47 locale.
+   */
+  label?: LocalizedText
+  /**
+   * Whether this is the headline score of its task: pages, leaderboards, and comparisons show it first. At most one
+   * score of a task is primary. A task with exactly one score uses it without this flag.
+   *
+   * @default false
+   */
+  primary?: boolean
+}
+
+/**
+ * Why a recorded score has its value, and what produced it.
+ */
+export interface ScoreRecordOptions {
+  /**
+   * What produced this value. Overrides the definition's `grader`.
+   */
+  grader?: Grader
+  /**
+   * Other details of the value, shown as an object tree.
+   */
+  metadata?: Record<string, unknown>
+  /**
+   * Why the score has this value: a judge's reasoning, or the rule that matched. Shown next to the value.
+   */
+  rationale?: string
+}
+
+/**
+ * One score: a definition from {@link defineScore}, or a bare score name without metadata.
+ */
+export type ScoreRef = ScoreDefinition | string
+
+/**
+ * Announces score definitions for one task and enforces one primary score per task.
+ *
+ * Use when:
+ * - a task starts (its `scores` option) or a case records a definition for the first time
+ *
+ * Expects:
+ * - `announcements` to be the task's tracker, shared by every case of the task
+ *
+ * Returns:
+ * - the `score.defined` payloads to emit, in order: inputs of a derived score come before it
+ *
+ * Throws:
+ * - when a second score of the task is primary
+ */
+export function announceScore(announcements: Announcements<ScoreDefinedData>, score: ScoreDefinition): ScoreDefinedData[] {
+  const emitted: ScoreDefinedData[] = []
+  const visit = (definition: ScoreDefinition) => {
+    for (const input of combinationInputs(definition.combine)) {
+      if (typeof input.score !== 'string') {
+        visit(input.score)
+      }
+    }
+    const defined = toScoreDefinedData(definition)
+    const otherPrimary = defined.primary === true
+      ? announcements.latest().find(existing => existing.primary === true && existing.name !== defined.name)
+      : undefined
+    if (otherPrimary != null) {
+      throw new Error(`Score "${defined.name}" is primary, but score "${otherPrimary.name}" of the same task already is.`)
+    }
+    if (announcements.announce(defined)) {
+      emitted.push(defined)
+    }
+  }
+  visit(score)
+  return emitted
+}
+
+/**
+ * Declares one score with the text viewers show for it and the rules that compute it.
+ *
+ * Use when:
+ * - a task measures a quality of each case as a value in `0..1`, where higher is better
+ * - a headline score combines other scores, for example "correct and natural"
+ *
+ * Expects:
+ * - `name` to be the stable score identifier
+ * - `combine` thresholds in `0..1` and weights of zero or more
+ *
+ * Returns:
+ * - a frozen definition; pass it to `context.recordScore(definition, value)`, or list it in the `scores` option of
+ *   `describeTask` so a derived or never-recorded score is known before the first case
+ *
+ * @example
+ * const correct = defineScore('formula.correct', { label: { 'en': 'Correct', 'zh-Hans': '正确' } })
+ * const natural = defineScore('formula.natural', { grader: { kind: 'model', id: 'judge' } })
+ * const pass = defineScore('formula.pass', { combine: { all: [correct, natural] }, primary: true })
+ * context.recordScore(correct, 1)
+ */
+export function defineScore(name: string, metadata: ScoreMetadata = {}): ScoreDefinition {
+  const combine = metadata.combine
+  if (combine != null) {
+    const inputs = 'all' in combine ? combine.all : 'mean' in combine ? combine.mean : combine.weighted
+    if (inputs.length === 0) {
+      throw new Error(`Score "${name}" combines no scores.`)
+    }
+    if ('all' in combine) {
+      for (const input of combine.all) {
+        if (typeof input === 'object' && 'atLeast' in input && (!Number.isFinite(input.atLeast) || input.atLeast < 0 || input.atLeast > 1)) {
+          throw new Error(`Score "${name}" has an "all" threshold outside 0..1: ${String(input.atLeast)}.`)
+        }
+      }
+    }
+    if ('weighted' in combine && combine.weighted.some(input => !Number.isFinite(input.weight) || input.weight < 0)) {
+      throw new Error(`Score "${name}" has a negative or non-finite weight.`)
+    }
+  }
+  return Object.freeze({ ...metadata, name })
+}
+
+/**
+ * Returns the name of a score reference.
+ */
+export function scoreNameOf(score: ScoreRef): string {
+  return typeof score === 'string' ? score : score.name
+}
+
+function combinationInputs(combine: ScoreCombination | undefined): Array<{ atLeast?: number, score: ScoreRef, weight?: number }> {
+  if (combine == null) {
+    return []
+  }
+  if ('all' in combine) {
+    return combine.all.map(input => typeof input === 'object' && 'atLeast' in input ? input : { score: input })
+  }
+  if ('mean' in combine) {
+    return combine.mean.map(score => ({ score }))
+  }
+  return [...combine.weighted]
+}
+
+/**
+ * Normalizes a score definition into its `score.defined` payload.
+ *
+ * Before:
+ * - `{ name: 'pass', combine: { all: [correct, { score: natural, atLeast: 0.5 }] }, primary: true }`
+ *
+ * After:
+ * - `{ name: 'pass', combine: { kind: 'all', inputs: [{ name: 'correct' }, { name: 'natural', atLeast: 0.5 }] }, primary: true }`
+ */
+function toScoreDefinedData(definition: ScoreDefinition): ScoreDefinedData {
+  const { combine, ...rest } = definition
+  if (combine == null) {
+    return { ...rest }
+  }
+  const kind = 'all' in combine ? 'all' : 'mean' in combine ? 'mean' : 'weighted'
+  return {
+    ...rest,
+    combine: {
+      inputs: combinationInputs(combine).map(input => ({
+        ...(input.atLeast == null ? {} : { atLeast: input.atLeast }),
+        name: scoreNameOf(input.score),
+        ...(input.weight == null ? {} : { weight: input.weight }),
+      })),
+      kind,
+    },
+  }
+}
diff --git a/packages/vieval/src/dsl/task.test.ts b/packages/vieval/src/dsl/task.test.ts
index 4dfd998..ba58b30 100644
--- a/packages/vieval/src/dsl/task.test.ts
+++ b/packages/vieval/src/dsl/task.test.ts
@@ -12,6 +12,7 @@ import { createRunEventBus } from '../core/runner/event-bus'
 import { createScheduler } from '../core/scheduler/runtime'
 import { caseEnded, caseEndedDataSchema, caseStarted, caseStartedDataSchema, metricRecorded, scoreRecorded } from '../events'
 import { defineMetric } from './metrics'
+import { defineScore } from './scores'
 import { caseOf, casesFromInputs, describeEval, describeTask } from './task'
 
 /**
@@ -152,7 +153,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       },
     })
 
-    expect(runResult?.scores[0]?.score).toBe(1)
+    expect(runResult?.passed?.mean).toBe(1)
   })
 
   it('supports builder-form case registration and casesFromInputs', async () => {
@@ -184,7 +185,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       },
     })
 
-    expect(runResult?.scores[0]?.score).toBe(1)
+    expect(runResult?.passed?.mean).toBe(1)
   })
 
   it('emits reporter hooks around each case without changing the score average', async () => {
@@ -244,7 +245,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       { errorMessage: 'boom', errorName: 'AssertionError', index: 1, name: 'case-2', state: 'failed', total: 2 },
       { index: 0, name: 'case-1', state: 'passed', total: 2 },
     ])
-    expect(runResult?.scores[0]?.score).toBe(0.5)
+    expect(runResult?.passed?.mean).toBe(0.5)
   })
 
   it('exposes scoped task matrix context with run, eval, and row ids in case callbacks', async () => {
@@ -277,7 +278,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       },
     })
 
-    expect(runResult?.scores[0]?.score).toBe(1)
+    expect(runResult?.passed?.mean).toBe(1)
   })
 
   it('isolates nested matrix objects between case callbacks', async () => {
@@ -316,7 +317,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       },
     })
 
-    expect(runResult?.scores[0]?.score).toBe(1)
+    expect(runResult?.passed?.mean).toBe(1)
   })
 
   it('throws when caseOf is called outside describeTask', () => {
@@ -355,21 +356,28 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       },
     })
 
-    expect(runResult?.scores[0]?.score).toBe(1)
+    expect(runResult?.passed?.mean).toBe(1)
   })
 
-  it('supports custom exact/judge scores while keeping describeTask case semantics', async () => {
+  /**
+   * @example
+   * correct 0.8 / 0.2 and natural 0.6 / 0.4 -> each score averaged by name; pass (all) derived 0 for both cases.
+   */
+  it('summarizes named scores and derives a combined primary score when each case try settles', async () => {
+    const correct = defineScore('correct')
+    const natural = defineScore('natural')
+    const pass = defineScore('pass', { combine: { all: [correct, { atLeast: 0.5, score: natural }] }, primary: true })
     const taskDefinition = describeTask('dsl-custom-scores', () => {
       caseOf('case-1', (context) => {
-        context.recordScore(0.8)
-        context.recordScore(0.6, 'judge')
+        context.recordScore(correct, 1)
+        context.recordScore(natural, 0.6)
       }, { input: {} })
 
       caseOf('case-2', (context) => {
-        context.recordScore(0.2)
-        context.recordScore(0.4, 'judge')
+        context.recordScore(correct, 0.2)
+        context.recordScore(natural, 0.4)
       }, { input: {} })
-    })
+    }, { scores: [pass] })
 
     const runResult = await taskDefinition.task?.run({
       cache: createTestTaskCacheRuntime(),
@@ -392,9 +400,11 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       },
     })
 
-    expect(runResult?.scores).toEqual([
-      { kind: 'exact', score: 0.5 },
-      { kind: 'judge', score: 0.5 },
+    expect(runResult?.primary).toBe('pass')
+    expect(runResult?.scores.map(summary => [summary.name, summary.mean, summary.n])).toEqual([
+      ['correct', 0.6, 2],
+      ['natural', 0.5, 2],
+      ['pass', 0.5, 2],
     ])
   })
 
@@ -494,7 +504,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
 
     const taskDefinition = describeTask('dsl-reporter-event-isolation', () => {
       caseOf('case-1', (context) => {
-        context.recordScore(0.42)
+        context.recordScore('quality', 0.42)
         context.recordMetric('customMetric', 0.9)
       }, { input: {} })
     })
@@ -518,7 +528,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
     })
 
     expect(onEvent).toHaveBeenCalledTimes(2)
-    expect(runResult?.scores).toEqual([{ kind: 'exact', score: 0.42 }])
+    expect(runResult?.scores).toEqual([{ mean: 0.42, n: 1, name: 'quality', sum: 0.42, total: 1, unscored: {} }])
   })
 
   /**
@@ -540,7 +550,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
 
     const taskDefinition = describeTask('dsl-case-telemetry', () => {
       caseOf('case-1', (context) => {
-        context.recordScore(0.75, 'exact')
+        context.recordScore('quality', 0.75, { grader: { id: 'judge', kind: 'model', model: 'gpt-5.5' }, rationale: 'mostly right' })
         context.recordMetric('benchmark.id', 'locomo')
         context.recordMetric('benchmark.tags', ['retrieval', 'qa'])
       }, { input: {} })
@@ -567,7 +577,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       telemetry,
     })
 
-    expect(runResult?.scores).toEqual([{ kind: 'exact', score: 0.75 }])
+    expect(runResult?.scores).toEqual([{ mean: 0.75, n: 1, name: 'quality', sum: 0.75, total: 1, unscored: {} }])
     expect(spanCalls).toEqual([{
       attributes: {
         'vieval.case.id': '0:case-1',
@@ -580,8 +590,10 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
     expect(reporterEvents).toContainEqual({
       caseId: '0:case-1',
       data: {
-        kind: 'exact',
-        score: 0.75,
+        grader: { id: 'judge', kind: 'model', model: 'gpt-5.5' },
+        name: 'quality',
+        rationale: 'mostly right',
+        value: 0.75,
       },
       event: 'score.recorded',
     })
@@ -594,7 +606,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       event: 'metric.recorded',
     })
     expect(telemetry.addEvent).toHaveBeenCalledWith('vieval.case.score', {
-      'vieval.score.kind': 'exact',
+      'vieval.score.name': 'quality',
       'vieval.score.value': 0.75,
     })
     expect(telemetry.addEvent).toHaveBeenCalledWith('vieval.case.metric', {
@@ -648,9 +660,9 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
 
     expect(rejectedErrors).toHaveLength(1)
     expect(errorMessageFrom(rejectedErrors[0])).toBe('case exploded')
-    // The errored case measured nothing: no exact score, and coverage counts it as settled but unmeasured.
+    // The errored case measured nothing: no score, and the completion rate counts it as settled without a result.
     expect(runResult?.scores).toEqual([])
-    expect(runResult?.coverage).toEqual({ measured: 0, total: 1 })
+    expect(runResult?.completion).toMatchObject({ n: 1, sum: 0 })
   })
 
   it('supports caseOf without input by exposing undefined inputs in context', async () => {
@@ -681,7 +693,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       },
     })
 
-    expect(runResult?.scores[0]?.score).toBe(1)
+    expect(runResult?.passed?.mean).toBe(1)
   })
 
   it('preserves task-local concurrency metadata on the produced task definition', () => {
@@ -943,7 +955,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       },
     ])
     expect(runResult?.scores).toEqual([])
-    expect(runResult?.coverage).toEqual({ measured: 0, total: 1 })
+    expect(runResult?.completion).toMatchObject({ n: 1, sum: 0 })
   })
 
   it('retries a failing case within the same task attempt until it passes', async () => {
@@ -993,7 +1005,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       { autoRetry: 2, retryIndex: 1 },
       { autoRetry: 2, retryIndex: 2 },
     ])
-    expect(runResult?.scores[0]?.score).toBe(1)
+    expect(runResult?.passed?.mean).toBe(1)
   })
 
   /**
@@ -1068,7 +1080,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
 
       expect(runs).toBe(3)
       expect(startedAt).toEqual([0, 500, 1500])
-      expect(runResult.scores[0]?.score).toBe(1)
+      expect(runResult.passed?.mean).toBe(1)
     }
     finally {
       vi.useRealTimers()
@@ -1148,7 +1160,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
 
       expect(retryDelayIndexes).toEqual([1, 2])
       expect(startedAt).toEqual([0, 100, 300])
-      expect(runResult.scores[0]?.score).toBe(1)
+      expect(runResult.passed?.mean).toBe(1)
     }
     finally {
       vi.useRealTimers()
@@ -1190,8 +1202,8 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
     await waitForExpectation(() => {
       expect(states).toEqual({ crash: 'errored', right: 'passed', wrong: 'failed' })
     })
-    expect(runResult.scores).toEqual([{ kind: 'exact', score: 0.5 }])
-    expect(runResult.coverage).toEqual({ measured: 2, total: 3 })
+    expect(runResult.scores).toEqual([])
+    expect(runResult.completion).toMatchObject({ n: 3, sum: 2 })
   })
 
   /**
@@ -1216,8 +1228,8 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       },
     })
 
-    expect(runResult.scores).toEqual([{ kind: 'exact', score: 0 }])
-    expect(runResult.coverage).toEqual({ measured: 1, total: 1 })
+    expect(runResult.scores).toEqual([])
+    expect(runResult.completion).toMatchObject({ n: 1, sum: 1 })
   })
 
   /**
@@ -1273,8 +1285,8 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       ])
     })
     // Only the final run of `flaky` counts: three results, all passed.
-    expect(runResult.scores).toEqual([{ kind: 'exact', score: 1 }])
-    expect(runResult.coverage).toEqual({ measured: 3, total: 3 })
+    expect(runResult.scores).toEqual([])
+    expect(runResult.completion).toMatchObject({ n: 3, sum: 3 })
   })
 
   /**
@@ -1312,7 +1324,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
     await waitForExpectation(() => {
       expect(requeues).toEqual([1, 2])
     })
-    expect(runResult.coverage).toEqual({ measured: 0, total: 1 })
+    expect(runResult.completion).toMatchObject({ n: 1, sum: 0 })
   })
 
   /**
@@ -1341,7 +1353,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
     })
 
     expect(runs).toBe(1)
-    expect(runResult.scores).toEqual([{ kind: 'exact', score: 0 }])
+    expect(runResult.scores).toEqual([])
   })
 
   /**
@@ -1412,7 +1424,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
 
     expect(erroredRuns).toBe(2)
     expect(failedRuns).toBe(1)
-    expect(runResult.scores).toEqual([{ kind: 'exact', score: 0.5 }])
+    expect(runResult.scores).toEqual([])
   })
 
   /**
@@ -1455,7 +1467,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
 
     expect(delayErrors).toEqual([failure])
     expect(runs).toBe(2)
-    expect(runResult.scores[0]?.score).toBe(1)
+    expect(runResult.passed?.mean).toBe(1)
   })
 
   /**
@@ -1502,7 +1514,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       { message: 'rate limited', retryIndex: 1 },
       { message: 'wrong answer', retryIndex: 2 },
     ])
-    expect(runResult.scores[0]?.score).toBe(0)
+    expect(runResult.passed?.mean).toBe(0)
   })
 
   it('starts the next auto attempt only after the current task attempt fully settles', async () => {
@@ -1571,25 +1583,28 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
 
     const runResult = await runPromise
 
-    expect(runResult.scores[0]?.score).toBe(0.75)
+    // Both cases end passed in the second attempt; the passed share counts each case once by its final state.
+    expect(runResult.passed).toMatchObject({ mean: 1, n: 2 })
   })
 
   /**
    * it('scores auto attempts as independent evidence') verifies task-level attempt aggregation.
    *
    * @example
-   * autoAttempt: 1 with first attempt failing and second attempt passing -> exact score 0.5.
+   * autoAttempt: 1 with first attempt failing and second attempt passing -> score 0.5 over 2 outcomes, while the
+   * case counts once by its final state: passed 1 of 1.
    */
   it('scores auto attempts as independent evidence instead of replacing earlier failures', async () => {
     let runs = 0
 
     const taskDefinition = describeTask('dsl-auto-attempt-evidence', () => {
-      caseOf('flaky-case', async () => {
+      caseOf('flaky-case', async (context) => {
         runs += 1
 
         if (runs === 1) {
           throw createAssertionError('first-attempt-failed')
         }
+        context.recordScore('correct', 1)
       }, {
         autoAttempt: 1,
         input: 'flaky',
@@ -1616,7 +1631,8 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
     })
 
     expect(runs).toBe(2)
-    expect(runResult?.scores[0]?.score).toBe(0.5)
+    expect(runResult?.scores).toEqual([{ mean: 0.5, n: 2, name: 'correct', sum: 1, total: 2, unscored: {} }])
+    expect(runResult?.passed).toMatchObject({ mean: 1, n: 1 })
   })
 
   /**
@@ -1662,7 +1678,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
     })
 
     expect(runs).toBe(2)
-    expect(runResult?.scores[0]?.score).toBe(1)
+    expect(runResult?.passed?.mean).toBe(1)
   })
 
   /**
@@ -1816,7 +1832,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
     })
 
     expect(peak).toBe(1)
-    expect(runResult.scores[0]?.score).toBe(1)
+    expect(runResult.passed?.mean).toBe(1)
   })
 
   /**
@@ -2065,7 +2081,7 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
       },
     })
 
-    expect(runResult?.scores[0]?.score).toBe(1)
+    expect(runResult?.passed?.mean).toBe(1)
   })
 
   /**
@@ -2165,3 +2181,165 @@ describe('describeTask DSL', { timeout: 10000 }, () => {
     expect(endedCases.map(data => [data.state, data.errorMessage])).toEqual([['errored', 'Run interrupted by SIGTERM.']])
   })
 })
+
+describe('describeTask named scores', () => {
+  async function runTask(taskDefinition: ReturnType<typeof describeTask>, emitted: Array<{ caseId?: string, data: unknown, event: string }>) {
+    return await taskDefinition.task!.run({
+      cache: createTestTaskCacheRuntime(),
+      events: createTestRunEvents({ onAny: envelope => emitted.push(toEmitted(envelope)) }),
+      models: createTestModels(),
+      task: {
+        entry: { description: 'd', directory: 'x', filePath: '/tmp/x.eval.ts', id: 'x', name: 'x' },
+        id: 'task-named-scores',
+        inferenceExecutor: { id: 'openai:gpt-4.1-mini' },
+        matrix: createScheduledTaskMatrix(),
+      },
+    })
+  }
+
+  /**
+   * @example
+   * scores: [pass] announces correct, natural, then pass (inputs first) before any case, without a case id.
+   */
+  it('announces the scores option before the first case, inputs of a derived score first', async () => {
+    const correct = defineScore('correct', { label: { 'en': 'Correct', 'zh-Hans': '正确' } })
+    const natural = defineScore('natural', { grader: { id: 'judge', kind: 'model' } })
+    const pass = defineScore('pass', { combine: { all: [correct, natural] }, primary: true })
+    const emitted: Array<{ caseId?: string, data: unknown, event: string }> = []
+
+    await runTask(describeTask('scores-announced', () => {
+      caseOf('case-1', (context) => {
+        context.recordScore(correct, 1)
+        context.recordScore(natural, 1)
+      })
+    }, { scores: [pass] }), emitted)
+
+    const defined = emitted.filter(event => event.event === 'score.defined')
+    expect(defined.map(event => (event.data as { name: string }).name)).toEqual(['correct', 'natural', 'pass'])
+    expect(defined.every(event => event.caseId == null)).toBe(true)
+    expect(defined[2]!.data).toEqual({ combine: { inputs: [{ name: 'correct' }, { name: 'natural' }], kind: 'all' }, name: 'pass', primary: true })
+    expect(emitted.findIndex(event => event.event === 'score.defined')).toBeLessThan(emitted.findIndex(event => event.event === 'case.started'))
+  })
+
+  /**
+   * @example
+   * the derived `pass` is emitted with `derived: true` after the recorded values and before `case.ended`.
+   */
+  it('emits derived scores when the try settles, before case.ended', async () => {
+    const correct = defineScore('correct')
+    const pass = defineScore('pass', { combine: { all: [correct] }, primary: true })
+    const emitted: Array<{ caseId?: string, data: unknown, event: string }> = []
+
+    const runResult = await runTask(describeTask('scores-derived', () => {
+      caseOf('case-1', context => context.recordScore(correct, 1))
+    }, { scores: [pass] }), emitted)
+
+    const caseEvents = emitted.filter(event => event.caseId === '0:case-1').map(event => [event.event, event.data])
+    expect(caseEvents.filter(([event]) => event === 'score.recorded' || event === 'case.ended').map(([event]) => event)).toEqual(['score.recorded', 'score.recorded', 'case.ended'])
+    expect(caseEvents.find(([event, data]) => event === 'score.recorded' && (data as { derived?: boolean }).derived === true)?.[1]).toEqual({ derived: true, name: 'pass', value: 1 })
+    expect(runResult.primary).toBe('pass')
+    expect(runResult.scores.find(summary => summary.name === 'pass')?.mean).toBe(1)
+  })
+
+  /**
+   * @example
+   * recordUnscored(natural, 'grader_failed') -> not counted, reason summed; derived pass unscored 'input_unscored'.
+   */
+  it('records unscored values without counting them and marks derived scores unscored', async () => {
+    const correct = defineScore('correct')
+    const natural = defineScore('natural')
+    const pass = defineScore('pass', { combine: { all: [correct, natural] }, primary: true })
+    const emitted: Array<{ caseId?: string, data: unknown, event: string }> = []
+
+    const runResult = await runTask(describeTask('scores-unscored', () => {
+      caseOf('graded', (context) => {
+        context.recordScore(correct, 1)
+        context.recordScore(natural, 1)
+      })
+      caseOf('judge-broke', (context) => {
+        context.recordScore(correct, 1)
+        context.recordUnscored(natural, 'grader_failed', { rationale: 'the judge answer was not JSON' })
+      })
+    }, { scores: [pass] }), emitted)
+
+    expect(emitted).toContainEqual({ caseId: '1:judge-broke', data: { name: 'natural', rationale: 'the judge answer was not JSON', unscored: { reason: 'grader_failed' } }, event: 'score.recorded' })
+    expect(runResult.scores.find(summary => summary.name === 'natural')).toEqual({ mean: 1, n: 1, name: 'natural', sum: 1, total: 2, unscored: { grader_failed: 1 } })
+    expect(runResult.scores.find(summary => summary.name === 'pass')).toEqual({ mean: 1, n: 1, name: 'pass', sum: 1, total: 2, unscored: { input_unscored: 1 } })
+    expect(runResult.passed).toMatchObject({ n: 2, sum: 2 })
+  })
+
+  /**
+   * @example
+   * recordScore(pass, 1) for a derived score -> the case errors with a message naming the score.
+   */
+  it('rejects recording a derived score', async () => {
+    const correct = defineScore('correct')
+    const pass = defineScore('pass', { combine: { all: [correct] } })
+    const ended: CaseEndedData[] = []
+
+    await describeTask('scores-derived-recorded', () => {
+      caseOf('case-1', context => context.recordScore(pass, 1))
+    }).task!.run({
+      cache: createTestTaskCacheRuntime(),
+      events: createTestRunEvents({ onCaseEnd: data => ended.push(data) }),
+      models: createTestModels(),
+      task: { entry: { description: 'd', directory: 'x', filePath: '/tmp/x.eval.ts', id: 'x', name: 'x' }, id: 'x', inferenceExecutor: { id: 'openai:gpt-4.1-mini' }, matrix: createScheduledTaskMatrix() },
+    })
+
+    await waitForExpectation(() => {
+      expect(ended.map(data => [data.state, data.errorMessage])).toEqual([['errored', 'Score "pass" is derived from other scores; the runtime computes it when the case try settles.']])
+    })
+  })
+
+  /**
+   * @example
+   * two primary scores in one task -> the task run throws when the second is announced.
+   */
+  it('rejects a second primary score in one task', async () => {
+    const first = defineScore('first', { primary: true })
+    const second = defineScore('second', { primary: true })
+
+    await expect(runTask(describeTask('scores-two-primaries', () => {
+      caseOf('case-1', () => {})
+    }, { scores: [first, second] }), [])).rejects.toThrow('Score "second" is primary, but score "first" of the same task already is.')
+  })
+
+  /**
+   * @example
+   * --case selects nothing -> no scores (the old implicit exact 1 is gone).
+   */
+  it('returns no scores for an empty case selection', async () => {
+    const runResult = await describeTask('scores-empty-selection', () => {
+      caseOf('case-1', () => {})
+    }).task!.run({
+      cache: createTestTaskCacheRuntime(),
+      models: createTestModels(),
+      selectsCase: () => false,
+      task: { entry: { description: 'd', directory: 'x', filePath: '/tmp/x.eval.ts', id: 'x', name: 'x' }, id: 'x', inferenceExecutor: { id: 'openai:gpt-4.1-mini' }, matrix: createScheduledTaskMatrix() },
+    })
+
+    expect(runResult).toEqual({ scores: [] })
+  })
+
+  /**
+   * @example
+   * an errored case with errors: 'zero' counts as 0; the default leaves it out; completion shows it.
+   */
+  it('applies errors: zero per score and reports the errored case in the completion rate', async () => {
+    const strict = defineScore('strict', { errors: 'zero' })
+    const lenient = defineScore('lenient')
+
+    const runResult = await runTask(describeTask('scores-errors', () => {
+      caseOf('ok', (context) => {
+        context.recordScore(strict, 1)
+        context.recordScore(lenient, 1)
+      })
+      caseOf('provider-down', () => {
+        throw new Error('fetch failed')
+      })
+    }, { scores: [strict, lenient] }), [])
+
+    expect(runResult.scores.map(summary => [summary.name, summary.mean, summary.n, summary.total])).toEqual([['strict', 0.5, 2, 2], ['lenient', 1, 1, 2]])
+    expect(runResult.completion).toEqual({ mean: 0.5, n: 2, name: 'completion', sum: 1, total: 2, unscored: { errored: 1 } })
+  })
+})
diff --git a/packages/vieval/src/dsl/task.ts b/packages/vieval/src/dsl/task.ts
index 7dad9b3..59d861b 100644
--- a/packages/vieval/src/dsl/task.ts
+++ b/packages/vieval/src/dsl/task.ts
@@ -1,23 +1,26 @@
 import type { TaskCaseState, TaskConcurrencyConfig, TaskExecutionPolicy, TaskRunContext, TaskRunOutput } from '../config'
-import type { RunScore, RunScoreKind } from '../core/runner'
 import type { Scheduler } from '../core/scheduler/types'
 import type { ScoredCaseOutcome } from '../core/scoring'
 import type { TelemetryAttributeValue } from '../core/telemetry'
-import type { RunEventDefinition, RunEventEmitter } from '../events'
+import type { MetricDefinedData, RunEventDefinition, RunEventEmitter, ScoreDefinedData, ScoreRecordedData } from '../events'
+import type { Announcements } from './announcements'
 import type { RecordArtifactOptions } from './case-artifacts'
-import type { MetricAnnouncements, MetricDefinition, MetricMetadata } from './metrics'
+import type { MetricDefinition, MetricMetadata } from './metrics'
+import type { ScoreDefinition, ScoreRecordOptions, ScoreRef } from './scores'
 
 import { errorMessageFrom, errorNameFrom, sleep } from '@moeru/std'
 
 import { defineEval, defineTask } from '../config'
 import { runWithCaseOutput } from '../core/runner/case-output'
 import { createScheduler } from '../core/scheduler/runtime'
-import { scoreCaseOutcomes } from '../core/scoring'
+import { deriveScores, scoreCases } from '../core/scoring'
 import { createNoopTelemetryRuntime } from '../core/telemetry'
-import { artifactRecorded, caseEnded, caseOutput, caseRequeued, caseStarted, metricDefined, metricRecorded, scoreRecorded } from '../events'
+import { artifactRecorded, caseEnded, caseOutput, caseRequeued, caseStarted, metricDefined, metricRecorded, scoreDefined, scoreRecorded } from '../events'
+import { createAnnouncements } from './announcements'
 import { createCaseArtifacts } from './case-artifacts'
-import { createMetricAnnouncements } from './metrics'
+import { resolveMetricAnnouncement } from './metrics'
 import { registerEvalDefinition } from './registry'
+import { announceScore, scoreNameOf } from './scores'
 
 /**
  * Per-case registration options for `caseOf`.
@@ -69,7 +72,7 @@ export interface CaseRunContext<TInput> extends TaskRunContext {
    * Records one custom case metric into report events.
    *
    * Use when:
-   * - tasks need structured benchmark metadata beyond exact/judge score families
+   * - tasks need structured benchmark data that is not a score: latency, cost, a category, a count
    *
    * Expects:
    * - `metric` to be a stable metric name, or a definition from `defineMetric`
@@ -83,15 +86,34 @@ export interface CaseRunContext<TInput> extends TaskRunContext {
    */
   recordMetric: (metric: MetricDefinition | string, value: TelemetryAttributeValue, metadata?: MetricMetadata) => void
   /**
-   * Records a custom normalized value for one case score family, replacing an earlier value of the same kind.
+   * Records the value of one score for this case try, replacing an earlier value or unscored record of the same name.
    *
    * Use when:
-   * - one case computes a benchmark-native score that should flow into run aggregation
+   * - the case measured a quality as a value in `0..1`, where higher is better
    *
    * Expects:
-   * - `score` to stay in the `0..1` range
+   * - `score` to be a definition from `defineScore` or a score name; a definition is announced on first use
+   * - `value` to be finite and in `0..1`
+   * - `score` not to be derived (`combine`); the runtime computes derived scores when the try settles
+   *
+   * Returns:
+   * - nothing; emits `score.defined` first when the definition is new for the task, then `score.recorded`
+   */
+  recordScore: (score: ScoreRef, value: number, options?: ScoreRecordOptions) => void
+  /**
+   * Records that the grader could not produce a value for one score in this case try.
+   *
+   * Use when:
+   * - the case produced a result but its grader failed, for example an unparseable judge answer; the value says
+   *   nothing about the test object, so it is not 0 and never counts
+   *
+   * Expects:
+   * - `reason` to be a short code; `grader_failed`, `invalid_response`, `no_response`, and `refusal` have labels in viewers
+   *
+   * Returns:
+   * - nothing; emits `score.recorded` with `unscored`
    */
-  recordScore: (score: number, kind?: RunScoreKind) => void
+  recordUnscored: (score: ScoreRef, reason: string, options?: ScoreRecordOptions) => void
   /**
    * Cooperative abort signal for the current case execution.
    */
@@ -179,6 +201,17 @@ export interface DescribeTaskOptions extends TaskExecutionPolicy {
    * Optional description override.
    */
   description?: string
+  /**
+   * Scores of this task, announced when the task starts.
+   *
+   * Use when:
+   * - a score is derived (`combine`), so no case records it
+   * - a score should count as missing for cases that never record it, or should be the task's primary score before
+   *   the first case records anything
+   *
+   * @default []
+   */
+  scores?: readonly ScoreDefinition[]
 }
 
 /**
@@ -194,8 +227,11 @@ interface CaseExecutionOutcome {
   error?: unknown
   errorMessage?: string
   output?: unknown
-  scoresByKind: Map<RunScoreKind, number>
+  /** Score values of the try by name, recorded or derived. */
+  scores: Record<string, number>
   state: TaskCaseState
+  /** Unscored records of the try by name, with the reason. */
+  unscored: Record<string, string>
 }
 
 interface RegisteredCase<TInput> {
@@ -207,6 +243,12 @@ interface RegisteredCase<TInput> {
   run: CaseRunner<TInput>
 }
 
+/** Metric and score definitions one task run announced. */
+interface TaskAnnouncements {
+  metrics: Announcements<MetricDefinedData>
+  scores: Announcements<ScoreDefinedData>
+}
+
 function assertNonNegativeInteger(value: number, label: string): void {
   if (!Number.isFinite(value) || !Number.isInteger(value) || value < 0) {
     throw new Error(`Invalid ${label}: ${String(value)}`)
@@ -487,8 +529,9 @@ async function executeRegisteredCase(
 
   return lastOutcome ?? {
     errorMessage: 'Unknown case failure.',
-    scoresByKind: new Map(),
+    scores: {},
     state: 'failed',
+    unscored: {},
   }
 }
 
@@ -571,11 +614,8 @@ function resolveCaseExecutionPolicy(
   }
 }
 
-function toScoredCaseOutcome(outcome: CaseExecutionOutcome): ScoredCaseOutcome {
-  return {
-    scores: { exact: outcome.scoresByKind.get('exact'), judge: outcome.scoresByKind.get('judge') },
-    state: outcome.state,
-  }
+function toScoredCaseOutcome(outcome: CaseExecutionOutcome, caseId: string): ScoredCaseOutcome {
+  return { caseId, scores: outcome.scores, state: outcome.state, unscored: outcome.unscored }
 }
 
 /**
@@ -603,23 +643,35 @@ async function waitUnlessAborted(milliseconds: number, signal: AbortSignal | und
 }
 
 /**
- * Metric announcements of each running task, keyed by the task's run context.
+ * Announcements of each running task, keyed by the task's run context.
  *
  * NOTICE:
  * Every case of one task run receives the same `TaskRunContext` object, so it identifies the task run without
  * threading a tracker through the attempt, retry, and case helpers. A `WeakMap` drops the tracker with the context.
  */
-const metricAnnouncementsByTask = new WeakMap<TaskRunContext, MetricAnnouncements>()
+const announcementsByTask = new WeakMap<TaskRunContext, TaskAnnouncements>()
 
-function metricAnnouncementsOf(context: TaskRunContext): MetricAnnouncements {
-  let announcements = metricAnnouncementsByTask.get(context)
+function announcementsOf(context: TaskRunContext): TaskAnnouncements {
+  let announcements = announcementsByTask.get(context)
   if (announcements == null) {
-    announcements = createMetricAnnouncements()
-    metricAnnouncementsByTask.set(context, announcements)
+    announcements = { metrics: createAnnouncements(), scores: createAnnouncements() }
+    announcementsByTask.set(context, announcements)
   }
   return announcements
 }
 
+/**
+ * Announces a score definition for the task once; definitions belong to the task, so they carry no case id.
+ */
+function announceTaskScore(context: TaskRunContext, score: ScoreRef): void {
+  if (typeof score === 'string') {
+    return
+  }
+  for (const defined of announceScore(announcementsOf(context).scores, score)) {
+    emitSafely(context.events, scoreDefined, defined)
+  }
+}
+
 async function runCaseOnce(
   context: TaskRunContext,
   taskCase: RegisteredCase<unknown>,
@@ -628,7 +680,8 @@ async function runCaseOnce(
   autoAttempt: CaseAutoAttempt | undefined,
 ): Promise<CaseExecutionOutcome> {
   const { classifyError, timeout } = policy
-  const customScoresByKind = new Map<RunScoreKind, number>()
+  const scores: Record<string, number> = {}
+  const unscored: Record<string, string> = {}
   const abortController = new AbortController()
   const telemetry = context.telemetry ?? createNoopTelemetryRuntime()
   const caseId = createCaseId(index, taskCase.name)
@@ -643,14 +696,51 @@ async function runCaseOnce(
   const completedOutcome = (output: unknown): CaseExecutionOutcome => context.signal?.aborted === true
     ? {
         errorMessage: errorMessageFrom(context.signal.reason) ?? 'Run interrupted.',
-        scoresByKind: customScoresByKind,
+        scores,
         state: 'errored',
+        unscored,
       }
     : {
         output,
-        scoresByKind: customScoresByKind,
+        scores,
         state: 'passed',
+        unscored,
+      }
+  // Shared by `recordScore` and `recordUnscored`: both replace the other record of the same name in this try.
+  const recordScoreEvent = (score: ScoreRef, record: ScoreRecordedData): void => {
+    const derived = announcementsOf(context).scores.latest().find(defined => defined.name === record.name)?.combine
+    if (derived != null || (typeof score !== 'string' && score.combine != null)) {
+      throw new Error(`Score "${record.name}" is derived from other scores; the runtime computes it when the case try settles.`)
+    }
+    announceTaskScore(context, score)
+    if ('value' in record) {
+      scores[record.name] = record.value
+      delete unscored[record.name]
+    }
+    else {
+      unscored[record.name] = record.unscored.reason
+      delete scores[record.name]
+    }
+    telemetry.addEvent('vieval.case.score', {
+      'vieval.score.name': record.name,
+      ...('value' in record ? { 'vieval.score.value': record.value } : { 'vieval.score.unscored': record.unscored.reason }),
+    })
+    emitSafely(events, scoreRecorded, record)
+  }
+  // Emits the derived scores of this try once it settled, before `case.ended`.
+  const settleScores = (): void => {
+    for (const derived of deriveScores(announcementsOf(context).scores.latest(), scores, unscored)) {
+      if ('value' in derived) {
+        scores[derived.name] = derived.value
       }
+      else {
+        unscored[derived.name] = derived.unscored
+      }
+      emitSafely(events, scoreRecorded, 'value' in derived
+        ? { derived: true, name: derived.name, value: derived.value }
+        : { derived: true, name: derived.name, unscored: { reason: derived.unscored } })
+    }
+  }
 
   try {
     return await telemetry.withSpan('vieval.case', {
@@ -689,7 +779,7 @@ async function runCaseOnce(
             return
           }
 
-          const { defined, name } = metricAnnouncementsOf(context).resolve(metric, metadata)
+          const { defined, name } = resolveMetricAnnouncement(announcementsOf(context).metrics, metric, metadata)
           // Metadata belongs to the task, not to this case, so it is announced without the case id.
           if (defined != null)
             emitSafely(context.events, metricDefined, defined)
@@ -699,18 +789,20 @@ async function runCaseOnce(
             telemetry.setAttributes({ [name]: value })
           }
         },
-        recordScore(score, kind = 'exact') {
+        recordScore(score, value, options) {
           if (abortController.signal.aborted || settled) {
             return
           }
 
-          assertValidScore(score)
-          customScoresByKind.set(kind, score)
-          telemetry.addEvent('vieval.case.score', {
-            'vieval.score.kind': kind,
-            'vieval.score.value': score,
-          })
-          emitSafely(events, scoreRecorded, { kind, score })
+          assertValidScore(value)
+          recordScoreEvent(score, { ...options, name: scoreNameOf(score), value })
+        },
+        recordUnscored(score, reason, options) {
+          if (abortController.signal.aborted || settled) {
+            return
+          }
+
+          recordScoreEvent(score, { ...options, name: scoreNameOf(score), unscored: { reason } })
         },
         signal: abortController.signal,
       })
@@ -730,16 +822,19 @@ async function runCaseOnce(
 
         const output = await Promise.race([runPromise, timeoutPromise])
         settled = true
+        settleScores()
         return completedOutcome(output)
       }
 
       const output = await runPromise
       settled = true
+      settleScores()
       return completedOutcome(output)
     })
   }
   catch (error) {
     settled = true
+    settleScores()
     // Abort errors raised by the interrupt carry generic messages ("The operation was aborted"); the run's reason is clearer.
     const interrupted = context.signal?.aborted === true
     const interruptMessage = interrupted ? errorMessageFrom(context.signal!.reason) : undefined
@@ -754,8 +849,9 @@ async function runCaseOnce(
     return {
       error,
       errorMessage: interruptMessage ?? errorMessageFrom(error) ?? (timedOut && timeout != null ? `Case timed out after ${timeout}ms.` : 'Unknown case failure.'),
-      scoresByKind: customScoresByKind,
+      scores,
       state,
+      unscored,
     }
   }
   finally {
@@ -853,10 +949,13 @@ export function describeTask(
       async run(context): Promise<TaskRunOutput> {
         // `--case` runs a subset; each case keeps the index (and so the case id) it has in the full task.
         const runCases = registeredCases.flatMap((taskCase, index) => context.selectsCase == null || context.selectsCase(taskCase.name) ? [{ index, taskCase }] : [])
+        // An empty selection measured nothing, so it has no scores.
         if (runCases.length === 0) {
-          return {
-            scores: [{ kind: 'exact', score: 1 }],
-          }
+          return { scores: [] }
+        }
+        // Declared scores are announced before any case, so the primary score and derived scores are known up front.
+        for (const score of options.scores ?? []) {
+          announceTaskScore(context, score)
         }
 
         const totalCases = runCases.length
@@ -890,7 +989,7 @@ export function describeTask(
             runCases.map(async ({ index, taskCase }) => {
               await executeCaseWithRequeue(context, taskCase, index, totalCases, taskExecutionPolicy, undefined, async execute => await scheduleCase(taskCase, index, 0, execute), (outcome) => {
                 emitCaseOutcome(context, taskCase, outcome, index, totalCases, undefined)
-                settledOutcomes.push(toScoredCaseOutcome(outcome))
+                settledOutcomes.push(toScoredCaseOutcome(outcome, createCaseId(index, taskCase.name)))
               })
             }),
           )
@@ -924,7 +1023,7 @@ export function describeTask(
 
               const { index, taskCase } = runCase
               emitCaseOutcome(context, taskCase, outcome, index, totalCases, autoAttemptOf(taskCase))
-              settledOutcomes.push(toScoredCaseOutcome(outcome))
+              settledOutcomes.push(toScoredCaseOutcome(outcome, createCaseId(index, taskCase.name)))
             })
 
             const shouldContinue = context.signal?.aborted !== true && attemptOutcomes.some((outcome, position) => {
@@ -948,19 +1047,7 @@ export function describeTask(
           }
         }
 
-        const { coverage, exact, judge } = scoreCaseOutcomes(settledOutcomes)
-        const scores: RunScore[] = []
-        if (exact != null) {
-          scores.push({ kind: 'exact', score: exact })
-        }
-        if (judge != null) {
-          scores.push({ kind: 'judge', score: judge })
-        }
-
-        return {
-          coverage,
-          scores,
-        }
+        return scoreCases(settledOutcomes, announcementsOf(context).scores.latest())
       },
     }),
   })
diff --git a/packages/vieval/src/events/index.ts b/packages/vieval/src/events/index.ts
index 0ea559f..fc1805e 100644
--- a/packages/vieval/src/events/index.ts
+++ b/packages/vieval/src/events/index.ts
@@ -1,6 +1,5 @@
 import type { Eventa } from '@moeru/eventa'
 
-import type { RunScoreKind } from '../core/runner/aggregate'
 import type { TelemetryAttributeValue } from '../core/telemetry/types'
 
 import { defineEventa } from '@moeru/eventa'
@@ -24,10 +23,11 @@ export { defineEventa } from '@moeru/eventa'
  *   keep accepting newer logs (payload schemas ignore unknown keys, and readers skip unknown events)
  * - renaming, removing, or retyping an envelope or payload field bumps the version
  *
- * Optional fields added within version 2: `case.started` / `case.ended` `attemptIndex` and `autoAttempt`;
- * `artifact.recorded` `label` and `description`. Events added within version 2: `metric.defined`.
+ * Version 3 replaced the `exact` / `judge` score kinds with named scores: `score.recorded` carries `name` and
+ * `value` (or `unscored`) instead of `kind` and `score`, and `score.defined` declares a score. Readers reject
+ * envelopes of older versions.
  */
-export const RUN_EVENT_SCHEMA_VERSION = 2
+export const RUN_EVENT_SCHEMA_VERSION = 3
 
 /**
  * One typed run event definition.
@@ -272,12 +272,80 @@ export const metricRecordedDataSchema = v.object({
   value: metricValueSchema,
 })
 
-/** Payload of {@link scoreRecorded}. */
-export const scoreRecordedDataSchema = v.object({
-  kind: v.picklist(['exact', 'judge'] satisfies RunScoreKind[]),
-  score: v.number(),
+/** What produced a score value. */
+export const graderSchema = v.object({
+  /** Stable name, for example `listener`, `judge`, or `grader.chem`; matches the `role` of the grader's inference events. */
+  id: v.optional(v.string()),
+  /**
+   * `rule`: deterministic code. `model`: an LLM or another model. `human`: a reviewer. `external`: the verifier of
+   * another harness, for example a Harbor `tests/test.sh`.
+   */
+  kind: v.picklist(['external', 'human', 'model', 'rule']),
+  /** Model id when `kind` is `model`. */
+  model: v.optional(v.string()),
+  /** How several votes were reduced to the value, for example `majority` or `median`. */
+  reducer: v.optional(v.string()),
+  /** Version of the grader: a prompt hash, a rule set version, or a package version. */
+  version: v.optional(v.string()),
+  /** Number of votes when the value reduces several answers. */
+  votes: v.optional(v.number()),
+})
+
+/** How a derived score combines other scores of the same case try. */
+export const scoreCombinationSchema = v.object({
+  inputs: v.array(v.object({
+    /** `all` only: the least value at which this input passes. */
+    atLeast: v.optional(v.number()),
+    name: v.string(),
+    /** `weighted` only: the weight of this input; weights are normalized. */
+    weight: v.optional(v.number()),
+  })),
+  /** `all`: 1 when every input reaches its `atLeast` (default 1), else 0. `mean`: the mean. `weighted`: the weighted mean. */
+  kind: v.picklist(['all', 'mean', 'weighted']),
 })
 
+/** Payload of {@link scoreDefined}. */
+export const scoreDefinedDataSchema = v.object({
+  /** Present when the runtime derives this score from other scores; a case cannot record it. */
+  combine: v.optional(scoreCombinationSchema),
+  /** What the score measures and how to read it. */
+  description: v.optional(localizedTextSchema),
+  /** How an errored or timed-out case counts: left out (`exclude`, the default) or as 0 (`zero`). */
+  errors: v.optional(v.picklist(['exclude', 'zero'])),
+  /** What usually produces the value; a record can override it. */
+  grader: v.optional(graderSchema),
+  /** Human-readable name shown instead of `name`. */
+  label: v.optional(localizedTextSchema),
+  /** Score name, the same as in `score.recorded`. */
+  name: v.string(),
+  /** Whether this is the headline score of its task. At most one score of a task is primary. */
+  primary: v.optional(v.boolean()),
+})
+
+const scoreRecordedEntries = {
+  /** Whether the runtime derived the value from other scores (`score.defined` `combine`). */
+  derived: v.optional(v.boolean()),
+  /** What produced this value; overrides the definition's `grader`. */
+  grader: v.optional(graderSchema),
+  /** Other details of the value. */
+  metadata: v.optional(v.record(v.string(), v.unknown())),
+  name: v.string(),
+  /** Why the score has this value: a judge's reasoning, or the rule that matched. */
+  rationale: v.optional(v.string()),
+}
+
+/**
+ * Payload of {@link scoreRecorded}: a value in `0..1`, or `unscored` when the grader could not produce one.
+ */
+export const scoreRecordedDataSchema = v.union([
+  v.object({ ...scoreRecordedEntries, value: v.number() }),
+  v.object({
+    ...scoreRecordedEntries,
+    /** The grader produced no value, for a reason such as `grader_failed`, `invalid_response`, `no_response`, or `refusal`. */
+    unscored: v.object({ reason: v.string() }),
+  }),
+])
+
 /** Payload of {@link artifactRecorded}. */
 export const artifactRecordedDataSchema = v.object({
   contentType: v.string(),
@@ -330,6 +398,7 @@ export type CaseEndedData = v.InferOutput<typeof caseEndedDataSchema>
 export type CaseOutputData = v.InferOutput<typeof caseOutputDataSchema>
 export type CaseRequeuedData = v.InferOutput<typeof caseRequeuedDataSchema>
 export type CaseStartedData = v.InferOutput<typeof caseStartedDataSchema>
+export type Grader = v.InferOutput<typeof graderSchema>
 export type InferenceFailedData = v.InferOutput<typeof inferenceFailedDataSchema>
 export type InferenceRequestedData = v.InferOutput<typeof inferenceRequestedDataSchema>
 export type InferenceRespondedData = v.InferOutput<typeof inferenceRespondedDataSchema>
@@ -341,6 +410,8 @@ export type ProjectStartedData = v.InferOutput<typeof projectStartedDataSchema>
 export type RunEndedData = v.InferOutput<typeof runEndedDataSchema>
 export type RunInterruptedData = v.InferOutput<typeof runInterruptedDataSchema>
 export type RunStartedData = v.InferOutput<typeof runStartedDataSchema>
+export type ScoreCombinationData = v.InferOutput<typeof scoreCombinationSchema>
+export type ScoreDefinedData = v.InferOutput<typeof scoreDefinedDataSchema>
 export type ScoreRecordedData = v.InferOutput<typeof scoreRecordedDataSchema>
 export type TaskEndedData = v.InferOutput<typeof taskEndedDataSchema>
 export type TaskQueuedData = v.InferOutput<typeof taskQueuedDataSchema>
@@ -381,7 +452,12 @@ export const caseOutput = defineEventa<CaseOutputData>('case.output')
 export const metricDefined = defineEventa<MetricDefinedData>('metric.defined')
 /** Case code recorded one metric. */
 export const metricRecorded = defineEventa<MetricRecordedData>('metric.recorded')
-/** Case code recorded one score. */
+/**
+ * Label, description, combination, and grader of one score; emitted by its task before the first value of the score,
+ * and again when they change. Viewers keep the latest per score name.
+ */
+export const scoreDefined = defineEventa<ScoreDefinedData>('score.defined')
+/** Case code recorded one score value or an unscored value, or the runtime derived one when the case try settled. */
 export const scoreRecorded = defineEventa<ScoreRecordedData>('score.recorded')
 /** Case code recorded one artifact file. */
 export const artifactRecorded = defineEventa<ArtifactRecordedData>('artifact.recorded')
diff --git a/packages/vieval/src/index.ts b/packages/vieval/src/index.ts
index 2ca024c..10aee29 100644
--- a/packages/vieval/src/index.ts
+++ b/packages/vieval/src/index.ts
@@ -2,6 +2,8 @@ export { defineConfig, loadEnv } from './cli/config'
 export { requiredEnvFrom } from './core/inference-executors/env'
 export { defineMetric } from './dsl/metrics'
 export type { MetricDefinition, MetricMetadata } from './dsl/metrics'
+export { defineScore } from './dsl/scores'
+export type { ScoreCombination, ScoreDefinition, ScoreMetadata, ScoreRecordOptions, ScoreRef } from './dsl/scores'
 export { caseOf, casesFromInputs, describeEval, describeTask } from './dsl/task'
-export type { LocalizedText } from './events'
+export type { Grader, LocalizedText } from './events'
 export { expect } from './expect'
diff --git a/packages/vieval/tests/projects/example-api-config-matrix/evals/model-comparison.eval.ts b/packages/vieval/tests/projects/example-api-config-matrix/evals/model-comparison.eval.ts
index faf2539..dda981f 100644
--- a/packages/vieval/tests/projects/example-api-config-matrix/evals/model-comparison.eval.ts
+++ b/packages/vieval/tests/projects/example-api-config-matrix/evals/model-comparison.eval.ts
@@ -1,4 +1,5 @@
 import { defineEval, defineTask } from '../../../../src/config'
+import { scoreCases } from '../../../../src/core/scoring'
 import { modelFromRun } from '../../../../src/plugins/chat-models'
 
 export default defineEval({
@@ -20,12 +21,13 @@ export default defineEval({
       const matchesSelectedModel = runModel != null && agentModel.model === runModel
       const stressPenalty = scenario === 'stress' && runModel === 'gpt-4.1-mini' ? 0.15 : 0
 
-      return {
-        scores: [{
-          kind: 'exact',
-          score: Math.max(0, (matchesSelectedModel ? 1 : 0) - stressPenalty),
-        }],
-      }
+      // A custom task runs no cases, so it reports its one measurement as one passed outcome and lets
+      // `scoreCases` build the summaries (`scores`, `primary`, `completion`, `passed`) of `TaskRunOutput`.
+      return scoreCases([{
+        scores: { 'model_comparison.match': Math.max(0, (matchesSelectedModel ? 1 : 0) - stressPenalty) },
+        state: 'passed',
+        unscored: {},
+      }])
     },
   }),
 })
diff --git a/packages/vieval/tests/projects/example-api-config-matrix/evals/rubric-sensitivity.eval.ts b/packages/vieval/tests/projects/example-api-config-matrix/evals/rubric-sensitivity.eval.ts
index f26f925..0e40e4e 100644
--- a/packages/vieval/tests/projects/example-api-config-matrix/evals/rubric-sensitivity.eval.ts
+++ b/packages/vieval/tests/projects/example-api-config-matrix/evals/rubric-sensitivity.eval.ts
@@ -1,4 +1,5 @@
 import { defineEval, defineTask } from '../../../../src/config'
+import { scoreCases } from '../../../../src/core/scoring'
 import { modelFromEval } from '../../../../src/plugins/chat-models'
 
 export default defineEval({
@@ -26,12 +27,13 @@ export default defineEval({
       const strictPenalty = rubric === 'strict' ? 0.05 : 0
       const hasJudge = judgeModel.id.length > 0
 
-      return {
-        scores: [{
-          kind: 'judge',
-          score: Math.max(0, (hasJudge ? 1 : 0) - strictPenalty),
-        }],
-      }
+      // A custom task runs no cases, so it reports its one measurement as one passed outcome and lets
+      // `scoreCases` build the summaries (`scores`, `primary`, `completion`, `passed`) of `TaskRunOutput`.
+      return scoreCases([{
+        scores: { 'rubric_sensitivity.rubric': Math.max(0, (hasJudge ? 1 : 0) - strictPenalty) },
+        state: 'passed',
+        unscored: {},
+      }])
     },
   }),
 })
diff --git a/packages/vieval/tests/projects/example-api-defining-new-task/evals/task-default.eval.ts b/packages/vieval/tests/projects/example-api-defining-new-task/evals/task-default.eval.ts
index 89dd2cc..c676907 100644
--- a/packages/vieval/tests/projects/example-api-defining-new-task/evals/task-default.eval.ts
+++ b/packages/vieval/tests/projects/example-api-defining-new-task/evals/task-default.eval.ts
@@ -40,8 +40,13 @@ export default defineEval({
 
       const score = defaultModel.id.length > 0 && rubricModel.id.length > 0 ? 1 : 0
 
+      // A custom task returns `TaskRunOutput` itself. Here the summaries are written by hand to show the shape;
+      // `scoreCases` from `vieval/core/scoring` builds the same object from case outcomes.
       return {
-        scores: [{ kind: 'exact', score }],
+        completion: { mean: 1, n: 1, name: 'completion', sum: 1, total: 1, unscored: {} },
+        passed: { mean: 1, n: 1, name: 'passed', sum: 1, total: 1, unscored: {} },
+        primary: 'task_default.models_resolved',
+        scores: [{ mean: score, n: 1, name: 'task_default.models_resolved', sum: score, total: 1, unscored: {} }],
       }
     },
   }),
diff --git a/packages/vieval/tests/projects/example-pattern-byoa-bring-your-own-agent/vieval.exec.config.cjs b/packages/vieval/tests/projects/example-pattern-byoa-bring-your-own-agent/vieval.exec.config.cjs
index e0f8cca..21af5db 100644
--- a/packages/vieval/tests/projects/example-pattern-byoa-bring-your-own-agent/vieval.exec.config.cjs
+++ b/packages/vieval/tests/projects/example-pattern-byoa-bring-your-own-agent/vieval.exec.config.cjs
@@ -8,7 +8,7 @@ module.exports = {
           id: task.id,
           matrix: task.matrix,
           providerId: task.inferenceExecutor.id,
-          scores: [{ kind: 'exact', score: 1 }],
+          scores: [{ mean: 1, n: 1, name: 'byoa.success', sum: 1, total: 1, unscored: {} }],
         }
       },
       include: ['evals/*.eval.ts'],
diff --git a/packages/vieval/tsdown.config.ts b/packages/vieval/tsdown.config.ts
index 5f6f02e..5d09018 100644
--- a/packages/vieval/tsdown.config.ts
+++ b/packages/vieval/tsdown.config.ts
@@ -9,7 +9,6 @@ export default defineConfig({
     'config': 'src/config/index.ts',
     'core/assertions/index': 'src/core/assertions/index.ts',
     'core/inference-executors/index': 'src/core/inference-executors/index.ts',
-    'core/processors/results/index': 'src/core/processors/results/index.ts',
     'core/runner/index': 'src/core/runner/index.ts',
     'core/scheduler/index': 'src/core/scheduler/index.ts',
     'core/scoring/index': 'src/core/scoring/index.ts',