feat: configurable retry policy as a top-level retry: root

CLAUDE.md

@@ -35,7 +35,8 @@ coder_eval/
 │   ├── criteria.py                # 14 success criterion types + base + union
 │   ├── experiment.py              # ExperimentDefinition, ExperimentVariant, ResolvedTask, result models
 │   ├── judge_defaults.py          # DEFAULT_JUDGE_MODEL constant (cycle-free leaf)
-│   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template/rephrase)
+│   ├── mutations.py               # PromptMutation variants (prefix/suffix/replace/template)
+│   ├── retry.py                   # RetryPolicy (top-level `retry:` block; -D root)
 │   ├── results.py                 # CriterionResult (+ ClassificationCriterionResult), TurnRecord, EvaluationResult, EarlyStopInfo/EarlyStopReason, CriterionAggregate, ThresholdCheck, SuiteRollup
 │   ├── routing.py                 # ApiRoute (DirectRoute/BedrockRoute)
 │   ├── sandbox.py                 # SandboxConfig, ResourceLimits
@@ -130,8 +131,8 @@ templates/                         # Sandbox template directories
 - **Separation of Concerns**: Data models (`models/`) are pure Pydantic; logic lives in `criteria/`, `evaluation/`, etc.
 - **Callback Streaming**: `StreamCallback` protocol with `TaskScopedCallback` wrapper for real-time LLM event output
 - **Experiment Layer**: Pre-processing config resolver (`ExperimentRunner`) that resolves task × variant combinations via 5-layer merge (default → experiment defaults → task → variant → CLI) before passing to `run_batch`. For running A/B comparisons (model vs. model, skill on vs. off, prompt vs. prompt), see [docs/AB_EXPERIMENTS.md](docs/AB_EXPERIMENTS.md).
-- **Single declarative merge resolver**: All five config layers merge through ONE engine (`orchestration/config_merge.py::resolve_root`) for the three `-D`-reachable roots (`agent`/`run_limits`/`sandbox`). Each field declares *how it merges* once, on the model, via `MergeField(strategy="deep"|"append"|"replace")` (or a type-aware default: nested `BaseModel`/free-form `dict` → `deep`; `list`/scalar → `replace`). `resolve_task_for_variant` (layers 1–4) and `apply_overrides` (layer 5) build `Layer` lists and call the same `resolve_root`, so a field merges identically regardless of which layer supplied it (the unification invariant, enforced by `tests/test_merge_unification.py`). Lint rule CE014 forces every list field to declare its strategy explicitly.
-- **Generic CLI overrides (`-D`/`--set`)**: Layer 5 is a thin wrapper (`orchestration/overrides.py`) over the resolver above. `coder-eval run -D agent.model=opus -D run_limits.max_turns=30` overrides any field on the resolved `TaskDefinition` (`agent`/`run_limits`/`sandbox` roots), schema-validated with did-you-mean. Only `--model` (→ `agent.model`) and `--driver` (→ `sandbox.driver`) survive as active thin aliases that emit the equivalent `-D` entry; an alias and `-D` targeting the same path is a hard error. `--type` (→ `agent.type`) is a separate, lighter alias that does NOT route through that collision check — `--type` and `-D agent.type=…` last-win rather than hard-error (the `-D` value wins). Tools, plugins, and SDK options are `-D`-only.
+- **Single declarative merge resolver**: All five config layers merge through ONE engine (`orchestration/config_merge.py::resolve_root`) for the four `-D`-reachable roots (`agent`/`run_limits`/`retry`/`sandbox`). Each field declares *how it merges* once, on the model, via `MergeField(strategy="deep"|"append"|"replace")` (or a type-aware default: nested `BaseModel`/free-form `dict` → `deep`; `list`/scalar → `replace`). `resolve_task_for_variant` (layers 1–4) and `apply_overrides` (layer 5) build `Layer` lists and call the same `resolve_root`, so a field merges identically regardless of which layer supplied it (the unification invariant, enforced by `tests/test_merge_unification.py`). Lint rule CE014 forces every list field to declare its strategy explicitly.
+- **Generic CLI overrides (`-D`/`--set`)**: Layer 5 is a thin wrapper (`orchestration/overrides.py`) over the resolver above. `coder-eval run -D agent.model=opus -D run_limits.max_turns=30` overrides any field on the resolved `TaskDefinition` (`agent`/`run_limits`/`retry`/`sandbox` roots), schema-validated with did-you-mean. Only `--model` (→ `agent.model`) and `--driver` (→ `sandbox.driver`) survive as active thin aliases that emit the equivalent `-D` entry; an alias and `-D` targeting the same path is a hard error. `--type` (→ `agent.type`) is a separate, lighter alias that does NOT route through that collision check — `--type` and `-D agent.type=…` last-win rather than hard-error (the `-D` value wins). Tools, plugins, and SDK options are `-D`-only.
 - **All core models importable from `coder_eval.models`** regardless of submodule
 - **Dataset fan-out**: `TaskDefinition.dataset` (inline rows or JSONL path) expands a single task into N row-tasks with `${row.<field>}` substitution in `initial_prompt` and `success_criteria` string fields. Expansion runs in `task_loader.expand_dataset` **before** variant resolution, so variants cannot override the dataset. Row sampling: CLI `--sample N` (fixed-seed uniform-random N over the whole dataset) overrides `--sample-per-stratum N` / `dataset.sample_per_stratum` (stratified random N-per-stratum, keyed on `stratify_field`, default `expected_skill` — for classification suites like activation). Stratified sampling (whether the N-per-stratum count comes from the **CLI** `--sample-per-stratum` flag or **YAML** `dataset.sample_per_stratum`) is **nondeterministic** by default — it re-draws each run (so the nightly activation suite broadens coverage over time). Set `dataset.sample_seed` to pin a reproducible sample; an explicit seed always wins. (Only `--sample N` uses a fixed seed, since a smoke test wants the same N rows each run.)
 - **Per-criterion aggregation**: Each `BaseCriterion` subclass exposes `aggregate(criterion, per_row_results) -> CriterionAggregate | None`. Default emits `count / mean / median / std / min / max` so every criterion is suite-thresholdable for free. Classification-style criteria return `ClassificationCriterionResult` (subclass of `CriterionResult`) and layer accuracy / P/R/F1 / confusion via the shared `overlay_classification_metrics` utility. `BaseSuccessCriterion.suite_thresholds` gates the suite on those metrics; CLI exits non-zero on any gate failure.
@@ -294,7 +295,7 @@ Tasks are YAML files. See [docs/TASK_DEFINITION_GUIDE.md](docs/TASK_DEFINITION_G
 
 **Runtime (always)**: pydantic, pydantic-settings, pyyaml, typer, rich, python-dotenv, anthropic, claude-agent-sdk, anyio, radon, tqdm, jmespath, jsonschema
 
-**Runtime (optional, `[uipath]` extra)**: uipath — the in-host `uipath` SDK (handy for local sandbox parity with tasks that invoke `uv run uipath eval ...`). Base installs without this extra still run end-to-end; UiPath-dependent paths fail at dispatch with a clear `pip install 'coder-eval[uipath]'` hint. The LLM judge and the `rephrase` mutation no longer use the LLM Gateway client — they route through the run's backend (Bedrock / Anthropic), so `uipath-llmgw-client` is no longer a dependency.
+**Runtime (optional, `[uipath]` extra)**: uipath — the in-host `uipath` SDK (handy for local sandbox parity with tasks that invoke `uv run uipath eval ...`). Base installs without this extra still run end-to-end; UiPath-dependent paths fail at dispatch with a clear `pip install 'coder-eval[uipath]'` hint. The LLM judge no longer uses the LLM Gateway client — it routes through the run's backend (Bedrock / Anthropic), so `uipath-llmgw-client` is no longer a dependency.
 
 **Dev**: pytest, pytest-asyncio, pytest-mock, pytest-cov, ruff, pyright, pip-audit, bandit, pre-commit, mcp
 

docs/AB_EXPERIMENTS.md

@@ -227,8 +227,8 @@ variants:
         text: "\n\nThink step by step and validate your work before finishing."
 ```
 
-The full mutation catalog (prefix / suffix / replace / template / rephrase) is
-defined in `coder_eval/models/mutations.py`.
+The full mutation catalog (prefix / suffix / replace / template) is defined in
+`coder_eval/models/mutations.py`.
 
 ## Recipe: Smoke vs. e2e Flavors (Early Stop)
 

docs/TASK_DEFINITION_GUIDE.md

@@ -103,6 +103,12 @@ run_limits:
   turn_timeout: 300                   # Optional: per-communicate() timeout in seconds
   task_timeout: 600                   # Optional: wall-clock cap across all iterations
 
+# Retry policy is NOT a cap — it lives at the task's top level, not under run_limits
+retry:                                # Optional: override the built-in retry policy
+  max_retries: 2                      #   attempts per retryable error category (0 = fail fast)
+  initial_delay: 5.0                  #   seconds before the first retry
+  backoff_multiplier: 2.0             #   exponential backoff factor
+
 agent:
   type: "claude-code"                 # Agent type — optional if supplied via experiment / --type
   permission_mode: "acceptEdits"      # Permission mode (see below)
@@ -184,6 +190,42 @@ model, etc.) is conceptually separate.
 > They must live under `run_limits:`. (A deprecation shim hoisted them
 > automatically until it was removed on 2026-06-01.)
 
+### `retry` (transient-error policy)
+
+`retry:` is a **top-level** task block, deliberately not under `run_limits:` —
+`run_limits` holds caps that *abort* a run, while `retry` is how hard the run
+tries to survive a transient error (an API 5xx, a rate limit, a sandbox setup
+hiccup).
+
+```yaml
+retry:
+  max_retries: 0        # fail fast — no retries on any category
+```
+
+```bash
+coder-eval run tasks/my_task.yaml -D retry.max_retries=0    # same, from the CLI
+```
+
+The baseline is a per-error-category table in `errors/categories.py::RETRY_CONFIG`.
+Every field here is optional and applies uniformly to the categories that are
+*already* retryable; omitting a field keeps the built-in value. Which categories
+are retryable at all is a property of the error, not the run: a non-retryable
+category (auth, billing) is **never** made retryable by an override.
+
+| Field | Default | Meaning |
+|-------|---------|---------|
+| `max_retries` | per-category | Attempts after the first failure. `0` disables retries entirely. |
+| `initial_delay` | per-category | Seconds before the first retry (exponential from there, plus jitter). |
+| `backoff_multiplier` | per-category | Exponential backoff factor between attempts. |
+
+**Retries resume in place.** A retried `communicate()` re-sends the prompt into
+the *same* sandbox and the *same* agent session — the working directory is not
+reset, so files the failed attempt already wrote survive into the retry. The
+crashed attempt's partial turn is also preserved on the result (for crash
+forensics), which means trajectory-observing criteria (`command_executed`,
+`commands_efficiency`, `skill_triggered`) see **both** attempts. Set
+`max_retries: 0` when you need a clean single-attempt trajectory.
+
 ### `expected_turns` (soft efficiency budget)
 
 `run_limits.expected_turns` is a **soft target**, not a cap: the run is never
@@ -275,11 +317,17 @@ sandbox:
     - "!node_modules"
     - "*.bak"                         #   bare entry adds an extra pattern
   limits:                             # Optional: resource limits
-    timeout: 300                       # Enforced via subprocess timeout
-    max_memory_mb: 512                 # NOT enforced (reserved for future use)
-    max_disk_mb: 1024                  # NOT enforced (reserved for future use)
+    timeout: 300                       # Enforced via subprocess timeout (both drivers)
+    max_memory_mb: 512                 # driver:docker -> `--memory`; ignored under tempdir
+    max_cpus: 2                        # driver:docker -> `--cpus`; ignored under tempdir
+    max_pids: 512                      # driver:docker -> `--pids-limit`; ignored under tempdir
+    max_disk_mb: 1024                  # NOT enforced (reserved: no portable docker knob)
 ```
 
+Under `driver: tempdir` only `timeout` is enforced — the agent can consume
+arbitrary host memory, CPU, and PIDs. Use `driver: docker` when you need the
+container limits above to actually bind.
+
 ## Template Sources
 
 Tasks can start with preset files instead of an empty sandbox. Multiple sources are applied sequentially (last wins for conflicts).
@@ -383,7 +431,7 @@ All criteria share these fields:
 | Field | Default | Description |
 |-------|---------|-------------|
 | `description` | — | Human-readable description (required) |
-| `weight` | 1.0 | Relative importance for weighted score |
+| `weight` | 1.0 | Relative importance for weighted score. `0` = **informational**: excluded from both the score and the pass/fail gate |
 | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass |
 | `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). See [`stop_early`](#stop_early-opt-in-early-stop). |
 
@@ -392,7 +440,20 @@ All criteria share these fields:
 - **Fractional** (0.0–1.0): `file_contains`, `file_check`, `json_check`, `command_executed`, `uipath_eval`
 - **Continuous** (0.0–1.0): `reference_comparison`, `llm_judge`, `agent_judge`
 
-**Task success:** ALL criteria must score >= their `pass_threshold`.
+**Task success:** all *gating* criteria must score >= their `pass_threshold`. A
+criterion with `weight: 0` is informational — it is still checked, stored, and
+rendered in reports, but it neither contributes to the score nor fails the task.
+(A `weight: 0` criterion may not set `stop_when` or `suite_thresholds`: arming a
+non-gating criterion for the early-stop or suite gate would let an
+"informational" check flip a run to failure.)
+
+Every surface labels it as such rather than as a failure: the terminal shows `○`
+instead of `✓`/`✗`, the HTML report tags the row "informational — not gated" and
+excludes it from the *n*/*m* passed header, the evalboard renders an `INFO` pill,
+and a below-threshold informational criterion is never sampled as the reason a
+suite row failed. The persisted `CriterionResult.gating` field carries this to
+every consumer, so no reader needs the original criterion to know whether a low
+score mattered.
 
 **Weighted score:** `weighted_score = sum(score * weight) / sum(weight)` — calculated regardless for quality assessment.
 

docs/USER_GUIDE.md

@@ -35,7 +35,7 @@ coder-eval run tasks/hello_date.yaml --stream full  # live LLM output
 | `--max-parallel, -j` | Concurrent tasks (default: 1) |
 | `--preservation-mode` | Sandbox persistence: `NONE` / `MOVE_ON_WRITE` / `DIRECT_WRITE`. Default is driver-derived (docker → `DIRECT_WRITE`, else `MOVE_ON_WRITE`); explicit value always wins. |
 | `--run-dir` | Custom run directory (default: timestamped in `runs/`) |
-| `-D path=value` / `--set` | Override any resolved task-config field (`agent`/`run_limits`/`sandbox` roots), e.g. `-D run_limits.max_turns=30 -D agent.permission_mode=plan -D agent.sdk_options.effort=high`. Repeatable; schema-validated. This is the way to set permission mode, turn/timeout limits, tools, plugins, and SDK options. |
+| `-D path=value` / `--set` | Override any resolved task-config field (`agent`/`run_limits`/`retry`/`sandbox` roots), e.g. `-D run_limits.max_turns=30 -D agent.permission_mode=plan -D agent.sdk_options.effort=high`. Repeatable; schema-validated. This is the way to set permission mode, turn/timeout limits, tools, plugins, and SDK options. |
 | `--model, -m` | Shorthand alias for `-D agent.model=…` (e.g., `claude-sonnet-4-20250514`) |
 | `--driver` | Shorthand alias for `-D sandbox.driver=…` (`tempdir` or `docker`) |
 | `--exclude-tags` | Skip tasks matching any of these tags (comma-separated) |

evalboard/app/runs/[id]/[...task]/_chips.tsx

@@ -26,15 +26,25 @@ export function Expandable({
     );
 }
 
-export function ResultPill({ passed }: { passed: boolean }) {
-    const cls = passed
-        ? "bg-green-50 text-green-700 border-green-200"
-        : "bg-red-50 text-red-700 border-red-200";
+// `gating: false` (weight: 0) criteria are informational — they cannot fail the
+// task, so rendering PASS/FAIL for them would contradict the task's own status.
+export function ResultPill({
+    passed,
+    gating = true,
+}: {
+    passed: boolean;
+    gating?: boolean;
+}) {
+    const cls = !gating
+        ? "bg-gray-50 text-gray-600 border-gray-200"
+        : passed
+          ? "bg-green-50 text-green-700 border-green-200"
+          : "bg-red-50 text-red-700 border-red-200";
     return (
         <span
             className={`inline-block px-2 py-0.5 text-xs rounded-full border font-medium ${cls}`}
         >
-            {passed ? "PASS" : "FAIL"}
+            {!gating ? "INFO" : passed ? "PASS" : "FAIL"}
         </span>
     );
 }

evalboard/app/runs/[id]/[...task]/_sections.tsx

@@ -108,16 +108,27 @@ export function CriteriaSection({ criteria }: { criteria: CriterionResult[] }) {
         <section className="space-y-2">
             <h2 className="text-sm font-semibold text-gray-900">
                 Success criteria ({criteria.length})
+                {criteria.some((c) => !c.gating) && (
+                    <span className="ml-2 font-normal text-gray-500">
+                        {criteria.filter((c) => !c.gating).length}{" "}
+                        informational
+                    </span>
+                )}
             </h2>
             <div className="space-y-2">
                 {criteria.map((c, i) => {
-                    const passed = c.score === 1;
+                    // Compare against the criterion's own threshold, not === 1:
+                    // fractional criteria pass below 1.0 (default threshold 0.9).
+                    const passed = (c.score ?? 0) >= c.passThreshold;
                     return (
                         <Expandable
                             key={i}
                             header={
                                 <div className="flex items-center gap-3">
-                                    <ResultPill passed={passed} />
+                                    <ResultPill
+                                        passed={passed}
+                                        gating={c.gating}
+                                    />
                                     <span className="text-sm text-gray-900">
                                         {c.description ??
                                             c.criterionType ??

evalboard/lib/runs.ts

@@ -103,6 +103,12 @@ export interface CriterionResult {
     score: number | null;
     details: string | null;
     error: string | null;
+    // Mirrors the Python CriterionResult fields. `gating: false` (weight: 0) means
+    // the criterion is informational — measured, but excluded from the score and
+    // the pass/fail gate, so it must not render as PASS/FAIL. Both default the way
+    // pre-existing task.json files behave: gating, threshold 0.9.
+    passThreshold: number;
+    gating: boolean;
 }
 
 export interface ElementExecution {
@@ -1823,6 +1829,8 @@ export async function readTaskDetail(
             score?: number;
             details?: string;
             error?: string | null;
+            pass_threshold?: number;
+            gating?: boolean;
         }>;
         iterations?: TurnEntry[];
         environment_info?: RawRunJson["environment_info"];
@@ -1836,6 +1844,8 @@ export async function readTaskDetail(
         score: c.score ?? null,
         details: c.details ?? null,
         error: c.error ?? null,
+        passThreshold: c.pass_threshold ?? 0.9,
+        gating: c.gating ?? true,
     }));
 
     const artifactRoot = path.join(contentDir, "artifacts");