UiPath · mohsen-uipath · Jul 24, 2026 · Jul 24, 2026 · Jul 24, 2026 · Jul 24, 2026
@@ -27,6 +27,22 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule
   reachable (needs a live run).
 - [ ] CE-rule: `type: Literal[...]` fields on models in `coder_eval/models/` must declare their tag default (`type: Literal["x"] = "x"`) — a member without the default degrades `validate_registry` diagnostics (PydanticUndefined in expected_types) and breaks direct construction. Nothing guards it today; needs a rule-design call (second violation class inside CE024 vs. a new CExxx at the next free id), and the failure is already double-caught by the MINIMAL_PAYLOADS parity test + direct-construction tests — caught in the 2026-07-03 top5-review-fixes run (Phase 1 quality review).
 
+## From 2026-07-23 stop_when:auto early-stop review
+
+- [ ] CE-rule: the early-stop watcher stop rule must decide polarity via the
+  resolved `_armed_polarities`, never a raw `criterion.stop_when` comparison —
+  forbid `.stop_when` attribute reads inside `EarlyStopWatcher._evaluate` /
+  `_resolve_armed_polarities`'s callers in `orchestration/early_stop.py`. This
+  diff *was* the fix for exactly that class of bug (the old rule compared
+  `stop_when in ("pass","decided")` and so vetoed every mixed `auto` pass-stop).
+  Deferred, not cheap: existing CE rules scope by file/module, not by a specific
+  method, so a method-scoped attribute-ban needs a new AST-walk shape (and risks
+  false positives on the legitimate `is not None` membership reads elsewhere in
+  the file). Claim the next free id in `tests/lint/rules/`. Caught in the
+  2026-07-23 stop_when:auto review; the behavior itself is guarded by
+  `test_auto_mixed_pass_stops_ignoring_undecided_distractors` +
+  `test_mixed_static_arming_pass_stops_ignoring_fail_armed`.
+
 ## From 2026-07-03 open-source docs cleanup
 
 - [ ] **Dead-relative-link checker for `docs/**/*.md`** — resolve every relative

@@ -141,7 +141,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block.
-- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided`; only criteria that can decide from a partial trajectory may arm (they declare a non-empty `live_stop_polarities` ClassVar and override `live_verdict` on `BaseCriterion` — currently `skill_triggered`, `command_executed`; CE025 enforces the two stay consistent). It uses a cooperative `should_stop` seam on the Claude agent's between-messages guard (tool-call granularity, no SIGKILL) driven by `orchestration/early_stop.py::EarlyStopWatcher` (own `EventCollector` + stop rule). Live verdicts only *trigger* the stop; the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use (non-observable criterion, non-Claude agent, wrong polarity, no armed criterion, simulation mode) is an error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (reason + deciding criterion + when), report notes/badges, `stopped_early` run.json rows, and `EarlyStopped`/`EarlyStopReason` telemetry dims. Defaults off ⇒ behavior byte-for-behavior unchanged.
+- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged.
 
 ## Success Criteria (14 types)
 
@@ -162,7 +162,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 | `llm_judge` | Continuous | LLM grades artifacts + optional trajectory + optional reference; routes through the run's backend (Bedrock / Anthropic) |
 | `agent_judge` | Continuous | Spawns a Claude Code SDK agent in an isolated sandbox copy; judge uses tools (Bash/Read/Grep/…) to investigate and returns a JSON verdict. Expensive; runs with evaluator credentials — see SECURITY note in the criterion docstring. |
 
-All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
+All criteria support `weight` (default 1.0) and `pass_threshold` (default 0.9), plus `stop_when` (`pass`/`fail`/`decided`/`auto`, default `null`) which arms the criterion for early stop when `run_limits.stop_early` is set (observable criteria only; `auto` arms the instance's own decidable polarities). On dataset-backed tasks, criteria may also set `suite_thresholds: {metric: min_value}` — the suite gate passes iff every listed metric (from the criterion's `aggregate()` output) meets its minimum.
 
 ## Evaluation Flow
 

@@ -292,11 +292,14 @@ flavor) and a `max_turns` generous enough for `e2e`; see
 [`stop_early`](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop). This recipe
 ships as `experiments/early-stop-ab.yaml`.
 
-Expect **identical pass/fail verdicts** between the two variants — an
-early-stopped run is gated on the armed subset only, and the non-armed criteria
-become advisory (clearly marked in the report), so the `smoke` flavor can't
-"pass for free" — with the `smoke` variant significantly lower on turns,
-duration, and tokens.
+Expect the `smoke` variant significantly lower on turns, duration, and tokens.
+Verdict parity with `e2e` is **one-sided**: an early-stopped run is gated on the
+armed subset only (non-armed criteria become advisory, clearly marked in the
+report), a **fail-stop** is verdict-preserving (it is deferred until every
+pass-armed criterion has resolved), but a **pass-stop** cuts the run once the
+positives are decided — a distractor that would only misfire on a later tool
+call is never observed, so the frozen row scores as a clean pass. Authoritative
+precision/recall therefore belongs on the `e2e` (`stop_early: false`) variant.
 
 ## Replicates (Statistical Power)
 

@@ -339,7 +339,7 @@ success_criteria:
   - type: skill_triggered
     skill_name: date-teller
     expected_skill: date-teller
-    stop_when: decided        # arm on pass OR definitive fail
+    stop_when: auto           # arm whichever polarity this instance can decide
   - type: file_exists         # not armed → advisory on an early-stopped run
     path: report.md
 ```
@@ -348,23 +348,46 @@ Semantics:
 
 - **Opt-in, per run.** With `stop_early: false` (the default) the run behaves
   exactly as before — `stop_when` is inert and every criterion gates normally.
-- **Polarity.** `stop_when: pass` stops the moment all armed criteria are decided
-  in the pass direction; `stop_when: fail` stops on a definitive wrong-signal
-  fail; `stop_when: decided` stops on either. Only criteria that can decide from a
-  partial trajectory (currently `skill_triggered`, `command_executed`) may be
-  armed — arming any other criterion is a hard error at resolution (plan *and*
-  run), never a silent no-op. Decidability can also depend on a criterion's own
-  fields: `command_executed` can live-**pass** only with `max_count` unset and
-  `min_count > 0`, and live-**fail** only with `max_count` set (which includes
-  the `min_count: 0, max_count: 0` "must-NOT-run" form). Arming a polarity the
-  configured criterion can never reach (e.g. `stop_when: pass` alongside a
-  `max_count`) is likewise a hard error at resolution, not a silent full run.
+- **Polarity.** `stop_when: pass` stops the moment all **pass-armed** criteria are
+  decided in the pass direction; `stop_when: fail` stops on a definitive
+  wrong-signal fail; `stop_when: decided` stops on either (the criterion instance
+  must be able to decide **both**). `stop_when: auto` arms whichever polarities
+  **this instance** can decide — use it when the decidable polarity is
+  instance-dependent, e.g. a `skill_triggered` activation suite where a positive
+  row (`skill_name == expected_skill`) can only live-pass and a distractor can only
+  live-fail, so one static value on a dataset-fanned criterion (whose
+  positive/distractor role flips per row) cannot fit every row. A **pass-stop**
+  needs every pass-armed criterion to pass — fail-armed distractors are not
+  required to, and a row with **zero** pass-armed criteria (e.g. a negative row)
+  never pass-stops; a **fail-stop** fires on the first fail-armed criterion that
+  live-fails, but is **deferred while any pass-armed criterion is still
+  undecided** — a distractor misfire on an early tool call must not cut a
+  positive row before its expected signal can appear (that would freeze a
+  would-be true positive as a false negative and deflate suite recall). The
+  misfire is latched, so the deferred fail-stop fires the moment every
+  pass-armed criterion decides; if none ever decides, the run simply continues
+  to the cap. Only criteria that can decide from a partial trajectory (currently
+  `skill_triggered`, `command_executed`) may be armed — arming any other criterion
+  is a hard error at resolution (plan *and* run), never a silent no-op.
+  Decidability can also depend on a criterion's own fields: `command_executed` can
+  live-**pass** only with `max_count` unset and `min_count > 0`, and live-**fail**
+  only with `max_count` set (which includes the `min_count: 0, max_count: 0`
+  "must-NOT-run" form). Arming a polarity the configured criterion can never reach
+  (e.g. `stop_when: pass` alongside a `max_count`, or `auto` on an instance that
+  can decide neither) is likewise a hard error at resolution, not a silent full
+  run.
 - **Verdict.** An early-stopped run is gated on the **armed subset only**; the
   non-armed criteria become **advisory** and are clearly marked (report badge +
   per-criterion note + `stopped_early` row). A run that completes naturally is
   gated on the **full** set, as always. This is what lets one file serve both a
-  `smoke` flavor (`stop_early: true`) and an `e2e` flavor (`stop_early: false`)
-  with identical verdicts — see [AB_EXPERIMENTS.md](AB_EXPERIMENTS.md).
+  `smoke` flavor (`stop_early: true`) and an `e2e` flavor (`stop_early: false`) —
+  see [AB_EXPERIMENTS.md](AB_EXPERIMENTS.md). Verdict parity between the flavors
+  is one-sided: a **fail-stop** is verdict-preserving (the deferral above
+  guarantees every pass-armed signal was allowed to resolve first), but a
+  **pass-stop** cuts the run once the positives are decided, so a distractor that
+  would misfire on a *later* tool call is not observed (the frozen row scores as a
+  clean pass) — the smoke flavor trades some precision completeness for budget, so
+  authoritative precision/recall belongs on the `stop_early: false` run.
 - **Fail-safe.** A live-verdict bug **fails open** to a full run (logged loudly) —
   it can never silently disable a criterion or cause a false early stop.
 
@@ -511,7 +534,7 @@ All criteria share these fields:
 | `description` | — | Human-readable description (required) |
 | `weight` | 1.0 | Relative importance for weighted score. `0` = **informational**: excluded from both the score and the pass/fail gate |
 | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass |
-| `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). See [`stop_early`](#stop_early-opt-in-early-stop). |
+| `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`/`auto`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). `auto` arms whichever polarity this instance can decide (for dataset-fanned criteria whose positive/distractor role flips per row). See [`stop_early`](#stop_early-opt-in-early-stop). |
 
 **Scoring types:**
 - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`, `classification_match`, `skill_triggered`

@@ -7,9 +7,11 @@ description: |
   decide when the `smoke` variant cuts off — as soon as the designated criteria
   are decided — while `e2e` runs every task to completion.
 
-  Expect identical pass/fail verdicts between the two variants (the armed subset
-  gates an early-stopped run; other criteria are advisory and clearly marked),
-  with the `smoke` variant significantly lower on turns, duration, and tokens.
+  Expect the `smoke` variant significantly lower on turns, duration, and tokens.
+  Verdict parity is one-sided: the armed subset gates an early-stopped run
+  (other criteria are advisory, clearly marked), a fail-stop preserves the e2e
+  verdict, but a pass-stop can miss a distractor that would only misfire later —
+  authoritative precision/recall belongs on the e2e variant.
 
 variants:
   - variant_id: e2e

@@ -123,12 +123,17 @@ class BaseSuccessCriterion(BaseModel, ABC):
         ),
     )
 
-    stop_when: Literal["pass", "fail", "decided"] | None = Field(
+    stop_when: Literal["pass", "fail", "decided", "auto"] | None = Field(
         default=None,
         description=(
             "Opt-in early-stop polarity for this criterion (membership in the run's 'armed set'). "
             "None (default) = not a stop criterion. 'pass' = a live PASS may contribute to a stop; "
-            "'fail' = a live definitive FAIL may trigger a stop; 'decided' = either. Inert unless "
+            "'fail' = a live definitive FAIL may trigger a stop; 'decided' = either (the instance "
+            "must be able to decide BOTH polarities). 'auto' = arm whichever polarities THIS instance "
+            "can actually decide (its live_decidable_polarities) - use it when the decidable polarity "
+            "is instance-dependent, e.g. skill_triggered under any-engagement, where a positive row "
+            "(skill_name == expected_skill) can only live-pass and a distractor can only live-fail, "
+            "so no single static polarity fits every fanned-out row. Inert unless "
             "run_limits.stop_early is True. Only valid on criteria observable mid-run (e.g. "
             "skill_triggered, command_executed); an unobservable armed criterion is rejected at "
             "resolution time."