fix: a missing upstream deployment is not a task failure

[shehabyasser-scale]

Stacked on #53. Refs #51.

The gap #53 left

#53 stopped a dead sandbox being scored as an informative zero. The same masking survives one layer up, for a cause that is both more common and easier to fix.

Full per-case terminal-exception census of the 2026-07-25 swe-atlas-qna run (469 cases, every one scored 0.0):

n	exception	classified as	correct?
165	NotFoundError: Modal Sandbox with container ID ta-… not found	transient_infra	✅ (thanks to #53)
145	NotFoundError: 404 … 'code': 'DeploymentNotFound'	task_failure	❌
102	RuntimeError: FetchSpec failed: loading container: file does not exist	transient_infra	✅
53	AddTestsDirError: Failed to add tests directory	transient_infra	✅
4	NotFoundError: Task has already finished with status failure	task_failure	❌ (see below)

Those 145 are the configured eval model not being provisioned on the Azure resource. The agent's every call 404s, it writes no answer, classify_case finds nothing it recognises in the signal and falls through to TASK_FAILURE, whose policy is is_informative_sample=True. Each case therefore became a genuine score of 0.0, and the harness reported that the candidate failed the task — for a model that does not exist.

This is exactly the failure mode the module's own docstring says it exists to prevent: "recorded 'nothing shipped' as a real score of zero".

The change

UPSTREAM_MODEL_UNAVAILABLE:

• is_informative_sample=False — the point of the fix; these stop polluting the aggregate.
• retryable=False — it will not heal.
• terminating=True — every remaining case fails identically, so there is nothing left to measure. Matches AUTH_FAILURE's shape; both are permanent operator-fixable misconfigurations.
• counts_toward_invalidity=True — unlike AUTH_FAILURE. If the terminating path is ever bypassed, the aggregate must still come out invalid rather than quietly averaging a shrinking set of survivors.

Ranked above TRANSIENT_INFRA in classify_case, so a case that saw both a dead sandbox and a missing deployment reports the deterministic, operator-fixable cause.

The precision that matters

The obvious pattern for this is "does not exist". That would be a bug: 102 cases in the same run fail with FetchSpec failed: loading container: file does not exist, which is genuine infrastructure and must stay transient_infra. The pattern therefore matches only the provider's own error codes (DeploymentNotFound, model_not_found) and the exact Azure sentence. test_missing_deployment_pattern_does_not_swallow_container_load_failures pins that boundary using the run's literal strings.

Not fixed here

The 4 NotFoundError: Task has already finished with status failure cases are Modal sandbox deaths that still land in task_failure, because neither the type name nor the message contains any word the infra pattern recognises. Small enough to leave rather than widen a regex on 4 samples, but worth knowing the taxonomy is still keyed off message text and will keep missing variants like this. A structural signal (the exception's module, e.g. modal.exception.*) would be a better long-term key.

Tests: test_a_missing_upstream_deployment_is_not_a_task_failure and test_missing_deployment_pattern_does_not_swallow_container_load_failures, both using the exact strings from the run. Full suite 386 passed / 1 environmental failure (Docker daemon down) / 5 skipped.

Complements #59, which stops the run from ever starting when a configured model is not deployed. This one handles the case where a deployment disappears mid-run.

Greptile Summary

Introduces UPSTREAM_MODEL_UNAVAILABLE as a new ErrorCategory to fix 145 of 469 swe-atlas-qna cases being wrongly treated as informative task failures when the Azure deployment for the evaluation model was simply not provisioned. The policy is is_informative_sample=False, terminating=True, retryable=False, and counts_toward_invalidity=True.

• Adds a precise signal pattern (deploymentnotfound, model_not_found, the exact Azure sentence) ordered before the TRANSIENT_INFRA block in both _SIGNAL_PATTERNS and the classify_case priority chain, so a case that saw both a dead sandbox and a missing deployment reports the deterministic, operator-fixable cause.
• Ships two tests using the run's literal exception strings: one verifying the full policy contract, and one pinning the boundary so FetchSpec failed: loading container: file does not exist cannot be misclassified by the new pattern.

Confidence Score: 5/5

Safe to merge — the change is well-scoped, backed by production data, and cannot misclassify existing categories

The signal pattern is deliberately narrow (provider-specific error codes and the exact Azure sentence) to prevent false positives, and the boundary test pins that distinction. Policy choices are coherent with the existing taxonomy. The only gap is a stale two-word docstring phrase with no runtime effect.

Files Needing Attention: No files require special attention

Important Files Changed

Filename	Overview
vero/src/vero/evaluation/scoring/error_taxonomy.py	Adds UPSTREAM_MODEL_UNAVAILABLE category with policy (non-informative, terminating, counts toward invalidity), a precise signal-matching pattern, and correct precedence in classify_case; docstring for classify_case is slightly stale
vero/tests/test_v05_error_taxonomy.py	Adds two focused tests using the run's exact production strings: one proving UPSTREAM_MODEL_UNAVAILABLE is classified and policy-checked correctly, one pinning the boundary so container-load failures are not swallowed by the new pattern

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[signals: list of str] --> B[classify_signal each signal]
    B --> C{Pattern match}
    C -->|budget/quota| D[INFERENCE_BUDGET_EXHAUSTED]
    C -->|auth/permission| E[AUTH_FAILURE]
    C -->|DeploymentNotFound / model_not_found / Azure sentence| F[UPSTREAM_MODEL_UNAVAILABLE]
    C -->|rate-limit / timeout / connection / 5xx| G[TRANSIENT_INFRA]
    C -->|sandbox / container / fetchspec| G
    C -->|no match / empty| H[None]
    D & E & F & G & H --> I[categories set]
    I --> J{Priority check}
    J -->|INFERENCE_BUDGET_EXHAUSTED in set| K[return INFERENCE_BUDGET_EXHAUSTED]
    J -->|AUTH_FAILURE in set| L[return AUTH_FAILURE]
    J -->|UPSTREAM_MODEL_UNAVAILABLE in set| M[return UPSTREAM_MODEL_UNAVAILABLE]
    J -->|TRANSIENT_INFRA in set| N[return TRANSIENT_INFRA]
    J -->|none matched| O[return TASK_FAILURE]

Loading

_{Reviews (3): Last reviewed commit: "fix: a missing upstream deployment is no..." | Re-trigger Greptile}