feat: preflight configured models before launching an optimizer trial

[shehabyasser-scale]

Refs #51. Independent of #58 — that one fixes swe-atlas-qna's config, this one makes the class of mistake self-reporting.

The failure mode this closes

A configured-but-undeployed model is invisible until the run is over:

1. The gateway happily forwards the request (the model is on its allow-list).
2. The upstream returns 404 DeploymentNotFound.
3. The agent makes no progress and writes no answer.
4. The verifier scores the case 0.0 and the harness labels it task_failure.

The result is a run that looks honest. It reports that the candidate failed the task. Nothing in result.json, reward.json or finalization.json says "there was no model". You only find out by extracting session.tar.gz and reading artifacts/inference/usage.json.

That is exactly how swe-atlas-qna produced 469 zero-score cases across two runs: 322/322 evaluation-scope and 79/79 finalization-scope requests returned DeploymentNotFound while the producer scope ran 113 requests clean. Two full trials, ~an hour, ~$4, to discover a config typo.

The change

vero harbor run sends one 16-token request per distinct allow-listed model (producer, evaluation, finalization) before compiling the task, and aborts if any returns 404.

Only a definitive 404 is fatal. A timeout, a 429 or a 503 is inconclusive — it gets reported and the run proceeds. A preflight that blocks runs on a transient upstream blip would be worse than no preflight, and that is also why there is no --skip-preflight escape hatch: the one blocking condition is never a false positive worth overriding.

Model names are probed with the provider prefix stripped (openai/gpt-4o → gpt-4o), which is the form the agents actually send after model_name.removeprefix("openai/").

Live verification against the real endpoint

Blocking path — gaia, which still points at the undeployed model:

$ vero harbor run --config ../harness-engineering-bench/gaia/baseline/build.yaml \
    --agent codex --model gpt-5.3-codex --yes -o /tmp/never
Error: these models are not deployed on https://mofdevoai5172b0.openai.azure.com/openai/v1
and every request to them would fail:
  - gpt-5.4-mini-2026-03-17 (evaluation scope): {
  "error": {
    "type": "invalid_request_error",
    "code": "DeploymentNotFound",
    "message": "The API deployment for this resource does not exist. ..."
  }
}
Note a model can appear in GET /models (the catalogue) and still have no deployment.
$ echo $?
1

About two seconds, before a single Modal sandbox is created.

Passing path — a gpt-5.3-codex producer with a gpt-4o evaluation scope (the shape #58 moves swe-atlas-qna to) clears preflight and proceeds.

That last note in the error message is worth keeping: GET /openai/v1/models on this resource returns 172 entries including gpt-5.4-mini-2026-03-17. It lists the catalogue, not the deployments. Anyone who checks the obvious way concludes the model is fine.

Tests

test_preflight_blocks_only_on_a_definitively_missing_deployment (404 aborts with the scope and upstream body; prefix stripped before probing) and test_preflight_does_not_block_on_inconclusive_upstream_failures (429 / 503 / timeout / missing credentials / no gateway all proceed). Both offline.

Greptile Summary

This PR adds a preflight model check to vero harbor run that sends a single 16-token request per configured model before compiling the task, aborting with a clear error if any returns a definitive DeploymentNotFound 404. It closes a silent failure mode where a mis-configured (but gateway-allowed) model would burn an entire optimizer trial producing only zero-score cases with no diagnostic signal.

• _preflight_models collects distinct model names across producer/evaluation/finalization scopes, probes each with its full configured name first and the bare (prefix-stripped) name as a fallback, and raises a ClickException only when a 404 definitively names the model — not when it names the route.
• _probe_model / _model_is_missing separate route-level 404s from model-level 404s by body inspection, and fall through to /chat/completions when /responses returns a route-level 404, so gateways that implement only one API shape are handled correctly.
• Four offline tests cover the blocking path, prefix-fallback logic, route/model 404 discrimination, and the full set of inconclusive cases (429, 503, timeout, missing credentials, no gateway).

Confidence Score: 5/5

Safe to merge. The preflight is purely additive — it only aborts when a deployment is definitively absent, and all inconclusive outcomes (timeouts, rate limits, missing credentials, no gateway) are non-blocking pass-throughs.

The implementation carefully distinguishes route-level 404s from model-level 404s, handles both OpenAI API shapes, provides a correct prefix-stripping fallback, and is fully tested offline. The placement of _preflight_models before tempfile creation ensures it exits early without side effects. No existing behavior is modified.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
vero/src/vero/harbor/cli.py	Adds _probe_route / _probe_model / _model_is_missing helpers and the _preflight_models entry point; hooks it into run_command before any Modal sandbox is created. Logic correctly separates route-level 404s from model-level 404s, tries both /responses and /chat/completions routes, falls back to the bare model name before declaring a deployment missing, and treats everything except a definitive 404 as inconclusive.
vero/tests/test_v05_cli.py	Adds four new tests covering: definitive 404 blocks with scope and body, prefix fallback before declaring missing, route-level vs model-level 404 discrimination, and non-blocking behavior for 429/503/timeout/no-credentials/no-gateway cases. Tests patch at the right level (_probe_model or _probe_route) to stay offline.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[vero harbor run] --> B[load config]
    B --> C{inference_gateway configured?}
    C -->|No| H[compile and launch]
    C -->|Yes| D{API key in env?}
    D -->|No| H
    D -->|Yes| E[collect distinct models across producer / evaluation / finalization]
    E --> F[for each model: probe full name, then bare name if prefixed]
    F --> G[_probe_model: try /responses then /chat/completions]
    G --> G1{HTTP 404 AND body names model?}
    G1 -->|Yes - definitive missing| MISS[mark model missing]
    G1 -->|No - route-level 404| G3[try next route]
    G3 --> G4{another route?}
    G4 -->|Yes| G
    G4 -->|No| INC[inconclusive - run proceeds]
    G1 -->|2xx / 429 / 503 / timeout| OK[probe passed or inconclusive]
    OK --> NEXT[next model]
    INC --> NEXT
    MISS --> NEXT
    NEXT --> NEXT2{all models probed?}
    NEXT2 -->|No| F
    NEXT2 -->|Yes| CHK{any missing?}
    CHK -->|Yes| ERR[ClickException: abort before compile]
    CHK -->|No| H

Loading

_{Reviews (4): Last reviewed commit: "fix: do not read a missing route as a mi..." | Re-trigger Greptile}

[shehabyasser-scale]

Restacked onto the new pr3-harness-bench, and fixed a defect in this PR that the rebase surfaced.

The defect: the probe only spoke the Responses API and treated any 404 as proof the deployment was absent. A route-level 404 (the upstream does not implement that path) and a model-level 404 (the deployment does not exist) share a status code and mean opposite things. Since 1b5ee0c / 7d57386 ported officeqa, tau3, swe-atlas-qna and browsecomp-plus to Chat Completions, an upstream serving only /chat/completions is now a realistic deployment, and this preflight would have hard-blocked a run that would have succeeded. That is the exact failure mode the PR exists to prevent, inverted.

Now it tries both routes and only calls a model missing when the 404 body names the model. Verified against the live Azure endpoint:

gpt-4o                  /responses 200   /chat/completions 200
deepseek-v4-flash       404 {"error":{"code":"DeploymentNotFound", ...}}   <- model
/no-such-endpoint       404 (no body)                                      <- route

Second fix in the same commit: the probe used to strip openai/ unconditionally. A provider prefix routes on a proxy and is meaningless to a single-provider endpoint, and fireworks_ai/... names must not be stripped at all. It now tries the configured spelling first and falls back to the bare one, so openai/gpt-5.3-codex passes preflight via the bare name instead of being reported missing.

Three new tests cover the route/model split and the name fallback, including the both-routes-404 case that cannot be reached against a real endpoint.

Not superseded by 800af32, though a reviewer will reasonably ask. That validator is a static, config-internal check that the target model appears in the right scope's allow-list. This is a live HTTP probe that the deployment exists upstream. 800af32 passes happily on a model nothing serves.

Full suite matches the base branch exactly: 11 pre-existing failures, none new.