[LEARN-02] Classify Deliveries on hardness dimensions, route model+effort, and check routing against telemetry #5248

Work level: Outcome · Track: Active delivery · Intended Status: In progress

Requested by

The owner (2026-11-02): "does learn … have classification machinery to classify deliveries on different complexity/hardness dimensions to decide what llm model with what effort to use; the results can then be checked against the telemetry?"

Current state

The baseline classifier, versioned routing and first real dispatch recording are accepted in closed Delivery #5249 through PR #5250. The original investigation is retained in native history and the owning design. That source acceptance does not close the complete evaluation and learning promise below.

The user's 2026-11-03 priority is now executing as Delivery #5273: store hardness as a visible Coordination card category and preserve completed-work model/effort provenance. Root owns integration and actual native adoption; hardness5273 owns the isolated implementation. Categories, routing safety floors, accepted launch configuration and provider-effective observations remain distinct; historical gaps stay Unknown.

Next action

Finish #5273 through independent source review, exact-head CI, supported native field/view adoption and evidence-based historical projection. Continue the remaining complete LEARN-02 evaluation and policy-learning acceptance separately; all children closed would not by itself accept this Outcome.

Done when

Primary acceptance, by the user’s latest direction: establish the validity of the hardness classification and whether it transfers across different models and reasoning-effort levels. This is the foundation; timeline prediction and routing optimization are secondary. Separate task difficulty from model/profile capability, safety risk and external waiting. Test profile-specific calibration, rank/order stability and task-by-model/effort differential functioning on held-out work. Check common-task/anchor overlap and identifiability; more unique issues alone cannot establish cross-profile invariance. Non-significance is not evidence of equivalence. Report uncertainty, non-invariance or inconclusive evidence honestly. Existing heuristic labels remain provisional; source/unit-test success is not statistical validity, and no learned routing activates on that basis.

1. Classifier. A versioned, pre-dispatch classification of each Delivery on explicit hardness dimensions, reusing and extending LEARN's rubric:

   • scope (repos and files, contract boundaries, cross-repo);
   • novelty and research need;
   • cross-cutting or invariant-heavy work (sizing, concurrency, lifecycle);
   • safety and data criticality (deletes, publishes, upgrades, credentials);
   • numeric and determinism rigour (Fable lockstep);
   • UI and browser-matrix cost;
   • review-round risk;
   • required evidence (live or native gates).

   Scored by fixed rules, plus an optional LLM-assisted scorer with a fixed prompt, recorded with provenance (rule vs model, version) before dispatch, and unknowns preserved.

2. Routing policy. Dimension scores map to a profile: model tier, effort level, critic tier (normal or best) and review depth. Guardrails: safety-critical classes never route below a floor, and Root or a human can override, with the override recorded. The policy is versioned, and only a reviewed PR changes it.

3. Recording in real dispatch. Root's dispatches (the Agent tool's subagent type and model and effort, or the coordinator equivalent) record classification, chosen profile and propensity on the Delivery and in telemetry, joined by Delivery identity.

4. Evaluation against telemetry:

   • Outcomes: tokens and wall time per Delivery, review rounds, blocking findings, repair commits, CI failures, acceptance, and 30-day regressions.
   • (a) Retrospective calibration on existing attributed telemetry: do hardness scores predict review rounds and tokens?
   • (b) Prospective learning feasible at our volume: a contextual bandit (Thompson sampling, from FS.GG.Algorithms N20) with a small, bounded exploration rate inside each safety class, logged propensities, and off-policy evaluation (IPW or doubly robust) for proposed policy changes.
   • (c) a periodic report that can propose a policy version; it can't activate itself.

5. Scope separation: LEARN-02 does not depend on the LEARN-01 pod and seal infrastructure. It reuses the telemetry store and the rubric, and records how it relates to LEARN-01's RCT. Coordination-specific models are captured in framework Atlas.

Deliveries

Filed as children.

Refs #4647, Refs #5171