[LEARN-02] Build semantic hardness skill and evaluate 50 items with fresh agents across model/effort profiles #5277

Work level: Delivery · Track: Active delivery · Intended Status: In progress

Owner

Root integrates; one accountable semantic-classifier pilot owner creates the skill, frozen cohort, isolated ratings and analysis. The #5276 owner retains the historical EDA/report; #5170 remains priority for archive deployment.

Done when

Create a meaning-aware agent skill for pre-work hardness assessment that separates task demand from safety consequences and external execution/wait conditions, cites evidence, represents uncertainty, and resists negated/out-of-scope/quoted-keyword inflation. Freeze its version and output schema before evaluation.

Freeze 50 actual FS-GG task snapshots with documented historical boundary, inclusion/strata and missingness; hide outcomes, earlier scores, current completion prose and native identities from rating agents. Preserve independent private outcome/time evidence for evaluation. Do not select examples for agreement or eventual success; open/failed/stopped work stays explicit.

Run each of the same 50 items through requested gpt-6.1-sol/medium, gpt-6.1-sol/high, gpt-6-astra/medium and gpt-6-astra/high using a NEW subagent for EACH item/profile pair with fork_turns=none and only the frozen skill and that item. No agent may rate a second item or inherit previous ratings. Preserve accepted launch identity/configuration, prompt/input/output hashes, timing and available usage; provider-effective facts stay Unknown unless observed. Original uncertain calls are not silently resubmitted. This is 200 primary classification evaluations, not execution of the 50 implementation tasks.

Use reproducible statistical-library analysis for agreement/ranking, profile and effort sensitivity, semantic controls, uncertainty, missingness, coverage and observed-time predictive validity against transparent baselines using task-group/time constraints. Agreement alone does not establish difficulty invariance across actual implementing agents. Keep any prespecified repeats/controls separate from the primary50 cohort and keep the assessment blind. No live card/learned-routing activation follows automatically.

Deliver independently reviewed skill, scripts, aggregate report in reports/, and normal PR/merge. Preserve private inputs/results, full #5248 and #5276 promises and existing authority; this pilot can report inconclusive or unfavorable results.

Current deliverable

The fixed228-call pilot is complete:200 planned primary judgments plus20 semantic and eight repeat controls, each using a distinct fresh context. Primary strict compliance is182/200;17 text-length failures and one controller wrong-input deviation are retained without retries, replacements or relabeling. All28 controls are strict-valid. Broad categories agree strongly on observed valid pairs, but missingness/range limit inference and subjective duration estimates do not establish held-out predictive benefit. The unchanged-content recovery is explicitly post hoc. Final skill, controller, statistical scripts and comprehensive report are independently reviewed at source3fbc7a4a. No live classification/routing policy is activated.

Next action

Root will compose the reviewed source onto the then-current protected base and take the skill, scripts and reports through normal PR/required checks/merge when the archive-priority lane permits. Capture the reusable, scoped findings in Atlas with immutable citations. Further independent validation remains separate from this overlapping development cohort; no additional calls are selected.

Blocked on

Normal source integration is queued behind the user-prioritized permanent archive rollout in #5170. Both hub managed slots are occupied by #5270 and #5278. This source-delivery hold is not a scientific prerequisite for archiving and does not imply completion of the broader #5248 or #5276 promises.

Acceptance evidence

Explicit user request and fresh-subagent instruction in this session; historical foundation #5276 and provisional metadata machinery #5273. Model selection uses currently exposed supported collaboration profiles; accepted requests do not prove provider-effective configuration.

Refs #5248