Repository navigation
perf(discovery): stop research verification looping on format rules it could check at write time #12517
Description
Refs: #6917
Problem
On 2026-11-02 a deep-tier /discovery:research run fanned out six topics (discovery:researcher, Budget: medium, Evidence use: internal). Every discovery:research-verifier pass failed: 6 of 6 first passes and 6 of 6 re-verifies after the one allowed rework. The verifiers re-found every quote at its link and re-derived every date. The failures were format and calibration rules, not wrong facts:
- row 12 restatements: summary lines that drop a qualifier or a confidence label, or widen a claim ("every source");
- row 4 single-source flag eligibility: content claim vs behaviour claim, third-party report vs first-party page;
- row 7 calibration: 2 Tier 1 + 1 Tier 2 is MEDIUM, not HIGH.
skills/research/context/artifact-shape.md already predicts the loop: "Each fresh verifier samples a different restatement, so rework that adds text fails somewhere new". The deterministic check-restatement-labels.py only checks labels, is advisory, and passed on runs whose verifier then failed row 12.
Measurements (seconds, from task notifications)
| Topic | Research | Verify | Rework | Re-verify | Verification share |
|---|---|---|---|---|---|
| slice size | 506 | 337 | 211 | 472 | 67% |
| behaviour representation | 578 | 446 | 205 | 406 | 65% |
| plugin suggestions | 608 | 387 | 150 | 383 | 60% |
| spec, plan, design | 686 | 467 | 94 | 578 | 62% |
| code in plans | 735 | 400 | 127 | 528 | 59% |
| interactive review | 734 | 660 | 188 | 484 | 64% |
About 60-67% of each topic's time went to verify, rework and re-verify, and none converged. Verifier passes that changed a decision: about six (a main-session wake bug, OpenAI ExecPlans' user-visible rule, a wrong install command, two upstream-practice corrections, a false follow-up avoided). Most other findings were label and restatement format.
Fix
- Make restatement drift impossible at write time, not caught at verify time: summaries, Settled lists and handoffs are ID pointers with labels only, and
check-restatement-labels.pyruns fail-closed at the researcher before hand-back. Extend it to flag sweep words ("every", "all", "none", "only") on lines that cite claim IDs. - Declare
claim_kind: content | behaviorand the source owner in each claim header, and check flag eligibility (row 4) and the tier mix for HIGH (row 7) with a script, so the researcher meets them before hand-back. - Scale verification to stakes: under
Evidence use: internal, the verifier grades only the claims a stated decision cites, and gets no rework round; keep the full loop forpublish. - A slice-root synthesis that adds no claims and cites by ID needs no verifier.
Verification
Re-run one topic from this set with the fixes and compare the verify share and the pass rate with the table above.
Related
Session record: .work/planning-chain-research/ (local). #12241 (review agents returning a pointer plus a bounded summary).
This was generated by AI.
Activity
github-actions commentedon Nov 2, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the discovery research verifier fails every topic on format and calibration rules (row 12 restatements, row 4 flag eligibility, row 7 tier mix), not on facts. Verification takes 60-67% of each topic's time and doesn't converge. The proposed fix has four parts. Move those checks to write time with a fail-closed script. Declare claim kind and source owner in claim headers. Scale verification to the stakes under
Evidence use: internal. Skip the verifier for pointer-only synthesis. Both measurements give a clear done-when: re-run a topic and compare the verify share and pass rate. That re-run is a measurement a person or host run would do. I found no likely duplicates among the open issues.Automated triage, applied by the lanes App.
kyle-sexton commentedon Nov 2, 2026
One more data point from the same run: the slice-root synthesis verifier (claim-free synthesis, IDs only) took 461 s and failed rows 12 and 14. Most findings were missing
HIGH (single source)labels on lines that only cite IDs, and judgment not labeled as judgment. One finding was substantive: a decision cited C5 for "types go to the design", but C5's qualifier says Pocock keeps interfaces in the spec. That catch changed the plan's stated basis for D3. Pass rate for the run: 0 of 13 verifier passes.This was generated by AI.
melodic-automation-lanes commentedon Nov 2, 2026
Not admitted to the issue lane: merged pull request #12603 names it with
Refs:after its newest decision or refinement comment, so its work may already be done: close the issue if it is, or post a comment whose first line is**Decision recorded.**saying what is left, which sends it back through intake. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 2, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 2, 2026
Other waste seen in the same 2026-11-02 session, beyond the research verifier loop:
/discovery:trace-intent, 47 claims). Each round failed on the same class: comments under the owner's login that agents wrote, graded as the owner's own words. The rule ("only quoted owner text counts") is clear but checked only by the verifier. Fix: the tracer tags each sourceauthor_kind: owner-quote | agent-relayat write time, and a script rejects a Direct claim whose only source is an agent relay..research-dispatchbaseline at the slice root, whichdeep-tier.mdallows for a fan-out, but each of six researchers reported "no baseline in my sub-slice". That is six open questions the parent had to read and dismiss. Fix: the envelope names the baseline path, and the researcher reads it from there.Paths forward are in the issue body (fixes 1 to 4) plus the three above.
This was generated by AI.
18 remaining items
github-actions commentedon Nov 3, 2026
The triage sweep dispatched intake triage for this issue (attempt 2 of 2).
kyle-sexton commentedon Nov 3, 2026
test(discovery): drop the new test's redundant test-scope header
fix(discovery): run the author-kind check on an inline trace too
fix(discovery): list the author-kind check's allow rules and pin the …
fix(discovery): call the author-kind check by direct path and match s…
kyle-sexton commentedon Nov 3, 2026
Claim. Backlog HITL host-run worker is taking this now.