Repository navigation
perf: cut the bodies of the most-invoked skills through progressive disclosure - part 1 of 2 #12243
Description
Part 1 of 2 of #12240, split at intake (owner Q93). The done-when below is copied from #12240 as written.
Done when
- Each of the six skills has its body cut through progressive disclosure: action-specific detail moves to reference files read on demand, and the hub keeps routing and gates. Each cut ships with eval evidence of no loss, as the skill criteria require.
Metadata
Metadata
Assignees
Labels
Type
Fields
Work status
confirmed
Activity
github-actions commentedon Nov 2, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: this is part 1 of 2 of #12240. It asks for progressive-disclosure cuts to six high-use skill bodies. Each cut needs eval evidence of no loss. The hub keeps routing and gates, and action detail moves to on-demand reference files.
The latest host-run comment says the worktree cut could not be measured by the current eval suite, so that skill's done-when item is still open. Until the suite can measure that cut, the before text stays. No duplicates found among open issues.
Automated triage, applied by the lanes App.
github-actions commentedon Nov 2, 2026
24 remaining items
kyle-sexton commentedon Nov 4, 2026
Host run: did not count. The negative control for #12387's worktree body cut, preregistered in #12243 (comment). No model ran, so this says nothing about whether the suite can detect a loss, and the cut's status is unchanged.
What ran (Claude Code 2.1.318, subscription login, the claude.slice scope, under the host's heavy-run gate at load 1.92):
git archivecopies of origin/main 284b530. The before arm has perf(source-control): move the worktree nesting measurement out of the skill hub #12387's change (339ce4a) reverted underplugins/source-control/skills/worktree/only. The patch applied with no fuzz, andSKILL.mdis byte-identical to the tree before the merge. docs: replace restated upstream facts with pointers in evals, source-control #12720 later changedcontext/audit.mdandfixtures/README.md; those stay as on the base in both arms. The degraded arm is the before copy with the worktreeSKILL.mdbody deleted and its frontmatter kept.diff -rqshows the two differ only in that file.claude plugin eval <copy> --case 'worktree-*' --runs 2 -j 2 --scaffold --ablation none --keep-temp --model claude-opus-5-5 --allow-tools Bash Write Edit --trust-plugin --no-publish --threshold 0 --max-cost-usd 5on both arms, launched together at 04:44:00Z. Both ended at 04:44:02Z with exit 0 and a reported cost of 0.run-validity.py --runs 2 --scaffoldedon before; the same plus--own-copy-search-warnson degraded.Verdicts:
verdict: INVALID (16 run errors; permission denials unchecked in 16 runs with no readable trace; skill firing unchecked in 16 runs)verdict: INVALID (16 run errors; permission denials unchecked in 16 runs with no readable trace; skill firing unchecked in 16 runs)a git directory of the plugin registers more linked worktrees than can be screenedCause: a setup defect, not a case or the skill. The copies sat under the repository's gitignored
.work/, so the CLI resolved this repository as the plugin's git directory, and this repository has more linked worktrees than the CLI will screen (449 listed bygit worktree list). The plugin-eval skill's route for this refusal is a copy outside any git repository.Result: did not count. No means, no case drops and no noise report, since none of the 32 runs took a turn. The preregistration's one re-run for a named setup defect has not been used. It needs the copies outside any git repository, with the suite, arms, flags and threshold unchanged.
Target labels: add none; remove none.
kyle-sexton commentedon Nov 4, 2026
Host run: did not count (second attempt). The negative control for #12387's worktree body cut. This used the one re-run the preregistration allowed, for the setup defect the first attempt named. The re-run is now used, and the cut's status is unchanged.
What ran (Claude Code 2.1.318, subscription login, launched together at 05:04:36Z, load 7.69, three other evals on the host):
diff -rqshows the arms differ only in the worktreeSKILL.md, and that file in the before arm is byte-identical to the tree before the merge.docker-links.py, joining the swap peers already held. No flag changed.Verdicts:
INVALID, for three reasons:cost_ceiling, reported cost above--max-cost-usd 5), soworktree-status-batched-factsnever ran.worktree-create-routes-through-helperrun 1 tried a Grep of the sandbox'sconfig/settings.json.gh-forge-path"not loaded" in 14 runs. This follows from the partial run: that plugin is the helper inside the case that never ran, and the degraded arm loaded it in all 16 runs.suite.pluginsmatches between the arms.INVALIDon one counted denial,worktree-cleanup-emits-branch-deleterun 1 (a compoundgrep ... /proc/self/mountinfo; git branch -d). The preregistration allows leaving that case out on this side.noise-report.pyrefused the compare:partial baseline (cost_ceiling): it did not finish, so no comparison is computed.Means, which do not count and route nothing: on the 7 shared cases, before 0.911 and degraded 0.821, a fall of 0.089; with the denial case left out, 0.896 and 0.833, a fall of 0.063.
Result: did not count. Neither preregistered outcome applies, so no restore and no close follow from this.
Cause of the ceiling: the 3.2 USD per arm estimate came from the plugin-eval skill's fresh-suite anchor, and
docs/upstream/opus-5-5-task-cost.md:283already records that those anchors predate the Opus 5.5 price change. A fresh preregistration follows.Target labels: add none; remove none.
kyle-sexton commentedon Nov 4, 2026
Negative control, second preregistration, posted before its first paid run. The attempts under the first preregistration did not count, and their means were seen: they are in the report above and do not count here.
Everything stands as in the first preregistration (suite, arms on base 284b530, model, threshold, compare, what counts, outcome), plus this attempt's setup: copies under
~/.ev/c12387, the CLI run throughdocker-links.py. One change:--max-cost-usd 12in place of5on both arms. Basis: the before arm passed 5 USD after 7 of 8 cases, so about 0.77 USD per case and over 6 USD for the suite, with headroom. This cap protects plan usage and bills nothing.Stop rule: this is the last attempt. If it does not count either (including a before-arm denial, which the first preregistration does not excuse), the cut counts as unmeasurable by this suite and the before text ships by PR under
rewrite-a-plugin.md:79. No further run follows. Detect (acase drop, or a mean fall of at least 0.15, holding with degraded-denial cases left out): #12243 closes after its comments are read in full. Not detect: the before text ships the same way.Target labels: add none; remove none.
kyle-sexton commentedon Nov 4, 2026
Host run: did not count (last attempt). The negative control for #12387's worktree body cut, under the second preregistration. Its stop rule applies: the cut counts as unmeasurable by this suite, and the before text ships by PR under
rewrite-a-plugin.md:79. No further run follows.What ran (Claude Code 2.1.318,
--max-cost-usd 12, everything else as preregistered; both arms finished in full):verdict: VALID. Every case scored 1.00 on both runs.verdict: INVALID (the skill did not fire in 1 with-arm run of a should-trigger case):worktree-not-pr-lifecyclerun 1. This arm's traces hold no denial, and the preregistrations excuse only denials, so leaving that case out after seeing the numbers would be the post-hoc exclusion they exist to stop.Numbers, which do not count:
noise-report.pycalls a case drop onworktree-cleanup-surfaces-removal-errors(1.00 to 0.50), and the mean falls 0.106 (1.000 to 0.894). In the second attempt the before arm scored that same case 0.5 on one run, so a 0.5 is inside its spread at 2 runs. These do not show the suite detects a loss.Result: did not count. The restore PR follows, and it links here.
What this leaves open: a worktree body cut the suite can measure, so #12243's done-when is not met for that skill.
Target labels: add none; remove none.
github-actions commentedon Nov 4, 2026
The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).
melodic-automation-lanes commentedon Nov 4, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 1 (eval: "Each of the six skills has its body cut through progressive disclosure: actio..."); no done-when item is left for the lane, so it goes to the host queue (
host-run). Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 4, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 4, 2026
Closing: five of the six skills meet the done-when, and the sixth moves to its own issue by the owner's decision.
/source-control:commit(#12704),/source-control:pull-request(#12739),/session-flow:handoff(#12869),/evals:plugin-eval(#12875),/discovery:research(#12876). These are as read earlier on this issue from each PR body and the clean re-runs; I did not re-run them./source-control:worktree. Its control did not count on any of three attempts, so the before text shipped in PR 13095. Owner, 2026-11-04: "Go with recomemndations", on keeping that restore. The cut this suite can measure is tracked on perf(source-control): cut the worktree skill body once its suite can detect a loss #13238.The after-figure re-measure on #12244 is no longer blocked by this issue. It measures five cut skills and one uncut.