Repository navigation
code-tidying: make the remove-comments body rules measurable #12703
Description
Refs: #12474
Problem
The eval suites can't detect a loss in the remove-comments and audit body rules, so #12549 shipped those rules with the gap disclosed, on the owner's decision on #12474.
Measured
Deleting each skill's procedure steps barely moved its outcome suite, against a 0.15 detection line:
dissolve-commentsbody control: mean fall 0.010, no case drop.audit-comment-residuebody control: mean fall 0.083, no case drop.
15 of 21 same-contract cases scored the same across before, after and the control. tree-sitter does not reach the eval sandbox, so every arm only proposes deletions, and an applied deletion is never measured.
Done when
- tree-sitter, or an equivalent the skill's deletion path accepts, runs in the eval sandbox, or the cases exercise applied deletions another way.
- A negative control on each skill's body detects a loss (a
case dropor a mean fall of 0.15 or more). - Before and after, measured with those controls, show no loss in the body rules feat(code-tidying): rename dissolve-comments to remove-comments with a comment-reviewer agent #12549 shipped. If they show a loss, it's fixed or the before text returns.
Activity
github-actions commentedon Nov 3, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: the eval suites for
dissolve-commentsandaudit-comment-residuecan't detect a loss in the body rules shipped in #12549. Deleting the procedure steps moved the outcome suites by less than the 0.15 line. The work is to get tree-sitter or an equivalent running in the eval sandbox, or to exercise applied deletions another way. It also needs a negative control on each skill's body that does detect a loss, and a before/after measurement with those controls. Nothing is missing that blocks triage. I found no duplicate among the open issues I checked. This is related to #12474.Automated triage, applied by the lanes App.
github-actions commentedon Nov 3, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
melodic-automation-lanes commentedon Nov 3, 2026
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 3, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: confirmed; its comment says why.kyle-sexton commentedon Nov 3, 2026
Claim. Lane 5 host run is building this now (R29, oldest unclaimed buildable host-run item after #12242, which is date-gated to 2026-11-04T19:13Z).
test(code-tidying): install the comment tooling for each interpreter …
test(code-tidying): require the tooling only for the scaffold's python3
test(code-tidying): skip a fallback python3 that does not run