Skip to content

refactor(tools): consolidate skill lint registries and checks - #203

Merged
mdsmithaustin merged 5 commits into
mainfrom
refactor/skill-lint-registries
Oct 13, 2026
Merged

mdsmithaustin merged 5 commits into
mainfrom
refactor/skill-lint-registries

Conversation

@mdsmithaustin

@mdsmithaustin mdsmithaustin commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

What Changed

  • Centralize skill-reference exemptions in tools/skill-references.py; make the command fail on violations and stale exemptions, and recognize a skill's own slash name in frontmatter descriptions.
  • Remove the duplicate auto-invocable registry, reuse the playbook registry, and check documented playbook counts and README skill rows against shipped files.
  • Expand reference-lint coverage and update porting and upstream-sync instructions for the consolidated checks.
Validation

Risk Assessment

✅ Low: This is a refactor of lint and test registries. On the real tree, violations() and stale_exemptions() both still return empty. The 15 own-name description hits that the removed "Use for /x" passages used to cover are now exempt only when they appear inside the parsed frontmatter description value. Every intent item is present: the description scoping including folded continuations, a stale_exemptions test, digit and word playbook counts, the PORTING row fixes, and the README row in the workflow list.

Testing

Inspected the change and configured tasks, passed the focused count-parser unit test, and confirmed a clean worktree. No scenarios were driven live and no product evidence artifacts were produced; previously declined restrictions remain in effect.

  • Live validation: ⚠️ inconclusive - 0 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Reference checking permits own-name frontmatter descriptions and folded continuations while rejecting body, sibling, and dollar references ⏸️ untested no Inspected the executable interface and disposable-repository test setup. Running the checker against disposable inputs would still execute a prohibited linter. This phase lacks authority to do so, and…
Reference checking detects stale exemptions and preserves results on the real tree ⏸️ untested no Both disposable-input validation and real-tree comparison require executing the prohibited reference checker. Earlier decisions declined lifting this restriction; changing the environment cannot suppl…
A mismatched implicit-invocation registry row or policy file is rejected ⏸️ untested no The real consumer is skill-ci check, which this phase prohibits as static analysis. Disposable policy files would not remove that restriction. Execution requires authorization already declined in earl…
Documentation registry checks recognize digit and spelled counts and reject incorrect counts, missing skill rows, and incorrect playbook links ⏸️ untested no The focused count-parser unit test passed, but it is not live validation. Driving the documentation checks against altered registries would execute static checks prohibited in this phase. Earlier deci…
Maintainers receive revised worklist substitutions and README registry maintenance instructions ⏸️ untested no These changes are documentation and workflow prompt text with no independently runnable product interaction in this phase. Reading the text does not prove an agent follows it, and executing the upstre…
  • Outcome: ⚠️ 1 warning across 1 run (1m16s)

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

⚠️ **Test** - 1 warning
  • ⚠️ live validation verdict: inconclusive (0 of 5 scenarios were driven live against the product); untested: Reference checking permits own-name frontmatter descriptions and folded continuations while rejecting body, sibling, and dollar references, Reference checking detects stale exemptions and preserves results on the real tree, A mismatched implicit-invocation registry row or policy file is rejected, Documentation registry checks recognize digit and spelled counts and reject incorrect counts, missing skill rows, and incorrect playbook links, Maintainers receive revised worklist substitutions and README registry maintenance instructions
  • Live validation: ⚠️ inconclusive - 0 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Reference checking permits own-name frontmatter descriptions and folded continuations while rejecting body, sibling, and dollar references ⏸️ untested no Inspected the executable interface and disposable-repository test setup. Running the checker against disposable inputs would still execute a prohibited linter. This phase lacks authority to do so, and…
Reference checking detects stale exemptions and preserves results on the real tree ⏸️ untested no Both disposable-input validation and real-tree comparison require executing the prohibited reference checker. Earlier decisions declined lifting this restriction; changing the environment cannot suppl…
A mismatched implicit-invocation registry row or policy file is rejected ⏸️ untested no The real consumer is skill-ci check, which this phase prohibits as static analysis. Disposable policy files would not remove that restriction. Execution requires authorization already declined in earl…
Documentation registry checks recognize digit and spelled counts and reject incorrect counts, missing skill rows, and incorrect playbook links ⏸️ untested no The focused count-parser unit test passed, but it is not live validation. Driving the documentation checks against altered registries would execute static checks prohibited in this phase. Earlier deci…
Maintainers receive revised worklist substitutions and README registry maintenance instructions ⏸️ untested no These changes are documentation and workflow prompt text with no independently runnable product interaction in this phase. Reading the text does not prove an agent follows it, and executing the upstre…
  • Inspected git diff 33d6cd05eca66ff8be64aaa9909871ac6236f7f1 99f9417293619a83a7fb75c05e272e609f0c3812 and repository task configuration.
  • PYTHONDONTWRITEBYTECODE=1 python3 -m unittest -v evals.promises.test_static.TestStaticPromises.test_a_stated_playbook_count_reads_digits_and_words completed successfully with one test executed; failure signal was a nonzero exit or unittest failure.
  • git status --short confirmed no transient worktree changes.
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@mdsmithaustin

Copy link
Copy Markdown
Owner Author

Verdict: FAIL
Head: 361e7c3
Patch-id: cd41048e84983e0d85da0ffcd8869a18a29a9d58
Docs: needs changes

Round 1 swarm at this head: gates + independent mutation proof + merge simulation (sonnet@high), structure review and test-quality review (opus@xhigh each; work model sonnet@high).

Gates: lint 5/5, skill-ci 0 failed, ledger audit 0 problems; only known host noise. Mutations: every removed or moved check still fails on its defect (self-load, stale passage, policy flip, wrong count, unlisted playbook, missing README row); the count and README-row checks are new coverage. Differential: on the real tree the head scanner classifies all 155 hits exactly as base. AUTO_INVOCABLE's rule is enforced by skill-ci on every PR. Merges clean with main and with #198, #201, #202, #205.

Fix-forward (in scope):

  1. tools/skill-references.py:73 own-name exemption keys on line text starting with description:, not on the frontmatter value: a body line description: then run /swarm again becomes exempt (base flagged it), and a folded description: >- (skills/make-bot-ui/SKILL.md:3-6 is folded today) with "Use for /make-bot-ui" on the continuation fails. Exempt the skill's own /name inside the parsed frontmatter description value only; tests for both shapes.
  2. stale_exemptions() has no test of its own (return [] passes the suite). Add one that fails on a stale passage.
  3. evals/promises/test_static.py:131-133 playbook count reads only spelled-out numbers: "24 playbooks" in README passes. Read digits too.
  4. PORTING.md new arena row says arena moves its phases under a ## Start heading; PR fix: isolate arena runs and clean up candidate artifacts #201 (landing first) restores upstream's one-line worklist sentence, so drop that clause. The worklist row also misses poteto-help:48 ("into the worklist", upstream "into the todo list") and figure-it-out:33 ("Add its steps to the todolist").
  5. .github/workflows/upstream-sync.yml:102-115 list of skill-naming lists omits the README all-skills table that the new README-row test enforces; add it.

Notes (no change): the playbook count fails loudly on some harmless rewordings; principle rows in README stay unchecked (pre-existing); violations() and stale_exemptions() each rescan the tree.

@mdsmithaustin

Copy link
Copy Markdown
Owner Author

Verdict: PASS+NOTES
Head: 10ce30b
Patch-id: fafd2ce06d67ec4e39731d0191ba6a0bcf012617
Docs: pass

Round 2 swarm at this head: gates + mutations + scanner differential + merges (sonnet@high), fix review and sync-instruction review (opus@xhigh each; work model sonnet@high).

Gates pass (only known host noise). Every round-1 mutation still fails on its defect; the four round-1 findings now behave as intended (body description: line flagged, folded description exempt, stale_exemptions tested, digit counts read). On the real tree the head and base scanners classify all 149 hits identically. The hand rebase lost nothing from main (#199's owners test intact). Merges clean with main and #201; on the merged tree PORTING's arena and worklist rows match the arena text. A simulated upstream sync adding a skill, a playbook and a principle fails loudly with fix-naming messages.

Notes (follow-ups, not blocking): #212 (principle exclusion wording in the sync bullet and test message; three lists step 3 does not name; untracked-file lint order; a hard-coded count in the workflow).

Merge order: after #201, so PORTING's arena row lands together with the text it describes.

…violations()

A skill's own /name is exempt only on its frontmatter description line, which
replaces 13 of the 17 self-reference passages. The CLI prints what the test
fails and drops the ignored --sigils flag.
skill-ci already fails when a skill's implicit_allowed row disagrees with its
agents/openai.yaml, so the set repeated the trigger file.
… README rows

The playbook list existed twice and the spelled count was unchecked. Every
non-principle skill now needs a README row.
…a differences

Reword the sibling-skill load row, drop AUTO_INVOCABLE from the notes and the
sync prompt, and add substitution rows for todolist to worklist and arena
Phase C's capability tier.
…ue and close three test gaps

The sigil lint exempted any line that began with description:, so a body line
passed and a folded description failed. It now exempts a skill's own /name only
inside the frontmatter description value. stale_exemptions() gets its own test,
the playbook count reads digits, PORTING.md records the todo list and todolist
substitutions, and the sync workflow lists the README all-skills table.
@mdsmithaustin
mdsmithaustin force-pushed the refactor/skill-lint-registries branch from 10ce30b to 99f9417 Compare October 13, 2026 03:43
@mdsmithaustin mdsmithaustin changed the title refactor(tools): consolidate skill lint registries refactor(tools): consolidate skill lint registries and checks Oct 13, 2026
@mdsmithaustin
mdsmithaustin merged commit ea8459b into main Oct 13, 2026
7 checks passed
@mdsmithaustin
mdsmithaustin deleted the refactor/skill-lint-registries branch October 13, 2026 03:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant