Repository navigation
test(build_tools): check every reference the procedure text makes - #2747
Merged
Merged
Conversation
build_tools/check_skill_refs.py scans CLAUDE.md, .claude/skills/*/SKILL.md,
.claude/skills/README.md, .claude/agents/*.md, docs/process/README.md and
docs/design/verification-contract.md, and goes red when a reference does
not resolve. build_tools/tests/test_check_skill_refs.py runs it in the
existing pytest job (build_tools/tests/).
Census of the reference kinds (counts on this tree):
- path 529: check_plan_refs.PATH_RE (reused, imported), plus what its
grammar cannot spell (extensionless files, directories, editors/), read
from code spans and fences only. Gitignored locations (git check-ignore,
dir form tried too) and family examples (audit-YYYY-MM-DD.md,
plan-a1b2c3.md: a sibling of the same shape exists) are not missing.
- skill 275: /name at a word start or opening a code span; a directory
under .claude/skills/ or BUILTIN_SKILLS = {simplify}, which the test
keeps minimal (each listed built-in must be cited, none a repo skill).
/name:x (old harness namespacing) is red, never skipped.
- section 111: TARGET › Name › Name[, step|item N]; TARGET Name §N;
TARGET item N; §Name and Name step|item|§ N (own file); shorthands
stage trail item N, README/Process item N, Rule N, Dimension N; a
target-less › chain resolves against /create-plan's fenced plan
template. Names are headings, bold lead-ins or numbered headings.
- contract 103: contract §N, §N–§M, verification-contract.md §N.
- link 17: relative targets and #anchors (GitHub slugs).
Out of scope, counted: bare §N/step N/item N (106), charter §N (4), URLs.
Each checked kind has a positive control (count > 0).
Already broken on 15209ec, one:
- .claude/agents/dogfood-analyst.md:21 cited /project:add-test-fixture;
now /add-test-fixture (shorter; not a PROCEDURE_FILES member).
Mutation record (Rule 12): each file copied to a mktemp -d scratch dir,
one reference broken, the tree test run, restored with cp, cmp identical,
run again green ("1 passed").
- path: push/SKILL.md check_commit_subjects.sh -> ..._subjectz.sh. Red:
"push/SKILL.md:26: [path] build_tools/check_commit_subjectz.sh -- no
such file (not gitignored, not a family example)". Restored: green.
- skill: plan-flow/SKILL.md `/verify-plan` -> `/no-such-skill`. Red:
"plan-flow/SKILL.md:7: [skill] /no-such-skill -- no skill
'no-such-skill' under .claude/skills/ and not a listed built-in".
Restored: green.
- section, heading renamed: docs/process/README.md "## The stage trail"
-> "## The stage path". Red: "CLAUDE.md:105: [section] Process item 6
-- no section 'The stage trail' in docs/process/README.md" (and every
other stage-trail citation). Restored: green.
- section, numbered heading: push/SKILL.md "### 1.25." -> "### 1.3.".
First run stayed GREEN: "1.25" prefix-matched "### 1.". Guard fixed
(numbers match whole only) and a fixture case added; rerun red:
"bump-version/SKILL.md:26: [section] `/push` › 1.25 -- no section
'1.25' in .claude/skills/push/SKILL.md". Restored: green.
- section, numbered item: implement-plan/SKILL.md "4. **Independent
review" -> "- **Independent review". Red: "health-watch/SKILL.md:60:
[section] `/implement-plan` › Cleanup and Report step 4 -- no
item/section 4 in ... › Cleanup and Report". Restored: green.
- section, bold lead-in: CLAUDE.md "**Layer ladder:**" -> "**Rung
order:**". Red: "create-plan/SKILL.md:122: [section] CLAUDE.md › Core &
Stdlib Conventions › layer ladder -- no section 'layer ladder' in
CLAUDE.md inside 'Core & Stdlib Conventions'". Restored: green.
- contract: CLAUDE.md "contract §11–§12" -> "§11–§99". Red: "CLAUDE.md:52:
[contract] contract §11–§99 -- no '## 99.' in
docs/design/verification-contract.md". Restored: green.
- link anchor: CLAUDE.md "#core-axioms" -> "#core-axiomz". Red:
"CLAUDE.md:61: [link] ...introduction.md#core-axiomz -- no heading with
anchor #core-axiomz". Restored: green.
- positive control: SKILL_RE, then CONTRACT_RE, made to match nothing
(r"(?!)"). Red: "AssertionError: no 'skill' reference found" / "no
'contract' reference found". Restored: green ("5 passed").
Measured @ this commit's tree on 15209ec: python -m pytest
build_tools/tests/ 2288 passed, 0 failed, 1 skipped (venv Python 3.15);
scorecard_metrics.py --check OK (procedure_bytes 143707, claude_md 9304,
both unchanged).
Co-Authored-By: Claude Fable 5.2 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LGMKZWuQhE7ZvVZDE3FdDs
Fix round on check_skill_refs.py after review (PR 2747); history kept.
1. Links resolve as GitHub does: against the citing file's directory, a
leading / at the repo root, to a file or a directory; never as a
/skill. Duplicate headings get GitHub's -1, -2 slugs.
2. Number lists and ranges are checked whole (Dimensions 2, 3 and 9,
Dimensions 1-5, steps 8 and 9, contract §1, §10, a chain then §N).
3. Prose is not a citation: Name step|item|§ N is read only when Name is
a section of the citing file (else counted bare; known limit in the
docstring: an own-file citation whose heading was renamed is counted,
not red). Contract/Charter are read at a sentence start too.
4. Skills: sentence-end /name., **/name**, ``/name`` are read.
5. `/x` §Name and /x Name step N resolve against /x.
6. step/item N: info-string fences do not count, a subsection's items do
not answer its parent, a list counts at its own indent, ### Step N:
headings count; a target with no Name takes an item of any section or
a numbered heading anywhere in it.
7. Info-string fences (```bash, ```markdown) are code: no skill, link,
section or shorthand is read there; links not in inline code. A bare
fence stays prose: in these files it is an agent prompt the reader
follows (/verify-implementation's `/ci-gate` › Control runs).
8. test_every_target_the_check_reads_starts_the_python_workflow: every
file the checker read (scanned files, targets, .gitignore) is matched
by RELEVANT_PATHS and push paths; a cited directory by one tracked
file in it. Patterns added to both lists: .gitignore,
docs/architecture.md, docs/design/**, docs/audits/**, the cited files
benchmarks/cross-language/run_benchmarks.py, editors/vscode/
{CHANGELOG.md,package.json,syntaxes/sharpy.tmLanguage.json},
src/Sharpy.Compiler.Tests/{Sharpy.Compiler.Tests.csproj,
CodeGen/EmitterCarrierOnlyConformanceTests.cs,
Project/ProjectConfigWithInMemorySourcesTests.cs},
src/Sharpy.Lsp.Tests/Conformance/FrontEndParityTests.cs, and one
anchor file per otherwise-uncovered cited directory: TestFixtures/
async/async_basic.spy, Properties/PropertyIterations.cs,
Benchmarks/Corpus/hello_world.spy, Sharpy.Playground.csproj,
Sharpy.TestInfrastructure.csproj, Spy/generated/approx_helper_tests.cs.
Each pattern is necessary (dropping any one turns the test red).
Mutation record (Rule 12): each fix reverted in place (the file copied
to a mktemp -d dir first), test_check_skill_refs.py run, restored with
cp, cmp identical, rerun green (92 passed) every time.
- 1 link resolved repo-root first: red
test_a_link_resolves_as_github_does_red[[p](docs/process/README.md#
the-stage-trail)-docs/design/verification-contract.md].
- 1 link resolved like a TARGET: red ...red[[b](/push)-CLAUDE.md] (+1).
- 1 no duplicate-heading suffix: red ...green[link-[x](.claude/skills/
push/SKILL.md#anchored-heading-1)].
- 2 numbers() keeps the first only: 5 red, e.g. ...red[section-Skip
Dimensions 1, 3 and 99.], ...[See `/push` › Cleanup and Report,
steps 8 and 9.], ...[contract-Per contract §1–§9.].
- 2 chain then §N not read: red ...red[section-See `/push` › Output §99.]
- 3 prose Name read as own section: red ...extracts_nothing[Repeat step
2. See §3 for it. ...], test_prose_numbers_fall_to_the_counted_bare_
bucket.
- 3 contract case-sensitive: red ...green[contract-Contract §2 holds.]
- 3 charter case-sensitive: red test_prose_numbers_fall_to_the_counted_
bare_bucket.
- 4 sentence-end . taken as a path: 3 red, ...red[skill-Then run
/no-such-skill.], test_a_builtin_cited_at_a_sentence_end_counts...
- 4 ** not a word start: red ...red[skill-Use **/nope** now.]
- 4 double backtick not a span start: red ...red[skill-See ``/nope``
now.] (+ the tree test).
- 5 `/x` §Name not a target form: red ...red[section-See `/push`
§Testing.]
- 5 bare /skill not a target: red ...red[section-Run /push Critical Rules
step 1.] and ...green[...Cleanup and Report step 8.]
- 6 numbers in info-string fences count: red ...[Cleanup and Report
step 3.]; subsection items count for the parent: red ...[step 4.];
nested items count: red ...[step 5.]; indent 0 only: red
...green[Indented, item 2.]; no Step N heading: red ...green[Steps,
step 2.]; no-name target whole-file only: red ...green[See `/push`
step 8.] and the tree test.
- 7 code fences read as prose: red ...extracts_nothing[```bash ...] and
[```markdown ...]; links read in inline code: red
...extracts_nothing[Index it as `a[i](x)`.]
- 8 docs/design/** dropped from both lists: red test_every_target_the_
check_reads_starts_the_python_workflow.
Real tree: green, 0 broken of 1028 (path 529, skill 268, section 111,
contract 103, link 17; bare 101, charter 4). No PROCEDURE_FILES change.
Co-Authored-By: Claude Fable 5.2 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LGMKZWuQhE7ZvVZDE3FdDs
Brings in #2746 (skill text) so the reference check runs against it. Co-Authored-By: Claude Fable 5.2 <[email protected]> Claude-Session: https://claude.ai/code/session_01LGMKZWuQhE7ZvVZDE3FdDs
Second review round on check_skill_refs.py (PR 2747); history kept.
A. The workflow-coverage test skips reads with no tracked file at or
under them (gitignored .claude/tmp/, .claude/plans/): no PR can
rename them. Reproduced first: with .claude/tmp/ present the test
failed on ".claude/tmp (RELEVANT_PATHS)"; now green with it present.
B. charter §N (any case) is claimed before the own-section pass, and an
own-file Name must equal a section name exactly ("Do" is not
"**Do not**"). Critical Rules step 9 stays red.
C. /word is no skill citation inside a link target ([d](/docs)), mid
code span (`msbuild /nologo`), or in NOT_SKILLS = {tmp} (a filesystem
root; pinned: each entry is a directory under /, none a skill).
D. has_number never reads a fenced line as a heading (`# Role` in a
prompt); bare-fence items still count.
E. `TARGET [Name] §N|step N|§Name` accepts a bare CLAUDE.md and a
markdown link; CLAUDE.md no longer matches build_tools/CLAUDE.md.
F. Dimensions 4-design, 6, 7 checks 4, 6, 7; step 1.25 checks 1.25;
Step/Item after a target or chain; contract §1–99, §1 and 99.
G. Links whose text wraps one line, or whose target has a "title".
H. New counted OUT_OF_SCOPE kind "filename": slash-less file names in
code spans (sharpy.sln, .python-version), not resolved.
Docstring known limits match the code (own-file renamed or shortened
heading, prefix-cut heading, family example, reference-style links,
code spans split by a hard wrap).
Real tree: 0 broken of 1019 (path 529, skill 262, section 108,
contract 103, link 17; bare 108, charter 4, url 0, filename 185). Lost
to the exact/span-start rules (now bare or not read, none were red):
implement-plan "Cleanup step 4", "Pre-Implementation §2" (x2),
verify-implementation "Cleanup steps 1, 2 and 6", "Cleanup step 4";
seven mid-span /skill mentions.
Mutation record (Rule 12): each fix reverted in place after copying the
file to a mktemp -d dir, test_check_skill_refs.py run, restored with
cp, cmp identical, rerun green (117 passed) every time.
- A untracked reads asserted: red test_coverage_skips_untracked_reads_green
- B charter not claimed first: 3 red, test_out_of_scope_forms_are_counted,
test_prose_numbers_fall_to_the_counted_bare_bucket, ..._and_named
- B own section by prefix: 2 red, ...extracts_nothing[Charter §3 says so.
Do step 3 first. See §2 of the plan. CHARTER §4.] (+1)
- C mid-span /word read: red ...extracts_nothing[Run `msbuild /nologo`,
write to `/tmp`.]
- C link target read as a skill: red
test_a_root_relative_link_is_a_link_not_a_skill
- C NOT_SKILLS ignored: red ...extracts_nothing[... `/tmp`.]
- D fenced # read as a heading: red ...red[See `/push` › Prompt step 7.],
...green[See `/push` › Prompt step 2.]
- E bare CLAUDE.md not a target ahead: red
test_a_target_ahead_is_the_named_file_green[As CLAUDE.md §Testing
says.], test_a_target_ahead_is_the_named_file_red
- E CLAUDE.md matches a path tail: red ...red[See build_tools/CLAUDE.md
› Testing.]
- E link not a target ahead: red ..._green[[c](../../../CLAUDE.md)
§Testing]
- F 4-design ends the list: red ...red[Run Dimensions 4-design, 6.]
- F 1.25 read as 1: red ...red[See `/push` step 1.3.]
- F Step N not read: red ...red[`/push` › Cleanup and Report, Step 9.]
- F contract continuation needs §: red ...red[contract §1–99.] and
[contract §1 and 99.]
- G link text may not wrap: red ...red[[a\nb](docs/missing.md)] (+1)
- G link title not read: red ...red[[x](docs/missing.md "a title")] (+1)
- H file names not counted: red
test_slash_less_file_names_in_code_are_counted
Co-Authored-By: Claude Fable 5.2 <[email protected]>
Claude-Session: https://claude.ai/code/session_01LGMKZWuQhE7ZvVZDE3FdDs
Third review round on check_skill_refs.py (PR 2747); history kept. 1. `step N.M` resolves to a heading numbered N.M; where the target has no N.x heading at all it is checked as N (property-stress step 4.2, Dimension 3.5, Rule 1.5). `/push` step 1.3 stays red (### 1.25). 2. After a contract citation only a tight range reads a §-less number (§1–99); `, 40` and `and 15` are prose again. 3. TARGET_AHEAD_RE is back on lowercase step/item (a `Step 7` on the next line is prose); the chain tail keeps Step/Item. 4. LINK_RE is LINK_TARGET: one pattern, "title" included, so a titled link works as a chain or §Name target. 7. Fixed (2 lines): the plan template's fenced body is read for numbers when resolve() asks for fences, so `Defect Class › Violated contract, item 1` can resolve. 8. Left: fence_lines and prose_of index lines alike only if Doc.lines and code_regions change too (three lines, not one). Tests only: NOT_SKILLS is pinned to a fixed tuple of POSIX root names; the dead half of the Critical Rules red case is cut. Docstring: missed forms vs. forms that can go red on fine text, and a gitignored path is any .gitignore match, not only a session's file. Real tree: 0 broken of 1019, counts unchanged (path 529, skill 262, section 108, contract 103, link 17). Mutation record (Rule 12): each fix reverted in place after copying the file to a mktemp -d dir, test_check_skill_refs.py run, restored with cp, cmp identical, rerun green (124 passed) every time. - 1 N.M never falls back to N: red ...green[See `/push` step 8.2; Dimension 3.5; Rule 1.5.] - 2 round-2 continuation restored: red test_fine_text_is_not_red[Per contract §2, 40 agents run.] and [Per contract §11 and 15 rows.] - 2 tight §-less range removed: red ...red[contract-Per contract §1–99.] - 3 STEP_ANY after a target: red test_fine_text_is_not_red[Use `/push` \nStep 7 of the build is next.] - 4 LINK_TARGET without the title: 4 red, incl. ...green[[p](docs/ process/README.md "t") › The stage trail.] - 7 template body skipped for numbers: red ...green[Read Defect Class › Violated contract, item 1.] Co-Authored-By: Claude Fable 5.2 <[email protected]> Claude-Session: https://claude.ai/code/session_01LGMKZWuQhE7ZvVZDE3FdDs
Docstring only: the module docstring already says it; the helper's own line still called every ignored path a session's file. Co-Authored-By: Claude Fable 5.2 <[email protected]> Claude-Session: https://claude.ai/code/session_01LGMKZWuQhE7ZvVZDE3FdDs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The skills, the agents,
CLAUDE.md, the process README and the verification contract cite each other by repo path, by skill name, by section (`/implement-plan` › Cleanup and Report step 4,contract §12,`/push` › 1.25) and by markdown link. Nothing checked that those targets still exist, so a renamed heading or a moved file left a reference pointing at nothing.build_tools/check_skill_refs.pymakes that red, andbuild_tools/tests/test_check_skill_refs.pyruns it in the existingbuild_tools/tests/pytest job.What it checks (census @ 2e7b52e: 0 broken of 1019)
check_plan_refs.PATH_RE(imported, not re-written), plus extensionless files and directories inside code spans and fences. A path any.gitignorerule matches, or a family example (plan-a1b2c3.md,audit-YYYY-MM-DD.md), is not a missing file; a glob or placeholder is never extracted./nameat a word start or opening a code span is a directory under.claude/skills/or the one listed harness built-in (/simplify); a test keeps that list minimal. A link target ([d](/docs)) and a filesystem root (`/tmp`) are not skills.TARGET › Name › Name[, step/item N],TARGET Name §N, the citing file's own§Name/Name step N, and four shorthands from the census (stage trail item N,README/Process item N,Rule N,Dimension N). A target is a backticked skill or.mdpath, a markdown link, or a bareCLAUDE.md. A name is a heading, a bold lead-in or a numbered heading; every number of a list or range is checked.contract §N,§N–§M,§N–M: a## N.heading of the contract./);#anchoris a heading slug.A fence with an info string is code; a bare fence is read as prose, because here it holds agent prompts whose steps are cited.
Out of scope, counted and named in the checker: a bare
§N/step N/item Nwith no target (108),charter §N(4), absolute URLs (0), a slash-less file name in a code span such assharpy.sln(185). A›the grammar cannot parse is a failure, never a skip, and each checked kind has a positive control (the test fails if its count is zero).Already broken on
dev, one.claude/agents/dogfood-analyst.md:21cited/project:add-test-fixture, the old namespaced spelling; now/add-test-fixture. Not a ratcheted file.Workflow trigger widened
The checker reads files the
Python Build Toolsworkflow did not start on, so a pull request renaming one would merge green and turn the next pull request red..github/workflows/python-build-tools.ymlgains, in bothRELEVANT_PATHSand the pushpaths:.gitignore,docs/architecture.md,docs/design/**,docs/audits/**,benchmarks/cross-language/run_benchmarks.py, threeeditors/vscode/files, and ten single files undersrc/that the procedure text cites by path. They are single-file patterns, not directory globs, so an ordinarysrc/change still does not start the Python job.test_every_target_the_check_reads_starts_the_python_workflowfails when the checker reads a tracked file neither list covers.Known limits (the module docstring carries the same two lists)
Missed, never red:
Cleanup step 4), counts as a bare number;[r]: path; none exist in the scanned files);, N/and Nwithout§after a contract citation;step N.Mwhere the target has noN.xheading is checked asN;/namethat does not open its code span;CLAUDE.mdfiles and the/batch-issuestemplates are not scanned.Can go red on text that is fine (the cure is to reword the text):
›chain whose target is outside the repository (~/.claude/CLAUDE.md › Testing);/alone, or with%20in its path.Mutation record (Rule 12)
Each file was copied to a scratch directory, one reference or one fix broken, the test run, the file restored with
cpand checked withcmp, and the test run again green. The commit bodies have the full red lines.### 1.25.→### 1.3.stayed green on the first run; the matcher now matches numbers whole), a numbered item, a bold lead-in, a contract range, a link anchor, and the positive controls.Measured
check_skill_refs.py@ 2e7b52e: 0 broken of 1019.python -m pytest build_tools/tests/@ f6d9ea2: 2364 passed / 0 failed / 1 skipped (the skip needs a builtsharpyc; no dotnet was run locally). 2e7b52e differs from it by one docstring line; the checker's and the workflow-relevance tests were re-run there, 178 passed.scorecard_metrics.py --check@ 2e7b52e: OK; this pull request touches no ratcheted file.Owner's words (2026-11-19, in this session)
Items 1 to 3 are #2746, merged.
Independent review
Independent review @ da5154c: merge after fixes; 8 applied, 9 declined (reasons below); fixes da5154c..2e7b52e read by the lead
Three reviews ran, each
/code-review(high) in a context that had the diff and not this session's reasoning. Each returns findings, not a verdict; "merge after fixes" is the lead's reading. The first two fix rounds each needed the next review (the first restructured the checker, the second changed 106 lines outside tests). The third changed 40, added no guard or abstraction, and had none. Its last commit, 2e7b52e, is one docstring line written by the lead; the rest of each fix range was written by a subagent and read by the lead.Third review @ da5154c (of the second fix round). Nothing wrong on the real tree. Applied:
step 4.2, meaning item 2 of step 4, went red after the second round.N.Mis a headingN.Mwhere the target numbers headingsN.x, otherwise it is checked asN.contract §2, 40 agents) was read as a section. Only a tight range (§1–99) continues without§.Step 7at the start of the line after a target was read as a citation. A capitalisedStepis read only in a chain's tail./create-plan's template could never resolve. The template's fenced body is read for numbers too.test_not_skills_stay_filesystem_rootsdepended on the machine's/. It compares against a fixed list.Declined:
~/.claude/CLAUDE.md › Testingis red. The target is outside the repository and cannot be checked; none exists in the scanned files; named in the limits./or with%20is red. None exists; named in the limits..gitignorepatterns pass a stale path underbin/orlib/. The rule is "ignored is never tracked"; the docstring now says how wide that is.build_tools/CLAUDE.mdand the/batch-issuestemplates are not scanned. See the first review's declined item./namein the middle of a code span is not read. Reading it mademsbuild /nologored; named in the limits.splitlines()andsplit("\n")disagree on\f,\vand U+2028. Inferred only; no scanned file holds one; the consistent change touches three places.Second review @ 5066544. Grouped into eight fixes, all applied in 761b1fe: the workflow-coverage test failed wherever gitignored
.claude/tmp/existed; prose read as an own-file citation;/wordthat is not a skill;#lines in a bare fence read as headings; a bareCLAUDE.mdor a link as the target of§Name; number forms (4-design,1.25,§1–99); wrapped and titled links; slash-less file names counted. Declined: reference-style link definitions (none exist), code-span pairing across a wrap, a narrower family-example rule.First review @ d07568a. Nine applied in 093f6f0: links resolved from the repo root before the citing file's directory; only the first number of a list checked; prose false reds; missed skill forms; the wrong target for
`/skill` §Name; the workflow-trigger gap above; loose number matching; fences read as prose; link forms. Declined, one: scanning nestedCLAUDE.mdfiles, becausebuild_tools/CLAUDE.md:42cites.claude/.sharpy_auto_builder/checkpoints/, which is neither tracked nor gitignored, and deciding which it should be is not this lane's call.🤖 Generated with Claude Code
https://claude.ai/code/session_01LGMKZWuQhE7ZvVZDE3FdDs