feat(docs-audit): audit, fix and gate a repo's documentation - #46
Merged
Conversation
Adds the `docs-audit` skill: grade a repository's docs and agent instructions against the house conventions, apply the mechanical fixes, propose the judgment calls, and optionally install the CI gate. The failure it targets is silent. An agent loads AGENTS.md from the repo root down to its working directory and stops at 32,768 bytes; everything past that offset is dropped with no warning. A rule at the bottom of a 33 KB file is not low-priority, it is absent — and the symptom is an agent that "ignores the conventions" while a human writes the rule a third time, further down. - `scripts/doc-audit.sh` — read-only, bash 3.2, ~1s on a typical repo: instruction-budget chains, CLAUDE.md/AGENTS.md wiring, the Skill profile, commands that no longer resolve, dead links, unindexed docs, spec convention, state-vs-record candidates, size outliers, whether anything is CI-enforced. Verified against 17 checkouts. - `references/conventions.md` — the rubric, with the failure each rule was bought with, so an edit can be argued for rather than asserted. - `templates/check-agents-md-budget.mjs` — the gate the skill installs into the target repo: root hard limit plus a per-chain ratchet, so existing debt freezes visibly instead of hiding. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
There was a problem hiding this comment.
Pull request overview
Adds a new docs-audit skill to this skills marketplace/plugin, intended to measure and improve a target repo’s documentation/agent-instruction health (budget overflow, CLAUDE/AGENTS wiring drift, stale commands, dead links, unindexed docs, spec conventions), with an optional CI “budget ratchet” gate template.
Changes:
- Adds the
docs-auditskill procedure (skills/docs-audit/SKILL.md) plus its rubric (references/conventions.md). - Adds a read-only auditing script (
skills/docs-audit/scripts/doc-audit.sh) and a CI gate template (skills/docs-audit/templates/check-agents-md-budget.mjs). - Updates top-level docs and plugin metadata (README skill table +
.claude-plugin/plugin.jsonversion bump).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| skills/docs-audit/SKILL.md | New skill instructions for running the audit, applying mechanical fixes, proposing judgment calls, and optionally installing the gate. |
| skills/docs-audit/templates/check-agents-md-budget.mjs | Node-based CI gate template enforcing root + chain budget limits with a baseline “ratchet”. |
| skills/docs-audit/scripts/doc-audit.sh | Read-only bash script that reports deterministic doc/instruction measurements and link/index findings. |
| skills/docs-audit/references/conventions.md | Rubric backing the audit with rationale tied to concrete failure modes. |
| README.md | Adds docs-audit to the skills table and repo-agnostic list. |
| .claude-plugin/plugin.json | Bumps plugin version to 1.1.0 and updates description to mention docs audit. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…oss-reference git ls-files exceeds execFileSync's 1 MB default on a large repo — 40k tracked files is ~2 MB of paths — and the gate died with ENOBUFS before it could measure anything. Reproduced, then fixed with an explicit 64 MB buffer. Also points the client-repo bullet at Hard rule 4 (deliberate conventions beat the rubric) instead of 5. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the
docs-auditskill — point it at any repo to grade its documentation and agentinstructions, apply the mechanical fixes, and leave the judgment calls as a ranked proposal.
The problem
Agent instructions fail silently. An agent loads the
AGENTS.mdfiles from the repo root downto its working directory and stops once the combined size hits its project-instruction
budget — Codex's default
project_doc_max_bytesis 32,768 bytes. Everything past that offsetis dropped from the prompt with no warning and no truncation notice.
So a rule at the bottom of a 33 KB file is not low-priority, it is absent — and because
the
Neverlist and the validation commands are conventionally at the bottom, those areexactly the rules that go missing. The symptom nobody connects to the cause: an agent that
"ignores the conventions", and a human who writes the rule a third time, further down.
Everything else the skill checks has the same shape — a documented command that no longer
exists, a
CLAUDE.mdthat forked from itsAGENTS.mdtwo bytes ago, a doc no index pointsat. All invisible until someone measures.
What the skill does
Seven phases: resolve the target and who owns it → measure → diagnose → fix what is mechanical
→ propose what is judgment → install the CI gate → report. Ownership decides how far it goes:
it fixes our repos, defers to a repo's own stated conventions where those are deliberate, and
for an upstream we don't own it proposes only — a drive-by reformat of a maintainer's
AGENTS.mdis how a contribution gets closed unread.Write by default (mechanical fixes applied, judgment calls proposed with a diff);
--audit-onlywrites nothing.
It hands off rather than duplicating: an unreadable spec goes to
/spec-polish, a ticket to/ticket-polish, shipping to/deliver.Files
scripts/doc-audit.sh— read-only, ~1s on a typical repo. Ten sections: instruction-budgetchains,
CLAUDE.md/AGENTS.mdwiring, the## Skill profile, commands that no longer resolve,dead links, unindexed docs, spec convention, state-vs-record candidates, size outliers, and
whether any of it is CI-enforced. Two sections are labelled heuristics; the other eight are
arithmetic.
references/conventions.md— the rubric, each rule carrying the failure it was boughtwith, so an edit can be argued for rather than asserted.
templates/check-agents-md-budget.mjs— the gate the skill installs into the target repo.Root hard limit plus a per-chain ratchet: chains under budget grow freely, a chain already
over may only shrink, a new over-budget chain fails outright. Existing debt freezes visibly
instead of hiding.
Verification
doc-audit.shran clean (exit 0) against 17 checkouts, from 2-markdown-file repos toone with 1,546. Targets bash 3.2 (the macOS default), so no nested
caseinside$( )and noassociative arrays.
AGENTS.mdwas being sized as the9-byte symlink rather than its target, which silently zeroed a whole repo's chain totals; and
sort | headwas taking SIGPIPE underpipefailand killing the run on large repos.spawnSync git ENOBUFSon a repo with ~40k tracked files —execFileSync's default 1 MB buffer, raised by review — reproduced, fixed, re-verified.record baseline → passes; nested file grows → fails; nested file shrinks → passes; clean repo →
passes with no baseline file.
unreachable), a root doc pair that had already drifted two bytes apart, ~90 dead links in one
repo, dozens of docs no index points at.
namematching itsdirectory, and the
version-checkworkflow's own logic simulated againstmain(1.0.0 → 1.1.0).Follow-ups (deliberately not in this PR)
Running the skill on this repo flags two things: no
## Skill profile, and no CI gate on itsown docs. Both worth a separate PR rather than widening this one.