Skip to content

feat(docs-audit): audit, fix and gate a repo's documentation - #46

Merged
jtomaszewski merged 2 commits into
mainfrom
jtomaszewski/docs-audit-skill
Aug 25, 2026
Merged

feat(docs-audit): audit, fix and gate a repo's documentation#46
jtomaszewski merged 2 commits into
mainfrom
jtomaszewski/docs-audit-skill

Conversation

@jtomaszewski

@jtomaszewski jtomaszewski commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Adds the docs-audit skill — point it at any repo to grade its documentation and agent
instructions, apply the mechanical fixes, and leave the judgment calls as a ranked proposal.

The problem

Agent instructions fail silently. An agent loads the AGENTS.md files from the repo root down
to its working directory and stops once the combined size hits its project-instruction
budget — Codex's default project_doc_max_bytes is 32,768 bytes. Everything past that offset
is dropped from the prompt with no warning and no truncation notice.

So a rule at the bottom of a 33 KB file is not low-priority, it is absent — and because
the Never list and the validation commands are conventionally at the bottom, those are
exactly the rules that go missing. The symptom nobody connects to the cause: an agent that
"ignores the conventions", and a human who writes the rule a third time, further down.

Everything else the skill checks has the same shape — a documented command that no longer
exists, a CLAUDE.md that forked from its AGENTS.md two bytes ago, a doc no index points
at. All invisible until someone measures.

What the skill does

Seven phases: resolve the target and who owns it → measure → diagnose → fix what is mechanical
→ propose what is judgment → install the CI gate → report. Ownership decides how far it goes:
it fixes our repos, defers to a repo's own stated conventions where those are deliberate, and
for an upstream we don't own it proposes only — a drive-by reformat of a maintainer's
AGENTS.md is how a contribution gets closed unread.

Write by default (mechanical fixes applied, judgment calls proposed with a diff); --audit-only
writes nothing.

It hands off rather than duplicating: an unreadable spec goes to /spec-polish, a ticket to
/ticket-polish, shipping to /deliver.

Files

  • scripts/doc-audit.sh — read-only, ~1s on a typical repo. Ten sections: instruction-budget
    chains, CLAUDE.md/AGENTS.md wiring, the ## Skill profile, commands that no longer resolve,
    dead links, unindexed docs, spec convention, state-vs-record candidates, size outliers, and
    whether any of it is CI-enforced. Two sections are labelled heuristics; the other eight are
    arithmetic.
  • references/conventions.md — the rubric, each rule carrying the failure it was bought
    with, so an edit can be argued for rather than asserted.
  • templates/check-agents-md-budget.mjs — the gate the skill installs into the target repo.
    Root hard limit plus a per-chain ratchet: chains under budget grow freely, a chain already
    over may only shrink, a new over-budget chain fails outright. Existing debt freezes visibly
    instead of hiding.

Verification

  • doc-audit.sh ran clean (exit 0) against 17 checkouts, from 2-markdown-file repos to
    one with 1,546. Targets bash 3.2 (the macOS default), so no nested case inside $( ) and no
    associative arrays.
  • Two real bugs found and fixed while testing: a symlinked AGENTS.md was being sized as the
    9-byte symlink rather than its target, which silently zeroed a whole repo's chain totals; and
    sort | head was taking SIGPIPE under pipefail and killing the run on large repos.
  • The gate additionally crashed with spawnSync git ENOBUFS on a repo with ~40k tracked files —
    execFileSync's default 1 MB buffer, raised by review — reproduced, fixed, re-verified.
  • The gate was fixture-tested through all five states: over budget with no baseline → fails;
    record baseline → passes; nested file grows → fails; nested file shrinks → passes; clean repo →
    passes with no baseline file.
  • Real findings on first contact with our own repos: chains up to 48 KB (a third of the file
    unreachable), a root doc pair that had already drifted two bytes apart, ~90 dead links in one
    repo, dozens of docs no index points at.
  • Repo checks: bash + node syntax, JSON validity, every skill's frontmatter name matching its
    directory, and the version-check workflow's own logic simulated against main (1.0.0 → 1.1.0).

Follow-ups (deliberately not in this PR)

Running the skill on this repo flags two things: no ## Skill profile, and no CI gate on its
own docs. Both worth a separate PR rather than widening this one.

Adds the `docs-audit` skill: grade a repository's docs and agent
instructions against the house conventions, apply the mechanical fixes,
propose the judgment calls, and optionally install the CI gate.

The failure it targets is silent. An agent loads AGENTS.md from the repo
root down to its working directory and stops at 32,768 bytes; everything
past that offset is dropped with no warning. A rule at the bottom of a
33 KB file is not low-priority, it is absent — and the symptom is an
agent that "ignores the conventions" while a human writes the rule a
third time, further down.

- `scripts/doc-audit.sh` — read-only, bash 3.2, ~1s on a typical repo:
  instruction-budget chains, CLAUDE.md/AGENTS.md wiring, the Skill
  profile, commands that no longer resolve, dead links, unindexed docs,
  spec convention, state-vs-record candidates, size outliers, whether
  anything is CI-enforced. Verified against 17 checkouts.
- `references/conventions.md` — the rubric, with the failure each rule
  was bought with, so an edit can be argued for rather than asserted.
- `templates/check-agents-md-budget.mjs` — the gate the skill installs
  into the target repo: root hard limit plus a per-chain ratchet, so
  existing debt freezes visibly instead of hiding.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new docs-audit skill to this skills marketplace/plugin, intended to measure and improve a target repo’s documentation/agent-instruction health (budget overflow, CLAUDE/AGENTS wiring drift, stale commands, dead links, unindexed docs, spec conventions), with an optional CI “budget ratchet” gate template.

Changes:

  • Adds the docs-audit skill procedure (skills/docs-audit/SKILL.md) plus its rubric (references/conventions.md).
  • Adds a read-only auditing script (skills/docs-audit/scripts/doc-audit.sh) and a CI gate template (skills/docs-audit/templates/check-agents-md-budget.mjs).
  • Updates top-level docs and plugin metadata (README skill table + .claude-plugin/plugin.json version bump).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
skills/docs-audit/SKILL.md New skill instructions for running the audit, applying mechanical fixes, proposing judgment calls, and optionally installing the gate.
skills/docs-audit/templates/check-agents-md-budget.mjs Node-based CI gate template enforcing root + chain budget limits with a baseline “ratchet”.
skills/docs-audit/scripts/doc-audit.sh Read-only bash script that reports deterministic doc/instruction measurements and link/index findings.
skills/docs-audit/references/conventions.md Rubric backing the audit with rationale tied to concrete failure modes.
README.md Adds docs-audit to the skills table and repo-agnostic list.
.claude-plugin/plugin.json Bumps plugin version to 1.1.0 and updates description to mention docs audit.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread skills/docs-audit/templates/check-agents-md-budget.mjs Outdated
Comment thread skills/docs-audit/SKILL.md Outdated
…oss-reference

git ls-files exceeds execFileSync's 1 MB default on a large repo — 40k
tracked files is ~2 MB of paths — and the gate died with ENOBUFS before
it could measure anything. Reproduced, then fixed with an explicit
64 MB buffer.

Also points the client-repo bullet at Hard rule 4 (deliberate conventions
beat the rubric) instead of 5.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@jtomaszewski
jtomaszewski merged commit 4db96e1 into main Aug 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants