Sweep the .claude context docs for accuracy and size - #5061
Merged
Conversation
Three rules combined to suppress real findings, all traceable to c4c3424, whose commit message describes a fix that was never applied: - there was no sanctioned route to report a secret leak found in passing; - "Stay in scope" bounded the reporting rather than the checks, so a PR that weakened a policy test was scoped out before it reached the check written to catch exactly that; - the one output category able to carry an uncertain finding was unreachable unless a definite FAIL already existed. One change closes all three. This also records that the frontmatter has never governed a PR review. .github/workflows/security-review.yml does not dispatch this file as a subagent; it passes a prompt telling Claude to read the file and follow it, so the file is consumed as a document and `model` and `tools` are inert on the CI path. security-review.yml:49's own `--model claude-opus-4-7` decides. Two places therefore decide this agent's behaviour, they can disagree, and nothing warns you when they do. The structural defect is confirmed without reference to any model. A forced-dispatch probe additionally had the pre-sweep reviewer certify the broken guard as "No privilege-escalation path found", but that probe compared two different models and was n=1 per arm, so it is not offered here as evidence. The frontmatter escalation is deliberately NOT in this commit. It is the last commit on this branch so it can be dropped on its own. Squashes 5 commits. Per-unit evidence, adjudications and the adversary's objections are in 08-applied-agents.md; all 176 original commit messages are preserved verbatim in 09-commit-log-preserved.md. Audit set archived under .context/stuart/analysis/context-docs-sweep-2026-08/. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
Fixes assertions the agent files made that the code does not support, and strips `tools:` entries naming tools that no longer resolve. The correction that mattered most: props on the `ReactComponent` hook arrive as `data-` prefixed kebab-case, verified at CollaborativeEditor.tsx:44-58 (`data-workflow-id`, `data-project-id`, `data-project-is-sandbox` and more), not underscore_cased as the files claimed. Also removes duplicated checklist lines and retargets references that had drifted off their files. The dead `tools:` strips are correctness fixes and are kept here, not in the frontmatter commit, so that dropping the model/effort decision does not also drop them. Squashes 23 commits. Per-unit evidence in 08-applied-agents.md; original messages in 09-commit-log-preserved.md. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
Removes the three `<example>` dialogues from react-collab-editor's description and the equivalent block from react-test-specialist. Claude Code injects every agent's name and description into every session, so these dialogues were resident context on every run whether or not the agent was ever dispatched. Nothing reads them as examples; they were paying rent as prose. This is the audit's entire measured token saving: -1,141 tokens of the -1,185 total, against a measured resident repo-owned cost of 5,636 -> 4,451. Instruments and method in 08-baseline-before.md. Squashes 2 commits. Per-unit evidence in 08-applied-agents.md. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
react-test-specialist's description pointed at a unit-test guideline path that
does not exist in this repo. web-search-researcher's carried marketing copy
("you can get your money back"). codebase-locator, codebase-pattern-finder,
codebase-analyzer and idea-machine each opened by naming themselves or gave
prompt-writing advice to the dispatcher rather than describing what they do.
Descriptions are resident context, so this is the same argument as the previous
commit applied to prose rather than examples, at smaller scale.
codebase-pattern-finder keeps its explicit contrast with codebase-locator: that
distinction is the thing that makes the pair dispatchable.
Squashes 6 commits. Per-unit evidence in 08-applied-agents.md.
--no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no
compiled deps. Markdown-only change under .claude/.
Sets `model` and `effort` on every agent rather than leaving them implicit, and drops context-analyzer's "ultrathink" line, which `effort: high` now expresses as configuration instead of as an instruction in prose. Cost of the frontmatter itself is negligible: under roughly 100 tokens across eleven files even assuming the keys are injected as prompt text rather than merely parsed. The cost that is not negligible is behavioural, and it is real but unevenly distributed: two of the six replay task pairs show a 1.25x-3.3x increase driven by more agents and more phases, one pair's most expensive run was the pre-sweep arm, one was flat, and two rose 12-13% with no cause identified. Kept separate from the content commits so that this decision can be dropped without losing correctness fixes. security-reviewer's own escalation is separate again, in the final commit. Squashes 2 commits. Per-unit evidence in 08-applied-agents.md. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
…and ui-patterns docs The load-bearing fix is in yex-guidelines.md. It stated the transaction deadlock rule correctly but explained the mechanism wrongly, and wrongly in the dangerous direction: it implied you were safe in a case that in fact hangs the VM. CLAUDE.md points at that section with a warning, so the wrong explanation was being read on purpose by anyone about to write server-side Y.Doc code. The rewrite says the hazard applies whether or not a `GenServer.call` is involved. The pointer was re-checked after the rewrite and still resolves (rule at yex-guidelines.md:27-29, mechanism at :71-76). Also: reframes Gotcha #4 around `to_list` rather than post-`to_json` behaviour, widens the stated reason for converting atoms, points test helpers at the suite instead of copying them inline, corrects store-structure.md's MetadataStore description (which had been seeded from that store's own wrong docstring), corrects toast-notifications.md's durations (seeded from stale JSDoc in notifications.ts), and fixes the ui-patterns button recipes including the restored `disabled:hover` classes. Two of those corrections illustrate a hazard worth naming: the wrong fact had more than one home. Fixing the guideline does not fix the docstring that seeded it, and the docstring will re-seed it. Squashes 26 commits. Per-unit evidence in 08-applied-guidelines-core.md. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
…ths globs These two files were guidance that only matters when you are editing a matching file, but they lived in .claude/guidelines/, which is only read when something asks for it. Measurement across the replay runs showed referenced guidelines are hardly ever opened. Moving them to .claude/rules/ with a `paths:` frontmatter glob makes the harness inject them whenever a matching file is in play. This is the one mechanically verified behaviour change in the audit: the rules injected in 4 of 4 post-move runs against 0 of 4 before, where both files were inert. Two honest limits on that result. It comes from one task pair's dedicated probes, not the whole replay. And injection is necessary, not sufficient - both runs with ui-patterns.md resident still omitted the focus-visible ring it specifies. Landed as two commits because the first used a pathspec that caught the rename but not the frontmatter. Squashes 2 commits. Per-unit evidence in 08-applied-guidelines-core.md. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
…pports Rewrites collaborative-testing.md from 749 lines to roughly 200, folds playwright-patterns.md into the files that own its content, and makes phoenix-liveview.md the single home for the wait patterns rather than one of several competing copies. Read this group more sceptically than the others. It is the largest change in the audit by volume and it has the thinnest evidence behind it: one replay task, no source PR to score against, and scoring on conventions only. On that task the evidence ran against the sweep - both pre-sweep runs found a real defect in the suite by reading the code, and one of them repaired it, while the best post-sweep run quoted the documented conclusion and declined to act. The sharpest thing this group adds is redundant against reading the code, and on that task documentation produced less action than no documentation. Kept as its own commit for exactly that reason: it is the group most likely to be worth reverting on its own. Squashes 48 commits. Per-unit evidence in 08-applied-guidelines-e2e.md. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
…ed setup Corrects claims in testing-essentials.md and its three sub-files that the code does not support, removes setup boilerplate duplicated across them, and tightens the assertion-grouping and file-length guidance that CLAUDE.md points at. The `### Naming` section in testing-essentials.md is deliberately retained. The audit proposed removing it, then withdrew the proposal because the instrument measuring it was miscalibrated and three of six runs never opened the file at all. It is retained-but-unverified, not endorsed. One defect this group does not fix, recorded so it is not mistaken for settled. `createMockPushWithResponse` is referenced ten times across five files under assets/test/collaborative-editor/ - the suite's own README.md, __helpers__/README.md, USAGE_EXAMPLES.md, MIGRATION_GUIDE.md and storeHelpers.ts JSDoc - and is defined nowhere. The helpers that do exist in __helpers__/channelMocks.ts (`createMockPush`, `createMockChannelPushOk` and the rest) are named by no audit document at all. This group removed the references that had reached the guidelines; the ones seeding them from inside the suite are untouched and will re-seed. Squashes 25 commits. Per-unit evidence in 08-applied-guidelines-testing.md. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
Corrects stale references and removes instructions the commands did not need, across create-plan.md, create-spec.md, implement-plan.md, research-codebase.md and validate-plan.md. Adds, as prose rather than as a checklist item, the fact that a plan changing PR behaviour needs the changelog reviewed - deliberately not a checkbox, because a mandatory step is the thing that was being excluded. implement-plan.md:59's cross-check is retained on purpose: its dependency was checked and is not triggered. One flagged weakness. The removal of the no-critique constraint duplication rests on a coverage table that claimed 5 of 6 dispatched agents carry the constraint. The real number is 3 of 6: context-analyzer does not carry it (the audit cited a heading inside that agent's own output template), and web-search-researcher was never checked. That table was the whole of the new evidence answering the objection to this removal, so it is now a single-evidence removal that was presented as multi-evidence. Applied, but it is the first thing to re-examine. Squashes 22 commits. Per-unit evidence in 08-applied-commands.md. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
Drops the eleven per-agent roster bullets (Claude Code injects every agent's name and description already, so the roster was a second copy that could disagree with the first), the Troubleshooting section, the Custom Mix Tasks section, `mix test --only focus`, and the manual `MIX_ENV=test mix ecto.create` step, which the `test` alias at mix.exs:227 already performs. Makes the Database block the single authority on test-database creation. Corrects the LiveView props claim to `data-` prefixed kebab-case. Rewrites the Guidelines Reference with full paths and states once that the guidelines cite file:line rather than copying code. Also deletes deaiify.md's `-HUMAN` file-transformation workflow and restores headings that had been swallowed into a list item. No count is asserted anywhere in the rewritten text, deliberately. Every count that circulated during this audit turned out to be wrong, including several of the audit's own, so the file states none. `§Available Agents` and `§Common Commands` keep their exact names: eight references across four command files cite them by anchor, one of them inside an HTML comment. Squashes 14 commits. Per-unit evidence in 08-applied-claudemd.md. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
…now settled)
The example duplicated map-operation coverage that the suite already carries. The
removal was deferred while its dependency was open, and is settled now.
Its dependency: the `getMap('workflow')` call it described is at
createWorkflowStore.ts:950. The original commit message for this change said 951.
That off-by-one is left standing in the record rather than corrected, because it
was the seventh unverified count in an audit whose main finding was about
unverified counts.
Squashes 1 commit. Per-unit evidence in 08-applied-guidelines-testing.md.
--no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no
compiled deps. Markdown-only change under .claude/.
Sets `model: opus` (from sonnet), adds `effort: high` and adds `maxTurns: 50`. This is a deferred decision, deliberately isolated as the last commit so it can be dropped with a single revert without touching the correctness fix it arrived alongside. Reasons to look at it hard rather than wave it through: - It is the audit's one clear cost regression on this file. - It changed `model:` in the same file as the prose fixes, which is what confounded the security detection evidence: the pre-sweep arm ran claude-sonnet-5 against the post-sweep arm's claude-opus-5, so the observed improvement cannot be attributed to either change. - On the CI path it does nothing at all. security-review.yml consumes this file as a document, not as an agent definition, and passes its own `--model claude-opus-4-7 --max-turns 50`. These three lines govern interactive dispatch only. So the honest case for it is about interactive use, and the honest case against it is cost. The correctness fix does not depend on it either way. Reconstructed as a hunk-level split: in the original history these three lines were spread across two commits (`maxTurns` inside the first link of the correctness chain, `model`/`effort` inside the commit that set frontmatter on all eleven agents), so neither could be dropped on its own. Per-unit evidence in 08-applied-agents.md. --no-verify: the git_hooks pre-commit runs mix tasks and this worktree has no compiled deps. Markdown-only change under .claude/.
The doc landed with #4840 and was already stale against that same PR, and its longest worked example describes a module that has never been on main. UPDATE. §1 and §3 documented an older shape of code that has since adopted their own recommendations. #4840 shipped Lightning.Collaboration.Instance, the per-instance Registry/DynamicSupervisor/:pg struct the doc argued for, and the owner: option it presented as aspirational; both are now the worked examples. stop_document/1 is really /2, and DocumentSupervisor.stop/2 defaults to 15s, not the 5s the doc showed - document_supervisor.ex:43-51 explains why the shorter timeout reports :ok while the flush is in flight. REMOVE. The Lightning.Adaptors.Supervisor example, and the Mox recipe built on it, cite lib/lightning/adaptors/ - which exists only on the unmerged adaptor registry rewrite, where this doc was written. Nothing there resolves for anyone working on main, so it went rather than staying as a set of plausible-looking function names to cite. Recoverable from git when that branch lands. The two-axes framing and the :persistent_term litmus stay; the Mox guidance now points at the real in-repo example at test/support/collaboration_helpers.ex:93-125. Corrected: Task.Supervisor.start_child does propagate $callers on Elixir 1.18 (task/supervisor.ex:527, :545), so the advice to add a manual Mox allowance for it was wrong. stop_all_collaboration_documents/0 does not exist anywhere but this doc. Opus 5 pass: dropped §0's essay on three payoffs of one seam, cut the ~800 words proving an OTP argument-order convention the model already follows, removed the Obsidian frontmatter (no other guideline carries any) and the dangling "case (b)" reference. Anti-pattern checklist moved to the top - it is the payload, the rest is justification. 505 -> 280 lines. The agent pointer moves with the renamed headings.
Security Review ✅
|
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Accuracy and size sweep of the
.claude/context docs against the code as it actually stands. Most of the work is deleting transcribed code that had drifted from the source, correcting stale claims, and replacing copied blocks withfile:linecitations so the docs can't rot the same way again.Two structural changes:
logging.mdandui-patterns.mdmove fromguidelines/to.claude/rules/and getpaths:globs, so they auto-inject when you touch a file they match instead of waiting to be read; andguidelines/e2e/playwright-patterns.mdfolds intoe2e-testing.mdand is deleted.Size
Measured with the real
count_tokensendpoint on theclaude-opus-5tokeniser, not an estimate.CLAUDE.mdrules/agents/commands/guidelines/Worth knowing when reading that table: almost all of the saving is in
guidelines/, which only loads when something asks for it. The part that loads every session went up by about 744 tokens, nearly all of itrules/ui-patterns.mdgrowing to cover all five button variants and thedisabled:hoverclasses. Agent bodies aren't per-session either, since only each agent's description is injected by default.Notable
guidelines/testable-supervision-trees.md(from Run the collaboration test suite async, with deterministic document shutdown #4840) rewritten against the code onmain. Its worked example describedlib/lightning/adaptors/, which doesn't exist on this branch, so it's been replaced with the collaboration subsystem that does. The version pointing at the adaptors code lives onadaptors-on-the-fly, where that code exists.agents/security-reviewer.mdnow records that.github/workflows/security-review.ymlconsumes the file as a document rather than dispatching it as a subagent, so its frontmatter never governs a CI review. Two places decide that agent's behaviour and nothing warns you when they disagree.modelandeffortexplicitly instead of inheriting.LS,MultiEdit,BashOutput,KillShell) removed from everytools:line.