From ecf09ee3b3949c6dbc0859c570a461239650ebe9 Mon Sep 17 00:00:00 2001 From: dch0202 Date: Wed, 12 Aug 2026 23:32:04 +0900 Subject: [PATCH 1/3] fix(wiki-lint): make check 2's prohibition rule mechanically enforceable Replaces the "don't outside Instead-of" judgment call with a ported, corpus-validated parser (scripts/wiki-lint-prohibitions.js): a bare prohibition alone in its directive item (table cell or bullet) is a violation; one paired with a replacement or mechanism, or living in an Instead-of row, is compliant; a bare 2-word cell is an undecidable blind spot surfaced at info (new check 11), never error. Closes #36. 0 violations / 61 directives on wiki/, matching the reference probe at .orchestration/evidence/i36-rule-probe.js. AGENTS.md rule 3 and wiki-lint check 2 restated to match, in the same words, both naming the script. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_0199ew7RNxbFAiiWaegbXuDP --- AGENTS.md | 19 ++-- scripts/wiki-lint-prohibitions.js | 171 ++++++++++++++++++++++++++++ skills/wiki-lint/SKILL.md | 11 +- tests/fixtures/prohibitions/bad.md | 11 ++ tests/fixtures/prohibitions/good.md | 22 ++++ tests/wiki-lint-prohibitions.bats | 119 +++++++++++++++++++ 6 files changed, 342 insertions(+), 11 deletions(-) create mode 100644 scripts/wiki-lint-prohibitions.js create mode 100644 tests/fixtures/prohibitions/bad.md create mode 100644 tests/fixtures/prohibitions/good.md create mode 100644 tests/wiki-lint-prohibitions.bats diff --git a/AGENTS.md b/AGENTS.md index 76a5ff9..0b48dbb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,11 +72,16 @@ Every page uses `templates/page.md`. Non-negotiable rules: 1. **One case per page.** A page answers one situation. If you are writing "and also…", split the page and cross-link under `related`. 2. **≤ 120 lines of body.** Precision beats coverage. Link, don't inline. -3. **Positive guidance only.** Every directive is "In situation X, do Y". - Anti-patterns may only appear in the `Instead of` table, where each row MUST pair - the anti-pattern with its replacement action. A "don't" without an "instead" is a - lint failure — a prohibition with no replacement invites the reader to improvise, - which is how hallucinations happen. +3. **Positive guidance; a prohibition must pair.** Every directive is "In situation X, + do Y". A prohibition (`don't` / `do not` / `never` / `avoid` / `must not`) may + appear anywhere in a page, as long as the same directive item also carries its + replacement action or the mechanism that makes the prohibition true. The + `Instead of` table remains the place for anti-pattern/replacement pairs; each row + there MUST still pair the anti-pattern with its replacement action. The unit of + pairing is the directive item — a table cell or a bullet — not the page as a + whole. A bare prohibition, alone in its item, is a lint failure: a prohibition + with no replacement invites the reader to improvise, which is how hallucinations + happen. Enforced by `node scripts/wiki-lint-prohibitions.js`. 4. **No vague qualifiers.** Words like "usually", "consider", "might want to", "generally", "as appropriate" are banned in directive sentences. State the condition that decides it: "When X, do A. When Y, do B." If you cannot state the @@ -126,8 +131,8 @@ Run these via the skill files, which contain the full step-by-step workflows: merge into existing pages before creating new ones, cite sources, update indexes and `log.md`. - **Query** (`skills/wiki-query/SKILL.md`) — answer a question from the wiki with citations; if the answer required synthesis across pages and is re-askable, file it as a new page. -- **Lint** (`skills/wiki-lint/SKILL.md`) — health check: unsourced claims, "don't"s without - "instead"s, banned vague qualifiers, orphan pages, broken links, stale `last_verified`. +- **Lint** (`skills/wiki-lint/SKILL.md`) — health check: unsourced claims, unpaired + prohibitions, banned vague qualifiers, orphan pages, broken links, stale `last_verified`. Two further skills use the wiki to run development work (rather than maintain the wiki): diff --git a/scripts/wiki-lint-prohibitions.js b/scripts/wiki-lint-prohibitions.js new file mode 100644 index 0000000..21c6740 --- /dev/null +++ b/scripts/wiki-lint-prohibitions.js @@ -0,0 +1,171 @@ +#!/usr/bin/env node +// wiki-lint check 2 — mechanically enforceable prohibition-with-replacement rule. +// +// SCOPE : page body only (no frontmatter), and NOT the `## Sources` section +// (citation text quotes prohibitions and is not addressed to the reader). +// UNIT : one directive item — a table cell, or a bullet/numbered item +// (multi-sentence allowed; wrapped lines rejoined). +// DIRECTIVE : a clause in that unit that BEGINS with a prohibition token AND is +// >= 3 words long (a 2-word cell like "Never read" is a state value +// in a data column, not an instruction to the reader). +// `never-fails` (hyphenated compound) is not a token. +// COMPLIANT : the unit is an `## Instead of` row, OR the unit carries at least one +// other clause (>= 1 word) — the replacement action, or the mechanism +// that makes the prohibition true. +// INFO : a unit that is nothing but a bare 2-word prohibition clause (no other +// content) — the declared blind spot (D5). Ambiguous by shape between a +// state value (`Never read`) and a real directive (`Never retry`), so it +// is reported for a human to look at, never as a checker error. +// +// Ported from the validated rule at .orchestration/evidence/i36-rule-probe.js — do +// not re-derive the parse; it was measured against the live corpus (61 directive +// units, 0 violations) before this script existed. +'use strict'; + +const fs = require('fs'); +const path = require('path'); + +const TOKEN = "(?:don't|do not|never|avoid|must not)"; +const STARTS = new RegExp(`^${TOKEN}(?![\\w-])`, 'i'); +const ANY = new RegExp(`(? s.replace(/[*`_]/g, '').trim(); +const words = (s) => s.split(/\s+/).filter(Boolean).length; + +function walk(dir, out = []) { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const p = path.join(dir, entry.name); + if (entry.isDirectory()) walk(p, out); + else if (entry.name.endsWith('.md')) out.push(p); + } + return out; +} + +// Reassemble a page body into directive items: one per table cell, or per +// bullet/numbered item with its wrapped continuation lines rejoined. +function collectItems(body) { + let section = ''; + let buf = []; + const items = []; + const flush = () => { + if (buf.length) { + items.push({ text: buf.join(' ').replace(/\s+/g, ' ').trim(), section }); + buf = []; + } + }; + + for (const raw of body.split('\n')) { + if (/^#{1,3}\s+/.test(raw)) { + flush(); + section = raw.replace(/^#{1,3}\s+/, '').trim(); + continue; + } + if (raw.trim().startsWith('|')) { + flush(); + if (/^\|[\s|:-]+\|$/.test(raw.trim())) continue; // table separator row + raw + .split('|') + .slice(1, -1) + .forEach((cell) => cell.trim() && items.push({ text: cell.trim(), section })); + continue; + } + if (!raw.trim()) { + flush(); + continue; + } + if (/^\s*(?:\d+\.|[-*])\s+/.test(raw)) { + flush(); + buf.push(raw.replace(/^\s*(?:\d+\.|[-*])\s+/, '')); + } else { + buf.push(raw.trim()); + } + } + flush(); + return items; +} + +function lint(files) { + let directiveUnits = 0; + let okInstead = 0; + let okPair = 0; + const violations = []; + const infos = []; + + for (const file of files) { + const raw = fs.readFileSync(file, 'utf8'); + const body = raw.replace(/^---\n[\s\S]*?\n---\n/, ''); + const items = collectItems(body); + + for (const { text, section } of items) { + if (/^Sources$/i.test(section)) continue; + const unit = strip(text); + if (!ANY.test(unit)) continue; + + const clauses = unit + .split(CLAUSE_SPLIT) + .map((c) => c.trim()) + .filter(Boolean); + const directives = clauses.filter((c) => STARTS.test(c) && words(c) >= 3); + + if (!directives.length) { + // Known blind spot (D5): a unit that is nothing but a bare 2-word + // prohibition clause is undecidable between a state value and a real + // directive. Report it at info; never treat it as a violation. + if (clauses.length === 1 && STARTS.test(clauses[0]) && words(clauses[0]) === 2) { + infos.push(`${file}:${section}: ${unit}`); + } + continue; + } + + directiveUnits++; + if (/^Instead of$/i.test(section)) { + okInstead++; + continue; + } + if (clauses.some((c) => !directives.includes(c) && words(c) >= 1)) { + okPair++; + continue; + } + violations.push(`${file}:${section}: ${unit}`); + } + } + + return { directiveUnits, compliant: okInstead + okPair, violations, infos }; +} + +function main() { + const target = process.argv[2] || 'wiki'; + + let stat; + try { + stat = fs.statSync(target); + } catch { + stat = null; + } + if (!stat) { + console.error(`wiki-lint-prohibitions: no such directory: ${target}`); + process.exit(2); + } + const files = stat.isDirectory() ? walk(target) : [target]; + + const { directiveUnits, compliant, violations, infos } = lint(files); + + if (violations.length) { + console.log('--- violations ---'); + violations.forEach((v) => console.log(v)); + } + if (infos.length) { + console.log('--- info: bare 2-word prohibition cells (state value or directive — undecidable by shape) ---'); + infos.forEach((i) => console.log(i)); + } + + console.log('--- summary ---'); + console.log(`directives: ${directiveUnits}`); + console.log(`compliant: ${compliant}`); + console.log(`violations: ${violations.length}`); + console.log(`info: ${infos.length}`); + + process.exit(violations.length ? 1 : 0); +} + +main(); diff --git a/skills/wiki-lint/SKILL.md b/skills/wiki-lint/SKILL.md index 1b54968..67ca333 100644 --- a/skills/wiki-lint/SKILL.md +++ b/skills/wiki-lint/SKILL.md @@ -21,7 +21,7 @@ Run all of these; report findings grouped by severity. | # | Check | Severity | |---|-------|----------| | 1 | Page with `confidence: verified` but empty/unverifiable `sources:` | error | -| 2 | "Don't/never/avoid" directive outside an `Instead of` table, or an `Instead of` row missing its replacement | error | +| 2 | A prohibition (`don't`/`do not`/`never`/`avoid`/`must not`) alone in its directive item (table cell or bullet), carrying no replacement action or mechanism — checked via `node scripts/wiki-lint-prohibitions.js`; `Instead of` rows must still pair the anti-pattern with its replacement | error | | 3 | Broken `related:` id or inline link | error | | 4 | Page not listed in its domain `index.md`, or index entry whose "load when" line no longer matches the page trigger | error | | 5 | Vague qualifiers in directive sentences (usually, consider, might, generally, as appropriate) | warn | @@ -30,13 +30,16 @@ Run all of these; report findings grouped by severity. | 8 | `last_verified` older than 12 months on `verified` pages (docs move, defaults change) | warn | | 9 | `contradiction` entries in `log.md` still unresolved | warn | | 10 | `gap` entries in `log.md` with no page created after 30 days | info | +| 11 | Bare 2-word prohibition cell (e.g. `Never read`) — undecidable by shape between a state value and a real directive, so it is surfaced rather than judged; reported by `node scripts/wiki-lint-prohibitions.js` | info | ## Fix protocol - Fix mechanical findings (3, 4, 6 splits, index lines) directly. -- For 1 and 2: fix when the correct source/replacement is known with certainty; - otherwise downgrade to `unverified` / move the prohibition into `Instead of` with a - `TODO replacement` marker and report it — do not invent sources or replacements. +- For 1: fix when the correct source is known with certainty; otherwise downgrade + to `unverified` and report it — do not invent sources. +- For 2: add the replacement action or the mechanism in place, in the same + directive item; moving the row into `Instead of` is one option, not the required + one. Do not invent a replacement — report it if none is known with certainty. - For 5: rewrite the sentence as a conditional ("When X, do A") only when the condition is stated elsewhere in the page; otherwise report it. - Append `## [YYYY-MM-DD] lint | errors fixed, reported` to `log.md`. diff --git a/tests/fixtures/prohibitions/bad.md b/tests/fixtures/prohibitions/bad.md new file mode 100644 index 0000000..ef53476 --- /dev/null +++ b/tests/fixtures/prohibitions/bad.md @@ -0,0 +1,11 @@ +--- +title: bad fixture (negative control) +--- + +# Bad fixture + +## Do this +- Never log secrets. + +## Sources +- Never log secrets, per the vendor incident report (accessed 2026-08-12). diff --git a/tests/fixtures/prohibitions/good.md b/tests/fixtures/prohibitions/good.md new file mode 100644 index 0000000..9d5d8fa --- /dev/null +++ b/tests/fixtures/prohibitions/good.md @@ -0,0 +1,22 @@ +--- +title: good fixture +--- + +# Good fixture + +## Do this +- Never retry a 500 without a backoff — the server may be transiently overloaded. +- Do not cache the response. The upstream marks it no-store. + +## Instead of +| Anti-pattern | Replacement | +|---|---| +| Retrying blindly | Never retry blindly — always confirm the response is safe to retry | + +## Notes +A function that never-fails still needs input validation. + +## State values +| Key | Behavior | +|---|---| +| Unknown key | Never read | diff --git a/tests/wiki-lint-prohibitions.bats b/tests/wiki-lint-prohibitions.bats new file mode 100644 index 0000000..e5b5b0d --- /dev/null +++ b/tests/wiki-lint-prohibitions.bats @@ -0,0 +1,119 @@ +#!/usr/bin/env bats +# Tests for scripts/wiki-lint-prohibitions.js (wiki-lint check 2). +# +# The checker's own report is not evidence it works until it has been shown to +# fail on something — a checker observed only ever returning 0 violations proves +# nothing (wiki/testing/quality/checks-that-cannot-pass.md). tests/fixtures/ +# prohibitions/bad.md is that negative control: a bare prohibition the checker +# must catch. good.md is the paired-case control: every shape the rule allows +# (em-dash pairing, sentence pairing, an `Instead of` row, a hyphenated +# `never-fails` compound, and the D5 bare-2-word-cell blind spot) must NOT be +# reported as a violation. + +setup() { + CHECKER="${BATS_TEST_DIRNAME}/../scripts/wiki-lint-prohibitions.js" + REPO_ROOT="${BATS_TEST_DIRNAME}/.." + FIXTURES="${BATS_TEST_DIRNAME}/fixtures/prohibitions" +} + +# --- normal: the real corpus is already compliant --------------------------- + +@test "real wiki: exits 0 with 0 violations and 61 directive units" { + cd "$REPO_ROOT" || return 1 + run node "$CHECKER" wiki + [ "$status" -eq 0 ] + [[ "$output" == *"directives: 61"* ]] + [[ "$output" == *"violations: 0"* ]] +} + +# --- error: the negative control (D6) --------------------------------------- + +@test "fixture dir: exits 1 and names bad.md as the sole violation" { + run node "$CHECKER" "$FIXTURES" + [ "$status" -eq 1 ] + [[ "$output" == *"bad.md"* ]] + # exactly one violation — the Sources quote (same file) must not inflate this. + [[ "$output" == *"violations: 1"* ]] +} + +# --- normal: every paired/permitted shape passes ----------------------------- + +@test "good.md alone: exits 0, no violations" { + run node "$CHECKER" "$FIXTURES/good.md" + [ "$status" -eq 0 ] + [[ "$output" == *"violations: 0"* ]] +} + +# --- boundary: `## Sources` is excluded (D4) --------------------------------- + +@test "a Sources line quoting a bare prohibition is not flagged" { + run node "$CHECKER" "$FIXTURES" + [ "$status" -eq 1 ] + # the Sources bullet repeats "Never log secrets" verbatim; if the Sources + # exclusion broke, violations would be 2, not 1. + [[ "$output" == *"violations: 1"* ]] +} + +# --- boundary: the 2-word blind spot is reported at info, never error (D5) -- + +@test "a bare 2-word prohibition cell is reported under info, exit still 0" { + run node "$CHECKER" "$FIXTURES/good.md" + [ "$status" -eq 0 ] + [[ "$output" == *"info: 1"* ]] + [[ "$output" == *"--- info:"* ]] + [[ "$output" == *"Never read"* ]] +} + +# --- explicit DoD case: the single most common corpus shape (D3) ------------ +# `wiki/backend/common/reliability/timeouts-and-retries.md` pairs a bare +# "Never retry" (2 words) with its reason via em-dash/semicolon. Because the +# prohibition clause itself is under the 3-word directive threshold, the unit +# falls out of `directives` entirely — it is never classified as a violation, +# compliant pair, or info row. rule 3 must still be read as PERMITTING this +# shape (it is never an error); this test is the checked-explicitly record for +# that DoD line, not a claim that the checker recognizes it as a directive. + +@test "the DoD's cited row (bare 2-word prohibition + reason via em-dash/semicolon) is not a violation" { + dir="${BATS_TEST_TMPDIR}/dod-example" + mkdir -p "$dir" + printf -- '---\ntitle: dod example\n---\n\n## Status codes\n| Codes | Behavior |\n|---|---|\n| 400/401/403/404/422 | Never retry — the request itself is wrong; the same bytes fail again |\n' > "$dir/row.md" + run node "$CHECKER" "$dir" + [ "$status" -eq 0 ] + [[ "$output" == *"violations: 0"* ]] +} + +# --- boundary: empty input --------------------------------------------------- + +@test "an empty directory: exits 0, no crash" { + empty_dir="${BATS_TEST_TMPDIR}/empty-wiki" + mkdir -p "$empty_dir" + run node "$CHECKER" "$empty_dir" + [ "$status" -eq 0 ] + [[ "$output" == *"directives: 0"* ]] + [[ "$output" == *"violations: 0"* ]] +} + +@test "a page with an empty body: exits 0, no crash" { + empty_body_dir="${BATS_TEST_TMPDIR}/empty-body-wiki" + mkdir -p "$empty_body_dir" + printf -- '---\ntitle: empty\n---\n' > "$empty_body_dir/empty.md" + run node "$CHECKER" "$empty_body_dir" + [ "$status" -eq 0 ] + [[ "$output" == *"directives: 0"* ]] + [[ "$output" == *"violations: 0"* ]] +} + +# --- error: bad directory argument ------------------------------------------- + +@test "a nonexistent directory: exits 2 with a stderr message" { + run node "$CHECKER" "${BATS_TEST_TMPDIR}/does-not-exist" + [ "$status" -eq 2 ] + [[ "$output" == *"no such directory"* ]] +} + +# --- syntax ------------------------------------------------------------------- + +@test "the script is syntactically valid" { + run node --check "$CHECKER" + [ "$status" -eq 0 ] +} From e94c066132b83ddb699a8a2cc8ad6d13e374754e Mon Sep 17 00:00:00 2001 From: dch0202 Date: Wed, 12 Aug 2026 23:32:31 +0900 Subject: [PATCH 2/3] fix(wiki): disjoint routing scopes for doc-gate and flaky-pair ambiguity Routing probes 6 and 7 found two AMBIGUOUS pairs where two domains claimed near-identical scope, forcing a router to open both pages to decide. Give each pair a disjoint predicate plus mutual inline cross-pointers so the index "load when" line alone resolves it. - Doc-gate cluster (testing/quality <-> qa/document-verification): split on mechanism ("does my check discriminate?") vs acceptance ("is passing the gate enough to accept the deliverable?"). qa/spec-document-gates no longer claims check-authoring or unwritten-target gate patterns. - Flaky pair (testing/flaky <-> debugging/concurrency): split on what the unreliable thing is - a test in your suite vs the system itself - after an initial reproduction-state axis proved undecidable from a bare symptom statement (probe 7) and was revised mid-implementation. - Two minor route fixes: INDEX/backend LLM phrasing no longer attracts ML-training queries; databases index cross-points to the backup/restore page. Closes #37. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01FU4PzRNVigcSwJCKxDTYrj --- INDEX.md | 2 +- log.md | 1 + wiki/backend/index.md | 2 +- wiki/databases/index.md | 3 ++- .../concurrency/intermittent-failures.md | 10 ++++++---- wiki/debugging/index.md | 2 +- .../spec-document-gates.md | 18 ++++++++++++++---- wiki/qa/index.md | 2 +- wiki/testing/flaky/diagnosing-flaky-tests.md | 13 ++++++++----- wiki/testing/index.md | 6 +++--- .../testing/quality/checks-that-cannot-pass.md | 7 +++---- wiki/testing/quality/spec-artifact-checks.md | 10 ++++++---- 12 files changed, 47 insertions(+), 29 deletions(-) diff --git a/INDEX.md b/INDEX.md index cb6cd09..9a24186 100644 --- a/INDEX.md +++ b/INDEX.md @@ -10,7 +10,7 @@ follow the cross-pointers in their index or take the next matching seeded domain | Domain | Status | Route here when | |--------|--------|-----------------| | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior, surveying live data to derive a rule, verifying additive migrations | -| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, call-site enumeration before a contract change, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM completion validation & context budgeting, consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | +| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, call-site enumeration before a contract change, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, consuming LLM APIs (completion validation, context budgeting), consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility | | [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting), per-environment/path-valued config, multi-agent orchestration (worker liveness signals, shared run state, tmux pane delivery, completion gates, worktree-isolated workers) | | [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) | diff --git a/log.md b/log.md index c930fc2..bd066cf 100644 --- a/log.md +++ b/log.md @@ -43,3 +43,4 @@ Append-only. Format: `## [YYYY-MM-DD]