feat(prior-art): research how others solve it, with every claim sourced - #48
Merged
Conversation
"How do other big players do this?" is the highest-confabulation question we ask an LLM: it returns fluent, sourceless prose about what Stripe/Postgres/ everyone does, and that prose then gets pasted into a spec as justification. No skill covered it — `brainstorm` and `zoom-out` both generate alternatives from inside our own head, and `om-spec-writing` gestures at "research against market leaders" without saying how. The skill's value is the discipline, not the prompt: - every claim about a named system is graded documented / inferred / folklore, and folklore may appear but may never justify a decision; - comparables are chosen by shared constraint, never by fame — the big-player fallacy is this question's main failure mode; - the format can express "they arrange things so the question never arises", "they do it and regret it", and "there is no consensus", which a fill-in-the- options table cannot; - our own repo, specs and tracker are searched first — we have often already decided it; - search queries are outbound to third parties, so they carry no client- identifying detail, same standing rule as the publishing skills; - it closes on a verdict against our constraints plus an explicit unknowns list, sized to drop into the spec/ADR/ticket the user asked for. Read-only throughout. Pairs with `zoom-out` as the other tool for seeing past your own frame — zoom-out gets a fresh context, prior-art gets the outside world.
Ran the skill on five problems on fresh contexts — a DB isolation setting, an e-commerce order-status model, durable background work, a financial ledger, and ERP module structure — and graded what came back. Five findings reproduced on all five runs; each maps to a change here. What held, and is now stated more strongly: - §2 "look inside before you look outside" was the highest-value step every single time — it found decisions we had already made and incidents we had already survived, which outrank anything published outside. Promoted. - "They avoid the situation", "they do it and regret it" and the licence to report no consensus all produced the best findings in at least one run. Kept, with the licence explicitly not an instruction to hedge: one run correctly reported a real consensus where one exists. - "Comparable by constraint, not fame" earned its place, but every run had to derive the ONE axis that actually sorted the evidence rather than being prompted for it. §3 now asks for the axis, not a constraint list. What broke: - The mandated subagent fan-out is the skill's most reliable failure. It exhausted the shared search budget in minutes on every run, leaving nothing for the orchestrator's own checks — in one run the dissent sweep, which a hard rule protects, issued zero queries and its silence was nearly reported as "no dissent found". §4 now budgets the fan-out, runs dissent first, requires a tool-failure to be distinguishable from an absence, and tells you to cross-check sweeps that disagree. - Hard rule 4 was unenforceable through the very mechanism the skill mandates. Sweeps mined the inherited repo CLAUDE.md, inferred the client's ERP product and vendor, and searched for them by name — despite being told to stay generic. The rule now says where the leak comes from and requires the sweep prompts to be sanitized explicitly. - The Documented/Inferred/Folklore ladder is a self-report, and passes a well-formatted wrong attribution straight through: one sweep credited one commerce platform's source code to another under a correctly-applied label. All three runs that hit this invented the same missing category. Grading is now two axes — strength × whether you opened it — and a new §5 requires you to open every source the verdict rests on. - The output contract asked for something "sized to drop into a spec section" and produced a ~4,500-word essay on all five runs, converging within 15%. One document that is both a verdict and an evidence file always becomes the evidence file. Split into two artifacts. Net +223 words: the additions are load-bearing, paid for partly by cutting rhetoric that argued for rules instead of stating them.
…anking
Graded 66 load-bearing citations from the five test reports by fetching every
one: 55 supported, 9 partial, 2 unsupported, 3 dead — 83% strict, 97% counting
partials. Per report, 100% down to 62%; the best run had every numeric default
and limit exact to the digit.
The important result is not the rate, it is that the labels did not predict it,
and for the strongest label the correlation is inverted. Both unsupported
claims and seven of the nine partials sat in the `documented` band. Every
`inferred (I read the source)` claim that could be diffed verified exactly, and
every hedge tested was honestly placed — including one report's hedged reading
of a platform's source, which is correct, while its confidently-cited docs
quote for the same rule does not exist on the page.
The mechanism: citing a page is cheap and looks authoritative, reading source
is expensive enough that nobody claims it falsely, and hedging costs the writer
something so it is rarely false. Labels record how a claim was gathered, not
whether it holds. So:
- Every documented claim in the verdict now carries a VERBATIM QUOTE, not just
a URL. You cannot quote a page that doesn't say it, and the reader checks it
in one click.
- State-bearing claims ("still open", "deprecated", "the default is") carry
what was seen and when. Two claims the reports said they had personally
verified were stale rather than invented — an issue described as "still open"
had been closed for years — and staleness is much harder for a reader to spot
than fabrication.
- §6 says plainly that the grades are not a confidence ranking, and that
reading the source outranks citing a doc page.
Also confirms the fix in the previous commit was aimed correctly but not
sufficient on its own: "opened by you" is itself a self-report, which is why it
now has to produce a quote. No cross-product misattribution survived into any
final text — the one surviving mix-up is docs-vs-source inside a single
product.
… weak ones for failures "Comparable by axis, not by fame" told you which systems' answers transfer, but said nothing about how much a given system's judgment is worth. Those are different questions and the skill conflated them. Maturity is the second weight, and it changes what you read a system FOR: - A team with a strong engineering record chose deliberately, so their choice is evidence about the design. - A legacy or poorly-built system's choice is evidence of nothing — but its failures are, and are often the best evidence available, because a mature system's bugs in the same area were found and quietly fixed years ago while a legacy one's are public, reproducible and still open. The test runs already showed this without naming it: the highest-value finding in the order-status run was two open bugs in the domain's legacy incumbent, where partial shipment plus partial refund silently strands fulfilable inventory. Under a fame-or-quality filter alone that source gets dropped; under this rule it is exactly where you go for failure data. Hence: read mature systems for what they chose, read legacy systems for what broke. A survey drawing only on the admirable ones carries no failure data at all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
"How do other big players do this?" is the highest-confabulation question we put to an LLM. It returns fluent, confident, sourceless prose about what Stripe / Postgres / "most teams" do — and that prose doesn't lose an argument, it gets cited, into a spec nobody can now re-check.
Nothing in the collection covered it.
brainstormandzoom-outboth generate alternatives from inside our own head. The closest thing anywhere is one unmethodical line inom-spec-writing— "research against open-source market leaders" — with no instructions for how.The skill
prior-art— read-only, repo-agnostic. Sharpen the question into a decision → look inside our own repo first → name the one axis that decides transferability → sweep four source classes on a budget → open and quote every source the verdict rests on → grade on two axes → emit a paste-ready verdict plus a separate evidence file.This was tested, not just written
The first draft was run on five problems on fresh contexts — a DB isolation setting, an e-commerce order-status model, durable background work (against open-mercato#5450), a financial ledger, and ERP module structure — then a sixth pass fetched 66 load-bearing citations from the resulting reports and graded them. Every finding below reproduced on all five runs.
What held
READPASTcursor is disallowed under the setting being proposed, so enabling it would partly re-introduce the locking it was meant to escape. No web search yields that.What broke, and what changed
CLAUDE.md, inferred a client's ERP product and vendor, and searched them by name despite being told to stay generic. The rule now names where the leak comes from and requires sanitized sweep prompts.documentedband, while everyinferred (I read the source)claim verified exactly and every hedge was honestly placed. Citing a page is cheap and looks authoritative; reading source is expensive enough that nobody claims it falsely. So: every documented claim now carries a verbatim quote, state-bearing claims ("still open", "deprecated") carry what was seen and when, and §6 says plainly the grades are not a confidence ranking.Side effect: the background-work run found an error in open-mercato#5450 itself — its DBOS claim ("recovers only via hosted Conductor") is too strong per DBOS's own architecture page.
A second weight: engineering maturity
"Comparable by axis, not fame" says whether an answer transfers; it says nothing about what a given team's judgment is worth. §3 now carries both, and the second one changes what you read a system for:
A strong team's choice is evidence about the design. A badly-built system's choice is evidence of nothing — but its failures often are the best evidence available anywhere, because a mature system's bugs in the same area were found and quietly fixed years ago while the legacy one's are public, reproducible and still open. The test runs had already shown this: the best finding in the order-status run was two open bugs in the domain's legacy incumbent.
Re-run on the fixed skill
Two follow-up runs on fresh questions (how mature orgs handle non-blocking OLTP reads; how they run multi-day background work) confirm the fixes fire:
Changes
skills/prior-art/SKILL.md— new skill (1,596 words).README.md— table row, added to the repo-agnostic list..claude-plugin/plugin.json—1.2.2→1.3.0(minor: new skill) + description.Open questions for review
openedis still a self-report — two claims a report said it had personally verified were stale. The quote requirement is the mitigation; is it sufficient?om-spec-writing's research step delegate here explicitly?Test artifacts (five reports + the citation grading) are in the run's
.context/prior-art-test/, not committed.