feat(scripts): read what a skill ships, not what it says - #56
Merged
Conversation
Every verdict reviewed so far rests on SKILL.md, which is the skill's description of itself. The behaviour lives in the scripts it invokes: buddy-card extracts a Claude OAuth token from the Keychain and then runs scripts/buddy-algorithm.js with it, and that script has never been read. A skill whose stated purpose differs from what its code does is precisely the supply-chain case worth finding, and reviewing only the prose cannot find it. --ls probes for a file listing (no documented endpoint, so try the plausible ones and report what answers). --fetch takes slug=path pairs so the referenced scripts can be read directly. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every verdict this project holds was formed from
SKILL.mdalone — the file where a skill describes itself. The behaviour is in the scripts it invokes, and until this branch there was no way to read them.buddy-cardwas the case that exposed it. Its SKILL.md extracts a Claude OAuth token from the macOS Keychain and pipes the resulting UUID intonode ${SKILL_DIR}/scripts/buddy-algorithm.js. I had asserted that token "only goes to Anthropic's own API" — on no evidence, because that script had never been fetched by anything.With
--fetchI read both scripts:scripts/buddy-algorithm.js— 40 lines, pure computation. Appends a salt, FNV-1a hash, mulberry32 PRNG, derives species/rarity/stats, prints JSON. No imports, no network, no filesystem. It even truncates the input touserId.substring(0, 8) + '...'.scripts/generate-image.ts— posts the card prompt togenerativelanguage.googleapis.comusing the user's ownGOOGLE_API_KEY. The OAuth token is never passed to either script.So the claim happened to be correct, and is now evidenced rather than assumed. A skill that says "I make a trading card" while its script does something else is exactly the supply-chain case worth finding, and reviewing prose cannot find it.
What this adds
--ls <slug>— probes for a file listing. Result: there isn't one./files,/tree,/contentsall 404;?include=filesignores the parameter. Worth recording so nobody re-derives it.--fetch <slug>=<path>— fetches any file a skill ships, so referenced scripts can be read directly.What it established
The registry serves every file. Our scanner reads one.
74,158 skills, every sweep since the project began, one file each.
buddy-algorithm.jsandgenerate-image.tsare real, published, executable code that no scan has looked at. This time they were clean; nothing in the pipeline could have told us that.Scope
Tooling only — no engine or rule changes. Measuring the size of that unread surface is PR #55; fetching and scanning those files is the change after it.
🤖 Generated with Claude Code
https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
Generated by Claude Code