feat(research): measure the file surface no scan has ever read - #55
Merged
Conversation
Every sweep fetches exactly one file per skill -- get_skill_file defaults to SKILL.md -- so all 74,158 verdicts describe the documentation, not the code. The registry serves the rest of the tree: buddy-card's two scripts both return real content, and neither has ever been scanned. A skill whose SKILL.md is clean and whose script beacons out is invisible to us, to a reviewer, and to the user. That is where real malware would be. Step one measures the surface. Extraction is deliberately conservative -- only paths a skill actually names as executed, read, or linked -- because guessing at conventional layouts would inflate the blind spot rather than measure it. The snapshot turned out to hold verdicts and hashes but no content, so text comes from the registry via --sample, which reports over skills actually fetched instead of extrapolating. Sampling seed is fixed and published so the number is reproducible. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
The dev sandbox cannot reach the registry, so the sample has to run where it can. Seed is an input and defaults to a fixed value so the figure is reproducible by anyone with the published snapshot. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
Every sweep this project has run fetches exactly one file per skill:
So all 74,158 verdicts — including every MALICIOUS one — describe the documentation, not the code.
The registry serves the rest of the tree.
buddy-card'sscripts/buddy-algorithm.jsandscripts/generate-image.tsboth return real content on request, and neither has ever been scanned. I read them manually: clean. Nothing in our pipeline could have told us that.A skill whose
SKILL.mdis clean and whose script beacons to a C2 is invisible to us, to a reviewer skimming the listing, and to the user. That is the definition of hidden behaviour, and it is where real malware would live.What this adds
scripts/extract-referenced-files.py— extracts the paths each SKILL.md names as executed (node scripts/x.js), read (Read `references/y.md`), or linked, and reports how many skills ship code no scan has read, split by extension with executables counted separately.Plus a workflow to run it, since the dev sandbox cannot reach the registry.
Two constraints on the number
Extraction is conservative. Only paths a skill actually names. Guessing at conventional layouts (
scripts/*,bin/*) would have produced a larger figure built partly on files that do not exist. An inflated denominator is how you publish a blind spot bigger than the one you have.Sampling is honest about its base. It reports over skills actually fetched, never extrapolated, and the seed is a fixed published default so anyone with the snapshot can redraw the same sample and check the figure.
Verification
Extractor unit-tested in both directions: it captures
node ${SKILL_DIR}/scripts/buddy-algorithm.js,npx tsx scripts/apply-skill.ts,Read `references/openclaw-workspace.md`, and markdown links; it rejectscurl … | shURLs,~/.bashrc,README.md, andSKILL.mditself.One thing I got wrong: the first version read SKILL.md text from the snapshot, which holds only verdicts and hashes. It returned
0 of 74,158, which is how I found it. Text now comes from the registry.Not in scope here
This measures the surface. Fetching and scanning those files, and isolating the cases where SKILL.md is clean but the script is not, is the next change.
🤖 Generated with Claude Code
https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
Generated by Claude Code