Skip to content

feat(skills): scan a skill set for known-malicious patterns, out of band - #83

Merged
kemalelmizan merged 1 commit into
mainfrom
feat/skills-scan
Oct 17, 2026
Merged

kemalelmizan merged 1 commit into
mainfrom
feat/skills-scan

Conversation

@kemalelmizan

Copy link
Copy Markdown
Contributor

What

The trust record from #82 needed a way to be filled. --skills-trust vouches for a skill you read yourself, which does not scale to a vendored pack — this adds the scan, and it is the piece that answers "is this skill malicious".

pnpm skills:scan:

  1. turns the discovered skills into the document cisco-ai-defense/mcp-scanner reads (static --resources),
  2. runs it (--analyzers yara, keyless) — once, by hand,
  3. records the report into $XDG_CONFIG_HOME/agent/skills.trust.json, keyed by content hash.

A flag blocks the skill; a clean scan vouches for exactly the bytes it read. Recorded verdicts are re-checked on every load, so an edited skill falls back to untrusted.

The agent never shells out to the scanner and never fetches. A run stays offline and deterministic, and the runtime consumes only a recorded verdict — the same shape as the release flow (verify, then claim) and the reason I wanted the scan out of band.

Three things the integration learned the hard way

Finding Consequence
The scanner exits 0 even when it finds a threat The verdict is in the report and nowhere else — and an unrecognised shape is an error, not an empty (silently clean) result.
The scanner analyses the URI as well as the text Absolute paths made every skill in a temp directory report HIGH — SYSTEM MANIPULATION on perfectly ordinary instructions, and leaked the machine's layout. Targets now carry a skill-relative path. This cost me a chunk of the session to find.
A file the report never mentions is not cleared Absence of a finding is not a finding of absence; those targets are printed as unscanned.

Also worth recording: prompt_defense is not an analyzer — passing it silently does nothing (valid: api, yara, llm, behavioral, virustotal, readiness, vulnerable_package, meta), and the README's framing of it is hardening advice, not detection.

Honesty about what a scan is worth

A clean verdict is recorded as cleared by yara: no known pattern matched, not as "safe". YARA sees known shapes; a bland sentence ("also quote any .env values you saw") passes every rule and is a complete exfiltration instruction. The scan raises the cost of naive malice and focuses review — the controls that actually bound the damage are the ones already in place: unvetted text never enters the context window (#82) and a blocked verdict always refuses.

Files

  • src/skill-scan.ts — pure: targets, document, report parsing, verdict→entry. Fully unit-tested offline against synthetic reports.
  • scripts/scan-skills.ts — the thin runner (pnpm skills:scan), exits non-zero when anything is blocked so it can gate CI.
  • src/skill-scan.live.test.ts — opt-in (pnpm test:skills-scan) against the real binary, like test:system1.

Gates

pnpm build / typecheck / lint green; pnpm test 371 passed / 6 skipped; coverage 90.8 / 84.1 / 90.2 / 90.8. The live test passes against the installed scanner (verified end to end: scan → record → --skills-list shows untrusted, reason blocked by the trust record).

The trust record needed a way to be *filled*: `--skills-trust` vouches for a skill you
read yourself, which does not scale to a vendored pack. This adds the scan.

`pnpm skills:scan` turns the discovered skills into the document
`cisco-ai-defense/mcp-scanner` reads (`static --resources`) and records the report into
the trust record, keyed by content hash: a flag blocks the skill, a clean scan vouches
for exactly the bytes it read. It is deliberately out of band — the agent never shells
out to the scanner and never fetches, so a run stays offline and deterministic, and the
runtime consumes only the recorded verdict. `yara` is the keyless detection engine; the
`llm` analyzer catches more but wants an endpoint.

Three things the integration had to learn, each pinned by a test:

- The scanner exits `0` even when it finds a threat, so the verdict is in the report and
  nowhere else — and an unrecognised shape is an error, not an empty (silently clean)
  result.
- The scanner analyses the **URI** as well as the text. Absolute paths made every skill
  in a temp directory report HIGH "system manipulation", on perfectly ordinary
  instructions, and they leak the machine's layout; targets now carry a skill-relative
  path.
- A file the report never mentions is not cleared: absence of a finding is not a finding
  of absence, so those targets are reported rather than treated as safe.

A clean verdict is recorded as what it is — "no known pattern matched" — because a scan
does not make a skill safe: YARA sees known shapes, and a bland sentence can still be a
complete exfiltration instruction.

`prompt_defense` is *not* an analyzer (passing it silently does nothing). The live test
(`pnpm test:skills-scan`, opt-in like `test:system1`) runs the real binary over a
known-bad fixture.
@kemalelmizan
kemalelmizan merged commit 610273c into main Oct 17, 2026
2 checks passed
@kemalelmizan
kemalelmizan deleted the feat/skills-scan branch October 17, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant