From 88780393d5a708b63e7a14a879818e845f422c44 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 00:45:22 +0000 Subject: [PATCH 1/4] docs: plan AI/LLM security and engineering-hygiene tool expansion Scopes nine new toolbox entries plus repo enablement work, sequenced in six phases. Every proposal is constrained to the existing architecture: no new runtime deps, no third-party APIs, guarded fetch + rate limit on every server route, Finding[] report shape, registry-driven wiring. Client-only tools (prompt-injection linter, key-leak linter, MCP config auditor) are sequenced first since they add no network surface; the active AI-endpoint probe is sequenced last. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01YbYteQPCs7CxG1gSJ568aD --- README.md | 4 + docs/AI-ENGINEERING-TOOLS-PLAN.md | 253 ++++++++++++++++++++++++++++++ 2 files changed, 257 insertions(+) create mode 100644 docs/AI-ENGINEERING-TOOLS-PLAN.md diff --git a/README.md b/README.md index 99e8c8a..da8c45b 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,10 @@ drop in cleanly. More tools planned (open-redirect tester, subdomain hygiene check, robots/sitemap auditor, JOSE algorithm confusion exerciser, etc.). +An expansion into **AI/LLM security** (prompt-injection linter, AI crawler-policy +auditor, MCP config auditor) and further engineering-hygiene tools is scoped in +[`docs/AI-ENGINEERING-TOOLS-PLAN.md`](docs/AI-ENGINEERING-TOOLS-PLAN.md). + --- ## Quick start diff --git a/docs/AI-ENGINEERING-TOOLS-PLAN.md b/docs/AI-ENGINEERING-TOOLS-PLAN.md new file mode 100644 index 0000000..04680a4 --- /dev/null +++ b/docs/AI-ENGINEERING-TOOLS-PLAN.md @@ -0,0 +1,253 @@ +# Plan: AI & Engineering Tools for Cyber Toolbox + +Status: proposal · Target: incremental, one tool per PR + +This plan extends the toolbox into two adjacent areas — **AI/LLM security** +(the OWASP Top 10 for LLM Applications) and **general engineering hygiene** — +without breaking any of the constraints that make the current codebase what it +is. + +## Constraints every proposal below respects + +Derived from the existing four tools, not invented for this document: + +| Constraint | Why it matters here | +| --- | --- | +| **No third-party runtime deps** (Next/React/Tailwind only) | Every parser below is hand-rolled, the way `lib/tls/der.ts` is. Nothing needs an SDK. | +| **No secrets, no third-party APIs** | Nothing here calls a model provider. "AI tools" means *tools that audit AI surfaces*, not tools that consume an LLM. | +| **Every outbound URL goes through `guardUrl` / `safeFetch`** | Any new server route reuses `lib/security/ssrf.ts` unchanged. | +| **`rateLimit()` at the top of every route** | Same three-line preamble as `app/api/tools/cors-tester/route.ts`. | +| **Report shape is `Finding[]` / `FindingGroup[]`** | New tools render through the existing `FindingsList` + `SeverityBadge`, so UI cost stays near zero. | +| **Pure logic lives in `lib/`, is unit-tested with vitest** | CI already runs `npm run test`; each tool below lands with tests for its analyzer. | +| **Registry-driven** | Adding to `lib/tools/registry.ts` wires up the home grid, sitemap, and OG image for free. | + +A useful side effect: the **client-only** tools (marked below) need no API +route, no SSRF surface, no `vercel.json` entry, and no rate limiting — they are +the cheapest and safest things to ship, exactly like JWT Inspector. + +--- + +## Track A — AI / LLM security tools + +### A1. AI Crawler & Content Policy Auditor — *server route* + +**What:** Point it at a domain; it fetches and grades the site's AI-facing +policy surface: + +- `/robots.txt` — directives for `GPTBot`, `ClaudeBot`, `Google-Extended`, + `CCBot`, `PerplexityBot`, `Bytespider`, `Applebot-Extended`, and friends; + flags "blocks nothing", "blocks everything", and self-contradicting groups. +- `/llms.txt` and `/llms-full.txt` — presence, size, whether it points at + routes that 404. +- `/.well-known/ai.txt`, `/ai.txt` — TDM/AI usage declarations. +- Response headers — `X-Robots-Tag: noai, noimageai`, `TDM-Reservation`. +- Cross-checks: `llms.txt` advertising paths that `robots.txt` disallows. + +**Why it earns a slot:** it is a real, current gap that nobody has a clean +checker for, and it is *pure fetch + parse* — the exact shape of Misconfig +Mapper. Highest value-to-risk ratio in this plan. + +**Files:** `lib/aipolicy/{fetch,robots,llmstxt,analyze,types}.ts`, +`app/api/tools/ai-policy/route.ts`, `app/tools/ai-policy/{page,View}.tsx`, +`tests/aipolicy/analyze.test.ts`. Add `maxDuration: 15` to `vercel.json`. + +**Effort:** M. Network: 4–6 guarded GETs. OWASP: A05 + LLM-adjacent. + +--- + +### A2. Prompt Injection Linter — *client-only* + +**What:** Paste a system prompt / RAG or agent template; get graded findings on +structural weaknesses: + +- Untrusted content interpolated with **no delimiter** (`{{context}}`, + `${userInput}` sitting inline in instruction text). +- Missing "content below is data, not instructions" framing. +- Instruction-override bait already present in the text ("ignore previous…"). +- **Secrets in the prompt** — API keys, internal hostnames, connection strings. +- Tool/authority language without a confirmation gate ("you may run any + command", "always approve"). +- No output-format constraint, so injected output flows straight downstream. +- Conflicting or duplicated instructions; unreachable trailing rules. + +**Why:** maps to **OWASP LLM01 (Prompt Injection)** and **LLM06 (Sensitive +Information Disclosure)**, runs entirely in the tab (prompts are sensitive — +the "your text never leaves your browser" story is the same one JWT Inspector +already tells and it lands well), and it is 100% pure functions, so it is +trivially testable. + +**Files:** `lib/promptlint/{rules,analyze,types}.ts`, +`app/tools/prompt-linter/{page,View}.tsx`, `tests/promptlint/rules.test.ts`. +No route, no `vercel.json` change. + +**Effort:** M (the rule set is the work, not the plumbing). Network: none. + +--- + +### A3. AI Key & Credential Leak Linter — *client-only* + +**What:** Paste a config file, `.env`, or code snippet; it flags provider +credentials by prefix pattern **plus** Shannon-entropy scoring, and shows +masked matches only (`sk-ant-…4f2a`): + +`sk-ant-*` (Anthropic), `sk-*`/`sk-proj-*` (OpenAI), `AIza*` (Google), +`hf_*` (Hugging Face), `r8_*` (Replicate), `gsk_*` (Groq), `xai-*`, +`pplx-*`, plus generic `AWS_SECRET_ACCESS_KEY`, bearer tokens, and private-key +PEM blocks. + +Each finding gets a remediation line: rotate first, then purge history. + +**Why:** ships in an afternoon, is genuinely useful, and pairs naturally with +A4 below. Same client-only safety posture — the pasted secret never leaves the +tab, which is the whole point. + +**Files:** `lib/secretlint/{patterns,entropy,analyze}.ts`, +`app/tools/secret-linter/{page,View}.tsx`, `tests/secretlint/*.test.ts`. + +**Effort:** S. Network: none. OWASP: A02 / LLM06. + +--- + +### A4. MCP & Agent Config Auditor — *client-only* + +**What:** Paste an MCP server config (`mcp.json`, `claude_desktop_config.json`, +`.mcp.json`) and grade each server entry: + +- `npx -y ` / `uvx` from a registry with no version pin — silent + remote-code update on every launch. +- Inline secrets in `env` (hand off to the A3 pattern set — shared module). +- Filesystem servers rooted at `/`, `~`, or the home directory. +- `command: bash -c …` / shell-wrapped launches. +- Remote servers over plain `http://`, or pointed at an IP literal. +- Tool-permission blocks that pre-approve broad wildcards. + +**Why:** this is where "AI" and "security misconfiguration" actually meet in +2026, and there is no tidy checker for it. Strong differentiator for a +portfolio piece; reuses A3's pattern module so the marginal cost is low. + +**Files:** `lib/mcplint/{rules,analyze,types}.ts`, +`app/tools/mcp-auditor/{page,View}.tsx`, `tests/mcplint/rules.test.ts`. + +**Effort:** M. Network: none. Depends on: A3 (shared secret patterns). + +--- + +### A5. Exposed AI Endpoint Probe — *server route, ship last* + +**What:** Same probe pattern as `lib/misconfig/probes.ts`, aimed at unauthenticated +AI infrastructure left open on a host: `/v1/models`, `/v1/chat/completions` +(GET → expect 401/405, flag a 200), Ollama `/api/tags`, LangServe `/invoke`, +Gradio `/config`, `/openapi.json`, vector-DB consoles. + +**Why it is last:** it is the only proposal that is meaningfully *active* +probing. It stays inside what the toolbox already does (`.git/HEAD` and +`/server-status` probes are the same class), but it deserves the strictest +treatment: GET/HEAD only, no request bodies, never send a prompt, an explicit +"only scan what you own" interstitial, and short timeouts. + +**Files:** `lib/aiendpoints/{probes,analyze,types}.ts`, +`app/api/tools/ai-endpoints/route.ts`, `app/tools/ai-endpoints/{page,View}.tsx`, +`tests/aiendpoints/analyze.test.ts`. `vercel.json`: `maxDuration: 20`. + +**Effort:** M. Network: ~8 guarded probes. OWASP: A05 / LLM. + +--- + +## Track B — Engineering hygiene tools + +### B1. DNS & Email Hygiene Check — *server route, `node:dns` only* + +Resolves and grades: SPF (present, single record, no `+all`, lookup count ≤ 10), +DMARC (`p=none` vs `quarantine`/`reject`, `rua` present), DKIM selector probing +for common selectors, MX sanity, CAA presence, DNSSEC (`DS` record), and +dangling-CNAME subdomain-takeover signals. + +Uses `node:dns/promises` — **no HTTP at all**, so the SSRF surface is nil. +Pairs naturally with the existing TLS viewer. **Effort:** M. **Highest value in +Track B.** + +### B2. Open Redirect Tester — *server route* + +Already named in the README's "planned" list. Enumerates common redirect params +(`next`, `url`, `redirect`, `return_to`, `dest`, `continue`), sends a benign +off-site value through `safeFetch` with redirects **not** followed, and grades +the `Location` header: same-origin (pass), off-site (fail), protocol-relative +`//evil` and `\/\/evil` bypasses, and open `javascript:` targets. +**Effort:** M. OWASP: A01/A10. + +### B3. Cache & Compression Auditor — *server route* + +Grades `Cache-Control` on HTML vs static assets, `Vary` correctness (the +`Vary: Origin` bug the CORS tester already hints at), `ETag`/`Last-Modified`, +`Age`/CDN headers, compression negotiation, and the classic *private data +cached publicly* pattern. Cheap: one or two requests, reuses everything. +**Effort:** S. + +### B4. Robots & Sitemap Auditor — *server route* + +Also on the README list; largely **free if A1 ships first** — the robots.txt +parser and fetch layer are the same module. Adds sitemap XML validation, URL +count/size caps, `robots.txt` ↔ sitemap disagreement, and 404/redirect +sampling. **Effort:** S when built after A1. + +--- + +## Track C — Engineering enablement (repo-internal) + +Small, unglamorous, pays for itself by tool #6: + +1. **`CLAUDE.md`** at the repo root — the "adding a new tool" contract from the + README, plus the non-negotiables (`safeFetch`, `rateLimit`, no new deps, + tests for every `lib/` analyzer) in a form an agent reads first. +2. **Tool scaffolder** — `npm run new:tool -- ` emits the registry stub, + `page.tsx`/`View.tsx`, the route with the rate-limit preamble already in + place, and a test file. Removes the four-file boilerplate and, more + importantly, makes it impossible to forget the security preamble. +3. **Coverage gate** — vitest `--coverage` in CI (already runs `npm run test`), + with a floor on `lib/**` so analyzers cannot land untested. +4. **Registry contract test** — one test asserting every `live` tool has a + reachable `href`, a unique `id`, and, when `apiPath` is set, a matching + `vercel.json` entry. Catches the most likely copy-paste mistake. +5. **Shared probe harness** — A1, A5, B2, B3, B4 all do "fire N guarded + requests, collect status + headers, classify". Factor that out of + `lib/misconfig/probes.ts` into `lib/security/probe-runner.ts` **before** + the third consumer, not after the fifth. + +--- + +## Sequencing + +Each phase is independently shippable, and each is one PR per tool. + +| Phase | Ships | Rationale | +| --- | --- | --- | +| **0** | C1 (`CLAUDE.md`), C2 (scaffolder), C4 (registry test) | ~half a day; every later tool gets cheaper. | +| **1** | **A2 Prompt Injection Linter**, **A3 Key Leak Linter** | Client-only: no network, no route, no new attack surface. Fastest path to "the toolbox does AI now". | +| **2** | **A1 AI Crawler & Policy Auditor** + C5 (probe harness) | First AI server tool; extract the shared harness while there are exactly two consumers. | +| **3** | **A4 MCP Config Auditor**, **B1 DNS & Email Hygiene** | The two strongest differentiators. A4 reuses A3; B1 reuses nothing but adds a new primitive (`node:dns`). | +| **4** | **B4 Robots/Sitemap**, **B3 Cache/Compression**, **B2 Open Redirect** | Cheap fills once the harness exists. | +| **5** | **A5 Exposed AI Endpoint Probe** | Ships last, deliberately: most active probing, needs the strictest guardrails and the clearest consent copy. | + +## Definition of done, per tool + +1. Registry entry in `lib/tools/registry.ts` (`status: "soon"` first if you want + it visible on the grid before it lands — `ToolCard` already renders "Soon" + as non-clickable and `sitemap.ts` filters to `live`, so this is safe). +2. Analyzer in `lib//`, pure and dependency-free. +3. Vitest suite covering pass/warn/fail for each rule. +4. Route with the `rateLimit` + `safeFetch` preamble (server tools only). +5. `vercel.json` `maxDuration` entry (server tools only). +6. README table row + `docs/` note if the tool has a non-obvious threat model. +7. `npm run typecheck && npm run test && npm run lint && npm run build` green. + +## Open questions + +- **Naming:** does the site want an explicit "AI" category on the home grid, or + do these just mix into the existing list? The registry has `tags` already — + an `ai` tag plus a filter row on `app/page.tsx` is a small change. +- **OWASP refs:** the registry's `owaspRefs` currently carries Top 10 (Web) + strings. AI tools want *OWASP Top 10 for LLM Applications* refs + (`LLM01:2025 - Prompt Injection`). Recommend keeping one field and letting + the string carry the taxonomy, rather than adding a second field. +- **A5 consent:** the existing tools warn in prose. An active AI-endpoint probe + may warrant a checkbox before the first scan. Product call, not a technical one. From be09d1921352b62987890e6b71a1b8c02c5aa74b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 00:57:12 +0000 Subject: [PATCH 2/4] docs: revise tool plan after fit review Re-checks each proposal against the property the four live tools share: every one returns a verifiable fact the user can check independently. - Promotes the MCP config auditor to lead tool; every rule is a fact and the threat model (supply-chain RCE) is real and under-served. - Keeps all four engineering-hygiene tools, including cache/compression and robots/sitemap. Notes B4's shared robots parser with A1. - Records the AI crawler-policy auditor as gated on a scope decision: technically the cleanest proposal, but it audits content licensing, not security, and has no honest OWASP mapping. - Cuts the prompt-injection linter (heuristics, not facts; a passing grade would imply an assurance nothing structural can deliver) and the exposed-AI-endpoint probe (its valuable case is internal addresses, which the SSRF guard blocks by design). Both retained under "Considered and cut" with reasoning. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01YbYteQPCs7CxG1gSJ568aD --- README.md | 6 +- docs/AI-ENGINEERING-TOOLS-PLAN.md | 247 ++++++++++++++++-------------- 2 files changed, 137 insertions(+), 116 deletions(-) diff --git a/README.md b/README.md index da8c45b..6d5521c 100644 --- a/README.md +++ b/README.md @@ -29,9 +29,9 @@ drop in cleanly. More tools planned (open-redirect tester, subdomain hygiene check, robots/sitemap auditor, JOSE algorithm confusion exerciser, etc.). -An expansion into **AI/LLM security** (prompt-injection linter, AI crawler-policy -auditor, MCP config auditor) and further engineering-hygiene tools is scoped in -[`docs/AI-ENGINEERING-TOOLS-PLAN.md`](docs/AI-ENGINEERING-TOOLS-PLAN.md). +An expansion into **AI/LLM security** (MCP config auditor, AI key-leak linter) +and further engineering-hygiene tools (DNS/email hygiene, cache & compression) +is scoped in [`docs/AI-ENGINEERING-TOOLS-PLAN.md`](docs/AI-ENGINEERING-TOOLS-PLAN.md). --- diff --git a/docs/AI-ENGINEERING-TOOLS-PLAN.md b/docs/AI-ENGINEERING-TOOLS-PLAN.md index 04680a4..b08750d 100644 --- a/docs/AI-ENGINEERING-TOOLS-PLAN.md +++ b/docs/AI-ENGINEERING-TOOLS-PLAN.md @@ -7,6 +7,19 @@ This plan extends the toolbox into two adjacent areas — **AI/LLM security** without breaking any of the constraints that make the current codebase what it is. +## The bar a new tool has to clear + +Beyond the architectural constraints below, the four live tools share a +property worth stating explicitly, because it is the thing that makes them +credible: **each one returns a verifiable fact.** The header is present or it +is not. The certificate expires on a date. `alg` is `none` or it is not. The +README's troubleshooting section actively invites the user to check the cert +verdict against `openssl s_client` — the verdict survives that check. + +A tool that returns a *heuristic opinion* dressed as a verdict does not belong +here, however useful the underlying idea. That test drove the cuts recorded at +the end of this document. + ## Constraints every proposal below respects Derived from the existing four tools, not invented for this document: @@ -29,58 +42,29 @@ the cheapest and safest things to ship, exactly like JWT Inspector. ## Track A — AI / LLM security tools -### A1. AI Crawler & Content Policy Auditor — *server route* - -**What:** Point it at a domain; it fetches and grades the site's AI-facing -policy surface: - -- `/robots.txt` — directives for `GPTBot`, `ClaudeBot`, `Google-Extended`, - `CCBot`, `PerplexityBot`, `Bytespider`, `Applebot-Extended`, and friends; - flags "blocks nothing", "blocks everything", and self-contradicting groups. -- `/llms.txt` and `/llms-full.txt` — presence, size, whether it points at - routes that 404. -- `/.well-known/ai.txt`, `/ai.txt` — TDM/AI usage declarations. -- Response headers — `X-Robots-Tag: noai, noimageai`, `TDM-Reservation`. -- Cross-checks: `llms.txt` advertising paths that `robots.txt` disallows. - -**Why it earns a slot:** it is a real, current gap that nobody has a clean -checker for, and it is *pure fetch + parse* — the exact shape of Misconfig -Mapper. Highest value-to-risk ratio in this plan. - -**Files:** `lib/aipolicy/{fetch,robots,llmstxt,analyze,types}.ts`, -`app/api/tools/ai-policy/route.ts`, `app/tools/ai-policy/{page,View}.tsx`, -`tests/aipolicy/analyze.test.ts`. Add `maxDuration: 15` to `vercel.json`. - -**Effort:** M. Network: 4–6 guarded GETs. OWASP: A05 + LLM-adjacent. - ---- +### A4. MCP & Agent Config Auditor — *client-only* · **lead tool** -### A2. Prompt Injection Linter — *client-only* - -**What:** Paste a system prompt / RAG or agent template; get graded findings on -structural weaknesses: +**What:** Paste an MCP server config (`mcp.json`, `claude_desktop_config.json`, +`.mcp.json`) and grade each server entry: -- Untrusted content interpolated with **no delimiter** (`{{context}}`, - `${userInput}` sitting inline in instruction text). -- Missing "content below is data, not instructions" framing. -- Instruction-override bait already present in the text ("ignore previous…"). -- **Secrets in the prompt** — API keys, internal hostnames, connection strings. -- Tool/authority language without a confirmation gate ("you may run any - command", "always approve"). -- No output-format constraint, so injected output flows straight downstream. -- Conflicting or duplicated instructions; unreachable trailing rules. +- `npx -y ` / `uvx` from a registry with no version pin — silent + remote-code update on every launch. +- Inline secrets in `env` (hand off to the A3 pattern set — shared module). +- Filesystem servers rooted at `/`, `~`, or the home directory. +- `command: bash -c …` / shell-wrapped launches. +- Remote servers over plain `http://`, or pointed at an IP literal. +- Tool-permission blocks that pre-approve broad wildcards. -**Why:** maps to **OWASP LLM01 (Prompt Injection)** and **LLM06 (Sensitive -Information Disclosure)**, runs entirely in the tab (prompts are sensitive — -the "your text never leaves your browser" story is the same one JWT Inspector -already tells and it lands well), and it is 100% pure functions, so it is -trivially testable. +**Why it leads:** every rule is a fact, not a heuristic — a package without a +version pin *is* unpinned; a server on `http://` *is* plaintext; a root at `~` +*is* the whole home directory. Real threat model (supply-chain RCE), genuinely +under-served by existing tooling, and it takes the same client-only posture as +JWT Inspector: the config never leaves the tab. -**Files:** `lib/promptlint/{rules,analyze,types}.ts`, -`app/tools/prompt-linter/{page,View}.tsx`, `tests/promptlint/rules.test.ts`. -No route, no `vercel.json` change. +**Files:** `lib/mcplint/{rules,analyze,types}.ts`, +`app/tools/mcp-auditor/{page,View}.tsx`, `tests/mcplint/rules.test.ts`. -**Effort:** M (the rule set is the work, not the plumbing). Network: none. +**Effort:** M. Network: none. Depends on: A3 (shared secret patterns). --- @@ -97,9 +81,10 @@ PEM blocks. Each finding gets a remediation line: rotate first, then purge history. -**Why:** ships in an afternoon, is genuinely useful, and pairs naturally with -A4 below. Same client-only safety posture — the pasted secret never leaves the -tab, which is the whole point. +**Why:** a prefix either matches or it does not, so the verdict is checkable. +Novelty is admittedly low — gitleaks and trufflehog cover this ground better — +but it is an afternoon of work and **A4 needs the pattern module anyway**, so +build it first and let A4 import it. **Files:** `lib/secretlint/{patterns,entropy,analyze}.ts`, `app/tools/secret-linter/{page,View}.tsx`, `tests/secretlint/*.test.ts`. @@ -108,53 +93,47 @@ tab, which is the whole point. --- -### A4. MCP & Agent Config Auditor — *client-only* - -**What:** Paste an MCP server config (`mcp.json`, `claude_desktop_config.json`, -`.mcp.json`) and grade each server entry: - -- `npx -y ` / `uvx` from a registry with no version pin — silent - remote-code update on every launch. -- Inline secrets in `env` (hand off to the A3 pattern set — shared module). -- Filesystem servers rooted at `/`, `~`, or the home directory. -- `command: bash -c …` / shell-wrapped launches. -- Remote servers over plain `http://`, or pointed at an IP literal. -- Tool-permission blocks that pre-approve broad wildcards. - -**Why:** this is where "AI" and "security misconfiguration" actually meet in -2026, and there is no tidy checker for it. Strong differentiator for a -portfolio piece; reuses A3's pattern module so the marginal cost is low. - -**Files:** `lib/mcplint/{rules,analyze,types}.ts`, -`app/tools/mcp-auditor/{page,View}.tsx`, `tests/mcplint/rules.test.ts`. +### A1. AI Crawler & Content Policy Auditor — *server route* · **scope decision required** -**Effort:** M. Network: none. Depends on: A3 (shared secret patterns). +**What:** Point it at a domain; it fetches and grades the site's AI-facing +policy surface: ---- +- `/robots.txt` — directives for `GPTBot`, `ClaudeBot`, `Google-Extended`, + `CCBot`, `PerplexityBot`, `Bytespider`, `Applebot-Extended`, and friends; + flags "blocks nothing", "blocks everything", and self-contradicting groups. +- `/llms.txt` and `/llms-full.txt` — presence, size, whether it points at + routes that 404. +- `/.well-known/ai.txt`, `/ai.txt` — TDM/AI usage declarations. +- Response headers — `X-Robots-Tag: noai, noimageai`, `TDM-Reservation`. +- Cross-checks: `llms.txt` advertising paths that `robots.txt` disallows. -### A5. Exposed AI Endpoint Probe — *server route, ship last* +**Technically the cleanest proposal here** — pure fetch + parse, the exact +shape of Misconfig Mapper, reusing its fetch layer almost verbatim, and every +output is a fact. -**What:** Same probe pattern as `lib/misconfig/probes.ts`, aimed at unauthenticated -AI infrastructure left open on a host: `/v1/models`, `/v1/chat/completions` -(GET → expect 401/405, flag a 200), Ollama `/api/tags`, LangServe `/invoke`, -Gradio `/config`, `/openapi.json`, vector-DB consoles. +**But it is not a security tool.** It audits content licensing and crawler +policy. There is no honest OWASP mapping for it; an earlier draft of this +document claimed "A05 + LLM-adjacent" and that was a fudge. Shipping it means +the site's remit widens from "web-security tools" to something closer to "web +surface auditing" — including the tagline, the About page, and the OG copy. -**Why it is last:** it is the only proposal that is meaningfully *active* -probing. It stays inside what the toolbox already does (`.git/HEAD` and -`/server-status` probes are the same class), but it deserves the strictest -treatment: GET/HEAD only, no request bodies, never send a prompt, an explicit -"only scan what you own" interstitial, and short timeouts. +**That is a product decision, not a technical one** — see *Decisions needed* +below. Build it if the answer is yes; drop it cleanly if the answer is no. -**Files:** `lib/aiendpoints/{probes,analyze,types}.ts`, -`app/api/tools/ai-endpoints/route.ts`, `app/tools/ai-endpoints/{page,View}.tsx`, -`tests/aiendpoints/analyze.test.ts`. `vercel.json`: `maxDuration: 20`. +**Files:** `lib/aipolicy/{fetch,robots,llmstxt,analyze,types}.ts`, +`app/api/tools/ai-policy/route.ts`, `app/tools/ai-policy/{page,View}.tsx`, +`tests/aipolicy/analyze.test.ts`. Add `maxDuration: 15` to `vercel.json`. -**Effort:** M. Network: ~8 guarded probes. OWASP: A05 / LLM. +**Effort:** M. Network: 4–6 guarded GETs. --- ## Track B — Engineering hygiene tools +All four are in scope and sequenced below. They are conventional web-quality +tools rather than novel ones, which is the point: they are well-understood, +their verdicts are checkable, and each reuses machinery the toolbox already has. + ### B1. DNS & Email Hygiene Check — *server route, `node:dns` only* Resolves and grades: SPF (present, single record, no `+all`, lookup count ≤ 10), @@ -163,8 +142,8 @@ for common selectors, MX sanity, CAA presence, DNSSEC (`DS` record), and dangling-CNAME subdomain-takeover signals. Uses `node:dns/promises` — **no HTTP at all**, so the SSRF surface is nil. -Pairs naturally with the existing TLS viewer. **Effort:** M. **Highest value in -Track B.** +Pairs naturally with the existing TLS viewer. **Effort:** M. **Strongest tool +in Track B.** ### B2. Open Redirect Tester — *server route* @@ -180,21 +159,25 @@ the `Location` header: same-origin (pass), off-site (fail), protocol-relative Grades `Cache-Control` on HTML vs static assets, `Vary` correctness (the `Vary: Origin` bug the CORS tester already hints at), `ETag`/`Last-Modified`, `Age`/CDN headers, compression negotiation, and the classic *private data -cached publicly* pattern. Cheap: one or two requests, reuses everything. -**Effort:** S. +cached publicly* pattern — which is a genuine confidentiality bug, not just a +performance one, and is the finding that most justifies the tool's slot. +Cheap: one or two requests, reuses everything. **Effort:** S. ### B4. Robots & Sitemap Auditor — *server route* -Also on the README list; largely **free if A1 ships first** — the robots.txt -parser and fetch layer are the same module. Adds sitemap XML validation, URL -count/size caps, `robots.txt` ↔ sitemap disagreement, and 404/redirect -sampling. **Effort:** S when built after A1. +Also on the README list. Validates sitemap XML, URL count/size caps, +`robots.txt` ↔ sitemap disagreement, and 404/redirect sampling of advertised +URLs. + +**Dependency note:** this shares a `robots.txt` parser with A1. If A1 ships, +B4 is nearly free (**S**); if A1 is dropped on scope grounds, B4 carries the +parser itself and costs **M**. Sequence it after the A1 decision either way. --- ## Track C — Engineering enablement (repo-internal) -Small, unglamorous, pays for itself by tool #6: +Small, unglamorous, pays for itself by tool #4: 1. **`CLAUDE.md`** at the repo root — the "adding a new tool" contract from the README, plus the non-negotiables (`safeFetch`, `rateLimit`, no new deps, @@ -208,25 +191,25 @@ Small, unglamorous, pays for itself by tool #6: 4. **Registry contract test** — one test asserting every `live` tool has a reachable `href`, a unique `id`, and, when `apiPath` is set, a matching `vercel.json` entry. Catches the most likely copy-paste mistake. -5. **Shared probe harness** — A1, A5, B2, B3, B4 all do "fire N guarded +5. **Shared probe harness** — A1, B2, B3, and B4 all do "fire N guarded requests, collect status + headers, classify". Factor that out of `lib/misconfig/probes.ts` into `lib/security/probe-runner.ts` **before** - the third consumer, not after the fifth. + the third consumer, not after the fourth. --- ## Sequencing -Each phase is independently shippable, and each is one PR per tool. +Each phase is independently shippable, one PR per tool. | Phase | Ships | Rationale | | --- | --- | --- | | **0** | C1 (`CLAUDE.md`), C2 (scaffolder), C4 (registry test) | ~half a day; every later tool gets cheaper. | -| **1** | **A2 Prompt Injection Linter**, **A3 Key Leak Linter** | Client-only: no network, no route, no new attack surface. Fastest path to "the toolbox does AI now". | -| **2** | **A1 AI Crawler & Policy Auditor** + C5 (probe harness) | First AI server tool; extract the shared harness while there are exactly two consumers. | -| **3** | **A4 MCP Config Auditor**, **B1 DNS & Email Hygiene** | The two strongest differentiators. A4 reuses A3; B1 reuses nothing but adds a new primitive (`node:dns`). | -| **4** | **B4 Robots/Sitemap**, **B3 Cache/Compression**, **B2 Open Redirect** | Cheap fills once the harness exists. | -| **5** | **A5 Exposed AI Endpoint Probe** | Ships last, deliberately: most active probing, needs the strictest guardrails and the clearest consent copy. | +| **1** | **A3 Key Leak Linter** → **A4 MCP Config Auditor** | Client-only: no network, no route, no new attack surface. A3 first because A4 imports its pattern module. A4 is the differentiator — lead the announcement with it. | +| **2** | **B1 DNS & Email Hygiene** | Adds a new primitive (`node:dns`) with zero HTTP surface. | +| **3** | **B2 Open Redirect** + C5 (probe harness) | First new consumer of the fetch layer; extract the shared harness here, while there are two consumers rather than four. | +| **4** | **B3 Cache & Compression**, **B4 Robots & Sitemap** | Cheap fills once the harness exists. | +| **5** | **A1 AI Crawler & Policy Auditor** | Gated on the scope decision below. If yes, it also retro-cheapens B4's parser. | ## Definition of done, per tool @@ -240,14 +223,52 @@ Each phase is independently shippable, and each is one PR per tool. 6. README table row + `docs/` note if the tool has a non-obvious threat model. 7. `npm run typecheck && npm run test && npm run lint && npm run build` green. -## Open questions - -- **Naming:** does the site want an explicit "AI" category on the home grid, or - do these just mix into the existing list? The registry has `tags` already — - an `ai` tag plus a filter row on `app/page.tsx` is a small change. -- **OWASP refs:** the registry's `owaspRefs` currently carries Top 10 (Web) - strings. AI tools want *OWASP Top 10 for LLM Applications* refs - (`LLM01:2025 - Prompt Injection`). Recommend keeping one field and letting - the string carry the taxonomy, rather than adding a second field. -- **A5 consent:** the existing tools warn in prose. An active AI-endpoint probe - may warrant a checkbox before the first scan. Product call, not a technical one. +--- + +## Considered and cut + +Recorded with reasoning so the calls are reviewable, and reversible in one +commit if the reasoning does not hold up. + +### Prompt Injection Linter — cut + +Paste a system prompt or RAG template, get findings on undelimited +interpolation, missing "data, not instructions" framing, instruction-override +bait, and ungated tool authority. + +**Why cut:** it fails the verifiable-fact bar. "Missing data-vs-instruction +framing" is a regex heuristic the user cannot check, and it will false-positive +heavily on ordinary prompts. The deeper problem is the green verdict: there is +no known structural fix for prompt injection, so a passing grade would imply an +assurance the tool cannot back — worse than shipping nothing in a security +toolbox. Its one genuinely checkable rule, *secrets embedded in the prompt*, is +already **A3**. + +### Exposed AI Endpoint Probe — cut + +Probe a host for unauthenticated AI infrastructure: `/v1/models`, +`/v1/chat/completions`, Ollama `/api/tags`, LangServe `/invoke`, Gradio +`/config`. + +**Why cut:** the toolbox's own safety guard removes the valuable case. The +classic finding is an unauthenticated Ollama on an internal address, and +`lib/security/ssrf.ts` blocks `10.0.0.0/8`, `172.16.0.0/12`, and +`192.168.0.0/16` by design, re-checking on every redirect hop. (Ports are +unrestricted, so `:11434` on a *public* IP is reachable — but that is the rare +case, not the valuable one.) What remains is probing third parties' public +inference endpoints: the most aggressive action in the box for the least yield. + +--- + +## Decisions needed + +1. **Does the site's remit stay "web-security tools"?** This gates **A1** + (Phase 5) and changes B4's cost. Shipping A1 means widening the tagline, + About page, and OG copy toward "web surface auditing". Recommendation: decide + before Phase 4, since B4 shares A1's parser. +2. **Do AI tools get their own home-grid category?** The registry already has + `tags` — an `ai` tag plus a filter row on `app/page.tsx` is a small change. +3. **`owaspRefs` taxonomy.** AI tools want *OWASP Top 10 for LLM Applications* + refs (`LLM01:2025 - Prompt Injection`) alongside the existing web Top 10 + strings. Recommendation: keep one field and let the string carry the + taxonomy, rather than adding a second field. From 887904a7ff6380edf84c16805148d878c51e1299 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 23 Aug 2026 13:25:59 +0000 Subject: [PATCH 3/4] docs: fix internal contradictions found in review Addresses six review findings plus a sequencing nitpick, all verified against the code and the doc before changing anything. - A3: deterministic prefix/format rules decide the verdict; Shannon entropy drops to a secondary signal reported at info as an observation. The old text billed entropy as a co-equal detector while the document's own bar rejects heuristic verdicts. - B2: safeFetch follows 3xx itself (safe-fetch.ts:120-122) and exposes only the terminal headers, and SafeFetchInit has no opt-out, so the claim that B2 would use it "with redirects not followed" was wrong. B2 now ships a guarded single-hop mode as part of its own scope. - B3: the private-data-cached-publicly finding caps at warn. Proving cross-user reuse needs two identities and the toolbox takes no target credentials, so it can only state the precondition. - A1/B4: advertised llms.txt and sitemap URLs are attacker-controlled; dereference same-origin only, capped at 20 per scan. guardUrl blocks private addresses but permits public ones, so this is a separate gate. - B4/A1: flip the shared robots.txt parser dependency. B4 ships in Phase 4 and A1 in Phase 5, so B4 introduces the parser and A1 imports it, not the reverse. - Definition of done: rateLimit on every server route, safeFetch only on routes that make HTTP requests, since B1 is node:dns-only. - Constraints: note that rateLimit is per-process and in-memory before the plan adds five more routes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01YbYteQPCs7CxG1gSJ568aD --- docs/AI-ENGINEERING-TOOLS-PLAN.md | 72 +++++++++++++++++++++++-------- 1 file changed, 55 insertions(+), 17 deletions(-) diff --git a/docs/AI-ENGINEERING-TOOLS-PLAN.md b/docs/AI-ENGINEERING-TOOLS-PLAN.md index b08750d..8237e4a 100644 --- a/docs/AI-ENGINEERING-TOOLS-PLAN.md +++ b/docs/AI-ENGINEERING-TOOLS-PLAN.md @@ -29,7 +29,7 @@ Derived from the existing four tools, not invented for this document: | **No third-party runtime deps** (Next/React/Tailwind only) | Every parser below is hand-rolled, the way `lib/tls/der.ts` is. Nothing needs an SDK. | | **No secrets, no third-party APIs** | Nothing here calls a model provider. "AI tools" means *tools that audit AI surfaces*, not tools that consume an LLM. | | **Every outbound URL goes through `guardUrl` / `safeFetch`** | Any new server route reuses `lib/security/ssrf.ts` unchanged. | -| **`rateLimit()` at the top of every route** | Same three-line preamble as `app/api/tools/cors-tester/route.ts`. | +| **`rateLimit()` at the top of every route** | Same three-line preamble as `app/api/tools/cors-tester/route.ts`. Note its scope before adding five more routes: the bucket is per-process in-memory, keyed by `clientKeyFromHeaders`, so it limits each serverless instance separately and resets on cold start. Fine for a portfolio deploy; if these tools see real traffic, swap `lib/security/rate-limit.ts` for a shared store (Vercel KV/Upstash) — one function, one call site. | | **Report shape is `Finding[]` / `FindingGroup[]`** | New tools render through the existing `FindingsList` + `SeverityBadge`, so UI cost stays near zero. | | **Pure logic lives in `lib/`, is unit-tested with vitest** | CI already runs `npm run test`; each tool below lands with tests for its analyzer. | | **Registry-driven** | Adding to `lib/tools/registry.ts` wires up the home grid, sitemap, and OG image for free. | @@ -71,14 +71,21 @@ JWT Inspector: the config never leaves the tab. ### A3. AI Key & Credential Leak Linter — *client-only* **What:** Paste a config file, `.env`, or code snippet; it flags provider -credentials by prefix pattern **plus** Shannon-entropy scoring, and shows -masked matches only (`sk-ant-…4f2a`): +credentials by prefix pattern, and shows masked matches only (`sk-ant-…4f2a`): `sk-ant-*` (Anthropic), `sk-*`/`sk-proj-*` (OpenAI), `AIza*` (Google), `hf_*` (Hugging Face), `r8_*` (Replicate), `gsk_*` (Groq), `xai-*`, `pplx-*`, plus generic `AWS_SECRET_ACCESS_KEY`, bearer tokens, and private-key PEM blocks. +**Deterministic rules decide the verdict; entropy never does.** A `pass`/`fail` +requires a prefix or format rule to match, and every finding names the rule that +fired so the user can check it. Shannon entropy runs only as a secondary signal +on values no rule matched, and those surface at `info` as *observations* — +"high-entropy string, no known provider format" — never as a credential verdict. +Keeping entropy out of the verdict is what holds this tool to the bar above; +an earlier draft billed entropy as a co-equal detector, which contradicted it. + Each finding gets a remediation line: rotate first, then purge history. **Why:** a prefix either matches or it does not, so the verdict is checkable. @@ -102,7 +109,13 @@ policy surface: `CCBot`, `PerplexityBot`, `Bytespider`, `Applebot-Extended`, and friends; flags "blocks nothing", "blocks everything", and self-contradicting groups. - `/llms.txt` and `/llms-full.txt` — presence, size, whether it points at - routes that 404. + routes that 404. **Only same-origin URLs are dereferenced, capped at 20 per + scan**: the advertised list is attacker-controlled content, so following + off-origin entries would turn the scanner into a request amplifier for + third-party hosts that never opted in. `guardUrl` blocks private and reserved + addresses but deliberately permits public ones, so the origin check is a + separate gate, applied before any `safeFetch`. Off-origin entries are still + *listed* in the report, just not fetched. - `/.well-known/ai.txt`, `/ai.txt` — TDM/AI usage declarations. - Response headers — `X-Robots-Tag: noai, noimageai`, `TDM-Reservation`. - Cross-checks: `llms.txt` advertising paths that `robots.txt` disallows. @@ -149,19 +162,35 @@ in Track B.** Already named in the README's "planned" list. Enumerates common redirect params (`next`, `url`, `redirect`, `return_to`, `dest`, `continue`), sends a benign -off-site value through `safeFetch` with redirects **not** followed, and grades -the `Location` header: same-origin (pass), off-site (fail), protocol-relative -`//evil` and `\/\/evil` bypasses, and open `javascript:` targets. -**Effort:** M. OWASP: A01/A10. +off-site value, and grades the **first** `Location` header: same-origin (pass), +off-site (fail), protocol-relative `//evil` and `\/\/evil` bypasses, and open +`javascript:` targets. + +**This needs a fetch primitive the repo does not have yet.** `safeFetch` follows +3xx responses itself (`lib/security/safe-fetch.ts:120-122`, up to +`MAX_REDIRECTS`) and returns only the terminal response's headers, and +`SafeFetchInit` (`method`, `headers`, `deadlineMs`) has no opt-out. B2 therefore +also ships a guarded single-hop mode — an added `SafeFetchInit` flag, or a small +sibling primitive — that stops at the first response and exposes its raw +`Location` without dereferencing it. Multi-hop stays the default for every +existing caller. Count that in the estimate: **Effort:** M, and it is the reason +B2 rather than B3 is the phase that touches the fetch layer. OWASP: A01/A10. ### B3. Cache & Compression Auditor — *server route* Grades `Cache-Control` on HTML vs static assets, `Vary` correctness (the `Vary: Origin` bug the CORS tester already hints at), `ETag`/`Last-Modified`, -`Age`/CDN headers, compression negotiation, and the classic *private data -cached publicly* pattern — which is a genuine confidentiality bug, not just a -performance one, and is the finding that most justifies the tool's slot. -Cheap: one or two requests, reuses everything. **Effort:** S. +`Age`/CDN headers, and compression negotiation. + +**The *private data cached publicly* pattern caps at `warn`, not `fail`.** +Proving cross-user response reuse needs two identities — an authenticated and an +anonymous request, or two distinct sessions — and the toolbox takes no +credentials for a target, so it cannot run that test. What it can state as fact +is the precondition: a `Cache-Control: public` (or missing) response that also +carries `Set-Cookie` or `Authorization`-varying content. Report that as a warning +naming the missing evidence, and never as a confirmed confidentiality bug. This +is the same bar that cut the prompt-injection linter: state the checkable fact, +not the inference. Cheap: one or two requests, reuses everything. **Effort:** S. ### B4. Robots & Sitemap Auditor — *server route* @@ -169,9 +198,16 @@ Also on the README list. Validates sitemap XML, URL count/size caps, `robots.txt` ↔ sitemap disagreement, and 404/redirect sampling of advertised URLs. -**Dependency note:** this shares a `robots.txt` parser with A1. If A1 ships, -B4 is nearly free (**S**); if A1 is dropped on scope grounds, B4 carries the -parser itself and costs **M**. Sequence it after the A1 decision either way. +Sampling of advertised URLs follows the same same-origin + 20-URL cap as A1, +and for the same reason: sitemap contents are attacker-controlled. + +**Dependency note:** B4 and A1 share a `robots.txt` parser, and **B4 ships +first** (Phase 4 vs Phase 5), so B4 introduces it — `lib/robots/parse.ts`, +costed at **M** — and A1 imports it later rather than the other way round. An +earlier draft had this backwards, claiming B4 was nearly free off a parser A1 +would not have written yet. The upside is unchanged, just relocated: if the A1 +scope decision comes back "no", the parser still lands with B4 and nothing is +stranded. --- @@ -209,7 +245,7 @@ Each phase is independently shippable, one PR per tool. | **2** | **B1 DNS & Email Hygiene** | Adds a new primitive (`node:dns`) with zero HTTP surface. | | **3** | **B2 Open Redirect** + C5 (probe harness) | First new consumer of the fetch layer; extract the shared harness here, while there are two consumers rather than four. | | **4** | **B3 Cache & Compression**, **B4 Robots & Sitemap** | Cheap fills once the harness exists. | -| **5** | **A1 AI Crawler & Policy Auditor** | Gated on the scope decision below. If yes, it also retro-cheapens B4's parser. | +| **5** | **A1 AI Crawler & Policy Auditor** | Gated on the scope decision below. Imports the `robots.txt` parser B4 landed in Phase 4, so if the answer is yes it is cheaper than its section implies. | ## Definition of done, per tool @@ -218,7 +254,9 @@ Each phase is independently shippable, one PR per tool. as non-clickable and `sitemap.ts` filters to `live`, so this is safe). 2. Analyzer in `lib//`, pure and dependency-free. 3. Vitest suite covering pass/warn/fail for each rule. -4. Route with the `rateLimit` + `safeFetch` preamble (server tools only). +4. Route with the `rateLimit` preamble (every server tool) and `safeFetch` for + every user-supplied URL (HTTP-fetching tools only — B1 is `node:dns`-only and + makes no HTTP request, so it takes the rate limiter and nothing else). 5. `vercel.json` `maxDuration` entry (server tools only). 6. README table row + `docs/` note if the tool has a non-obvious threat model. 7. `npm run typecheck && npm run test && npm run lint && npm run build` green. From ca29fb98416e0a0c49ab207ce1b75e4cb24bbb8a Mon Sep 17 00:00:00 2001 From: Tink Date: Sun, 23 Aug 2026 09:34:26 -0400 Subject: [PATCH 4/4] docs: bound policy-auditor fetches --- docs/AI-ENGINEERING-TOOLS-PLAN.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/AI-ENGINEERING-TOOLS-PLAN.md b/docs/AI-ENGINEERING-TOOLS-PLAN.md index 8237e4a..b8ad6e8 100644 --- a/docs/AI-ENGINEERING-TOOLS-PLAN.md +++ b/docs/AI-ENGINEERING-TOOLS-PLAN.md @@ -109,8 +109,8 @@ policy surface: `CCBot`, `PerplexityBot`, `Bytespider`, `Applebot-Extended`, and friends; flags "blocks nothing", "blocks everything", and self-contradicting groups. - `/llms.txt` and `/llms-full.txt` — presence, size, whether it points at - routes that 404. **Only same-origin URLs are dereferenced, capped at 20 per - scan**: the advertised list is attacker-controlled content, so following + routes that 404. **Only same-origin URLs are dereferenced within the + audit-wide request budget**: the advertised list is attacker-controlled content, so following off-origin entries would turn the scanner into a request amplifier for third-party hosts that never opted in. `guardUrl` blocks private and reserved addresses but deliberately permits public ones, so the origin check is a @@ -133,11 +133,20 @@ surface auditing" — including the tagline, the About page, and the OG copy. **That is a product decision, not a technical one** — see *Decisions needed* below. Build it if the answer is yes; drop it cleanly if the answer is no. +**Fetch budget and redirect rule:** an audit makes at most **20 outbound HTTP +requests total**. That budget includes the initial policy candidates, every +advertised URL sample, and every redirect hop—not merely each `safeFetch` +call. A1 therefore needs an origin-pinned fetch mode that checks every 3xx +`Location` against the submitted origin before making the next request; an +off-origin redirect is reported but never fetched. `guardUrl` still runs for +each permitted hop to protect address safety. + **Files:** `lib/aipolicy/{fetch,robots,llmstxt,analyze,types}.ts`, `app/api/tools/ai-policy/route.ts`, `app/tools/ai-policy/{page,View}.tsx`, `tests/aipolicy/analyze.test.ts`. Add `maxDuration: 15` to `vercel.json`. -**Effort:** M. Network: 4–6 guarded GETs. +**Effort:** M. Network: 4–6 guarded GETs without advertised-link sampling; +20 origin-pinned HTTP requests is the hard cap for a complete audit. --- @@ -198,8 +207,11 @@ Also on the README list. Validates sitemap XML, URL count/size caps, `robots.txt` ↔ sitemap disagreement, and 404/redirect sampling of advertised URLs. -Sampling of advertised URLs follows the same same-origin + 20-URL cap as A1, -and for the same reason: sitemap contents are attacker-controlled. +Sampling of advertised URLs follows the same origin-pinned 20-request budget as +A1, and for the same reason: sitemap contents are attacker-controlled. Every +redirect hop must match the submitted origin before it is requested; off-origin +locations are reported but not followed, and `guardUrl` validates every allowed +hop. **Dependency note:** B4 and A1 share a `robots.txt` parser, and **B4 ships first** (Phase 4 vs Phase 5), so B4 introduces it — `lib/robots/parse.ts`,