gitleaks parity: diff findings, explain every delta

[aksOps]

Question

Where do siloscan and gitleaks actually disagree when run over the same trees, and is every disagreement explained?

Parent: #48

secrets.yaml is mechanically generated from gitleaks v8.30.1 - 214 of its 222 rules convert. That "96% parity" is arithmetic: rules counted, not findings compared. The conversion has never been diff-tested, so any fidelity bug in scripts/convert_gitleaks.py - a mistranslated character class, a dropped entropy threshold, a keyword prefilter that suppresses more than gitleaks' does - is invisible.

Run gitleaks detect --no-git (directory mode, matching siloscan's depth-1 scope) and siloscan over the same corpora, then diff findings by file, line and secret value. Suggested trees: the expanded corpus from the corpus ticket, a public leaky-repo fixture, and a large real checkout.

Each disagreement lands in exactly one bucket, and the report must place every one of them:

• siloscan misses what gitleaks finds - conversion bug or an over-aggressive allowlist. Fix.
• siloscan finds what gitleaks misses - the generic rules doing their job, or a false positive. Decide which, per shape.
• Both find it, different span or rule id - cosmetic, unless the fingerprint changes.

Unblocked and independent - the corpus ticket makes it more informative but is not required to start. This is the cheapest high-information work on the map: it validates the conversion, calibrates the generic rules against a tool with years of field exposure, and produces the evidence the parity claim currently lacks.

Deliverable: a reproducible comparison script and a written disposition for every delta. Rule fixes it justifies go in their own tickets.

[aksOps]

Resolved by grilling 2026-08-07 (map #48).

Pin gitleaks v8.30.1 - the exact conversion source. Deltas against newer gitleaks are upstream rule drift, a separate question from conversion fidelity.

Method: checksum-pinned binary, harness in scripts/, run manually (optionally workflow_dispatch), never PR CI. gitleaks detect --no-git (directory mode, matching siloscan's depth-1 scope) and siloscan over the same trees: the committed corpus, a leaky-repo clone, and the local cargo-registry checkout (targets fixed in #49's resolution).

Output: machine-readable TSV from the script; the written disposition for every delta lands as a comment here, per the repo convention that reports live on the tracker. Each delta must land in exactly one bucket: siloscan-misses (conversion bug or over-aggressive allowlist - fix), siloscan-extra (generic rules working, or FP - decide per shape), or cosmetic (span/rule-id difference; matters only if the fingerprint moves).

[aksOps]

First full parity run, 2026-08-07. Harness: scripts/parity_gitleaks.py (PR #54), gitleaks 8.30.1 pinned, directory mode, joined by (file, line). TSVs preserved in the session measurement directory; both targets re-scan reproducibly.

Delta counts

target	agree	siloscan-missing	siloscan-extra	both-different-rule
cargo registry (1.8 GB)	296	449	130	8
leaky-repo	(rest)	4	12	14

Dispositions - registry

449 siloscan-missing decomposes into exactly three gitleaks rules:

• 380 x generic-api-key - the rule the converter deliberately excludes (oversized regex; generic.yaml is its narrower replacement, decision recorded on Recover the 8 gitleaks rules the converter skips #53). This is the exclusion's measured cost, and it argues for the exclusion: the samples are overwhelmingly crate test fixtures (age testdata, aws-lc-rs cipher examples) - shapes a scanner should arguably stay quiet on. The real cost hides in the leaky-repo rows below. Disposition: exclusion stands; the narrow real shapes it covers get targeted rules, not a 380-finding import.
• 60 x age-secret-key - a conversion-fidelity candidate, the first this diff has produced. siloscan's converted rule reports 33 in the same tree where gitleaks reports 93; the misses cluster in age's own deliberately-malformed testkit fixtures (hmac_bad, header_crlf, fail_*). Suspects: the converted entropy threshold or keyword prefilter rejecting what gitleaks accepts, or binary-sniff skipping fixture files. Disposition: investigate - the one open conversion-fidelity question. Needs a per-file trace before any rule change.
• 9 x pkcs12-file - committed .p12/.pfx keystores (openssl, azure_identity test certs). Path-only rule, already dispositioned to Recover the 8 gitleaks rules the converter skips #53 (walk-layer findings). Disposition: closes when Recover the 8 gitleaks rules the converter skips #53 lands; these 9 files are its acceptance test.

130 siloscan-extra (generic-secret-assignment 71, generic-authorization-header 27, generic-credentialed-url 27, generic-password-assignment 4, private-key 1): our generic rules firing where gitleaks has nothing. Consistent with the 434-finding precision baseline on #49 - mostly fixtures, upper bound on FPs, and the number future rule changes are measured against. Disposition: generic rules working as designed; no action from this diff.

8 both-different-rule: generic-api-key vs secrets.generic-secret-assignment - both tools report the same line under their generic umbrella. Disposition: cosmetic.

Dispositions - leaky-repo

• 4 siloscan-missing, all generic-api-key: the .docker/.dockercfg and .docker/config.json base64 auth blobs. Unlike the registry's 380, these are real credentials (base64 of user:password). This is the excluded rule's genuine cost, it is four lines large, and it is precisely the Base64-encoded credentials: decode, or document as out of reach #52 gate's territory: an encoded credential missed by every pattern rule. The Base64-encoded credentials: decode, or document as out of reach #52 trigger condition is now met on an external target - Base64-encoded credentials: decode, or document as out of reach #52 activates when its blockers (Corpus expansion: cover the shapes the rules are judged on #49, Recover the 8 gitleaks rules the converter skips #53) close, with the docker auth blob as its named test case. A targeted docker-auth rule may satisfy it without any decode layer; that is Base64-encoded credentials: decode, or document as out of reach #52's call to make.
• 12 siloscan-extra: sftp.json, logins.json, .bash_profile shapes gitleaks misses - generic rules ahead of gitleaks here. Disposition: working as designed; candidates for corpus positives.
• 14 both-different-rule: same lines, generic-vs-generic naming. Cosmetic.

Summary

Conversion fidelity is in better shape than the raw 449 suggested: one genuine investigation (age-secret-key), one already-scheduled recovery (#53's pkcs12), one deliberate exclusion whose cost is now measured and acceptable, and a four-line real-world miss that formally arms the #52 trigger.

[aksOps]

Resolved: parity harness merged in PR #54, first full run dispositioned in the comment above. Outcomes: generic-api-key exclusion stands (measured), pkcs12-file recovery already scheduled (#53, with 9 named acceptance files), the docker auth-blob miss formally arms the #52 trigger, and the one open fidelity question is now #55. Re-run the harness after any rule change.