test: per-family detection corpus and gitleaks parity harness (#49, #50) - #54
Merged
Conversation
…ty harness Five new shape families under tests/corpus/tree: xml (probing the markup allowlist), k8s (including base64 data: values), urls (all-vocabulary passwords), noise-code (amalgamations, generated bindings) and noise-artifacts (lockfiles, minified, certificates). 551 manifest rows total, 258 new. The harness now enforces a recall floor per family, set to the value measured at introduction, with precision a global 1.0 on classified rows. Known gaps start low by design: urls 0.4783 (#44), xml 0.5926 (#51), k8s 0.8333 (#53). No rule file changes. scripts/parity_gitleaks.py diffs siloscan against a checksum-pinned gitleaks 8.30.1 in directory mode and buckets every delta.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First two tickets of the v1.6 map (#48). Measurement only - no rule file changes, enforced and verified.
Corpus (#49)
Five new families, 258 new manifest rows (551 total, 60 files). Floors are now per-family, set to the measured value at introduction; precision stays a global 1.0 on classified rows:
The low floors are the point: they are the measured shape of the deferred debts, and the tickets that close them now have a number to move.
Verified independently of the authoring agents: 903 workspace tests, corpus suite 6/6, fmt and clippy clean, zero rule files touched, and the committed corpus scans raw-clean (0 findings on unmaterialized markers - push-protection-safe, same as before).
Parity harness (#50)
scripts/parity_gitleaks.py: checksum-pinned gitleaks 8.30.1, directory mode, joins findings by (file, line), buckets every delta (siloscan-missing / siloscan-extra / both-different-rule / agree). stdlib only, manual execution per the #50 resolution.
First leaky-repo results are already interesting: the .docker auth-blob misses appear as siloscan-missing against gitleaks' generic-api-key - the rule we deliberately excluded now has a measured cost. Disposition lands on #50, not here.
Baselines recorded on #49: registry tree 434 findings / 84,063 files / 241s; leaky-repo 17 of 43 seeded files detected.