Repository navigation
fix(gitleaks): scan markdown-escaped lines un-escaped - #39
Merged
Merged
Conversation
Markdown exporters, Google Drive's included, backslash-escape ASCII punctuation, so ghp_... is written as ghp\_... and matches no rule. The Scan step passed a GitHub token committed that way. Add a step that takes every added line carrying a backslash escape, un-escapes all ASCII punctuation (repeated backslashes included), and runs it through gitleaks stdin with the same config. Findings are mapped back to commit and file. The step: - covers merge-only content via --diff-merges=combined - uses byte semantics (LC_ALL=C, awk) so a NUL or invalid UTF-8 byte cannot silently empty the stream - runs even when the Scan step failed - refuses stdin-style .gitleaksignore entries, whose line-based fingerprints would silence the next secret on that line tests/gitleaks-md-escaped.test.sh runs the step's exact run: block against throwaway repos (13 cases).
The ok check was an always-pass echo, and tests/commit-email.test.sh was run only by hand. Run both test scripts inside ok, against the gitleaks version read from gitleaks.yml's input default so the two cannot drift. The tests run in ok itself rather than in a job it needs: a required check whose dependency failed reports as skipped, which GitHub counts as passing.
List the test scripts by glob rather than one step each, so a test added later cannot land without running. The loop runs all of them and fails if any failed.
# Conflicts: # .github/workflows/gitleaks.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This adds a Scan markdown-escaped lines step to the reusable gitleaks workflow, and makes CI run the workflow tests.
Markdown exporters backslash-escape ASCII punctuation, so
ghp_…is written asghp\_…and matches no rule. A GitHub token committed to company-docs this way passed the existing Scan step and GitHub secret scanning.How
The step takes every added line carrying a backslash escape and un-escapes all 32 ASCII punctuation characters, including repeated backslashes. It then runs those lines through
gitleaks stdinwith the same config (repo-local[extend]configs are honoured), and maps each finding back to its commit and file.Choices it makes, each covered by a test:
$ : = &and more, and the set has grown over time.password \= <value>slips past a narrower class.--diff-merges=combined. Without itgit log -pshows no diff for merges, so content introduced only by a merge resolution is never seen.LC_ALL=C, awk rather than grep). A NUL byte or invalid UTF-8 could otherwise empty the stream silently, and empty input passes.if: ${{ !cancelled() }}. One plain finding in Scan must not hide the escaped ones.:rule:lineentries in.gitleaksignore. stdin findings are fingerprinted by stream line, which shifts every commit, so such an entry would silence the next secret landing on that line. None of the org's existing ignore files has one.Known limit: stdin mode has no file paths, so the org
[allowlist] paths(tests/, fixtures/) don't apply to this step. A fixture holding an escaped token needs to build it at runtime, astests/gitleaks-md-escaped.test.shdoes. That affects 0 repos today.The CI commits make
okrun everytests/*.test.sh(so a later test cannot land unrun), against the gitleaks version read fromgitleaks.yml. The tests run inokitself, not in a job itneeds, because a required check whose dependency failed reports as skipped and counts as passing.Testing
tests/gitleaks-md-escaped.test.sh(13 cases) runs the step's exactrun:block under GitHub's shell flags: 13/13 pass with both gawk and mawk.LC_ALL=Cunder mawk, which reads input as bytes anyway; the setting stays for gawk.tests/commit-email.test.shpasses.Rollout: callers pin this workflow by digest. Renovate moves most of them within a week. company-docs' digest PR has been blocked by its
app-only-mainruleset since 2026-08-28 and will be merged by hand.