Skip to content

fix(gitleaks): scan markdown-escaped lines un-escaped - #39

Merged
CameronBrooks11 merged 4 commits into
mainfrom
fix/gitleaks-markdown-escapes
Oct 20, 2026
Merged

CameronBrooks11 merged 4 commits into
mainfrom
fix/gitleaks-markdown-escapes

Conversation

@CameronBrooks11

@CameronBrooks11 CameronBrooks11 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

This adds a Scan markdown-escaped lines step to the reusable gitleaks workflow, and makes CI run the workflow tests.

Markdown exporters backslash-escape ASCII punctuation, so ghp_… is written as ghp\_… and matches no rule. A GitHub token committed to company-docs this way passed the existing Scan step and GitHub secret scanning.

How

The step takes every added line carrying a backslash escape and un-escapes all 32 ASCII punctuation characters, including repeated backslashes. It then runs those lines through gitleaks stdin with the same config (repo-local [extend] configs are honoured), and maps each finding back to its commit and file.

Choices it makes, each covered by a test:

  • All ASCII punctuation, not a hand-picked set. The exporter escapes $ : = & and more, and the set has grown over time. password \= <value> slips past a narrower class.
  • --diff-merges=combined. Without it git log -p shows no diff for merges, so content introduced only by a merge resolution is never seen.
  • Byte semantics (LC_ALL=C, awk rather than grep). A NUL byte or invalid UTF-8 could otherwise empty the stream silently, and empty input passes.
  • if: ${{ !cancelled() }}. One plain finding in Scan must not hide the escaped ones.
  • Refuses :rule:line entries in .gitleaksignore. stdin findings are fingerprinted by stream line, which shifts every commit, so such an entry would silence the next secret landing on that line. None of the org's existing ignore files has one.

Known limit: stdin mode has no file paths, so the org [allowlist] paths (tests/, fixtures/) don't apply to this step. A fixture holding an escaped token needs to build it at runtime, as tests/gitleaks-md-escaped.test.sh does. That affects 0 repos today.

The CI commits make ok run every tests/*.test.sh (so a later test cannot land unrun), against the gitleaks version read from gitleaks.yml. The tests run in ok itself, not in a job it needs, because a required check whose dependency failed reports as skipped and counts as passing.

Testing

  • tests/gitleaks-md-escaped.test.sh (13 cases) runs the step's exact run: block under GitHub's shell flags: 13/13 pass with both gawk and mawk.
  • A mutation sweep reverted each fix in turn and confirmed its test fails. Every mutation was caught except dropping LC_ALL=C under mawk, which reads input as bytes anyway; the setting stays for gawk.
  • tests/commit-email.test.sh passes.
  • The step over the full history of every FlowMatrix-AI repo found 0 findings in about 25 s total. A synthetic positive control fired on every run.
  • Both scans pass on this repo's own history, including the new test file.

Rollout: callers pin this workflow by digest. Renovate moves most of them within a week. company-docs' digest PR has been blocked by its app-only-main ruleset since 2026-08-28 and will be merged by hand.

Markdown exporters, Google Drive's included, backslash-escape ASCII
punctuation, so ghp_... is written as ghp\_... and matches no rule.
The Scan step passed a GitHub token committed that way.

Add a step that takes every added line carrying a backslash escape,
un-escapes all ASCII punctuation (repeated backslashes included), and
runs it through gitleaks stdin with the same config. Findings are
mapped back to commit and file. The step:

- covers merge-only content via --diff-merges=combined
- uses byte semantics (LC_ALL=C, awk) so a NUL or invalid UTF-8 byte
  cannot silently empty the stream
- runs even when the Scan step failed
- refuses stdin-style .gitleaksignore entries, whose line-based
  fingerprints would silence the next secret on that line

tests/gitleaks-md-escaped.test.sh runs the step's exact run: block
against throwaway repos (13 cases).
The ok check was an always-pass echo, and tests/commit-email.test.sh
was run only by hand. Run both test scripts inside ok, against the
gitleaks version read from gitleaks.yml's input default so the two
cannot drift.

The tests run in ok itself rather than in a job it needs: a required
check whose dependency failed reports as skipped, which GitHub counts
as passing.
List the test scripts by glob rather than one step each, so a test
added later cannot land without running. The loop runs all of them
and fails if any failed.
# Conflicts:
#	.github/workflows/gitleaks.yml
@CameronBrooks11
CameronBrooks11 merged commit a374e37 into main Oct 20, 2026
2 checks passed
@CameronBrooks11
CameronBrooks11 deleted the fix/gitleaks-markdown-escapes branch October 20, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant