Fix flaky tests caused by gen_workflow_hash leaking the logger level - #5060
Open
lmac-1 wants to merge 1 commit into
Open
Fix flaky tests caused by gen_workflow_hash leaking the logger level#5060lmac-1 wants to merge 1 commit into
lmac-1 wants to merge 1 commit into
Conversation
lmac-1
marked this pull request as ready for review
August 12, 2026 08:36
Security Review ✅
|
The task lowers the global Logger level to keep boot chatter out of the printed hash, and never put it back. Called in-process — from a test, or an IEx session — that silenced every later log assertion, so unrelated tests failed depending on run order. Save the level and restore it in an `after` block. That covers every caller rather than only the ones that boot the Repo, and keeps the already_started clause meaningful. The new test pins the level before the call instead of reading it, so a leak from an earlier test in the file can't make the assertion trivially true.
lmac-1
force-pushed
the
fix-logger-level-leak-in-tests
branch
from
August 12, 2026 13:40
9acc2fc to
a02bcc3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes a source of test flakiness in
mix lightning.gen_workflow_hashand adds a safety net so it can't happen again unnoticed.start_repo/0inlib/mix/tasks/gen_workflow_hash.exalways calledLogger.configure(level: :error)before doing its work. That's fine when the task runs as a real CLI command, since it boots a fresh app and exits right after.The problem is that the task is also called in-process, from
test/mix/tasks/gen_workflow_hash_test.exs. In that case,Logger.configure/1changes the logger level for the entire test run, not just this task, and it never gets reset. Since Logger level is global, any later test that usescapture_logcould see the wrong logs and pass or fail depending on test order. That's exactly the kind of bug that looks like random flakiness because it only shows up depending on which tests happen to run afterwards.The fix checks whether
Lightning.Repois already running (Process.whereis/1). If it is, we're already inside a running app (a test or an IEx session), so there's nothing to boot and no reason to touch the logger. If it isn't running, we're in a real CLI invocation and everything works as before.To make sure this can't quietly happen again,
test/test_helper.exsnow records the logger level before the suite runs and checks it again after. If it's different, the whole run fails loudly with a message pointing at the likely cause, instead of just letting things get flakier over time.Validation steps
mix test test/mix/tasks/gen_workflow_hash_test.exs. It should pass, and with the fix in place,start_reposkips theLogger.configurecall because the test runs inside an app that's already booted.mix testand confirm it finishes without the new after-suite warning firing.start_repoguard (put back the unconditionalLogger.configure(level: :error)) and rerun the full suite. It should now fail loudly with the "Logger level was left at..." message instead of passing silently.Additional notes for the reviewer
AI Usage
Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):
You can read more details in our
Responsible AI Policy
Pre-submission checklist
/reviewwith Claude Code)
(e.g.,
:owner,:admin,:editor,:viewer)