Skip to content

Fix flaky tests caused by gen_workflow_hash leaking the logger level - #5060

Open
lmac-1 wants to merge 1 commit into
mainfrom
fix-logger-level-leak-in-tests
Open

Fix flaky tests caused by gen_workflow_hash leaking the logger level#5060
lmac-1 wants to merge 1 commit into
mainfrom
fix-logger-level-leak-in-tests

Conversation

@lmac-1

@lmac-1 lmac-1 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Description

This PR fixes a source of test flakiness in mix lightning.gen_workflow_hash and adds a safety net so it can't happen again unnoticed.

start_repo/0 in lib/mix/tasks/gen_workflow_hash.ex always called Logger.configure(level: :error) before doing its work. That's fine when the task runs as a real CLI command, since it boots a fresh app and exits right after.

The problem is that the task is also called in-process, from test/mix/tasks/gen_workflow_hash_test.exs. In that case, Logger.configure/1 changes the logger level for the entire test run, not just this task, and it never gets reset. Since Logger level is global, any later test that uses capture_log could see the wrong logs and pass or fail depending on test order. That's exactly the kind of bug that looks like random flakiness because it only shows up depending on which tests happen to run afterwards.

The fix checks whether Lightning.Repo is already running (Process.whereis/1). If it is, we're already inside a running app (a test or an IEx session), so there's nothing to boot and no reason to touch the logger. If it isn't running, we're in a real CLI invocation and everything works as before.

To make sure this can't quietly happen again, test/test_helper.exs now records the logger level before the suite runs and checks it again after. If it's different, the whole run fails loudly with a message pointing at the likely cause, instead of just letting things get flakier over time.

Validation steps

  1. Run mix test test/mix/tasks/gen_workflow_hash_test.exs. It should pass, and with the fix in place, start_repo skips the Logger.configure call because the test runs inside an app that's already booted.
  2. Run the full suite with mix test and confirm it finishes without the new after-suite warning firing.
  3. To see the safety net actually catch something, temporarily undo the start_repo guard (put back the unconditional Logger.configure(level: :error)) and rerun the full suite. It should now fail loudly with the "Logger level was left at..." message instead of passing silently.

Additional notes for the reviewer

  1. This only changes behaviour for the in-process path (tests, IEx). Running the task from a real CLI works exactly the same as before.
  2. The after-suite check isn't specific to this one task. Any test or mix task that lowers the logger level and forgets to restore it will now get caught by this check too.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

  • I have used Claude Code
  • I have used another model
  • I have not used AI

You can read more details in our
Responsible AI Policy

Pre-submission checklist

  • I have performed an AI review of my code (we recommend using /review
    with Claude Code)
  • I have implemented and tested all related authorization policies.
    (e.g., :owner, :admin, :editor, :viewer)
  • I have updated the changelog.
  • I have ticked a box in "AI usage" in this PR

@github-project-automation github-project-automation Bot moved this to New Issues in Core Aug 12, 2026
@lmac-1
lmac-1 marked this pull request as ready for review August 12, 2026 08:36
@lmac-1 lmac-1 changed the title Stop gen_workflow_hash from leaking Logger level into other tests Fix flaky tests caused by gen_workflow_hash leaking the logger level Aug 12, 2026
@github-actions

Copy link
Copy Markdown

Security Review ✅

  • S0 (project scoping): N/A — diff only touches CHANGELOG.md, a mix-task logger fix in lib/mix/tasks/gen_workflow_hash.ex, and test/test_helper.exs; no new queries or web-layer entrypoints.
  • S1 (authorization): N/A — no new create/read/update/delete actions or LiveView/controller handlers introduced.
  • S2 (audit trail): N/A — no Repo.insert/update/delete on config resources; changes are test-hygiene and CLI logger-level scoping only.

The task lowers the global Logger level to keep boot chatter out of the printed
hash, and never put it back. Called in-process — from a test, or an IEx session
— that silenced every later log assertion, so unrelated tests failed depending
on run order.

Save the level and restore it in an `after` block. That covers every caller
rather than only the ones that boot the Repo, and keeps the already_started
clause meaningful.

The new test pins the level before the call instead of reading it, so a leak
from an earlier test in the file can't make the assertion trivially true.
@lmac-1
lmac-1 force-pushed the fix-logger-level-leak-in-tests branch from 9acc2fc to a02bcc3 Compare August 12, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New Issues

Development

Successfully merging this pull request may close these issues.

1 participant