Repository navigation
Stop repeating persistent failure alerts and digest routine ones - #10
Merged
Merged
Conversation
A failure that persisted (a stale KMS status receipt) was queued again on every check once its previous occurrence was delivered, so one condition sent 380 emails in a day. Remember active failures and re-queue the same failure at most once a day until a successful check clears it. Send critical alerts (severity error, tls-regression) at once as before. Hold the rest until the oldest has waited an hour, then deliver them as one digest with its own durable receipt that acknowledges every member together.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production sent 380
monitor-failureemails between Oct 13 14:00 and Oct 14 11:34 UTC for one condition: the KMS status receipt stayed stale while scheduled issuance was blocked.enqueue_failureonly deduplicated while an occurrence was pending, so each delivered one was followed by a new one on the next check.activeFailures). The same failure is queued again at most once a day while it persists; a successful check clears it.error, which covers every monitor failure, andtls-regression) still go out immediately, one per message.queuedAt; older alerts without it are treated as due.Tests: full suite 181 run, 180 pass, 1 skip (pinned qScan integration, which CI runs). New tests cover the persistent-failure dedup and daily reminder, the hourly digest with fixed members, and critical alerts bypassing the digest.