Add a recurrence check to ops-report and narrow the Unhealthy threshold - #201
Merged
Conversation
Every flagged anomaly carries a line saying whether the same thing has happened recently, so a burst that clears itself is not read as an isolated event on its sixth appearance. Step 6 records each anomaly's predicate: expression, direction, and a threshold resolved to an absolute number, since recomputing a median-relative rule over the lookback would measure a different quantity. Step 6b re-evaluates that predicate over max(14 days, twice the report window), aggregated hourly, keeping the by(...) breakdown. The check fires in two batched rounds: an instant count per predicate, then range queries only for predicates returning 24 hours or fewer, so one that matches most hours cannot flood the context. Anomalies with no predicate behind them report as not checkable rather than asserting a first occurrence that was never measured. Step 8 re-runs its log queries over the prior occurrence's hour to compare signatures. The report template and Slack summary carry the line. Recurrence does not affect the health assessment. Claude-Session: https://claude.ai/code/session_01DFUnAKiPTqRrxMr2pMvygC
Severity measures a spike's peak and duration measures whether the service recovered. A critical spike confined to one or two consecutive data points, on a window whose success rate stays above 99%, lands at Degraded; three or more consecutive points is Unhealthy. The old rule scored feature flags Unhealthy in both regions on Aug 26, a day that served 99.9987% of requests successfully with a single 5-minute bucket above the threshold and every neighbouring bucket in single digits. Healthy tightens from "no sustained error spikes" to "no error spikes" so the three rows are disjoint. Claude-Session: https://claude.ai/code/session_01DFUnAKiPTqRrxMr2pMvygC
The count round returned one series per label set, and the procedure summed those counts and reported the total as a number of hours. On a pod-labeled predicate that sum reaches 1756 against a 336-hour lookback where the true answer is 99, and it pushes rare anomalies past the 24 cutoff into "not checkable". Wrapping the comparison in count() makes the result a bucket count and collapses the round to one row. Bucket size now follows the report window as max(1h, step). An hourly bucket cannot hold a month report's two-hour data point, so a spike split across two hours cleared neither bucket and the report claimed a first occurrence it never measured. The percentile subquery samples at the rate interval Step 5 used rather than a fixed 15m, which left two thirds of each hour unsampled on a day report. Also records Critical run lengths in Step 6, widens the no-predicate carve-out to trends and bidirectional drift rules, moves the promoted log pattern case to Step 8, and budgets tokens against Step 8's log queries. Claude-Session: https://claude.ai/code/session_01DFUnAKiPTqRrxMr2pMvygC
There was a problem hiding this comment.
🟢 Approval recommended
Changes are documentation/template-only updates that are internally consistent, well-integrated with the existing step structure, and introduce no apparent formatting or logic issues.
Pull request overview
This PR updates the ops-report skill documentation and report template to support a recurrence check for anomalies (so repeats are written up as patterns) and to refine the overall health classification so brief critical spikes don’t automatically mark a service Unhealthy.
Changes:
- Adds a “Recurrence” line to both action items and anomaly writeups in the report template.
- Documents Step 6b (recurrence check) including how to record anomaly predicates, how to count/bucket occurrences safely, and when recurrence is “not checkable”.
- Narrows the Unhealthy threshold so critical error spikes confined to 1–2 consecutive points (with >99% success rate) are Degraded, while 3+ consecutive points remain Unhealthy.
File summaries
| File | Description |
|---|---|
| ai/skills/ops-report/templates/report-template.md | Adds placeholders for recurrence reporting in action items and anomaly sections. |
| ai/skills/ops-report/SKILL.md | Documents predicate capture + recurrence evaluation workflow and refines status criteria around critical spike duration. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a recurrence check to the ops-report skill so a repeat anomaly gets written up as a pattern instead of a fresh event, and narrows when one short error spike makes a service Unhealthy.
Step 6 now records the test that flagged each anomaly: the expression, the direction, and a resolved absolute threshold. Step 6b re-runs that test over the past 14 days, or 60 for a month report, and writes an occurrence count into the action item, the anomaly section, and the Slack digest. Step 8 re-queries logs over the most recent prior occurrence so the two signatures can be compared. Anomalies with no threshold behind them (trends, bidirectional drift rules, log patterns with no counter) report as not checkable rather than guessing at a query.
The status table no longer sends every Critical-severity spike to Unhealthy. A spike confined to one or two consecutive data points, on a window whose success rate stays above 99%, lands at Degraded. Three or more consecutive points is still Unhealthy, and Healthy now requires no error spikes at all so the three rows stay disjoint.
Two details in the recurrence queries are worth a look, since both decide the occurrence number the report prints:
The count round wraps its comparison in
count(...). Without that wrapper the query returns one series per label set, and summing those counts gives label-buckets rather than buckets. On a pod-labeled predicate against prod-us that sum reaches 1756 over a 336-hour lookback where the true answer is 99, which both prints an impossible number and pushes rare anomalies past the 24 cutoff into "not checkable".Bucket size is
max(1h, {step})rather than a fixed hour. A month report's data point is two hours, so an hourly bucket cannot hold one: a spike split 250/250 across two hours clears neither bucket, and the report would claim a first occurrence it never measured.Test plan
npx markdownlint-cli2 ai/skills/ops-report/SKILL.md ai/skills/ops-report/templates/report-template.mdpasses./ops-report feature-flagson a day with a known anomaly. The Recurrence line appears in both the action item and the anomaly section.(N occurrences in 14 days)only when N is above 1, and omits it on a first occurrence or a not-checkable one.https://claude.ai/code/session_01DFUnAKiPTqRrxMr2pMvygC