MON-4059: split TelemeterClientFailures into send/retrieve alerts - #550
MON-4059: split TelemeterClientFailures into send/retrieve alerts#550slashpai wants to merge 1 commit into
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@slashpai: This pull request references MON-4059 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
Bot
added
the
approved
|
@slashpai: This pull request references MON-4059 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
simonpasquier
left a comment
simonpasquier
left a comment
There was a problem hiding this comment.
I'd recommend to create 2 alerting rules: one for the federate_from client and another one for the federate_to client. They can have the same name but different descriptions.
| sum by (namespace) ( | ||
| rate(federate_requests_total{job="telemeter-client"}[15m]) | ||
| ) > 0.2 | ||
| sum by(client, status_code) (rate(metricsclient_http_requests_total{status_code!~"200"}[15m])) > 0 |
There was a problem hiding this comment.
(suggestion) we need to preserve the namespace label. Let's also evaluate the rate of failed requests.
| sum by(client, status_code) (rate(metricsclient_http_requests_total{status_code!~"200"}[15m])) > 0 | |
| sum by(client, status_code,namespace) (rate(metricsclient_http_requests_total{status_code!~"2..",job="telemeter-client"}[15m])) | |
| / | |
| on(client, namespace) group_left() sum by(client, namespace) (rate(metricsclient_http_requests_total{job="telemeter-client"}[15m])) > 0.2 |
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
openshift-ci
Bot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle rotten |
openshift-ci
Bot
added
lifecycle/rotten
|
Rotten issues close after 30d of inactivity. Reopen the issue by commenting /close |
|
@openshift-bot: Closed this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/reopen |
|
@simonpasquier: Reopened this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@slashpai: This pull request references MON-4059 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
remove-lifecycle rotten |
|
/remove-lifecycle rotten |
openshift-ci
Bot
removed
the
lifecycle/rotten
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: slashpai The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
openshift-ci
Bot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle rotten |
openshift-ci
Bot
added
lifecycle/rotten
|
Rotten issues close after 30d of inactivity. Reopen the issue by commenting /close |
openshift-ci
Bot
removed
the
lifecycle/rotten
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
openshift-ci
Bot
added
the
lifecycle/stale
|
/remove-lifecycle stale |
openshift-ci
Bot
removed
the
lifecycle/stale
|
/test e2e-aws-ovn |
openshift-ci
Bot
added
the
needs-rebase
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📝 WalkthroughWalkthroughThe change replaces Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to The directional Telemeter alerts are covered for threshold, duration, and successful-request behavior, with no current merge-blocking risk identified. Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 7d316c0c-febf-473e-8265-7c5f5b2ef4e6
📒 Files selected for processing (1)
test/rulestests.yaml
Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
openshift-ci
Bot
removed
the
needs-rebase
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: ca877c23-d779-4084-b583-6cfd716d2211
📒 Files selected for processing (1)
test/rulestests.yaml
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.
|
Tested the alert with wrong value set for 
To test 
|
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
Replace deprecated federate_requests_* based alert with two focused
alerts using metricsclient_http_requests_total{status_code!~"2.."}:
- TelemeterClientSendErrors (client="federate_to")
- TelemeterClientRetrieveErrors (client="federate_from")
Each alert description distinguishes 4xx (user-actionable) from
5xx (Red Hat service issue). Threshold: >20% error rate for 1h.
Also extend check-rules to validate alert rules and add promtool
unit tests in rulestests.yaml.
Signed-off-by: Jayapriya Pai <[email protected]>
|
@slashpai: This pull request references MON-4059 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.1.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
slashpai
changed the title
|
@slashpai: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@simonpasquier I have revived this PR and updated per suggestions. Included tests and locally tested as above. Take a look when you get chance |
| The telemeter client in namespace {{ $labels.namespace }} has {{ $value | humanizePercentage }} error rate when sending metrics to the telemeter service. | ||
| 4xx errors typically indicate an authentication or authorization issue — check that the global pull secret is valid and up to date: | ||
| oc get secret pull-secret -n openshift-config -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | ||
| 5xx errors indicate a server-side issue on Red Hat telemeter service and are not actionable by the cluster administrator. |
There was a problem hiding this comment.
🤔 do we even need to alert in this case? maybe we just need an info alert for 5xx responses.
simonpasquier
left a comment
simonpasquier
left a comment
There was a problem hiding this comment.
what do we do with the runbook? obviously we need to keep it for supported OCP releases but we can either extend it to cover the new alerting rules or create dedicated runbooks for each (my preference would be option 2 as it's more obvious for customers and we could eventually remove the old runbook).
Ya. Since we can
We can add new ones since we currently one runbook per alert |
5 participants
Replace
TelemeterClientFailureswith two alerts onmetricsclient_http_requests_total:federate_to) — upload failures to telemeterfederate_from) — federation failures from PrometheusBoth fire when non-2xx rate > 20% for 1h. Descriptions distinguish 4xx (cluster-admin actionable) from 5xx (Red Hat service).
Ref: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/TelemeterClientFailures.md#diagnosis