Update contrib/mixin alerting thresholds #20917
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: tjungblu The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
With the help of @dgoodwin we were able to identify better threshold based on our fleet telemetry. Along with it, I wanted to contribute some minor improvements we did to our alerts over the years. Here's a summary by Claude: Alert Severity Changes - etcdMembersDown: Increased severity from warning to critical (alerts/alerts.libsonnet:10) Improved Alert Descriptions - etcdInsufficientMembers: Enhanced description with detailed troubleshooting guidance about control plane nodes, network connectivity, and the impact on Kubernetes APIs (alerts/alerts.libsonnet:20-21) Alert Query Improvements - etcdHighNumberOfLeaderChanges: Rewrote query to use changes(etcd_server_is_leader) instead of increase(etcd_server_leader_changes_seen_total), changed time window from 15m to 10m (alerts/alerts.libsonnet:30) More Aggressive Disk Performance Thresholds - etcdHighFsyncDurations (warning): Lowered threshold from 0.5s to 0.05s (alerts/alerts.libsonnet:47) - etcdHighFsyncDurations (critical): Lowered threshold from 1s to 0.07s (alerts/alerts.libsonnet:56) - etcdHighCommitDurations (warning): Lowered threshold from 0.25s to 0.08s (alerts/alerts.libsonnet:65) - etcdHighCommitDurations (critical): Added new critical alert at 0.1s threshold (alerts/alerts.libsonnet:74-87) Database Quota Alerts - etcdDatabaseQuotaLowSpace: Added tiered alerts at 65% (info), 75% (warning), and lowered critical from 95% to 85% (alerts/alerts.libsonnet:89-121) Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
tjungblu
force-pushed
the
upstream_alerts
branch
from
54e9e68 to
3d537d1
Compare
|
@tjungblu: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted filessee 72 files with indirect coverage changes @@ Coverage Diff @@
## main #20917 +/- ##
==========================================
+ Coverage 66.22% 70.07% +3.85%
==========================================
Files 422 406 -16
Lines 34839 34289 -550
==========================================
+ Hits 23071 24029 +958
+ Misses 10357 8867 -1490
+ Partials 1411 1393 -18 Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
I am not sure whether this will increase the volume of noises with such lower thresholds. What's the real effect in your production? |
| alert: 'etcdHighNumberOfLeaderChanges', | ||
| expr: ||| | ||
| increase((max without (%(etcd_instance_labels)s) (etcd_server_leader_changes_seen_total{%(etcd_selector)s}) or 0*absent(etcd_server_leader_changes_seen_total{%(etcd_selector)s}))[15m:1m]) >= 4 | ||
| avg by (job) (changes(etcd_server_is_leader{%(etcd_selector)s}[10m])) > 5 |
There was a problem hiding this comment.
I think etcd_server_leader_changes_seen_total might be a little better, it's exactly designed for this alert.
3 participants
With the help of @dgoodwin we were able to identify better threshold based on our fleet telemetry. Along with it, I wanted to contribute some minor improvements we did to our alerts over the years.
Here's a summary by Claude:
Alert Severity Changes
Improved Alert Descriptions
Alert Query Improvements
More Aggressive Disk Performance Thresholds
Database Quota Alerts
Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.