Many outages never announce themselves as outages. They show up as rising latency or an error rate that creeps from 2% to 4% over an afternoon while everyone’s busy with something else. The site is up. Nothing has paged. Something is still wrong, and by the time it’s obvious, it’s been wrong for an hour.
Catching the server that falls over is the easy case. The hard one is deciding which of these slow, quiet changes should pull someone out of bed. Most teams get that wrong, and they get it wrong by monitoring too specifically.
Start From a Baseline
A CPU spike on one box out of a hundred running the same workload isn’t worth a phone call. The same spike on your only server might be. It depends entirely on the case, and that’s the part people skip.
The number on its own barely tells you anything. Take a 2% error rate. If it’s been flat at 2% all year, that’s your baseline. It’s what normal looks like for you, and it doesn’t need an alert. It needs one when it moves, and moves by more than your usual variation.
Monitor the Outcome
The same problem shows up in how most teams watch an application: Every part in isolation. A check on each endpoint, each page, each box. All of them can return a clean 200 while the one thing your customer came to do is quietly broken.
Monitoring the outcome works better. Log in the way a user would, and measure whether the login completes, how long it takes, and what error you get when it fails. Or walk the whole path: Open the site, drop a couple of products in the basket, go to checkout, and confirm you reach the payment screen. Now you’re testing what the user actually experiences.
It also tells you what to ignore. If a few of your hundred servers drop out but login still works, send a note to a non-urgent channel and look in the morning. If nothing the user touches is affected, it doesn’t warrant a page.
When Outages Actually Happen
The timing is worth taking seriously. Across 1.8 million outages on thousands of sites, Oh Dear found that about 68% started outside the 9-to-6 workday, on weeknights and weekends. The same data is a caution about averages. The median outage was over in under two minutes, but the mean was close to 22, because a thin tail of multi-hour failures pulls it up.
So design your alerting around the rare long outage, the one that runs for hours, and assume it lands outside working hours, because most do. The common short one usually clears before you’ve even read the page.
Alert Fatigue
Alert fatigue wrecks more teams than any single outage. Monitor every detail and you drown in notifications, and after a while, they stop meaning anything. The on-call engineer glances, dismisses, goes back to work, and one day dismisses the alert that actually mattered along with the rest. It’s the boy who cried wolf.
The fix I trust most is dull: At a hosting company where I ran engineering, the team sat down every Monday and went through the past week’s alerts, urgent and not. For each one, we asked out loud whether it was useful, whether it warranted an action, and whether the severity was right. The point was to bring the off-hours noise down deliberately, and to move the work earlier — to catch the warning sign at 2 in the afternoon and deal with it then, instead of being paged for it at 3 in the morning.
Anomaly detection helps, but only with a duration attached. A 2% baseline that jumps to 10% for thirty seconds and settles again probably isn’t worth waking anyone. See it in the morning, write a short postmortem, tune the alert. A spike that’s already gone is tomorrow’s job. Get that distinction right and you shed a lot of pages without missing anything that counts.
One Question for Your Team
There’s one question worth putting to a team out loud: Would a customer notice this before we do? People in the middle of using your product spot a slow page or a failed checkout fast. If they’re the ones telling you, your monitoring missed it.
from DevOps.com https://ift.tt/1xgqwAX
Comments
Post a Comment