Handling Alert Storms and Outages in ControlUp Scoutbees: Current Best Practices and Upcoming Features

During an internet connectivity outage in Italy, a large volume of Scoutbees alerts was triggered due to multiple login request failures, resulting in an alert storm. This situation prompted a request for a dependency model that would suppress downstream alerts when a critical component like a gateway is detected as unavailable. Alternatively, the user inquired about automatically triggering a maintenance window to temporarily disable alerts for affected scouts during such outages.

To address this, the current recommended approach involves tuning alert settings to reduce noise while preserving important signals. Key adjustments include configuring alerts to notify only when test results return to normal, thereby limiting notifications to one alert at the start of an issue and one upon resolution. Additionally, increasing the threshold for consecutive failed tests before sending an alert helps ignore brief transient failures but still catches sustained outages. These simple changes significantly reduce alert volume during connectivity issues without missing critical events.

Looking ahead, the ControlUp team is actively developing a feature to enable maintenance windows by tag. This will allow users to silence all scouts associated with a specific tag—such as “italy”—with a single action during known problems or outages. This functionality aims to streamline manual alert silencing in complex environments without risking important signals being suppressed automatically. Automatic suppression of downstream alerts in the event of gateway failure is discouraged since synthetic monitoring is designed to reflect real user experience, and connectivity failures are essential to surface.

Until the tag-based maintenance feature becomes available, the user opted to implement manual maintenance windows that disable alerts during outages to avoid missing issues outside planned maintenance periods. The approach balances the need to reduce alert fatigue while ensuring critical incidents are not overlooked. The ControlUp team remains open to direct conversations with users to refine upcoming features for scenarios such as this.

For further detail on tuning Scoutbees alerts and maintenance windows, see the ControlUp documentation at https://docs.controlup.com. The upcoming maintenance-by-tag feature will enhance flexibility in managing alert noise during large-scale or regional disruptions.

Read the entire ‘Handling Alert Storms and Outages in ControlUp Scoutbees: Current Best Practices and Upcoming Features’ thread below:

Last night we experienced internet connectivity problems in Italy. As a consequence, login requests failed and we received hundreds of Scoutbees alerts.
I was wondering whether it would be possible to implement a dependency model, so that when a critical component such as a gateway is detected as unavailable, Scoutbees would stop generating additional downstream alerts for all affected checks.
Alternatively, could a maintenance window be triggered automatically in such situations, temporarily disabling the relevant scouts until the underlying connectivity issue has been resolved?
How would you recommend handling this kind of alert storm?

Cc

Hi,

Thanks for sharing this, and sorry about the alert storm. Here’s how I’d handle it today, plus what’s coming.

1. Tune the alert settings (quickest win): In the alert configuration, you have a few levers that cut the noise a lot (screenshot attached):
a. Set “After that, notify me again” to “when test results return to normal.” You get one alert when the problem starts and one when it recovers, instead of a notification every time the conditions are met. During an outage like last night’s, this alone should reduce the volume dramatically.

b. Raise “Consecutive tests meeting conditions before notification is sent.” At 1, a single failed run triggers an alert. Setting it to 2 or 3 means a brief blip won’t notify you, and a sustained outage still will. The trade-off is slightly slower detection, since you’re effectively alerting only after X minutes of continuous failure.

2. Maintenance by tags: Putting scouts into maintenance by tag is on our roadmap. That would let you silence all scouts tagged italy in one action during a known issue.

A note on automatic suppression
We’d be careful about automatically suppressing downstream alerts when a gateway fails. Synthetic monitoring exists to show you what your users experience, so a connectivity failure is exactly the kind of thing you want to know about. Our goal is to help you cut the noise while keeping the signal, which is why we’re focusing on smarter thresholds and tag-based maintenance rather than silencing checks automatically.

One more thing! Since your use case is so relevant to the maintenance-by-tag work, would you be open to a short chat with me? I’d love to make sure what we build really covers your situation. Just let me know and I’ll reach out directly.

Thanks again for the detailed feedback!

Thank you for your detailed response and suggestions.
As we’re concerned that setting the notifications to “when test results return to normal” could cause us to miss an issue if only a single notification is sent, we’ve decided to create a maintenance window that disables all alerts.
It does require manual activation, but at least this way we can make sure we don’t miss any issues outside of planned maintenance periods.

Feel free to send me a DM if you have any questions or if there’s anything I can clarify that might help you with the maintenance-by-tag feature.

Thanks again!

Continue reading and comment on the thread ‘Handling Alert Storms and Outages in ControlUp Scoutbees: Current Best Practices and Upcoming Features’.  Not a member? Join Here!