Handling Alert Storms and Outages in ControlUp Scoutbees: Current Best Practices and Upcoming Features
During an internet connectivity outage in Italy, a large volume of Scoutbees alerts was triggered due to multiple login request failures, resulting in an alert storm. This situation prompted a request for a dependency model that would suppress downstream alerts when a critical component like a gateway is detected as unavailable. Alternatively, the user inquired about automatically triggering a maintenance window to temporarily disable alerts for affected scouts during such outages.
To address this, the current recommended approach involves tuning alert settings to reduce noise while preserving important signals. Key adjustments include configuring alerts to notify only when test results return to normal, thereby limiting notifications to one alert at the start of an issue and one upon resolution. Additionally, increasing the threshold for consecutive failed tests before sending an alert helps ignore brief transient failures but still catches sustained outages. These simple changes significantly reduce alert volume during connectivity issues without missing critical events.
Looking ahead, the ControlUp team is actively developing a feature to enable maintenance windows by tag. This will allow users to silence all scouts associated with a specific tag—such as “italy”—with a single action during known problems or outages. This functionality aims to streamline manual alert silencing in complex environments without risking important signals being suppressed automatically. Automatic suppression of downstream alerts in the event of gateway failure is discouraged since synthetic monitoring is designed to reflect real user experience, and connectivity failures are essential to surface.
Until the tag-based maintenance feature becomes available, the user opted to implement manual maintenance windows that disable alerts during outages to avoid missing issues outside planned maintenance periods. The approach balances the need to reduce alert fatigue while ensuring critical incidents are not overlooked. The ControlUp team remains open to direct conversations with users to refine upcoming features for scenarios such as this.
For further detail on tuning Scoutbees alerts and maintenance windows, see the ControlUp documentation at https://docs.controlup.com. The upcoming maintenance-by-tag feature will enhance flexibility in managing alert noise during large-scale or regional disruptions. Read the entire article here...
