Alarm Storm
Designing an ML-assisted monitoring system that helps network engineers detect and diagnose alarm storms faster
Project snapshot
Designed Fujitsu’s network alarm analysis workflow to help network operators detect root causes during alarm storms affecting 10k+ alarms across enterprise networks.
Role
Impact
Timeline
Team
Users
Lead Product Designer
Reduced alarm analysis time with an experience built around ML algorithm
12 weeks
R&D ML team, Network engineers,
Product owner
Network Operations Center Engineers
The Problem: Alarm Storms
Large telecom networks generate thousands of alarms across devices, links, and services.
When infrastructure failures occur, alarms spike rapidly across the network - creating what engineers call alarm storms.
During these events engineers must quickly answer:
-
Is this a real failure or noise?
-
Which alarms are related?
-
What triggered the event?
Alarm storms can generate 5,000–20,000 alarms within minutes, making manual root cause analysis extremely difficult.
During major incidents, engineers often have to manually scan thousands of alarms to understand what was happening in the network
New Capability Introduced by R&D
Introducing Machine Learning Storm Detection
The R&D team developed a machine learning algorithm capable of detecting abnormal alarm spikes and identifying patterns in alarm activity.
However, the algorithm alone was not enough.
Engineers still needed a way to:
• Configure and train the detection model
• Understand when a storm event was detected
• Interpret alarm patterns surfaced by the algorithm
The algorithm alone was not enough,
engineers still needed a way to interpret alarm patterns
The UX Challenge
Designing for machine learning systems introduces a unique challenge.
Users must trust and interpret algorithmic outputs before acting on them.
This project focused on two key UX problems:
Problem 1 - Training the Model
Engineers needed a way to continuously train how storm detection should behave across different network environments.
This required an interface to define thresholds, training windows, and detection parameters.
Problem 2 - Interpreting ML Signals
Once the system detected storm events, engineers needed to understand:
-
Which alarms belong to the storm
-
What the root cause might be
-
What could be the possible resolution
Without a clear interpretation, ML output would become another source of noise.
The design challenge was to translate ML detection into a usable operational workflow
Designing the Experience
Training the Storm Detection Model
The first step in the workflow allows engineers to configure and train the ML model using historical alarm patterns.
This interface allows engineers to:
• Define training time windows
• Annotate with possible root cause and solutions
Instead of requiring precise timestamps, engineers could scrub across the alarm timeline and mark segments as training data.
Detecting Alarm Storm Events
Once trained, the system continuously monitors alarm streams across the network.
When alarm activity deviates significantly from normal patterns, the system detects a storm event.
Instead of listing alarms individually, the interface highlights storm activity on a timeline. This allows engineers to immediately identify abnormal behaviour.
Grouping Alarms Into Meaningful Clusters
After detecting a storm event, the system groups related alarms into clusters named ‘Events’ based on:
• Temporal proximity
• Affected devices
• Alarm type similarity
(+24 more parameters used by the ML algorithm)
This reduces hundreds of alarms into a smaller set of structured clusters.
Supporting Root Cause Analysis
Once events are identified, engineers can drill down into individual groups to investigate potential root causes.
The interface highlights:
-
Dominant alarm types
-
Affected nodes
This helps engineers quickly move from detection to diagnosis.
Confidence-based ranking balanced algorithmic guidance with engineer judgment, allowing users to investigate likely causes while maintaining control over the final diagnosis.
Translating the ML capability into a usable workflow required several key design decisions.
Key Design Decisions
Designing the experience around the ML system required aligning user workflows around how the algorithm behaved. Several key decisions helped simplify complex operations while ensuring engineers could trust and interpret the system’s outputs.
Segment-Based Training Instead of Precise Timestamp Selection
Context
Early explorations required engineers to manually define exact timestamps when selecting training data for the ML storm detection model.
However, discussions with the R&D team revealed that the algorithm did not require precise timestamps to learn alarm patterns.
Decision
Instead of forcing engineers to enter exact time values, the interface allows users to scrub across the alarm timeline and mark a broader segment as training data.
Concept - 1
Concept - 2
Outcome
• Reduced effort required to prepare training datasets
• Faster model configuration
• Interaction aligned with how engineers explore alarm patterns
Evolving the Root Cause Analysis Module
Context
Once clusters were identified, engineers needed guidance to determine what triggered the storm. Designing this module required balancing initial insight with engineer’s current judgment.
Iteration 1 - Single Root Cause
Exploration
​The first concept surfaced one predicted root cause for the storm event
Challenge
Engineers were hesitant to trust a single initial diagnosis.
Iteration 2 - Multiple Possible Causes
Exploration
​The second iteration surfaced multiple potential causes that can be added by the users, improving transparency
Challenge
This module created ambiguity for the engineers as a team
Iteration 3 - Confidence-Based Root Cause Ranking
Exploration
​This iteration presents multiple root cause hypotheses ranked by confidence, allowing engineers to investigate all likely causes
Outcome
• Increased trust in algorithmic recommendations
• Faster and more confident root cause investigation
Workflow Transformation
Before
Operators manually scanned thousands of alarms.
After
Storm segmentation + clustering surfaces root causes faster
Impact
The new workflow transformed how operators analyse alarm storms.
By structuring thousands of incoming alarms into storm segments and clusters, the system made patterns easier to detect and significantly reduced the effort required for manual alarm correlation.
Operators could now:
• Isolate high-activity storm segments quickly
• Reduce thousands of alarms into a few meaningful clusters
• Investigate multiple potential root causes within a single incident
This shifted the workflow from manual alarm scanning to structured, pattern-based investigation, enabling faster diagnosis during network outages.
Reflection & Next Steps
What This Project Reinforced
-
Structuring information - such as transforming thousands of alarms into clusters — reduces cognitive load far more effectively than interface simplification alone.
-
AI/ML-assisted workflows are most effective when they augment expert decision-making rather than replace it.
What I’d Do Next
-
Explore richer ways to document root causes and resolutions, moving beyond basic text fields to structured inputs and reusable patterns.
-
Collaborate with the ML team to predict alarm storms earlier, surfacing signals before a full storm develops.
-
Introduce cluster explainability, helping operators understand why alarms were grouped together and what signals influenced the root cause suggestion.
-
Design collaborative investigation workflows, allowing multiple operators to annotate clusters and share findings during large outages
Strategic Insight