Grouping & deduplication

An unhealthy system rarely sends one alert. Firefight applies three layers of noise control so that repeated firings update instead of multiply, flapping alerts do not mint duplicates, and a storm of related alerts lands in a single incident.

Deduplication by fingerprint

Every alert has a fingerprint that identifies “the same alert”. When a provider sends its own fingerprint, such as Prometheus Alertmanager, Firefight uses it. Otherwise the fingerprint is computed from the source’s Deduplicate by fields, which default to service and title and are editable per source in Settings → Alert Sources.

While an alert with a given fingerprint is open, every further firing with the same fingerprint updates that alert. Its fire count goes up and its last-seen time moves forward. No new alert is created, no new routing runs, and no new message is posted. The alert’s Slack digest message updates in place, at most once per minute, so a hundred firings do not produce a hundred pings. When the provider later sends a resolved status for the fingerprint, the open alert is marked resolved.

Deliveries are also idempotent by external id. A provider retrying the exact same event never creates a second alert.

The flap window

An alert that resolves and immediately fires again is flapping, not a new problem. If a firing arrives within the source’s Flap window after the same fingerprint resolved, Firefight reopens the resolved alert instead of creating a fresh one. The window is configurable per source from 0 to 60 minutes and defaults to 5.

If the reopened alert’s incident has already closed, the alert detaches and routes again as a fresh episode, so the regression becomes visible somewhere new instead of updating a message in a closed channel.

Grouping into incidents

Deduplication handles the same alert repeating. Grouping handles different alerts that belong to the same problem.

When a routing rule creates an incident from an alert, Firefight opens a grouping window for that alert’s content signature. The signature is built from the Group by fields in Settings → Alert Routing, which default to service. While the window is open, any alert whose values for those fields match attaches to the same incident instead of creating another one. This applies to alerts matching a Create incident rule, and it is the mechanism behind the Attach to open incident action, which only ever attaches into an open group.

Setting	Default	Range
Window (minutes)	10	5 minutes to 7 days
Group by fields	service	Any comma-separated list of alert fields

The window is measured from when the incident is created. Once it expires, or once the incident closes, the next matching alert that hits a Create incident rule starts a new incident with a new window.

Grouping settings live on the same page as your routing rules, on the workspace default and on each source’s own routing.

What happens to grouped alerts

A grouped alert attaches to the incident. It appears in the Alerts panel on the incident page with its firing status, source, fire count, and last-seen time, and the attachment is recorded on the incident timeline. When a grouped alert resolves, that shows on the timeline too. The incident itself is unaffected, alerts never close an incident for you.

How this prevents alert storms

Put together, a bad deploy that fires 50 related alerts across a service plays out like this. The first alert matches a rule and creates one incident. The other 49 either share a fingerprint with an open alert and fold into it, or carry the same service and attach to the incident through the open grouping window. Slack sees one incident channel and a handful of digest messages that update in place. Nothing else is created until the window expires.

Tip

Pick group-by fields at the level you would actually run one incident for. Grouping only by service is a good default. Adding environment splits production and staging into separate incidents when both fire.

Note

Grouping fields are compared exactly. Two alerts group only when every group-by field has the same value on both. An alert missing one of the fields groups only with alerts that also lack it.

Related pages: How alerts work, Routing rules.