Halon on call, with paging through PagerDuty and Opsgenie

Halon can now take the first look at an alert when nobody is awake to. Everything is set on the new On-call page under Halon, which admins can open, and each setting is off until you turn it on. See Halon on call.

Start Halon when an alert opens an incident. Every incident an alert opens gets an investigation as soon as its channel is ready. Alerts that group into one incident start Halon once, and a spending ceiling per hour covers a storm of alerts that do not group.

Page whoever is on call. At the end of such an investigation, Halon pages the person on call for what broke, with its answer and the fix it proposes. With PagerDuty connected it raises an incident on the right service, and with Opsgenie an alert for the right team or schedule, so the tool's own escalation wakes them. Without an on-call tool, Halon escalates the incident in Firefight instead. You can also ask Halon to page someone during an incident, and agents use page_on_call.

Pages stay in step, both ways. Resolving the incident in Firefight resolves its open pages, and acknowledging an escalation acknowledges them. Acknowledging, escalating or adding a note in PagerDuty or Opsgenie shows on the incident's timeline, and the incident page's Pages panel lists every page and where it stands.

Unattended rules. Decide ahead which changes Halon may make on its own after an alert, such as "Restart checkout when its 5xx responses over the last 10 minutes average above 50". Halon acts only when every change in the fix matches a rule, it holds the permission, and a fresh reading is above the threshold, and then it posts what it did and how to undo it. Agents manage rules with upsert_unattended_rule, list_unattended_rules and delete_unattended_rule.