Halon's safe changes
Some changes need more than a yes before Halon makes them. If Halon proposes a change to rows in a database, a change your customers feel for a while, or stopping something someone else started, Firefight adds a safeguard around it. You see it on the request before you press Approve, and what it found sits under Details.
These safeguards come on top of the request every change makes, and approval rules still apply. See Tools and approvals and Approvals.
Changes to rows in a database
Section titled “Changes to rows in a database”When Halon proposes an UPDATE, DELETE or INSERT through a connected database, such as PlanetScale, Supabase or Neon, Firefight counts the rows it would touch before you are asked. Under Details, the request shows:
| Row | What it says |
|---|---|
| Rows it changes (or Rows it adds) | How many rows of which table, such as 42 rows of orders |
| Rows still wrong now | How many rows Halon’s check finds wrong before the change |
| A few of them | Up to five of the rows as they are now |
| Before it runs | That Firefight keeps a copy of these rows for 30 days, or that nothing is overwritten when the change only adds rows |
Each of these changes asks on its own. Allow for the rest of this chat is not offered, since each one touches different rows.
Right before the change runs, Firefight counts the rows again. If the count has moved since you approved, the change does not run and Halon tells you. Otherwise Firefight keeps a copy of the rows as they were, runs the change, then runs Halon’s check again and tells you how many rows are still wrong.
The chat then shows the change with Download the rows as they were, which gives you the copy as a file for 30 days. You can download it as long as you may read that database, and each download is in the activity log.
A change touches at most 1,000 rows in all, and each statement changes one table. In a Neon transaction, every statement in it is counted and copied, and the whole transaction is refused if one of them cannot be. Firefight refuses a statement it cannot count, such as one that joins other tables, updates rows already there while inserting, packs several statements in one, or runs TRUNCATE, and Halon rewrites it. A change to the schema, such as adding a column, asks like any other change, with nothing counted.
Temporary changes
Section titled “Temporary changes”A change your customers feel for a while is undone by default. That covers turning a feature flag on or off in PostHog or Mixpanel, adding or changing a block, rate limit or firewall rule in Cloudflare, pausing a service in Northflank, a Convex deployment, a QStash schedule in Upstash, or a queue or concurrency limit in Trigger.dev, or scaling a service to fewer instances than Firefight last saw it running.
The request asks when, under Undo automatically when Firefight puts it back itself, or Ask Halon to put it back when Halon does it in the chat. For a service scaled down it reads Scale it back up automatically. Pick a time before you press Approve:
| Choice | What happens |
|---|---|
| 1 h | The default. It is put back an hour after it runs |
| 15 min, 4 h or 24 h | A shorter or longer time |
| Never | It is never put back |
In Slack, the same choice is a menu beside that line.
An unattended rule never covers one of these changes, since a person chooses how long it lasts.
The time starts when the change runs. Fifteen minutes before the time is up, Firefight reminds you in the chat’s Slack thread, or in your Slack direct messages for a dashboard chat. When the time is up, it is put back in one of two ways:
- A fixed undo. A service scaled down, without changing its plan, is scaled back to the count it had before, and a paused Trigger.dev queue or concurrency limit is resumed. Firefight runs this itself, as the person who asked for the change, and says how it went. If an approval rule covers the undo, it waits for someone to approve it, and Firefight says so where it reminded you.
- In the chat. Any other change, such as a feature flag, a firewall rule or a paused Convex deployment, is put back by Halon in the chat, with a call that asks or runs like any change Halon makes. The change’s card then says Halon was asked to put this back. Mark it once it’s back. Once you see it is back, press It’s back. Firefight never takes Halon’s word for it, so the change counts as in place until someone presses it. If nobody has after 15 minutes, Firefight tells you the change is due to be undone by hand, and Undo now asks Halon again.
Pressing Undo now before the time is up works the same way.
Until then, the change shows in the chat, and on every message about it in Slack, with three buttons:
- Keep it stops it being put back.
- One more hour moves the time an hour later.
- Undo now asks first, then puts it back straight away, or asks Halon to in the chat.
Once Halon has been asked to put a change back, the card offers It’s back and Undo now, to ask Halon again. Anyone who could press Undo now can press It’s back. It asks first, “Mark this change as back? Firefight stops tracking it.”, since it cannot be taken back.
In a dashboard chat only you can press them. In a Slack thread, anyone in the channel can, since you may be away when the time runs out.
Stopping something someone else started
Section titled “Stopping something someone else started”Before Halon stops something, Firefight reads who started it where the provider says. That covers cancelling a GitHub Actions run, a GitLab pipeline or job, a Bitbucket pipeline, an Azure DevOps pipeline run, a CircleCI workflow, a Render, Railway or Vercel deploy, and stopping a GitLab environment or a Railway deployment. Before it stops a Modal app, Firefight reads who last deployed it, and before it cancels a PlanetScale deploy request, who opened it. If that is someone else, the request names them under Started by, Last deployed by or Created by in its Details.
When Firefight can match them to a member of your workspace by the email address the provider shows for them, they are asked once you approve. They get a message in Slack saying who asked Halon to stop what, and why, with Agree and Say no. Only they can answer. The run is cancelled only once they agree. If they say no, nothing is cancelled and Halon tells you. While you wait, the chat and the Slack message show that it is waiting for them to agree.
When Firefight cannot match or reach them, the request names them and asks you to check with them before you approve. Nobody else is asked. Bitbucket, CircleCI, Modal and PlanetScale do not give an email address, so there you are always asked to check with them yourself.
In a plan you approved to run later, nobody is there to approve each step, so the person who started the run is asked as soon as the step comes up.
If you started it yourself, or the provider does not say who did, Halon asks you as for any other change. Stopping a Modal container, cancelling a Cloudflare Workers build and cancelling a Render cron run, one-off job or task run name nobody, since the provider does not say who started them.