Investigations

An investigation is Halon working out on its own why something is broken. It reads what Firefight already knows and what your connected tools show, writes down its theories, tests them, and ends with an answer, the evidence behind it and a fix you can apply.

An investigation can be about an incident, or about a problem nobody has declared an incident for yet. Halon works on it in the background for a few minutes, and you can follow every step while it runs. An investigation has at most an hour, see Time limit.

Starting an investigation

Starting one, stopping one, adding to one and applying its fix all need Investigations with create. Every member has it unless an admin took it away under Gateway → Permissions, and agents and service keys need it granted. See Permissions.

In Slack

Where	What you do	What Halon investigates
An incident channel	/ff investigate, or choose Investigate on the incident’s pinned message	The incident
Any other channel	/ff investigate checkout returns 500 since 14:00	What you said is wrong
Any channel Firefight is in	Mention @Firefight with a message whose first word is investigate, such as @Firefight investigate billing is slow	The incident, in an incident channel. Otherwise what you said

Whatever you write after investigate is what Halon starts from, so say what is failing, roughly when it started, and any service names or error text you have. Attach a screenshot, a PDF or a log to the mention and Halon reads it before its first step, see Giving Halon a file. Outside an incident channel you must say what is wrong, and Firefight asks you to if you leave it out. In a private channel Firefight is not in, the answer comes to you as a direct message instead.

Only the first word counts. In an incident channel, a mention that talks about investigating later in the message is answered as an ordinary question. Outside one, Halon answers only a mention that starts with investigate.

In the dashboard

Open Chat under Halon in the sidebar and ask Halon to investigate. Halon starts an investigation only when you ask for one, and may ask when the problem started if nobody has said. In a chat about an incident it investigates that incident. Anywhere else it investigates what you described. The run reads the files sent in the chat. See Chat.

The run shows in the chat as a card with its steps as they happen, then its answer and any charts it drew. Choose Open the run while it works, or See how it got there once it has finished, to read the whole run beside the chat.

On an incident’s page, choose Investigate beside the channel link at the top. Halon starts on the incident, Firefight says so, and the run opens beside the incident. While a run is working on the incident, the button is greyed out, and pointing at it says so with Open the run. On an incident that is over, or one whose channel is still being created, pointing at it says why it cannot start. The button is there only when Halon is on for your workspace and you can start investigations.

Over MCP

An AI agent connected over MCP starts one with start_investigation. Give an incident, or a symptom saying what is wrong when there is no incident. It also takes started_around, names and error_text. Follow the run with get_investigation, by its id or by the incident for its newest run.

When Firefight says no

Message	Why
Already investigating INC-42, I will post here when I have something.	One investigation runs on an incident at a time. Questions without an incident are not limited this way
INC-42 is closed, so there is nothing left to investigate.	The incident is resolved or cancelled
INC-42 has no channel yet, so there is nowhere to post what it finds.	The incident’s channel is still being created. Try again in a moment
Say what is wrong, for example: investigate checkout returns 500 since 14:00.	You asked outside an incident channel without saying what is wrong

What Halon does

Halon starts from the signal, not from a guess about which service is at fault. It reads when the trouble began, taking the earliest real sign such as an alert’s first seen time, and the clues in the alerts and in what you said: service and repository names, stack traces, error names and commits.

On an incident it also starts from the resource map. It looks up where the incident’s services run and what changed on those resources in the day before the trouble began, then follows the map from there.

When a code change could explain what broke, Halon asks your code host what changed before the start, across every repository it can see. It ranks the changes the way an engineer would, with its reasons shown: a change touching a file in the stack trace, in a repository the clues name, to migrations, configuration, flags or dependencies, a deploy rather than a merge, and how close it landed before the start. When nothing in that window explains it, it says to look past the code.

Halon reads code in a sandbox of its own for each investigation, from GitHub, GitLab, Bitbucket and Azure DevOps. It reads the commit that was running, with the whole history, finds where something is defined and everywhere it is used, and can run the repository’s tests. The sandbox holds no credential and closes when the investigation ends.

Beyond that, Halon calls only the tools the question needs. A question about latency reads metrics, and an error in a log leads to the code that raised it. It reads through the tools your workspace connected and the access granted to the Halon agent under Gateway → Permissions. An investigation you start also does nothing you could not do yourself. It reads no more of the resource map than you can see, uses a tool only when you have access to it, and never calls a tool in an environment you cannot see on the map. The same goes for one you start by pressing Run now on a monitoring check, or through a workflow you run or test yourself. An investigation an agent or a service key starts, over MCP or through a workflow it runs, is held to what that agent or key was granted, so it saves what it learns to memory only when that agent or key may create and update memories. One that started from an alert, a scheduled check or a workflow’s own trigger, with nobody behind it, reads everything the Halon agent was granted. Every call it makes goes through the gateway and is in Activity. See Tools and approvals.

Before an answer goes out, Halon tries to prove itself wrong, and drops any claim that the steps it cites do not show. Results from a connected tool carry a link back to the page they came from, when that tool has one, so you can check a claim at its source.

Halon also uses what your team taught it. It starts from your handbook, memories and instructions, and from past incidents that fired the same alert, with the answer each one got and whether your team said it was right.

Following it in Slack

A run on an incident posts Investigating INC-42 in the incident’s channel, with who started it. A question is posted in the channel it was asked in, titled by its own words. That message is a live checklist of what Halon has looked at so far, grouped by the kind of system it read, each marked as waiting, working, done or failed, with the answer as the last item:

Item	What Halon read
Gathered what Firefight knows	The incident, its alerts and what your team already recorded. Always first
Checked recent deploys	Deploys, CI runs and what changed
Read the logs and metrics	Logs, metrics, traces and errors
Read the code	Your repositories and the code sandbox
Checked the database	Your connected databases
Checked cloud resources	Your cloud accounts
Read the team’s notes	Runbooks, the handbook, memories, issues and docs
Checked alerts and on-call	Alerts, past incidents and who is on call
Checked the resource map	The map and the catalog
Checked the page in a browser	A page in Halon’s browser
Find the cause	The answer, until Halon has one

Only the kinds Halon actually read appear, at most five, with the rest counted under Ran other checks. A kind where some reads failed says how many, such as 1 of 3 lookups failed. The checklist updates in place as Halon works, so anyone watching the channel can follow along without opening anything. Once Halon is done it reads Investigated INC-42, with the kind of answer as its last item, or Investigation of INC-42 stopped. Several investigations on one incident each have their own checklist.

Halon works in that message’s thread. Slack shows each step there as Halon takes it, with Slack’s own stop button. Stopping ends the run, and the checklist says it stopped. When Slack’s agent features are not available in your Slack workspace, you see no live steps in the thread, and the checklist still follows along. When your Slack workspace cannot show Slack’s checklist layout, the same checklist arrives as a plain list, one line per item.

The answer is posted in the thread and also sent to the channel, short enough to read at a glance:

• What kind of answer it is, Root cause found, Likely cause or No cause found yet, and how sure Halon is: Very sure, Fairly sure, Not very sure or A guess. Halon says a root cause is found only when it is sure enough, and otherwise calls it a likely cause
• What is wrong, in one sentence
• Up to three facts behind it
• Suggested fix, in one sentence
• Apply fix, when the fix has a step Firefight can run, and Open in Firefight, which opens the whole run in the dashboard
• What Halon could not check, in one line, when there was something

The full answer follows in the thread only:

• What Halon found, in its own words
• Why I think so, each claim with the results it rests on
• How to fix it, the fix’s steps in order (see below)
• Could not check, what Halon could not reach and why it matters
• Charts Halon drew while it worked
• Right, Partly right and Wrong, to rate the answer

When there is no incident yet and the answer says users are hurt now, it also offers Declare incident. The incident starts named after your question, its timeline shows when the investigation started and answered, and its new channel opens with the short answer, with the full answer in its thread.

A run that stops without an answer says Stopped without an answer and why, in the thread and on its checklist. When it stopped because of a problem on Firefight’s side, it offers Run again, which asks the same thing in the same place.

Time limit

An investigation has an hour from when Halon starts working on it. When the hour is up, Halon is asked to answer with what it has found so far, and an answer it gives then is posted as usual. An investigation picked up again after the hour also gets that chance to answer. If it still has none, the run stops with Stopped at its time limit before it could answer, in the thread, on its checklist and on its page. The thread also lists what Halon did before it stopped, and every step and theory stays on the run’s page. A step still running a few minutes after the hour is stopped, so a stuck read never holds the incident.

Adding to a run

Mention @Firefight in the thread of a run that is still working, and what you write is added to the run, with any files you attach. In the dashboard, each file added to a run is listed under what was added. Choose an image to see it full size, or any other file to download it. Anyone who can read the investigation can open them. Halon reads it at its next step and changes course if it should, so you never have to stop it and start again. In the dashboard, write in Tell Halon something at the bottom of the run and choose Add to the run. Once the run has finished, a mention in its thread is an ordinary question.

Reading it in the dashboard

A run on an incident opens over the incident’s page, in a side panel. The incident’s timeline shows when each investigation started, found an answer or stopped, and choosing one opens it. An answer shows on the timeline in a few plain lines (the kind of answer and how sure, what is wrong and the suggested fix), with Show the full answer for the rest. Once someone applies the fix, the timeline also shows who applied it, each step’s result with its link, such as Opened a pull request: PR #856, a step a person marked done, and how the fix ended or who cancelled it. A question with no incident has a page of its own, which Open in Firefight in Slack leads to.

Every run reads the same way, top to bottom:

Part	What it shows
Header	The status (Queued, Investigating, Answered or Stopped), what was asked, Asked by, From (such as /ff investigate, Investigate button, Incident page, Chat or Outside agent), Started and Took
Answer	What Halon found. While it works, the step it is on. When it stopped without one, No answer and why
Theories	Every theory Halon wrote down, as Open, Supported or Ruled out, with how sure it was and the steps that decided it
How it got there	The run in order: what was asked, every step, each theory where it was written down and where it was confirmed or ruled out, and anything people added

Under the answer you see its parts:

• Cause, the cause Halon settled on
• Why it thinks so, each claim with links to the steps it rests on
• How to fix it and, once written, How to undo it
• Could not check, what Halon could not reach
• The team says, how the answer was rated
• Was it right?, with Right, Partly right and Wrong to rate it yourself

Each step in How it got there says whether the gateway allowed it and when. Choose Show what it returned to read the result itself. Reads of Firefight’s own records are not logged, and say so.

When a run with no incident says users are hurt now, the answer shows Halon thinks this is hurting users now. with Declare incident.

Stopping a run

While a run works, the last row of How it got there says Still working. This updates as it goes. with Stop. A chat’s card has Stop too. Halon ends the run at once, even in the middle of a step, as stopped by a responder. Stop in Slack and in the dashboard do the same thing.

Fixes

Every answer that names a cause comes with a fix. A fix is a short summary, ordered steps, how to undo each step, and how to tell it worked. A step says what it is:

Kind	What it is
Code change	A change in one repository, opened as a pull request
Change through a tool	A call to one of your connected tools, or one of Firefight’s own, that changes something, such as rerunning a CI run or inviting someone to the incident channel. The step shows the tool, and What it sends shows its exact arguments
For a person	Something no tool can do. It names who does what and where, and says what Firefight is missing to do it, under Needs
Page check	A look at a page in Halon’s browser once the steps before it are done, to confirm the fix works for people. What it opens shows the address and steps, and the step shows the screenshots it took. It changes nothing, so a fix of page checks alone is not applied

Halon makes a step Firefight’s whenever a tool can do it, Firefight’s own included, so inviting people, assigning a role, paging or posting an update are never left to you. It also checks how things stand first where it can, and leaves out a step that is already done, such as making someone lead who already is.

A step that waits on others says so, such as After steps 1 and 2, and runs only once they are done.

Halon only proposes a fix. Nothing in it runs until a person applies it.

Applying a fix

Choose Apply fix on the answer in Slack or in the dashboard. Firefight asks first, saying how many steps run through your connections and how many are for a person. Applying runs the steps in order, as you, through your own permissions. If you have no access to a tool a step needs, Firefight tells you which step and why, and nothing starts.

Each step then shows where it got to: Running, Waiting for approval, Approved, waiting for someone to run it, Done, Failed, Declined, or Skipped, since a step it waits on did not go through. A step held by an approval rule waits for its approver, and approving it never runs it. Halon reads how things stand now, and the step shows what it found with Run and Dismiss, on the run page and in the run’s thread. Whoever applied the fix can run it, and so can someone who may use the step’s tool themselves, and it runs as whoever applied the fix. Dismiss skips it, so the steps that wait on it do not run. The approval lasts an hour, after which the step offers Ask again. When the fix has no thread, whoever applied it gets a direct message in Slack when a step is approved or declined. A step that fails or is declined stops the steps that wait on it, and the rest carry on.

A step that went through shows what it produced, with its link, such as Opened a pull request: PR #856. A step whose change was already made when it ran is marked done without changing anything, and says why, such as Already done: Uros Nikolic is already in #inc-003.

A step for a person, or a code change Firefight cannot open itself, says it is yours, such as Your step: Invite Uros Nikolic to #inc-003. Then press Done., with Done. Choose it once you have done the step, and whatever waits on it goes ahead. Steps go in order, so you cannot mark one done before the steps it waits on.

In Slack, applying posts one checklist in the run’s thread that is also sent to the channel, named for the incident, such as Fix for INC-003. Each step is named in a few words, such as Open a pull request with the change, with where it got to and what it produced. A step waiting on a person, for an approval or for someone to run it, is shown as not started and says who it waits on. The full instructions for each step are in the full answer and on the run page. Under the checklist, a line says how the fix stands, such as Applying the fix · 1 of 2 steps done, then come Your step with Done where a person is needed, Run and Dismiss on an approved step, Try again on a failed one, and who applied the fix. It updates in place and is posted only once. Once every step is done it reads Fix applied · 2 of 2 steps done, or, when a step did not go through, Fix stopped with how many failed, such as Fix stopped: 1 of 2 steps failed. The dashboard shows the same steps.

A step that fails says why in plain words, such as Couldn’t open a pull request with the change: the code sandbox didn’t start. What the tool itself said is on the run page. Try again runs the step again, in Slack and on the run page. Firefight asks first in Slack. It runs as whoever applied the fix, and the same people who can Run an approved step can use it. The steps that waited on it carry on once it is done. A fix that was cancelled, or one Halon applied on its own under an unattended rule, has no Try again.

A fix with no step Firefight can run is never applied. Each step is marked done by hand.

Code changes

A code change is written by Firefight’s own agent by default. It writes the change in the investigation’s sandbox and opens the pull request through your GitHub connection. Merging is always a person’s decision. A change that touches a path your connection lists under Paths Halon may not change, more than 100 files or more than 4 MB is not opened. A change to a CI workflow opens with a warning at the top of the pull request to review it carefully.

While the agent works, the step shows what it is doing, the same way a code fix in a chat does: its last few steps, how long it has been working and the files it has changed so far. The agent starts from what the investigation was asked, the notes people added and the results the finding rests on, and it reads your connected tools as the person who applied the fix. When it asks a question, the step shows it with a box to answer, and only the person who applied the fix can answer. Once it is answered, that person can choose Change answer while the agent is still writing, the same way as in a chat. The change is checked and reviewed before the pull request opens. Once it opens, the step sums up the review, the files, the tests and checks with their results, and links to the pull request. In Slack, the fix’s checklist shows how far the agent got, as how many steps it took, how many files it changed and whether its tests passed, updated every few seconds.

An admin can hand code changes to a coding agent you connected instead, Devin, Cursor or Factory. Under Settings → Workspace, in the Halon card, choose it in Write code fixes with. Firefight hands it the change and the evidence, that agent opens the pull request itself, and the step shows how it is going until it finishes or reaches its time limit. Firefight asks it to open the pull request even when your tests cannot run in its environment, with what could not run listed in the description, so your repository’s own CI runs them. When the chosen agent is removed or switched off, code steps wait for a person rather than falling back to Firefight’s own agent, and the setting says why. When the agent’s pull request changes a path your connection lists under Paths Halon may not change, or Firefight cannot read what it changed, the step says a person must review it before it is merged, on the run page and in the fix’s checklist in Slack, and Firefight comments on the pull request.

When neither can open the pull request for a repository, the code change is marked done by hand. An investigation never starts a code change on its own.

Cancelling a fix

While a fix is being applied, Cancel fix stops it. Firefight asks first. Steps that have not run never will, and approvals they wait on are withdrawn. A step already running finishes, and whatever went through can be undone. The fix then reads Cancelled with who cancelled it.

Undoing a fix

Once a fix has been applied, partly applied or cancelled with something done, Undo fix asks Halon to write the steps that put back what it changed, from what each step actually did. Asking changes nothing. The undo shows under How to undo it in the dashboard, and is posted in the run’s thread in Slack, to apply with Apply undo like any fix. It can be cancelled the same way. If Halon could not write one, the fix says why and you can ask again.

Rating answers

Every answer in Slack ends with Right, Partly right and Wrong, and Firefight tells you privately what you chose. In the dashboard, the answer has the same three under Was it right?, with your own choice highlighted. Anyone who can read investigations can rate it, and change their mind later in either place. The answer counts as rated only while everyone who rated it agrees. The team says shows the agreed rating, or how many said what while people disagree.

Halon learns from what it got wrong:

• When an incident ends, an answer marked wrong and a fix that was applied and then undone are read with the rest of the incident, so a lesson can say what the symptom really meant
• An answer marked wrong after its incident ended teaches at most one lesson about the mistake, posted in the incident’s channel to confirm
• The next investigation that meets the same alert reads a past answer your team confirmed as a worked example, and one marked wrong as a mistake not to repeat

Lessons are memories like any other, unconfirmed until someone confirms them. See Halon’s memory. To see how Halon has done over time, open Performance.