An AI incident review should change what the product can do, what it checks, or how people recover. If the only output is a beautifully formatted document, congratulations. The document is now more reliable than the product.
The useful question is simple: what will happen differently when this situation returns? An AI incident postmortem needs an answer that can be tested. My proposed method joins three things teams often keep apart: the incident record, a regression test and a named release decision.
Start with the outcome, not the apology
Consider an illustrative support agent that changes a customer’s delivery address. The tool times out. The agent says the update failed, then retries. Meanwhile, the first request has already succeeded. Now the customer receives contradictory confirmations, and support has to work out which address the warehouse actually received.
This is a fictional example, not a reported incident or a test result. Its point is that a convincing explanation tells you little about external state. Before changing a prompt, establish what the downstream service committed, what the agent observed and what the customer was told.
Anthropic’s agent evaluation guidance distinguishes the transcript from the outcome in the environment. That distinction belongs in incident reviews too. A chat log saying “done” is evidence of a statement. It is not proof of a completed operation.
Make the failure reproducible without exporting the customer
Preserve the relevant versions: model, instructions, tool schema, permissions, retrieval inputs and workflow configuration. Record timestamps and request identifiers where available. Keep unknowns visible. Missing evidence is a finding, not an invitation to let the model fill in the scene.
Then build a sanitised reproduction. Replace personal information and credentials with synthetic values, retaining the timing and state transitions that caused the problem. Restrict the original evidence separately. A shared test fixture should not become a souvenir shop for customer data.
For the delivery example, simulate a successful server-side update followed by a lost response. The test should inspect the final address and number of committed updates, not merely reward the agent for sounding appropriately sorry. Run it in an isolated environment with controlled tool responses.
A fix needs a counterexample
“Never retry” might stop this particular failure. It might also abandon every legitimate request after a temporary outage. That is why I would pair the incident reproduction with a neighbouring case: the first request really failed before any change was committed.
The proposed behaviour is to reconcile an uncertain outcome before trying again, using an operation identifier where the service supports it. If the outcome cannot be established, surface uncertainty and route the case to someone who can resolve it. Whether this design works depends on the downstream service; it is not a universal prompt trick.
Add another case where an ordinary successful update completes without unnecessary escalation. Otherwise a system that refuses everything can look wonderfully safe. It is also a very expensive out-of-office reply.
These checks extend the argument in Evals are the product: evaluate useful completion, permissions and recovery together. If an LLM grades the result, calibrate that judge against evidence rather than letting the agent mark its own homework.
Separate recovery from permission to release
Restoring service and approving a durable change are different decisions. An emergency workaround may need a narrow scope, an owner and an expiry. It should not silently become the expected answer in the test suite. Write the intended product behaviour first, then ask whether the workaround meets it.
Google’s SRE postmortem guidance stresses concrete, owned, prioritised action items with verifiable outcomes. For an AI product, I would make the closure record link to the reproduction, the changed control, the test results and the person accepting remaining risk. “Improve the prompt” is a task description with the important half missing.
The NIST AI RMF Playbook’s Manage guidance connects post-deployment monitoring, incident response, recovery and change management. This is guidance, not a claim that this workflow satisfies a legal obligation. A proportionate review still needs the people affected by the failure, not just the people who can edit the model configuration.
Keep watching after the ticket closes
Before release, name the signal that would reopen the incident: another contradictory confirmation, an unresolved operation, or a recovery queue that grows beyond the team’s capacity. Choose thresholds for the actual service. I cannot give you a responsible universal number from a blog.
Not every incident needs a new benchmark or a six-week committee. A deterministic defect may need a small regression test and a targeted fix. The test is whether the next person encountering the problem gets a better outcome. They will never read your postmortem. They will notice whether they still need to call support.
