Engineering Delivery
Incidents
The Incidents page answers one question: when a release breaks, how fast do you recover? It shows failed deployment recovery time and, once an incident source is connected, every incident in the period by service, with how long each took to restore. Everything on it is read per team or per service, never per person.
New to DORA? Start with What is DORA?, then come back here.
Where to find it
Open Engineering Delivery, choose Delivery on the left, then the Incidents tab. The page opens with "When a release breaks, how fast do you recover. Team and service level only." Below that:
- Incident measures: the failed deployment recovery time tile.
- How these are measured: guides to recovery time and deployment rework rate.
- Incidents by service: the "Where incidents happen" table, once incidents are read.
An incident is an unplanned problem in production that affects users, such as an outage or a serious bug. Teams usually record incidents in a tool, and Engineering Delivery reads them from there.
Throughout this page we follow a fictional team of six who build an online booking app.
Failed deployment recovery time
The tile shows how long it takes to recover once a release has gone wrong. That is how long customers live with the problem.
What a reading means. A short recovery time means the team notices problems quickly and can undo or repair a release fast. A long one usually means one of three things: the problem was noticed late, nobody was sure who should act, or the fix had to wait for the normal, slow path to production.
How to improve it, in order:
- Make rolling back one step, tested and routine.
- Keep changes small so the cause of a problem is easy to find.
- After each recovery, write down what slowed it (noticing, deciding or shipping the fix), then fix that one thing.
How it is measured. It depends on whether an incident source is connected:
- With an incident source: the median time to restore the resolved incidents opened in the period.
- Without one: the median time from a release that needed intervention (rolled back or hotfixed) to the change that fixed it.
The tile says which method it is using. Read the full Failed deployment recovery time guide for DORA's definition and figures.
Deployment rework rate
Rework rate is the share of deploys that were not planned, but shipped because of a problem in production. Its tile sits on the Quality page, and its guide is linked here because it is read from the same incidents.
What a reading means. A high rework rate means a lot of your deploying is firefighting rather than moving forward. Each unplanned deploy is time the team meant to spend on planned work.
How to improve it:
- Open the deploys made under an incident and look for the services they share.
- Name the service in a label on each incident, so its deploys are linked by service, not by team.
How it is measured. A deploy counts as unplanned when it shipped while an incident was open: to the service the incident names, or, when it names none, to a service of the incident's team. Only a resolved incident links deploys, so one left open never marks every deploy after it. Without an incident source, rework rate is not measured, and never estimated.
Read the full Deployment rework rate guide.
Incidents by service
The "Where incidents happen" table lists the incidents opened in the period, most first, with the median time to restore the ones that were resolved.
| Column | What it shows |
|---|---|
| Service | The service the incident affected. "Not tied to a service" when no label or incident tool names one. |
| Incidents | How many incidents were opened for it in the period. |
| Still open | How many of those are not resolved yet. |
| Time to restore | The median time from opened to resolved. "None resolved yet" when none of its incidents has been resolved. |
A service is known when a label or the incident tool names it. Under the table, See every incident opens every incident of the period, with how long each took to restore.
How to read it. Look for the service with the most incidents, then at its time to restore. In the booking team, the payments service had 4 incidents with a median time to restore of 3 hours, while the website had 1, restored in 20 minutes. Payments is where to look first.
Questions the app raises
It says "Incidents are not read yet". What does that mean?
Incident counts, incidents by service and the rework rate are read from your incident source, and the app has none to read yet. The panel says why: either no incident source is chosen in Settings, or the one chosen has not delivered incidents yet. Until then, recovery time is measured from releases that needed a fix. See releases by service in Quality takes you to the releases instead.
How do I connect an incident source?
An admin chooses it in Settings, under Incident source. The choices are Jira (Jira Service Management), Linear and PagerDuty. See Setting it up.
Rework rate says its incidents arrive after the next pipeline run. Why?
The app pulls data from your tools on a schedule. Once you choose an incident source, its incidents arrive with the next run, and rework rate is measured from then. Nothing is estimated in the meantime.
We use Opsgenie. What now?
Opsgenie is being retired by Atlassian, so the app no longer offers it. Choose Jira (Jira Service Management), Linear or PagerDuty as your incident source instead.
Why are so many incidents "Not tied to a service"?
The app only knows an incident's service when a label or the incident tool names it. Add the service as a label on each incident, using the same name your deploys use. That also makes rework rate link deploys by service rather than by team.
Recovery time changed after we connected an incident source. Is that a mistake?
No. The method changed. Without incidents, recovery time runs from a release that needed a fix to the change that fixed it. With incidents, it is the time to restore the incidents themselves, which starts when the problem was recorded and ends when service was back. The tile says which one it uses. Compare periods measured the same way.
How do we recover faster?
Notice sooner, decide faster, ship the fix faster. Good monitoring and alerts help you notice. A clear on-call owner helps you decide. A one-step rollback and a quick deployment pipeline help you ship. The Failed deployment recovery time guide covers each.
The same kind of incident keeps happening. What should we do?
Hold a short review after each incident and keep it blameless: the goal is to understand what in the system let the problem through, not who made the change. Then add the check that would have caught it, such as an automated test or an alert. If incidents cluster on one service, look at its change fail rate by service too.
Learn more
Official sources:
- DORA's software delivery performance metrics
- DORA capability: monitoring and observability
- DORA capability: proactive failure notification
Related guides: