Incident policy
Learn how to set the timings that turn a problem into an incident and an alert, and when to change them.
This page explains the three settings that decide when a problem becomes an incident and an alert, and how to choose them. The policy is per project, under the gear on the Incidents page, and can be changed by the project's owner or an admin.
#The settings
| Setting | Default | Range | What it does |
|---|---|---|---|
| Open after | 5 minutes | 0 to 1440 minutes | How long a confirmed problem has to last before the incident opens and the alert is sent. |
| Close grace | 10 minutes | 0 to 1440 minutes | How long a recovery has to hold before the incident closes and the all-clear is sent. |
| Trigger | Down + slow | Down only, or Down + slow | Whether a slow monitor counts as a problem. With Down only, slow checks never open an incident or alert. |
The clock starts when the monitor's status is confirmed, after its failure threshold, and each setting is evaluated when a check runs. So an Open after of 5 minutes on a 5 minute monitor fires on the first check at or past the five minute mark.
The policy applies to HTTP, Keyword, API, Port, and DNS monitors. Heartbeat and scheduler monitors open and close their incidents at once.
On every plan
#Choosing values
- Open after 0 alerts on the same check that confirms the status. Combine it with a failure threshold of 1 for the fastest possible page, at the cost of being paged for single blips.
- A longer Open after absorbs short outages. With the default 5 minutes, a problem that fixes itself within five minutes never alerts and never appears as an incident.
- A longer Close grace keeps a flapping service in one incident instead of a run of alerts. Each return to down during the grace is folded into the open incident.
- Down only suits monitors with a tight slow threshold, where slow is information rather than a page.
#On-call escalation
Team projects can escalate an incident that nobody acknowledges. Turn it on under Escalation on the project's Team page and add up to 10 steps, each a team member to page or a channel to post to, with a delay from the previous step. The first step fires as soon as the incident opens. Set a repeat interval and the last step repeats until someone acknowledges the incident. A paged member receives a push notification and an email.
Escalation runs for every open incident, automatic or manual, and ends when the incident is acknowledged or resolved. It is not silenced by maintenance windows.
#What's next?
- When alerts fire: the full path from a failed check to an alert.
- Incident lifecycle: what happens once an incident is open.