When alerts fire
The exact path from a failed check to an alert, the default timings, and what never pages you.
This page explains the exact path from a failed check to an alert, and why a single blip never pages you.
#The three steps
For HTTP, Keyword, API, Port, and DNS monitors, an alert is the result of three steps in order.
- A check fails. A timeout, a refused connection, a wrong status, or a failed content rule makes the check down. A check that passes but takes at least the slow threshold is slow.
- A streak confirms the status. The monitor is confirmed down after as many consecutive failed checks as its failure threshold (3 by default), and confirmed up again after as many consecutive passes as its recovery threshold (1 by default). A pass in the middle of a failing streak resets it. Moving between up and slow takes a streak as long as the failure threshold.
- The incident policy decides when to alert. A confirmed problem has to last for the project's Open after setting (5 minutes by default) before the alert is sent. After recovery, the all-clear waits for the Close grace (10 minutes by default). The Trigger setting decides whether slow counts as a problem. By default, Down + slow, it does.
Heartbeat and scheduler monitors skip steps 2 and 3. A heartbeat alerts the moment a ping is later than its interval plus its grace. A scheduler alerts on any run that does not return 2xx. Both alert again on recovery, at once.
#What that means in time
With the defaults (failure threshold 3, open after 5 minutes, close grace 10 minutes), measured from the first failed check:
| Interval | Confirmed down | Down alert sent | Recovery alert sent after the first passing check |
|---|---|---|---|
| 1 minute | about 2 min | about 8 min | about 10 min |
| 5 minutes | about 10 min | about 15 min | about 10 min |
Each step is evaluated when a check runs, so times round up to the next check. Set Open after to 0 and the alert goes out on the same check that confirms the status.
#What does not alert
- A problem that recovers before Open after has passed. No incident opened, so no alert was sent.
- Going down again inside the Close grace. The incident stays open and the all-clear is delayed. You get one alert for the whole flap, not one per bounce.
- A slow monitor that then goes down while its incident is already open. The open incident already alerted.
- A new monitor's first result.
- Any check for a paused monitor or a monitor under a maintenance window, since those checks do not run.
#Who gets it
Each alert goes first to push, on the devices the project's owner is signed in to, then to every enabled channel in the project. The Down, Slow, and Recovery switches under Notification settings turn each event on or off for all of them. Delivery is attempted once. A destination that rejects the message is not retried.
#The all-clear
The recovery alert says how long the monitor was down, measured from the confirmed down to the first passing check. The close grace is not included. A recovery from a slow incident is reported the same way.
#Expiry warnings
Certificate and domain expiry warnings are separate: a push notification, once per expiry date, from 14 days out. They do not go to channels and do not follow the incident policy. See SSL and domain expiry.
Changing the timing
#What's next?
- Incident policy: the settings and how to choose them.
- Notification channels: where alerts go and what they say.
- Maintenance windows: silence alerts for planned work.