When alerts fire

The exact path from a failed check to an alert, the default timings, and what never pages you.

This page explains the exact path from a failed check to an alert, and why a single blip never pages you.

#The three steps

For HTTP, Keyword, API, Port, and DNS monitors, an alert is the result of three steps in order.

  1. A check fails. A timeout, a refused connection, a wrong status, or a failed content rule makes the check down. A check that passes but takes at least the slow threshold is slow.
  2. A streak confirms the status. The monitor is confirmed down after as many consecutive failed checks as its failure threshold (3 by default), and confirmed up again after as many consecutive passes as its recovery threshold (1 by default). A pass in the middle of a failing streak resets it. Moving between up and slow takes a streak as long as the failure threshold.
  3. The incident policy decides when to alert. A confirmed problem has to last for the project's Open after setting (5 minutes by default) before the alert is sent. After recovery, the all-clear waits for the Close grace (10 minutes by default). The Trigger setting decides whether slow counts as a problem. By default, Down + slow, it does.

Heartbeat and scheduler monitors skip steps 2 and 3. A heartbeat alerts the moment a ping is later than its interval plus its grace. A scheduler alerts on any run that does not return 2xx. Both alert again on recovery, at once.

#What that means in time

With the defaults (failure threshold 3, open after 5 minutes, close grace 10 minutes), measured from the first failed check:

IntervalConfirmed downDown alert sentRecovery alert sent after the first passing check
1 minuteabout 2 minabout 8 minabout 10 min
5 minutesabout 10 minabout 15 minabout 10 min

Each step is evaluated when a check runs, so times round up to the next check. Set Open after to 0 and the alert goes out on the same check that confirms the status.

#What does not alert

  • A problem that recovers before Open after has passed. No incident opened, so no alert was sent.
  • Going down again inside the Close grace. The incident stays open and the all-clear is delayed. You get one alert for the whole flap, not one per bounce.
  • A slow monitor that then goes down while its incident is already open. The open incident already alerted.
  • A new monitor's first result.
  • Any check for a paused monitor or a monitor under a maintenance window, since those checks do not run.

#Who gets it

Each alert goes first to push, on the devices the project's owner is signed in to, then to every enabled channel in the project. The Down, Slow, and Recovery switches under Notification settings turn each event on or off for all of them. Delivery is attempted once. A destination that rejects the message is not retried.

#The all-clear

The recovery alert says how long the monitor was down, measured from the confirmed down to the first passing check. The close grace is not included. A recovery from a slow incident is reported the same way.

#Expiry warnings

Certificate and domain expiry warnings are separate: a push notification, once per expiry date, from 14 days out. They do not go to channels and do not follow the incident policy. See SSL and domain expiry.

Changing the timing

Open after, Close grace, and Trigger are the incident policy, under the gear on the project's Incidents page. The Incidents page is part of Pro and Team. On the free plan the defaults apply.

#What's next?