Why we page on two regions, not one

For the first year, Tideline paged the moment any probe saw a failure. It felt like the responsible default: a customer somewhere could not reach the service, so somebody should know. What we actually built was a machine for waking engineers up because a transit provider in one metro had a bad thirty seconds.

We went back through eleven months of check results and sorted every failure by how many regions saw it. The split was not close.

• Failures seen by exactly one region: 41,802
• Of those, resolved without any action within 60 seconds: 40,914
• Failures seen by two or more regions: 3,177
• Of those, resolved without any action within 60 seconds: 214

Put another way: a single-region failure was a real incident about two percent of the time. A multi-region failure was a real incident ninety-three percent of the time. We had built an alerting system whose most common output was noise, and then we had asked our customers to tune it themselves with retry counts and cooldowns.

A single region failing is evidence about the region. Two regions failing is evidence about you.

What confirmation actually costs

The objection to confirmation is latency. If you wait for a second region to agree, you detect real outages later. That is true, and the size of the delay is the whole question.

Probes are staggered rather than synchronised, so the second region's next check lands somewhere in the interval window. At a ten second interval, the expected wait for confirmation is about five seconds, with a worst case of ten. Against a median time-to-acknowledge of four minutes, that is not a number anyone can feel.

// a failure opens a window instead of paging
on check_failed(region, check):
    window = open_or_get(check, ttl=30s)
    window.add(region)

    if window.distinct_regions >= check.confirm_from:
        page(check, window)          // real
    else:
        record(check, window)        // visible, not paged

The unconfirmed failures do not disappear. They land in the check timeline with the region that saw them, which turned out to be the more useful product: several customers found a regional peering problem they would never have noticed, because our old system had trained them to ignore the alerts entirely.

Where the default is wrong

Confirmation assumes your service is reachable from everywhere. Two cases break that assumption, and both are worth setting confirm_from: 1 for.

The first is a geo-restricted endpoint that only answers in one region by design. The second is an internal agent check, where there is exactly one probe and no second opinion available. In both, waiting for a second region means waiting forever, so we skip confirmation and say so in the alert.

What we would do differently

We should have looked at the data a year earlier. The argument for single-region paging was entirely about what could theoretically happen, and eleven months of results answered it in an afternoon. If you are running your own monitoring, the query is worth writing: group your failures by how many vantage points saw them, then check how many of the single-vantage ones ever needed a human.