Incidents

An incident is one stretch of time when a monitor was down or degraded. CuliPulse opens it when the monitor goes down or degraded, and closes it when the monitor recovers. Each monitor keeps its own list of past incidents, and your uptime figures and reports are calculated from them.

This page explains the details that decide when an incident starts and ends, so the numbers in your alerts and reports match what you see in your own logs.

When an incident opens

A monitor doesn’t open an incident on its first failed check. It waits for several failed checks in a row, so a one-off blip doesn’t wake you. How many is the Alert me after setting (1 to 5, 2 unless you changed it). The alert is sent once the outage is confirmed, but the incident, and the “down since” time in the alert, start at the first failed check of that run of failures. So an incident is dated from when CuliPulse first saw the problem, and the alert reaches you a little later.

Example. A monitor on the Free plan checks every 5 minutes and alerts after 2 failed checks. Your site breaks at 12:03 and you fix it at 12:22.

Time	What CuliPulse sees	Incident
12:05	First failed check	Waiting for a second failure
12:10	Second failed check in a row	The outage is confirmed. The incident is recorded from 12:05, and you get the down alert now
12:15, 12:20	Still failing	Stays open
12:25	First good check	Closes at 12:25 (with the default Mark healthy again after of 1)

The incident reads 12:05 to 12:25, 20 minutes, while your site was really broken from 12:03 to 12:22. Checks run on a schedule, so the incident is measured at check granularity: it begins at the first check that failed, not at the exact moment things broke, and it can also run on after you fixed the problem, up to one check with the default Mark healthy again after of 1. A faster check interval makes that gap smaller. The only thing the Alert me after setting delays is the alert itself. See Wait for repeated failures.

The same goes for a monitor that is slow rather than down: a degraded incident starts at the first slow check of the run that led to it.

If a monitor is already degraded and then starts failing, the incident stays the same one. Its degraded time ends, and its down time begins, at the first failed check.

Outages shorter than the wait are not recorded

If a good check arrives before the required number of failed checks in a row, the count starts over and nothing is recorded: no incident, no alert, no downtime in your reports. In the example above, a problem that fixes itself at 12:07 leaves no trace. An incident that closes on recovery lasts at least one check interval.

When only one location fails

When a monitor is watched from more than one checkpoint, you can require more than one of them to fail before it counts as down. See More than one checkpoint. Two things to know about this setting:

• Where it starts on. In the app, the Only alert if more than one location fails checkbox appears once a monitor has two or more locations, and it starts unticked on a new monitor. A monitor created through the API or Terraform without that setting is treated as “confirm from 2 locations” when it has two or more.
• What it doesn’t record. With this setting on, if only one location fails and the others are fine, the monitor stays up. No incident is opened, no alert is sent, and nothing is kept in the history. The failure shows only on the monitor’s Checkpoints tab while it is happening.
• When a location isn’t reporting. A location that can’t run the check at all (for example an agent that is offline) is never counted as a failure, and the monitor no longer waits for it. The setting then needs agreement only from the locations that are reporting, so with two locations and this setting on, if one is offline the other alone can mark the monitor down. The monitor page says which location isn’t checking in (for example “Checking from 1 of 2 locations”), so you know that, for now, one location is deciding alone. If every location stops reporting, the monitor becomes unknown instead.

Down and degraded in one incident

A monitor is down when the check fails and degraded when it works but badly (slow, close to a limit you set, or a certificate about to expire). See The four states.

There is one incident per episode. If a down monitor starts answering but slowly, the same incident switches to degraded instead of closing and opening another; if a degraded monitor goes down, the same incident switches to down. The incident keeps separate totals for the time spent down and the time spent degraded.

Going from degraded back to up is immediate. Going from down back to up takes the number of good checks in a row set by Mark healthy again after (1 to 5, 1 unless you changed it).

When an incident closes

An incident closes when:

• The monitor recovers. You get a “recovered” alert that shows how long the outage lasted and names the last failure seen before it recovered. If the failure changed during the outage (say from HTTP 503 to HTTP 502), the incident updates to show the latest failure within a few minutes.
• You pause the monitor. The incident closes at the moment you pause, with the time up to then counted. No alert is sent. See Pause and resume.
• Nothing can check the monitor any more. If you delete its only agent, or every checkpoint watching it stays unreachable for three checks in a row, the monitor turns unknown and the open incident closes at that point, with no “recovered” alert. If the target is still failing once checking resumes, a new incident opens with a fresh down alert; if it is fine, nothing is sent. The same applies to vendor, domain and DNS monitors when CuliPulse can’t read the status page or run the lookup three times in a row.
• You delete the monitor. Its incidents are deleted with it.

An unknown monitor never opens an incident of its own: CuliPulse can’t tell whether the target is up, so it records nothing about the time it couldn’t see.

CuliPulse also checks once a day that a monitor that is down or degraded still has its open incident, and recreates it (dated from when the monitor went down, with no new alert) if it was lost.

Where to find incidents

• The monitor’s Incidents tab lists its past incidents, newest first, with start time, duration and cause. Load older reaches back as far as your plan’s history, listed in Plans and limits.
• Overview lists the open incidents right now (the first 20, with the total count shown).
• Alert buttons. The View incident button in an alert opens the monitor with that incident highlighted. Webhook events carry the incident in detail.incident_id; see Webhooks.

An incident is a single row: start, end, duration and the cause. There is no per-incident timeline, no comments, and no “acknowledge” or “resolve by hand”: it closes only the ways listed above.

How incidents count toward uptime

Uptime in the monitor’s uptime figure and in reports counts down time only, taken from incidents:

• The seconds spent down are subtracted from uptime. Seconds spent degraded are shown separately and are never subtracted.
• An incident that is still open counts up to the present moment.
• An incident that crosses the edge of a report period counts its share inside the period.
• An incident closed by pausing, or because the monitor couldn’t be checked, counts up to the moment it closed.
• A monitor that was created partway through a period is measured from when it was created.
• Because an incident starts at the first failed check, and short outages are not recorded (see above), your uptime figure can still read slightly higher than a comparison against your own logs: an outage that clears before the required number of failed checks leaves no trace, and the time between the real failure and the next scheduled check isn’t visible to CuliPulse. Checking more often narrows the difference.

Heartbeat incidents

Heartbeat monitors work differently, because nothing is checked: your job checks in, and CuliPulse notices when it doesn’t.

• No waiting for repeated failures. A heartbeat goes down at the first missed check-in, and back up at the next check-in. Alert me after and Mark healthy again after don’t apply.
• When the incident opens. CuliPulse expects the next check-in within the schedule you set plus the grace period, counted from the last check-in (or from when you last changed the schedule or grace period, if that is later). When that time passes with no check-in, the incident opens at that moment, with the cause No check-in (dead-man’s switch tripped). In plain words: the job didn’t report in on time (email alerts say “No check-in received on time”). The time before that deadline is not counted as down.
• What the alert says. The went-quiet alert gives the last check-in and when the next one was expected (for example “Expected by 03:26 UTC”), in UTC in email and Telegram; Slack shows your local time. It doesn’t print an “overdue” figure, because the alert fires at the moment the deadline passes, so it would always be about zero.
• A job that reports failure. If your job calls the failure URL, the monitor goes down right away with the cause Job reported failure, without waiting for the grace period.
• A run that is still going at its max run time. If your job reports when it starts (/start) and you set a max run time, CuliPulse expects the finishing check-in within that limit. If the limit passes first, the monitor goes degraded (not down) at that moment, with a cause such as Run started 14:03 UTC, no finish check-in within 1h: when the start report arrived, and the limit. It says only what CuliPulse saw, not why the job hasn’t finished. The incident closes when the run’s finishing check-in arrives, even if late. If the job reports failure instead, the monitor goes down as below. If no check-in at all arrives by the normal deadline, the same incident turns down with the cause No check-in (dead-man’s switch tripped).
• A job that ran too long. If a run finishes over the max run time without having been flagged while it ran (for example, it started before the heartbeat’s first check-in), the monitor goes degraded with the cause Job ran too long (123s), where the number is how long the run took. The incident closes at the next check-in that finishes within the limit, or at any on-time check-in that comes without a start report.
• What the recovery alert says. If the incident was only ever degraded, the recovery alert says Back to normal, not “back up” or “was down”, because the monitor never went down. For a run that was still going at its max run time and then finished, it also says Slow for 26s. For a run that finished over the limit and was recovered by the next on-time check-in, it names the cause (“Job ran too long (71s)”) without a duration, because the time between those two check-ins is not how long the run took. If the incident did go down at some point (for example the run never finished and the monitor went down), the recovery says Back up and Was down for the time it was actually down. Other monitors follow the same rule: a response-time breach reads Slow for …, anything else that was degraded (an expiring certificate, packet loss, a domain near expiry) reads Degraded for …, so an alert never names a cause we didn’t see.
• A heartbeat that has never checked in has no deadline yet, so it opens no incident (unless your job calls the failure URL, which goes down right away). The schedule starts at the first check-in. The same is true after you resume a paused heartbeat: the first check-in is a quiet one, with no “recovered” alert, because there was nothing to recover from.

What the cause text means

The cause is a short plain-language reason that CuliPulse puts on the incident and in every alert. It is what CuliPulse saw, taken from the check, not a diagnosis of why. The wording below is the text you will see.

Website (HTTP)

Cause	What it means	What to check
HTTP 503 (service unavailable), HTTP 502 (bad gateway), HTTP 504 (gateway timeout), HTTP 500 (server error)	Your server, or something in front of it, answered with an error code. Other codes show as the plain code, such as HTTP 520.	Your server or load balancer logs at the incident time.
HTTP 403 (forbidden), HTTP 429 (rate limited), HTTP 401 (unauthorized), HTTP 404 (not found)	The address answered, but refused or didn’t find the page.	See Why does it say down when my site is fine?
HTTP 200 — expected 3xx (or a similar “expected” note)	The page answered with a code that isn’t the one the monitor was told to accept.	The monitor’s Expected status, or what the page should return.
Request timed out	No answer arrived before the monitor’s timeout.	Whether the server is slow or unreachable from the internet.
Connection failed (refused / unreachable)	The server could not be connected to.	The server is running and its firewall allows CuliPulse’s addresses.
Domain not found (expired, mistyped, or no DNS record)	The name doesn’t exist in DNS.	The address is spelled correctly and the domain hasn’t expired.
Couldn’t look up the domain (DNS error or timeout)	The name lookup failed or took too long.	Your DNS provider’s status.
TLS handshake failed, TLS certificate expired	The secure connection could not be set up, or the certificate is out of date.	The certificate and its chain.
Slow response (SLA breach)	Degraded: the page answered slower than the response-time limit you set.	Server load.
TLS certificate expiring soon	Degraded: the certificate is close to the expiry warning you set.	Renew the certificate.
Page missing expected text, Page contained forbidden text, Page didn’t match expected pattern, Response body didn’t match expected, JSON body didn’t match expected, Response header didn’t match expected	The page answered, but a content check you set didn’t pass.	The check’s expected text or value, and what the page returns.
Invalid request configuration	The monitor’s own settings can’t be turned into a request.	Edit the monitor.
Address is on a private network — shared checkpoints can’t reach it	The shared locations never connect to private addresses.	Use your own agent. An HTTP monitor on a private IP address can use only your own agents.
Redirected to an unencrypted http:// address — not followed, so the check’s secrets aren’t sent	The site redirected an authenticated check from https to http, so CuliPulse stopped before sending your credentials in clear text.	Make the redirect stay on https.

Port (TCP and UDP)

Cause	What it means
TCP check failed, UDP check failed	The check failed in a way that doesn’t fit the cases below.
Couldn’t connect — port closed or refused	Something answered that the port is closed. Agent running in Docker on the same machine? See Checking services on the same machine.
Couldn’t connect — no answer before the timeout	Nothing answered before the timeout.
Connected, but the server sent nothing	The port is open but the server sent no greeting you expected.
Connected, but the reply didn’t match	The server answered with something other than the expected text.
No reply to our test message (UDP)	Many UDP servers ignore a message they don’t understand. If yours is running, it may expect a different payload.
The server replied, but the reply didn’t match (UDP)	The server answered, but not with the expected reply.

See Why is my port monitor down?.

Ping

Cause	What it means
100% packet loss (no reply)	No ping got an answer.
High latency / packet loss	Degraded: replies are slow or some are lost, past the limit you set.
Probe socket error (agent)	The agent couldn’t open the connection it needs to send pings. On your own agent this almost always means it lacks ping permission: start it with --cap-add=NET_RAW (see Installing an agent). If it shows on a shared location, contact support.

DNS, domain and vendor monitors

Cause	What it means
Domain does not resolve (NXDOMAIN), Expected DNS record missing, DNS record didn’t match expected, DNS lookup failed	A DNS monitor’s lookup failed, or the record isn’t what you expect.
Domain registration lapsed	The domain’s registration has expired.
Domain registration expiring soon	Degraded: the domain is within your warning window.
Vendor reports an outage, Vendor reports degraded service	The vendor’s own status page reports a problem. The incident can also name what the vendor reported.

Heartbeat

See Heartbeat incidents above: No check-in (dead-man’s switch tripped), Job reported failure, Run started 14:03 UTC, no finish check-in within 1h (the start time and your limit; a limit of a day or more also shows the date), Job ran too long (…s).

When CuliPulse can’t judge

The alerts for an unknown state say so plainly. When an agent stopped checking in, they name it and when it was last heard from, for example “Agent office hasn’t checked in since 14:32 UTC — check that it’s running” (for a shared CuliPulse agent they say CuliPulse runs it, so there is nothing to fix on your side). Otherwise chat and webhook alerts say “All checkpoints unreachable” and email says “No location could run the check”, with “This is a coverage gap, not a confirmed outage.” It isn’t an incident: CuliPulse couldn’t run the check, so it doesn’t say whether your target is up.

Why does it say down when my site is fine?

An incident means the check CuliPulse ran failed. It doesn’t prove your site was unreachable to everyone. If you can open the page in your browser, work through this list.

The cause is HTTP 403 or HTTP 429. The address answered, but with “forbidden” or “slow down”. The shared locations run from cloud data centers, and some firewalls, CDNs and bot protections turn away traffic from those networks. That may be what is happening; CuliPulse can’t see from the incident alone why the answer was refused. To find out and fix it:

1. Run Diagnose on the monitor (see Run a check on demand) to see whether the refusal repeats right now.
2. Look in your firewall, CDN or bot-protection logs at the incident time for refused requests. Checks from the shared locations come from a short, published list of addresses, and identify themselves with a User-Agent that contains culipulse-agent (unless you set your own).
3. Allow CuliPulse’s checks in your protection settings: see Allow CuliPulse through your firewall for the addresses and the safest ways to do it. Or run your own agent from an address you control.
4. If the status you got is the right answer for that address (for example a login-protected page that always answers 401), set the monitor’s Expected status to match.

The cause shows “HTTP 200” or another success code under Down. Check the monitor’s Expected status. A page that answers 200 is down if the monitor expects a redirect (3xx). See the cause table above.

Only one location sees a failure. On the monitor’s Checkpoints tab, compare the locations. A failure from one location only is often a network problem near that location. If you want one location’s failure to count, turn off Only alert if more than one location fails; if you don’t, see When only one location fails.

A timeout or “Connection failed” only at certain times. Your server may be limiting connections or is briefly overloaded. A higher Alert me after setting filters short episodes out, at the cost of later alerts.

Still stuck. Send the incident’s start time and the monitor’s name to [email protected].