A cron job failed and nobody noticed: how to catch the ones that fail quietly

Cron's error reporting is a mail to the crontab's owner. On a modern server there is usually no MTA, no mail spool and nobody reading root@localhost — so the output goes nowhere. A job that has been failing every night for three weeks looks exactly like one that has been succeeding.

Worse is the job that runs, exits zero, and does nothing useful: a backup writing a zero-byte file, a sync that silently skipped every record.

Report success, and only success

# Wrong: pings whether or not the job worked
0 3 * * * /usr/local/bin/backup.sh; curl -fsS https://hb.alertkite.com/p/TOKEN

# Right: && only pings on exit code 0
0 3 * * * /usr/local/bin/backup.sh \
  && curl -fsS -m 10 https://hb.alertkite.com/p/TOKEN

# Better: report the failure too, so you know it ran and broke
0 3 * * * /usr/local/bin/backup.sh \
  && curl -fsS -m 10 https://hb.alertkite.com/p/TOKEN \
  || curl -fsS -m 10 https://hb.alertkite.com/p/TOKEN/fail

Setting it up

Chain the ping with && so it only fires when the job exits zero.
Add a || branch to /fail so a job that ran and broke is distinguishable from one that never started.
For long jobs, ping /start first — then a job that hangs is visible as started-but-never-finished.
Set the expected period slightly longer than the real interval, so a slow night is not an alert.

The semicolon is the bug

`command; curl` runs the ping regardless of whether the command worked. It is the single most common mistake in heartbeat setups, and it produces a monitor that reports green for a job that has been failing for months — worse than no monitoring, because it is actively reassuring.

Exit zero is not the same as success

Plenty of scripts exit zero after doing nothing. Where that is a risk, assert on the result rather than the exit code: check the backup file exists and is larger than a threshold, then ping.

Cron's environment is not your shell's

No PATH to speak of, no profile sourced, a different working directory. A script that runs perfectly by hand and fails under cron is almost always this. Use absolute paths and test with `env -i`.