Cron Job Failure Modes and How to Detect Them

Failure Mode 1

The script exits 0 but does nothing

This is the most insidious failure. Your crontab says 0 2 * * * /opt/backup.sh. The script runs, exits with code 0, and cron logs "success." But the backup file is 0 bytes, or the database dump is truncated, or the script hit an error path that exit 0'd instead of failing properly.

Cron has no way to know. It trusts the exit code. If the script says "I'm fine," cron believes it.

How to detect it: Heartbeat monitoring catches the "script never ran" case, but not this one — the script did run, it just didn't do its job. The fix is to have your script verify its own output before pinging. For example:

0 2 * * * /opt/backup.sh && [ -s /backups/db.sql.gz ] && curl -s https://cronwatch.steveco-ai.com/ping/YOUR-SLUG

The [ -s ... ] check ensures the backup file exists and is non-empty before sending the success ping. If the file is missing or 0 bytes, no ping is sent, and CronWatch alerts you after the grace period.

Failure Mode 2

The cron daemon is stopped

Sometimes crond isn't running. A package update stopped it and didn't restart it. A misconfigured systemctl disable cron from a hardening script. A container restart that didn't bring cron back up. Your scripts are scheduled, the clock is ticking, but nothing executes.

This is silent. There's no error log because nothing ran to produce one. The only signal is the absence of pings.

How to detect it: Heartbeat monitoring is purpose-built for this. If your job is supposed to ping every hour and CronWatch doesn't hear from it within the grace period, you get an alert. The grace period matters here — set it to 1.5x your expected interval so a single missed run doesn't page you, but two consecutive misses do.

# Job runs every hour, grace = 90 minutes
curl -X POST https://cronwatch.steveco-ai.com/api/v1/jobs \
  -H "Authorization: Bearer ***" \
  -d '{"name": "hourly-report", "expected_interval": 3600, "grace_seconds": 5400}'

Failure Mode 3

Server rebooted, cron not running

After a reboot, crond should start automatically if it's enabled. But "should" is doing a lot of work. If cron isn't enabled (someone ran systemctl disable cron during a security audit and forgot to re-enable it), or if the server uses a container where cron isn't in the entrypoint, your jobs won't run after the next reboot.

This is a subset of Mode 2, but the detection window is longer — you might not reboot for weeks, so the first missed ping after a reboot could be days late.

How to detect it: Same as Mode 2 — heartbeat monitoring. But also add a boot-time ping so you know the server came back up and cron is running:

# /etc/cron.d/boot-check
@reboot sleep 60 && curl -s https://cronwatch.steveco-ai.com/ping/BOOT-SLUG

If you don't get the boot ping within an hour of a known reboot, cron isn't running. If you don't get it and didn't reboot, something else is wrong.

Failure Mode 4

The script hangs and never exits

Your backup script connects to a database, the database is unresponsive, and the script blocks on recv() forever. It never exits, never pings, and never produces an error. Cron won't kill it (there's no built-in timeout). The process sits there indefinitely, and the next scheduled run either stacks up behind it or is skipped.

This is harder to detect than a stopped cron because the process is running — it's just not doing anything.

How to detect it: Two layers. First, add a timeout to your script so it can't hang forever:

# In your crontab — kill after 30 minutes
0 2 * * * timeout 1800 /opt/backup.sh && curl -s https://cronwatch.steveco-ai.com/ping/YOUR-SLUG \
  || curl -s https://cronwatch.steveco-ai.com/ping/YOUR-SLUG/fail

Second, heartbeat monitoring catches the "never pinged" case. If the script hangs and times out, the || branch sends a failure ping. If it hangs and the timeout kills it, same thing. If somehow neither fires, the missing heartbeat still triggers an alert.

Failure Mode 5

Resource exhaustion — the OOM killer

Your script runs fine for months, then one day the server is under memory pressure (a second process spiked, or the dataset grew), and the Linux OOM killer terminates your script. The script doesn't get a chance to exit cleanly — it receives SIGKILL (signal 9), which cannot be caught or handled. No cleanup, no error message, no exit code in your script's log.

Cron will log something like ("backup.sh" terminated by signal 9) in /var/log/syslog, but if you're not watching that log, you won't know.

How to detect it: The /fail endpoint won't help here — the script was killed before it could send any ping. Heartbeat monitoring is your detection mechanism. The ping never arrives, the grace period expires, and you get alerted. To confirm it was the OOM killer, check:

sudo dmesg | grep -i "killed process"

Or check the kernel log for OOM events. If this happens repeatedly, either add memory, add swap, or split the job into smaller pieces.