How our own product caught a 19-day backup failure

For nineteen days, CaneCron had no automated database backups. Every scheduled run failed silently. No error, no email, nothing — because the failure happened before the part of the script that sends errors.

Here's the story, because it's a good lesson in why silent failures are the dangerous ones.

The one-character bug

We rewrote our backup script and, in doing so, left a blank line above the #!/bin/bash shebang. When cron executed it, the kernel couldn't find the interpreter on line one, fell back to a minimal shell, and that shell choked on the first bash-specific line — set -o pipefail. The script died immediately, before it ever reached the code that would have emailed us the error.

Manual runs always worked, because an interactive shell is more forgiving. So every time we checked by hand, backups looked fine. They weren't.

What caught it

CaneCron monitors its own backup job with a cron ping — the script pings a monitor only on success. After the rewrite, those pings stopped. Nineteen days later, the "last ping 2 weeks ago" alert on the phone is what surfaced it.

The irony isn't lost on us: our monitoring product caught a failure in our own infrastructure that every manual check had missed. That's exactly the case cron monitoring exists for — a job that stops running doesn't announce itself. Something has to notice the absence.

What we changed

• The backup script now pings only after a verified, size-checked backup.
• Off-site backups are now a priority, not a nice-to-have.
• And we added a rule: after any script rewrite, verify the shebang is on line one. A blank line above it is a silent killer.

If you run cron jobs that matter, monitor them for silence — not just for failure. The quiet failures are the ones that hurt.