Forem: Jasper Brookers

Cron Job Monitoring for Backups: What Actually Goes Wrong

Jasper Brookers — Tue, 27 Jan 2026 04:08:10 +0000

Backups are the most trusted cron jobs in any system.
Once they are set up, everyone assumes:
“Backup is running daily, no worries.”

But in reality, backup cron jobs fail more dangerously than any other job — because they often look successful until the day you actually need them.

Let’s talk honestly about what actually goes wrong with cron-based backups, and how to monitor them properly.

Why Backup Cron Jobs Are Special (And Risky)

Unlike many other cron jobs:

Backups are rarely checked
Failures are discovered only during recovery
A “successful run” does not mean a usable backup

This makes backup monitoring critical.

Common Backup Failures You Will Eventually Face

1. Backup Runs, But File Is Empty

This is very common.

What happens:

Database connection fails
Dump command exits early
Permissions issue

Result:

Backup file exists
File size is 0 bytes

Cron thinks job succeeded.

Your restore will not.

Why this is dangerous:

Most people only check if a file exists, not whether it contains data.

2. Backup Stops Mid-Way

What goes wrong:

Network drops
Disk fills up
Database connection resets

Result:

Partial backup file
Script exits without clear error
No alert sent

This is a false sense of security failure.

3. Backup Never Runs at All

This is the worst one.

Reasons:

Server rebooted and cron didn’t start
Cron daemon stopped
Crontab overwritten during deployment
Timezone misconfiguration

Result:

No backup for days or weeks
No logs
No alerts

4. Backup Runs, But Upload Fails

Very common with cloud backups.

What goes wrong:

S3 credentials expire
Network timeout
Storage quota exceeded

Result:

Local backup exists
Remote backup missing
Nobody notices

You think you have offsite backups. You don’t.

5. Backup Is Too Old (But Nobody Notices)

Backup runs fine, but:

Retention job fails
Rotation logic breaks
Old backups silently deleted

Result:

Only very old backups remain
Recent data is gone

6. Backup Takes Longer and Longer Over Time

As data grows:

Backup duration increases
Job overlaps with next run
Server load increases

Result:

Slowdowns
Failed backups
Corrupted files

Cron does not warn you about runtime drift.

Why Logs Don’t Save You

Many teams rely on logs to monitor backups.

But logs fail when:

Disk is full
Job never starts
Script hangs before logging
Nobody checks logs regularly

A backup that fails silently is worse than no backup at all.

What You Should Actually Monitor for Backups

Monitoring backups is not about checking cron ran.

It’s about verifying outcomes.

Here’s what actually matters:

1. Did the Backup Job Run?

If it didn’t run:

Something is fundamentally broken
You need to know immediately
Use execution confirmation with alerting.

2. Did the Backup Job Finish?

A started backup is not a finished backup.

Monitor:

Start vs completion
Maximum expected duration

Alert if the job hangs or runs too long.

3. Did It Produce a Valid Backup?

Don’t just check existence.

Monitor:

File size
Timestamp freshness
Basic integrity checks

Empty or tiny backups are failures.

4. Did the Backup Reach Safe Storage?

Local backup is not enough.

Monitor:

Upload success
Remote storage presence
Storage quota issues

If offsite copy fails, backup is incomplete.

5. Is Backup Health Degrading Over Time?

Watch trends:

Increasing runtime
Increasing failures
Increasing storage usage

These are early warning signals.

Heartbeat vs Workflow Monitoring for Backups

Backups are not simple jobs. They need more than a single “ping”.

Heartbeat Monitoring

Good for:

“Did the job run?”

Bad for:

Partial backups
Hung uploads
Multi-step failures

Workflow Monitoring (Recommended)
Better approach:

Signal when backup starts
Signal when backup completes
Signal when upload completes
Alert if any step fails or times out

This gives you real confidence, not hope.

Tools That Help Monitor Backup Cron Jobs

Tool	Detect Missed Backups	Detect Hung Backups	Track Duration	Workflow Steps	Backup-Friendly
Cronbee	✅	✅	✅	✅	Very Good
Cronitor	✅	✅	✅	⚠️ (limited)	Good
Custom Scripts	✅	✅	✅	✅	Powerful but risky
Dead Man’s Snitch	✅	❌	❌	❌	Basic
Healthchecks.io	✅	⚠️ (timeouts)	⚠️ (basic)	❌	Good (simple)

One Hard Truth About Backups

A backup you don’t monitor is not a backup.

If nobody knows it failed, then when the day comes:

Data is already lost
The incident already happened

Final Thoughts

Cron is good at running backup commands.

It is terrible at telling you whether backups are usable.

The solution is not replacing cron — it’s adding visibility:

Did it run?
Did it finish?
Did it produce valid data?
Did it reach safe storage

Answer these questions automatically, and your backups will finally be reliable.

10 Cron Jobs That Silently Fail (And How to Detect Them)

Jasper Brookers — Wed, 21 Jan 2026 11:57:15 +0000

Cron jobs are everywhere. They run backups, sync data, generate reports, clean databases, and keep systems alive.

Yet some of the most critical cron jobs fail silently, sometimes for weeks, before anyone notices.

Here are 10 common cron jobs that silently fail, why they fail, and how to detect them reliably.

1. Database Backups

What goes wrong:

Disk is full
Credentials expire
Backup command exits early
Backup file is created but empty

Why it’s silent ?

Cron only runs the command. It doesn’t verify backup integrity.

Detection

Check backup size
Monitor expected execution time
Use heartbeat monitoring to ensure the job actually completed

2. Log Rotation Jobs

What goes wrong:

Permissions change
Path no longer exists
Script runs but rotates nothing

Impact:

Disks fill up
Applications crash later

Detection

Alert if job doesn’t run

Alert if disk usage keeps increasing after rotation

3. Data Sync / ETL Jobs

What goes wrong:

API rate limits
Partial data sync
One step fails but script exits 0

Detection

Workflow monitoring (start / step / complete)
Validate row counts or checksums

4. Cleanup Jobs (Temp Files, Old Records)

What goes wrong:

Query condition changes
Script becomes a no-op
Job runs but deletes nothing

Detection

Track execution duration
Alert on sudden runtime drops

5. SSL Certificate Renewal (Certbot)

What goes wrong:

Renewal fails silently
Cron runs but certificate not replaced

Impact:

Website outage days later

Detection

Monitor expiration dates
Alert if renewal job doesn’t report success

6. Email or Notification Jobs

What goes wrong:

SMTP credentials expire
Mail provider blocks IP
Emails fail but script continues

Detection

Monitor success events
Track actual sent counts

7. Report Generation Jobs

What goes wrong:

Data source unavailable
Script generates empty reports

Detection

Validate output files
Alert if report size is suspiciously small

8. Cache Warmup Jobs

What goes wrong:

Job runs before dependency is ready
Cache never populated

Detection

Workflow monitoring with dependency checks

9. Payment Reconciliation Jobs

What goes wrong:

API changes
Partial failures
Currency mismatches

Detection

Alert on missing execution
Compare expected vs actual transaction counts

10. “Temporary” Cron Jobs That Become Permanent

What goes wrong:

Nobody remembers they exist
They keep failing unnoticed

Detection

Centralized cron monitoring
Ownership tracking

How to Detect Silent Failures (Without Relying on Luck)

Silent failures happen because cron answers only one question:

“Was the command triggered?”

It does not tell you whether the job actually did what it was supposed to do.

Detecting silent failures requires adding signals and expectations around execution, not just running the task.

Here are the most effective detection strategies.

1. Execution Confirmation (Did the Job Run at All?)

The most basic silent failure is non-execution:

Server was down
Cron daemon stopped
Crontab was overwritten
Timezone or schedule changed

Detection approach:

Define an expected execution window
Trigger an alert if the job does not report within that window

This detects:

Missed runs
Infrastructure-level failures
Scheduling mistakes

2. Completion Confirmation (Did the Job Finish?)

Some jobs start but never finish:

Processes hang
Network connections stall
Deadlocks occur
Scripts block waiting for input

Detection approach:

Distinguish between “job started” and “job completed”
Alert if completion is not reported within an expected duration

This detects:

Hung processes
Infinite loops
Long-running degradations

3. Duration Anomalies (Did It Take Too Long or Too Little Time?)

Sudden runtime changes are a strong signal of silent failure:

Jobs that run much faster may be skipping work
Jobs that run much longer may be stuck or retrying endlessly

Detection approach:

Track historical execution durations
Alert on abnormal deviations

This detects:

Partial execution
Skipped data
Performance regressions

4. Output Expectations (Did the Job Produce Something?)

Many cron jobs are expected to:

Generate a file
Send data
Update records
Produce side effects

Detection approach:

Validate that expected outputs exist
Watch for anomalies in size, count, or freshness

This detects:

Empty backups
Missing reports
Failed exports

5. Workflow Visibility (Did Every Step Run?)

Complex jobs often have multiple steps:

Fetch data
Transform data
Store results
Notify downstream systems

Detection approach:

Track progress through defined stages
Alert if a job stops mid-workflow

This detects:

Partial failures
Broken dependencies
Mid-pipeline crashes

6. Ownership & Accountability

Silent failures often persist because:

Nobody “owns” the job
Alerts go nowhere
Failures are ignored

Detection approach:

Assign ownership per job
Route alerts to people who can act
Track recurring failures

This detects:

Long-running neglect
“Zombie” cron jobs

Tools That Help Detect Silent Failures

Tool	Detect Missed Runs	Detect Hung Jobs	Duration Tracking	Workflow Visibility	Output Validation	Complexity	Notes
Cronbee	✅	✅	✅	✅	⚠️ (via workflow logic)	Medium	Strong focus on execution state and workflows
Cronitor	✅	✅	✅	⚠️ (limited)	❌	Medium	Good dashboards and historical trends
Custom Monitoring	✅	✅	✅	✅	✅	Hard	Maximum control, high maintenance cost
Dead Man’s Snitch	✅	❌	❌	❌	❌	Very Easy	Focused purely on missed executions
Healthchecks.io	✅	⚠️ (timeouts only)	⚠️ (basic)	❌	❌	Very Easy	Excellent for simple heartbeat monitoring