Forem: Kriss

Output assertions: the cron job check most monitoring tools skip

Kriss — Tue, 28 Apr 2026 21:28:25 +0000

Output assertions: the cron job check most monitoring tools skip

A follow-up to A reader comment made me realise I'd only solved half the problem — this is a deeper reference guide on output assertions specifically.

"Did it run?" is the wrong question.

Every monitoring tool asks it. Heartbeat monitors, cron schedulers, even purpose-built tools like Cronitor and Healthchecks.io — they all fundamentally ask: did the job check in? If yes, green. If no, red.

It's a useful question. But it's not the useful question.

The failure mode that looks like success

Imagine a nightly job that syncs user records from your CRM into your database. It runs at midnight, takes about 90 seconds, and exits cleanly. Your heartbeat monitor sees the ping at 12:01:34am and marks it healthy.

What it doesn't see: the job synced 0 records. It has been syncing 0 records for eight days, since someone rotated the CRM API credentials and forgot to update the environment variable. The job connects, gets a 401, logs a warning, falls back to a no-op, and exits 0.

All monitoring: green. Business: broken for eight days.

This is not a hypothetical. Variants of this failure happen constantly. The job ran. That fact is true and also completely useless.

What "did it do anything?" looks like

Output assertions flip the question. Instead of only checking that the job pinged in, you also check what it reported.

A job that processes records should report how many it processed. A job that generates a file should report the file size. A job that sends emails should report how many it sent. You instrument the job to emit a count — one number representing meaningful work done — and your monitoring layer validates it falls within expected bounds.

The failure modes this catches:

Zero when non-zero expected: sync runs, processes nothing, exits clean
Suspiciously low counts: normally syncs 500 records, today synced 3
Count drift over time: weekly report used to include 10k rows, now consistently 200

None of these trip a heartbeat check. All of them are real problems.

Why most tools don't do this

Heartbeat monitoring is architecturally simple: job pings URL, URL records timestamp, alerting checks timestamp age. The data model is just "last seen at".

Output assertions require more: the job must emit structured data, the tool must store it, and the alerting logic must understand what "normal" looks like for that specific job. That's a significantly more complex product to build.

Most tools solve the simpler problem because it covers the obvious failure mode and is much easier to ship.

How to instrument your jobs

The instrumentation is lightweight. Pick a number that represents meaningful work and emit it at the end:

# Database backup — report dump file size
result = subprocess.run(["pg_dump", "-Fc", "mydb", "-f", "/backups/mydb.dump"])
dump_size = os.path.getsize("/backups/mydb.dump")
ping_monitor(count=dump_size)

# CRM sync — report records synced
synced = sync_from_crm()
ping_monitor(count=len(synced))

# Email campaign — report emails sent
sent = send_campaign(campaign_id)
ping_monitor(count=sent)

Three extra lines per job. The return is knowing your job didn't just run — it did something. (ping_monitor is a wrapper around your monitoring call — implementation below.)

Sending the count to your monitor

DeadManCheck accepts a count parameter with each ping:

curl -fsS "https://deadmancheck.io/ping/YOUR-TOKEN?count=1547" > /dev/null

You configure the assertion on the monitor: "alert if count is 0" or "alert if count drops below threshold". If the job checks in but reports zero records, you get alerted — even though the job technically ran fine.

It also does duration monitoring with rolling average anomaly detection. If your 90-second job starts taking 45 minutes, that gets flagged too. Jobs that hang are a separate silent failure mode that output counts don't catch on their own.

The right question

Monitoring that only asks "did it run?" will eventually lie to you at the worst possible moment.

The right question is "did it do anything useful?" Output assertions are how you ask that question automatically, at 2am, every night, without anyone having to check.

Start with your backup jobs. That's where the answer matters most.

A reader comment made me realise I'd only solved half the problem

Kriss — Sat, 25 Apr 2026 13:20:52 +0000

A reader comment made me realise I'd only solved half the problem

Last month I wrote about the cron job failure mode nobody talks about: the job that doesn't die, it just drags.

The short version: a nightly ETL job at a previous employer took four hours instead of forty minutes for six days before anyone noticed. It ran. It completed. It exited zero. Every dashboard showed green. Downstream data was silently wrong.

The fix I described was duration anomaly detection — once you have a few weeks of run history, you know what "normal" looks like. A job that takes 4x its baseline is a signal even if it succeeded. I built DeadManCheck partly because I couldn't find a tool that combined silence detection with duration tracking.

The article got some traction. Then someone left a comment that stopped me in my tracks - https://dev.to/krissv/the-cron-job-failure-mode-nobody-talks-about-3p1a

The comment

The failure mode I keep seeing: the job runs, logs "complete," and the output silently goes nowhere.

No error. No alert. Just a cron that appeared healthy while accomplishing nothing for days.

The fix that actually works is external verification. Don't check that the job ran; check that the downstream artifact exists. A job that succeeds but doesn't write the expected DB record is the same as a failed job.

They were right. And I hadn't covered it.

Duration anomaly detection catches "job ran slow." Silence detection catches "job didn't run." Neither catches "job ran fine, on time, but produced nothing."

That's a third failure mode entirely.

What this looks like in practice

Here's a simplified backup script:

import psycopg2
import csv
import os

conn = psycopg2.connect(os.environ["DATABASE_URL"])
cur = conn.cursor()

cur.execute("SELECT * FROM orders WHERE exported = false")
rows = cur.fetchall()

with open("/backups/orders.csv", "w") as f:
    writer = csv.writer(f)
    writer.writerows(rows)

cur.execute("UPDATE orders SET exported = true WHERE exported = false")
conn.commit()

print(f"Backup complete. {len(rows)} rows exported.")

Can you spot the bug?

The script runs. It prints "Backup complete. 0 rows exported." It exits cleanly.

The bug is in a migration from three weeks earlier. A developer renamed the exported column to is_exported. The WHERE clause now silently returns nothing. Every night: zero rows fetched, empty CSV written, nothing marked, exit code 0.

Exit code: 0. Monitoring alert: none.

This is exactly what the commenter was describing. A job that succeeds but produces nothing is functionally the same as a failed job. Your monitoring just doesn't know that yet.

Why the standard fix is hard to scale

The commenter suggested checking the downstream artifact — verify the DB record exists, check the file isn't empty. That's the correct instinct, but it requires custom verification logic for every job. Each job writes to a different place, in a different format, with different expectations about what "something" looks like.

What I wanted was a generalised version: tell the monitoring service what your job produced, and let it decide if that's suspicious.

That's what I built into DeadManCheck as output assertions.

How output assertions work

The idea is simple. When your job pings the monitoring service at completion, it includes a count of what it actually did:

curl -fsS "https://deadmancheck.io/ping/YOUR-TOKEN?count=0" > /dev/null

You configure a rule: "alert if count is 0 more than once in a row" or "alert if count drops more than 80% below the rolling average."

The job ran. It just did nothing. Now you know.

In Python:

import requests
import os

def ping_deadmancheck(count=None):
    token = os.environ["DEADMANCHECK_TOKEN"]
    params = {"count": count} if count is not None else {}
    try:
        requests.get(f"https://deadmancheck.io/ping/{token}", params=params, timeout=5)
    except requests.RequestException:
        pass  # never let monitoring break the job

rows_processed = do_the_work()
ping_deadmancheck(count=rows_processed)

Ten lines. The complexity stays in the service, not in your scripts. And unlike checking a downstream artifact, it works the same way regardless of what your job actually produces.

The full picture: three failure modes

After that comment, I updated my mental model. There are three distinct ways a cron job can fail silently:

Failure mode	What happens	What catches it
Job doesn't run	Silence. No ping arrives.	Dead man's switch (silence detection)
Job runs slow	Ping arrives late or after too long	Duration anomaly detection
Job runs, produces nothing	Ping arrives on time, output is empty	Output assertions

Most tools only cover the first row. Some cover the first two. The third is almost always a blind spot.

What I do now

Every background job I write now has three things:

A counter variable tracking records processed
A guard clause that exits non-zero if zero is never a valid outcome
A heartbeat ping that includes the count

rows_processed = do_the_work()

if rows_processed == 0:
    raise RuntimeError("Processed 0 records — investigate before marking success")

ping_deadmancheck(count=rows_processed)

For jobs where zero is sometimes valid (quiet periods, weekends), skip the guard clause and let the monitoring service decide based on historical patterns.

Credit where it's due

I wouldn't have built output assertions without that comment. Sometimes the feature request hiding in a code review or a reply thread is the most valuable one you'll get.

If you've got a background job running right now, ask yourself three questions:

Will I know if it silently stops running?
Will I know if it starts taking 4x longer than normal?
Will I know if it ran perfectly but accomplished nothing?

If any of those is "no" — that's your monitoring gap.

→ Try DeadManCheck free at deadmancheck.io

The cron job failure mode nobody talks about

Kriss — Sun, 29 Mar 2026 19:04:31 +0000

A few months ago, a nightly ETL job at a previous job nearly cost us a major client. Not because it failed. Because it took four hours instead of forty minutes — and nobody noticed for six days.

The job ran. It completed. It exited zero. Every monitoring dashboard showed green. Meanwhile, the downstream data pipeline was ingesting half-processed records, and reports were silently wrong. By the time a client flagged it, we had six days of corrupted reporting to unpick.

This is the failure mode nobody talks about: the job that doesn't die, it just... drags.

Why your existing monitoring misses it -

If you're using Healthchecks.io, Better Uptime, or a similar dead man's switch tool, here's how it works: your cron job pings a URL at the end of each run. If the ping doesn't arrive within a grace window, you get an alert.

That's genuinely useful. It catches jobs that crash, hang indefinitely, or never start. But what it doesn't catch is a job that completes in 240 minutes when it should take 45. The ping arrives. The check passes. Everything looks fine. The tool has no idea what "normal" looks like for that job — it only knows silence vs. noise.

Duration anomaly detection is the missing piece.

What duration anomaly detection actually means

The concept is simple: instead of only checking whether a job completed, you also check how long it took.

Once you have a few weeks of run history, you know that your nightly job usually takes 40–50 minutes. So when it takes four hours, that's a signal — even if it succeeded. Something changed: the dataset grew, a dependency got slow, a query plan degraded, a network hop started timing out and retrying.

Catching this early means you can investigate before it causes damage downstream.

The /start + /finish pattern -

Job begins

curl -s "https://deadmancheck.io/ping/abc123/start"

... your actual job logic ...

Job ends

curl -s "https://deadmancheck.io/ping/abc123"

Now the monitoring service knows: this run started at T, it ended at T+4h. It compares that against the rolling average of previous runs and alerts if the duration exceeds a configurable threshold — say, 2x the usual runtime. Two curl calls. The complexity lives in the service, not in your scripts.

Why this matters more as systems age -

New jobs are fast. As systems mature, things get slower in ways that creep up on you. Rows accumulate. Indexes bloat. Third-party APIs introduce latency. Your job that took 8 minutes in January takes 35 minutes in October.

Without duration tracking, you have no visibility into this degradation. With it, you have a canary. The alert fires at 70 minutes, you investigate, you find the index that needs rebuilding. Crisis averted before the downstream effects compound.

So I built this -

After looking for a tool that combined silence detection with duration anomaly detection and not finding one, I built DeadManCheck (deadmancheck.io). It supports the /start + /ping pattern, tracks rolling run history, and alerts you when a job takes significantly longer than its baseline. Standard silence detection is included too, so both failure modes are covered in one place.

Free tier available, no credit card required.

The checklist -

Next time you wire up a cron job, ask yourself:

Will I know if this job silently stops running?
Will I know if this job starts taking 4x longer than normal?
Will I know before my users do?

If the answer to any of those is "no", you have a monitoring gap. It's a small one to close.

→ Try DeadManCheck free at deadmancheck.io