Monitoring automated workflows: how to catch silent failures

The worst automation failure isn't the one that crashes loudly. It's the nightly invoice sync that stopped running three weeks ago, or the lead-routing flow that still runs every hour but has been sending everything to an inbox nobody reads. Nothing errored, so nothing alerted. You find out when a customer asks why they were never billed.

Most small automations get built with a happy path and maybe a retry. Monitoring is the part that gets skipped, because it's invisible when things work. This post is about the handful of checks that would have caught those failures, and how to set them up without drowning in alerts.

Why automations fail quietly#

A web app has users who complain when it breaks. A background workflow usually doesn't. It runs on a schedule or a webhook, talks to two or three other systems, and writes results somewhere a person only looks at occasionally.

That creates a few failure shapes that error alerts alone won't catch:

• The job never starts. The scheduler was paused, the server was rebuilt without the cron entry, or a webhook subscription expired on the other side.
• The job runs and "succeeds" but does nothing. An API now returns an empty list because a filter field was renamed, so the loop processes zero records and exits cleanly.
• The job does the wrong thing confidently. An LLM step starts classifying every email as "general enquiry" after a prompt change, and every downstream step happily follows.
• The job is fine but slow. A step that took two minutes now takes forty, and it's about to collide with the next scheduled run.

Only the first of these looks like a failure to the machine. The rest look like normal, successful runs. So the useful question isn't "did it throw an error?" but "did the thing I care about actually happen?"

Watch the outcome, not just the process#

Google's SRE book draws a line between two kinds of monitoring. White-box monitoring is "based on metrics exposed by the internals of the system", while black-box monitoring means "testing externally visible behavior as a user would see it." The same chapter separates symptoms ("what's broken") from causes ("why"), and argues it's better to spend much more effort catching symptoms than causes.

For an automation, the symptom is the missing outcome. Some examples of outcome checks:

These checks live outside the automation and look at the destination system. That's the point. If the job silently stops, the check still runs and notices. A check written inside the job dies with it.

The SRE chapter's "four golden signals" (latency, traffic, errors and saturation) are written for user-facing services, but they translate well. Latency becomes run duration. Traffic becomes records processed per run. Errors are failed runs or failed items. Saturation is how close you are to a limit that matters, such as an API quota or the gap before the next scheduled run. Records processed per run is the one people most often skip, and it's the one that catches the "succeeds but does nothing" case. A run that handled zero records on a weekday should look suspicious, even if it exited cleanly.

Heartbeats for jobs that might not run at all#

The simplest defence against "the job never started" is a heartbeat, sometimes called a dead man's switch. The job pings a monitoring service when it finishes, and the service raises the alarm when the ping doesn't turn up. You're alerting on an absence, which a job that never started can't do for itself.

Healthchecks.io is one service built around this pattern, and its docs describe the mechanics clearly. Each job sends an HTTP request on success, and "when Healthchecks.io does not receive the HTTP request at the expected time, it notifies you." You set a grace time, "the amount of extra time to wait when a cron job is running late before declaring it as down." In their example a job starts at 14:00 with 30 minutes' grace; if there's no ping by 14:30, the job is declared failed and alerts go out.

Two details from those docs are worth copying whatever tool you use. First, their example only sends the ping if the script exits with code zero (your_script && curl …), so a crash also shows up as a missing ping. Second, you can send a "start" signal at the beginning of the run to track how long it takes. With a start signal in place, the grace time also caps how long a run can take before the check is marked failed, which covers the slow-job case as well.

For a simple interval schedule, the configuration docs define the two settings you'll tune: "Period is the expected time between pings" and grace time is "the additional time to wait before sending an alert when a check is late." Set the grace time from real run durations, not a guess. If the job normally takes 5 to 12 minutes, 30 minutes of grace avoids false alarms without leaving you blind for half a day.

Put the ping at the very end, after the last write to the destination system. A heartbeat sent at the start only proves the scheduler fired.

Logs you can actually use at 7am#

When an alert does fire, the log is what tells you why. Workflow tools tend to log either nothing useful or everything, including full API payloads.

The OWASP Logging Cheat Sheet is written for application security, but its basics apply directly. It says logs must record "when, where, who and what" for each event: the time in an international format, the application name and version, the identity involved, and the type, severity and description of the event. It recommends structured formats such as JSON, and sanitising event data (carriage returns, line feeds and delimiter characters) so that input can't forge log lines.

It's equally clear on what to keep out. Things that "should usually not be recorded directly in the logs" include access tokens, authentication passwords, database connection strings, encryption keys, payment card data and sensitive personal information. Automations are especially prone to breaking this rule, because the quickest debugging trick is to log the whole request, headers and all. If you've already worked through least-privilege API keys, a log file full of bearer tokens undoes a lot of that work.

For a multi-step workflow, the most useful single field is a run ID that appears on every log line and is passed along to each system the run touches. OpenTelemetry formalises this as context propagation: when one service calls another, "Service A includes a trace ID and a span ID as part of the context", and Service B uses them to create a new span in the same trace. You don't need a full tracing stack for a three-step Zapier-style flow. A UUID generated at the start, written into every log line and stored on the records you create, gets you most of the benefit. When the accounting system shows a strange invoice, you can search for its run ID and see exactly what produced it.

A minimal log line for each item might look like this:

{"time":"2026-10-24T06:02:11Z","workflow":"invoice-sync","version":"1.4.0",
 "run_id":"9f2c…","step":"create_invoice","customer_id":"C-1042",
 "result":"created","invoice_id":"INV-20931","duration_ms":840}

There's a customer ID but no customer name or email, an invoice ID but no amounts or line items, and nothing secret. That's enough to trace a problem and little enough that the log isn't itself a privacy risk.

Alerts people won't learn to ignore#

The quickest way to make monitoring useless is to alert on everything. The SRE chapter puts it bluntly: "When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts." Its rules for what should wake a person up are a good filter for a small team too. Every page should be actionable. Every page response should require intelligence, and "if a page merely merits a robotic response, it shouldn't be a page."

That last rule is useful for automations. If the fix for an alert is always "re-run the job", that's a sign the job should retry itself (with the care described in our post on retries and idempotency), and the alert should only fire when the retries are exhausted.

A practical split for a small business:

• Immediate alert (phone or chat ping): heartbeat missed, outcome check failed, or an error that stops the whole run. Someone needs to look today.
• Daily digest: individual items that failed and were skipped, runs that took noticeably longer than usual, volume outside the normal range. Someone should glance at it.
• Logs only: retries that succeeded, expected skips, routine detail.

The SRE book also says the rules that catch real incidents most often "should be as simple, predictable, and reliable as possible." Resist clever anomaly detection at the start. "No successful run in 26 hours" and "zero records processed on a weekday" are crude, but you'll understand them when they fire, and you'll trust them.

Make sure each alert says which workflow, what was expected, what was seen, and where to look. "Invoice sync: no heartbeat since 06:30 (expected by 06:30). Last run log: <link>" can be acted on from a phone. "Job failed" can't.

AI steps need their own checks#

When a workflow includes an LLM, a run can succeed technically and still be wrong. The model returns valid JSON with a plausible but incorrect category, and every check above passes.

The outcome-check idea still works, it just needs a statistical flavour. Track the distribution of the model's outputs over time (share of each category, average confidence if you have it, how often a human reviewer overrides it) and alert when it shifts sharply after a prompt or model change. Keep a small labelled sample and re-run it whenever something changes, as covered in how to evaluate an AI automation. Log the model name and prompt version on every item, so a jump in overrides can be tied to the change that caused it.

When TheAILAB hands over an automation, the monitoring is usually the part we spend the most time agreeing with the client: who gets which alert, and what they're expected to do about it. The code is the easy bit.

A starting checklist#

If you have an automation running today with no monitoring, add these in order. Each one takes an hour or two at most.

1. A heartbeat ping at the end of every scheduled run, with grace time based on real durations.
2. One outcome check per workflow that looks at the destination system, not the job.
3. A run ID on every log line, and a scan of your logs for tokens, passwords and personal data.
4. A "records processed" count per run, with an alert on zero when zero isn't normal.
5. A sorted alert list: what pages someone now, what goes in a daily digest, what stays in the logs.

Then break something on purpose. Pause the schedule, or point the job at an empty test account, and confirm the right person hears about it within the time you expected. A monitor you've never seen fire is a guess, not a safeguard.

Sources

1. Monitoring Distributed Systems (Site Reliability Engineering, chapter 6) — Google, Site Reliability Engineering book (checked 2026-10-24)
2. Logging Cheat Sheet — OWASP Cheat Sheet Series (checked 2026-10-24)
3. How to Monitor Cron Jobs with Healthchecks.io — Healthchecks.io documentation (checked 2026-10-24)
4. Configuring Checks — Healthchecks.io documentation (checked 2026-10-24)
5. Context propagation — OpenTelemetry documentation (checked 2026-10-24)