Retries and idempotency: making automations safe to run twice

Your automation calls the accounting system to create an invoice. The request hangs for 30 seconds and the connection drops. Did the invoice get created? You don't know. If you retry, the customer might get two invoices. If you don't, they might get none.

That uncertainty is the normal state of any automation that talks to other systems over a network, and it gets worse once an LLM agent is the one deciding to call tools. The fix has two halves. Retry only the failures that are worth retrying, in a way that doesn't make an outage worse. And make the operations themselves safe to repeat, so a retry can't do the same work twice.

What idempotent actually means#

An operation is idempotent if doing it twice leaves things in the same state as doing it once. Setting a customer's status to "paid" is idempotent. Adding a fixed amount to their balance isn't.

HTTP gives you a head start. The IETF draft on the Idempotency-Key header restates the rule from RFC 9110: GET, HEAD, OPTIONS, PUT and DELETE are idempotent by definition, while POST and PATCH aren't. That's what the spec promises, though. Whether a particular API honours it is something you check in its docs.

Google's Cloud Storage retry strategy is a useful model because it sorts every operation into three buckets. Some are always idempotent, like listing or reading objects. Some are only idempotent if you send a precondition, such as ifGenerationMatch when writing an object, which makes the write fail rather than overwrite if someone else changed the object first. And some are never idempotent, like creating a new Pub/Sub notification. Google's page says plainly that "retrying requests that are not idempotent can lead to race conditions and other conflicts."

Do the same exercise for your own automation. List every step that writes something (creates a record, sends a message, moves money, posts to a channel) and put it in one of those three buckets. The "never" bucket is where duplicate invoices come from.

Which failures are worth retrying#

Retrying only helps when the failure is temporary. A bad API key, a malformed payload or a record that doesn't exist will fail the same way every time. Google's SRE book puts it bluntly in its chapter on cascading failures: "Don't retry permanent errors or malformed requests in a client, because neither will ever succeed."

The vendors broadly agree on what counts as temporary. Google Cloud Storage lists HTTP 408, 429 and 5xx responses, plus socket timeouts and dropped TCP connections, as worth retrying. The AWS SDKs reference guide treats timeouts, connection resets and throttling errors as retryable, while errors such as AccessDeniedException, ValidationException and ResourceNotFoundException "are returned to your code immediately."

A practical rule for most automations:

• Retry: timeouts, connection errors, 429 Too Many Requests, 500, 502, 503 and 504.
• Don't retry: 400, 401, 403, 404, 422, or anything the API documents as a validation error.
• Only retry a write if you've made it idempotent (more on that below).

That last point matters more than the status code. A timeout on a GET is harmless to retry. A timeout on a POST that creates a payment is the exact case where you can't tell whether it worked.

How to retry without making things worse#

The naive retry loop (try, fail, try again immediately, forever) is how a brief blip at a supplier turns into a sustained outage. Every client that failed hammers the service again at the same moment.

The SRE book's advice is short and worth following as written. "Always use randomized exponential backoff when scheduling retries." Limit retries per request: "Don't retry a given request indefinitely." Consider a retry budget for the whole process, and "if the retry budget is exceeded, don't retry; just fail the request."

Exponential backoff means each wait is longer than the last, for example 1, 2, 4 then 8 seconds. Randomising it (jitter) matters because without it, every client that failed at the same instant retries at the same instant too. Google's client libraries for Cloud Storage mostly start with a 1 second delay and double it each time, with a maximum wait that varies by language. AWS's SDKs default to three attempts in standard mode, which their guide spells out as "one initial request and up to two retries", and add a retry quota. That quota is a token bucket: each retry spends tokens, successes refill it, and when it's empty the SDK stops retrying and fails fast.

Here's a small Python version you can adapt. It retries only the errors you mark as temporary, caps the delay, and uses full jitter (a random wait between zero and the current backoff):

import random
import time

class TemporaryError(Exception):
    """Raise this for timeouts, 429s and 5xx responses."""

def call_with_retries(fn, max_attempts=4, base=1.0, cap=30.0):
    for attempt in range(max_attempts):
        try:
            return fn()
        except TemporaryError:
            if attempt == max_attempts - 1:
                raise  # out of attempts: let the caller see the failure
            backoff = min(cap, base * 2 ** attempt)
            time.sleep(random.uniform(0, backoff))

If the API sends a Retry-After header with a 429 or 503, respect it instead of your own schedule.

Retry in one place only#

This one is easy to miss. The HTTP library retries. The workflow tool or queue worker retries the step. The scheduler reruns the whole job. Each layer looks sensible on its own. Together they multiply.

The SRE book gives the arithmetic: if three layers each make up to four attempts, "a single user action may create 64 attempts (4^3) on the database." Pick the one layer that knows enough to decide (usually the step closest to the external call) and switch retries off, or right down, everywhere else.

Set timeouts on purpose#

A retry policy is only as good as the timeout that triggers it. Without a deliberate timeout, a hung request can tie up your worker long after it should have given up. The SRE book warns that "having deadlines several orders of magnitude longer than the mean request latency is usually bad." Look at how long the call normally takes, set a timeout comfortably above that, and make the timeout of the whole job longer than the sum of its retries.

Idempotency keys: making writes safe to repeat#

For writes that aren't naturally idempotent, the standard tool is an idempotency key. You generate a unique ID for the operation, send it with the request, and the server remembers what it did the first time. If the same key arrives again, it returns the original result instead of doing the work again.

Stripe's idempotent requests documentation is the clearest worked example. You send an Idempotency-Key header on a POST. Stripe suggests a V4 UUID or another random string with enough entropy, allows keys up to 255 characters, and asks you not to use sensitive data like email addresses as keys. For its v1 API, Stripe saves the status code and body of the first request once the endpoint has started executing, and returns that saved response for retries with the same key, "including 500 errors." It compares the parameters of a retry against the original and errors if they differ. Keys can be removed once they're at least 24 hours old, after which a reused key is treated as a new request.

Three details in there catch people out.

First, the key has to be created before the first attempt and stored with the job, so every retry sends the same value. Generating a fresh UUID inside the retry loop gives you no protection at all.

Second, keys expire. A retry that happens two days later, after a long outage or a manual rerun, may be treated as brand new. If your reruns can be that late, you need your own record of what's already been done (see the next section).

Third, the same key with a different payload is an error, not a silent update. The IETF draft takes the same line: reusing a key with a different payload should get a 422, and retrying while the original request is still being processed should get a 409. As of October 2026 that header is still an Internet-Draft, not a published RFC, so treat it as a description of good practice and check what each API you use actually supports.

Plenty of APIs don't support idempotency keys. For those, look for a natural key you can check first: an external reference field on the invoice, an order number, a message ID. Search for it before creating, or use an upsert endpoint if one exists. It isn't airtight if two workers race, but it closes most of the gap.

Deduplicating what comes in#

Retries happen on the inbound side too. A sender may resend an event it thinks failed, or someone may press "redeliver" after an outage. Your handler will see some events more than once, so it needs to recognise them.

GitHub's webhook best practices are a good template. Your server should respond with a 2XX within 10 seconds, or GitHub treats the delivery as failed, so the advice is to put the payload on a queue and process it in the background. Each delivery carries an X-GitHub-Delivery header, and a redelivery keeps the same value as the original. That gives you a dedupe key for free: record each delivery ID when you process it, and skip any you've already seen.

It also works the other way. GitHub's guide on failed deliveries says it "does not automatically redeliver failed deliveries." So you can't assume a sender will retry for you either. Read each provider's docs, because some resend aggressively and some never do.

A simple pattern that works for most small automations is a table of processed event IDs with a unique constraint. Insert the ID in the same database transaction as the work it triggers. If the insert fails on the constraint, you've seen the event before and can acknowledge it and stop.

Where AI agents make this harder#

An LLM agent that calls tools is a retry engine with opinions. If a tool call times out, the model may simply call it again, or call a slightly different tool that does the same thing. Don't rely on the prompt to prevent that.

Put the protection in the tool layer instead. Have the tool generate and store an idempotency key for each logical action (for example, one per "send invoice for order 1042"), check for an existing result before acting, and return that result if there is one. Then it doesn't matter how many times the model asks. At TheAILAB we treat any tool that sends, pays or deletes as needing this before an agent gets access to it.

A checklist before you switch on retries#

Go through your automation step by step and answer these:

1. Is this step a read, an idempotent write, or a write that's unsafe to repeat?
2. For unsafe writes, does the API accept an idempotency key? If not, is there a natural key you can check first?
3. Is the key created once, stored with the job, and reused on every attempt?
4. Which errors does this step retry, and which does it fail on straight away?
5. Is there exactly one layer doing the retrying?
6. Is there a timeout, a cap on attempts, and jittered backoff between them?
7. Does the inbound side record event IDs so repeats are skipped?

If you can answer all seven for every step that writes, a dropped connection becomes a non-event. Start with the steps that move money or contact customers, because those are the duplicates people notice.

Sources

1. Addressing Cascading Failures (Site Reliability Engineering, chapter 22) — Google, Site Reliability Engineering book (checked 2026-10-22)
2. Retry strategy — Google Cloud Storage documentation (checked 2026-10-22)
3. Retry behavior — AWS SDKs and Tools Reference Guide (checked 2026-10-22)
4. Idempotent requests — Stripe API reference (checked 2026-10-22)
5. The Idempotency-Key HTTP Header Field (draft-ietf-httpapi-idempotency-key-header) — IETF HTTPAPI Working Group, Internet-Draft (Jena, Dalal, Tishkin) (checked 2026-10-22)
6. Best practices for using webhooks — GitHub Docs (checked 2026-10-22)
7. Handling failed webhook deliveries — GitHub Docs (checked 2026-10-22)