API pagination for automations: stop missing and duplicating records

Your nightly job copies customers from one system into another. It runs green every night. Then someone notices that a customer created last Tuesday never arrived, and another one shows up twice. Nothing errored. The job asked for "all customers", got back what looked like all of them, and moved on.

A lot of the time the culprit is pagination. Almost every API that returns a list hands it back one page at a time, and the loop that walks through those pages is usually a few lines someone wrote in a hurry. Those few lines decide whether your sync sees every record exactly once. This post covers how the common pagination styles work, the specific mistakes that cause gaps and duplicates, and a small Python loop that guards against them.

Three styles you'll meet#

There are really three shapes, and a single vendor can use more than one.

Offset or page-number pagination asks for "items 200 to 299" or "page 3". It's easy to understand and easy to get wrong when the data changes underneath you.

Cursor pagination asks for "the next items after this one". Stripe's list endpoints work this way: per the Stripe pagination guide, you take the ID of the last object on the page and pass it as starting_after on the next call, and you keep going while has_more is true. Objects come back newest first.

Page-token pagination gives you an opaque string to send back. Google's API design rules in AIP-158 say page tokens "must be opaque (but URL-safe) strings, and must not be user-parseable". Slack's next_cursor is used the same way: you send back exactly what you were given.

The GitHub row comes from GitHub's pagination docs, which also note that a given endpoint might use page, before/after or since parameters, so check the reference page for the endpoint you're calling. The Slack row is from Slack's pagination page.

Mistake 1: treating a short page as the last page#

The most common loop looks like this: ask for 100 items, and if fewer than 100 come back, stop. It feels obviously right. It isn't.

Slack's docs say it directly: "It's possible to receive fewer results than your specified limit, even when there are additional results to retrieve." AIP-158 says the same thing for Google-style APIs, that the API "may return fewer results than the number requested", and that an empty next_page_token is the only way the server signals the end. Neither page explains when it happens, and you shouldn't need to know. A short page is allowed, so your loop has to survive one.

The fix is to stop on the API's own end signal, whatever it is: no next link, has_more false, an empty cursor. Never on page size.

A related trap is asking for a page size the API won't honour. GitHub's docs note that for most endpoints the maximum per_page is 100, and "If you specify a value greater than the maximum, GitHub does not return an error." AIP-158 tells servers to quietly "coerce down" to their maximum. So if you ask for 500, get 100 back, and use "fewer than I asked for" as your end test, you'll stop after the first page every time.

Mistake 2: offset pagination over data that's changing#

Offsets describe a position, and positions move. Say you're reading a list sorted newest first, 100 at a time. You read items 1 to 100. Before you fetch the next page, five new records are created. They go to the top, so everything shifts down five places. When you ask for items 101 to 200, the first five of those are records you already saw on page one. You get duplicates.

Run it the other way, with deletions, and you get gaps. If five records from page one are deleted between requests, everything shifts up and the five records that used to sit at positions 101 to 105 are now at 96 to 100. You've already read that page, so you never see them. That's the dangerous direction, because a gap leaves nothing behind to notice.

It gets worse if the sort order isn't unique. The PostgreSQL documentation on LIMIT and OFFSET warns that without an ORDER BY that "constrains the result rows into a unique order", you'll get "an unpredictable subset of the query's rows", and that different LIMIT/OFFSET values "will give inconsistent results" unless the ordering is predictable. Sorting by created_at alone isn't unique if two records share a timestamp. The same page drawn twice can differ.

Cursor pagination largely avoids the shifting problem because "after object X" doesn't depend on how many rows sit above X. If an API offers both, use the cursor. If it only offers offsets, de-duplicate by ID on your side and run a periodic reconciliation (more on that below) to catch gaps.

Mistake 3: building the next request yourself#

When an API gives you the next page's URL or token, use it as given. GitHub's docs show the link header carrying full URLs for next, prev, first and last, and say "You can use the URLs from the link header to request another page of results." Constructing ?page=N+1 yourself works until the endpoint turns out to use after instead.

Don't decode tokens either. AIP-158 makes them opaque on purpose, so the server can change what's inside without telling you.

And keep every other parameter the same while you page. AIP-158 says all other parameters "must match the call that provided the page token", and that an API should return an INVALID_ARGUMENT error if they don't. If your filter includes something like "updated since now minus one hour", compute that timestamp once at the start of the run and reuse it for every page.

Mistake 4: saving a cursor for tomorrow's run#

It's tempting to store the last cursor and resume from it next time. Cursors aren't built for that. Slack's docs say cursors "expire and are meant to be used within a reasonable amount of time", and warn against keeping them for hours or days; an old one gets you an invalid_cursor error. AIP-158 allows servers to expire tokens and suggests three days as "a good rule of thumb", which still won't help a weekly job.

For incremental syncs, store your own high-water mark instead: the newest created or updated time you've fully processed, or the newest ID. Next run, filter on that value (with a small overlap to cover clock skew and records that landed mid-run), page through the result from scratch, and let your ID-based de-duplication absorb the overlap. If the API supports it, filtering on an "updated" timestamp catches edits as well as new records.

Mistake 5: syncing from a search endpoint#

Many APIs have both a list endpoint and a search endpoint, and search is handy because it takes a query. But search indexes usually lag behind the primary data. Stripe's customer search reference says: "Don't use search in read-after-write flows where strict consistency is necessary." It adds that data is normally searchable in under a minute, but during outages propagation "can be up to an hour behind."

That's fine for a dashboard. It's a problem for a job that creates a record and then searches to confirm it's there, or a sync that runs every five minutes and assumes search shows everything up to now. Use list endpoints for anything that has to be complete, and treat search as a convenience.

A pagination loop with guard rails#

Here's a cursor-style loop in Python. It's written against a generic "fetch one page" function so you can adapt it to any of the styles above. It stops only on the API's end signal, refuses to loop forever if the server hands back the same cursor twice, caps the number of pages so a bug can't run up your rate limit, and drops duplicate IDs.

def fetch_all(fetch_page, max_pages=10_000):
    """fetch_page(cursor) -> (items, next_cursor). next_cursor is None or "" at the end."""
    seen_ids = set()
    seen_cursors = set()
    cursor = None
    for _ in range(max_pages):
        items, next_cursor = fetch_page(cursor)
        for item in items:
            if item["id"] in seen_ids:
                continue
            seen_ids.add(item["id"])
            yield item
        if not next_cursor:          # the API's own end signal
            return
        if next_cursor in seen_cursors:
            raise RuntimeError(f"Cursor repeated: {next_cursor!r}")
        seen_cursors.add(next_cursor)
        cursor = next_cursor
    raise RuntimeError(f"Stopped after {max_pages} pages without reaching the end")

For Stripe, fetch_page would call the list endpoint with starting_after=cursor and return the last item's ID as the next cursor when has_more is true, or None when it's false. (Stripe's client libraries also have auto-pagination helpers that do this for you; use them when you can.) For GitHub, the "cursor" is simply the next URL from the link header. For Slack, it's response_metadata.next_cursor.

We tested the loop against a fake API that returns uneven page sizes, including an empty page in the middle and an overlapping record between pages. It returned all eight records once each and only stopped at the real end. A version that stopped on the first short page quit after page two and missed half the records.

The loop doesn't handle retries or rate limits. Wrap fetch_page in your usual retry logic, and make sure a retried request reuses the same cursor rather than advancing.

Check the count, not just the exit code#

Pagination bugs don't crash. They produce a slightly wrong number of records and a green tick. So add one cheap check to any sync that matters: compare what you received with an independent count. Some APIs return a total; otherwise, compare against yesterday's count plus new records, or run a weekly full sync and diff the IDs against what the incremental runs collected. When we build integrations at TheAILAB, we add that reconciliation step by default, because it's the only part of the job that notices a missing record.

If you're the one designing an API, give clients a cursor or opaque token, sort on something unique, and add pagination to list endpoints from day one. AIP-158 notes that "Adding pagination to an existing RPC is a backwards-incompatible change", because old clients will read the first page and assume it's everything.

Sources

1. AIP-158: Pagination — Google API Improvement Proposals (checked 2026-10-26)
2. Using pagination in the REST API — GitHub Docs (checked 2026-10-26)
3. How pagination works — Stripe Documentation (checked 2026-10-26)
4. Search customers (API reference) — Stripe Documentation (checked 2026-10-26)
5. Pagination — Slack Developer Docs (checked 2026-10-26)
6. 7.6. LIMIT and OFFSET — PostgreSQL Global Development Group, PostgreSQL 18 documentation (checked 2026-10-26)