Pagination stopping rules that prevent missing records

A crawler that stops making requests is not necessarily finished. It may have reached the last page, repeated a cursor, loaded a blank intermediate screen or exhausted its own safety budget. Your collector should report which outcome occurred instead of calling every stopped run complete.

Identify the pagination model

Inspect the public page or permitted interface before implementing a loop. A numbered page parameter, a next-page link, an opaque cursor and a load-more button need different continuation logic. Do not assume that incrementing a number will reproduce a flow that is actually driven by a cursor.

For cursor-based collection, preserve the returned token exactly. Keep filters, sorting and intended market consistent across the sequence. If the interface documents an expiration or a snapshot boundary, account for it when resuming. A cursor is not evidence that the underlying dataset stays unchanged during the entire run.

Require a positive completion signal

Choose the end condition from the interface's behavior: a documented terminal cursor value, a disabled next control or another explicit indicator. An empty extraction is a warning until you know why it is empty. A changed selector, loading failure or unexpected page can all produce zero records without marking the true end of the collection.

Keep completion distinct from a safety stop. A page budget or time budget should bound work, but reaching it means the job needs review or continuation. Record that outcome as incomplete. Do not publish a partial dataset with the same completion label as a verified end of sequence.

Stop repeated cursors

The helper below is for a fixture interface whose continuation token is either a string or null. Call it after successfully reading and validating a page. The null rule is specific to this example, not a universal API convention.

def continuation(next_cursor, seen, pages_read, max_pages=100):
    if max_pages < 1:
        raise ValueError("max_pages must be positive")
    if next_cursor is None:
        return "complete"
    if not isinstance(next_cursor, str):
        raise ValueError("unexpected cursor type")
    if next_cursor in seen:
        raise ValueError("repeated cursor: incomplete")
    if pages_read >= max_pages:
        raise ValueError("page budget reached: incomplete")
    seen.add(next_cursor)
    return "continue"

Keep the seen set for the active collection sequence. An empty string remains a string here; if your interface uses it as an end marker, implement that contract explicitly. This function does not fetch pages, retry failed requests or validate extracted records.

Wait for the data you need

For browser-driven pagination, clicking a next button is only one step. Wait for an observable change that proves the next batch is ready, such as a changed page marker or the expected response followed by updated item content. Avoid treating a fixed delay as proof of readiness.

A load event does not establish that a dynamic list is complete. Network-idle waiting is also a poor universal completion rule: pages can keep background connections active or render useful data independently of that event. Use locators and explicit checks tied to the list you are collecting.

If you follow a next-page URL, resolve relative links against the current page and validate the resulting destination. A URL-joining helper can accept an absolute link that changes the host. URL parsing alone is not a security policy; keep the fetcher's destination and redirect restrictions in place.

Reconcile records and state

Track pages read, distinct item identifiers, repeated items and the final stop reason. If a next-page action returns the same item sequence, investigate rather than looping indefinitely. Repeated items can be legitimate across overlapping pages, so define a deduplication rule appropriate to your dataset instead of deleting every repetition blindly.

Test a one-page result, a normal multi-page sequence, a repeated token and a failed intermediate request. Also test an empty extraction without an explicit end marker. The expected outcome for each fixture should say whether the collection is complete, incomplete or awaiting recovery.

Store the next continuation state only after the associated records have been acknowledged by your output destination. Coordinate that decision with your export checkpoints. If the sequence changes while you collect it, report that limitation rather than claiming a perfectly consistent snapshot.

Use the setup guide for your client to establish routing, then keep pagination, record validation and retry decisions explicit in the collector.