Paging Under One Budget That Retries Also Spend

By the end of this lesson you can collect every page of a result set while respecting a wait the service asks for, with pages and retries drawing on one shared request budget, and return a stop reason that says whether the collection is complete or was cut short.

Pagination and retries are each simple. Together their separate limits multiply, and the product is easy to miss.

A job pages through results with a cap of twenty pages. It allows up to five attempts per page. Both numbers look careful. The run could make up to a hundred requests if each of twenty pages succeeds only on its fifth attempt. Those separate caps do not limit the total, and that worst case would add load to a service already rate-limiting requests.

The second problem is the wait. Some rate-limit responses publish how long to wait before trying again. Ignoring that and retrying immediately can trigger another limit response; sleeping a fixed five seconds when the service said sixty is still too early. When the response publishes a duration, use it before an allowed retry.

A retry path tied directly to real sleep makes tests slow unless the delay is replaced or mocked. Inject a wait function and a test can record requested delays instantly and assert them exactly.

One budget, spent by everything. The run has a number of requests it may make. A page costs one. A retry costs one. A retry of a page you have already tried twice costs one more. There is no separate retry allowance, because a separate allowance is what lets the two limits multiply.

The budget is checked before each attempt and spent whether the attempt succeeds or fails. The caller counts every attempt it makes, even when no records come back. The service's own rate-limit accounting may differ; that does not increase this job's allowance.

Honour the wait the service publishes. When a response says how long to wait and another attempt is permitted, use that duration before retrying. Stop without waiting after the final allowed attempt. Two details:

• The published value can be larger than you expect. Sixty seconds against a five-minute deadline is fine; six hundred seconds is the service telling you this run is not going to happen, and the right response is to stop with that reason rather than to sleep through the deadline.
• When no wait is published, back off on your own schedule — doubling from a small base — and stop after a bounded number of attempts. Doubling with no bound is not a bound.

Inject the clock. Two functions, passed in: one that reports the current time and one that waits. In production they are the real ones. In a test they are a counter and a recorder, so a run that "waits" ninety seconds finishes instantly and the test can assert that it waited ninety.

def make_recording_clock():
    """A clock that records waits instead of performing them."""
    state = {"now": 0.0, "waits": []}
    def now():
        return state["now"]
    def wait(seconds):
        state["waits"].append(seconds)
        state["now"] += seconds
    return now, wait, state

Recording the waits rather than discarding them is what makes "it backed off correctly" an assertion instead of an opinion. The worked sample below injects only wait; it does not implement this clock's deadline check or a fallback schedule for a response without a published wait.

Return a reason for each handled stop. This scripted page loop has three ordinary endings and they are not interchangeable:

• complete — the service said there are no more pages.
• budget spent — the run stopped early and the result is partial.
• gave up — a page failed more times than allowed.

Only the first means the records are all of them. Transport exceptions are outside this scripted example and still propagate; a production client needs an explicit exception or error-result policy. A function that returns a list with no reason gives its caller no way to tell a complete result from a truncated one, and the caller may write it into a report as though it were complete. That report will be wrong in the direction that matters: it will under-report, quietly, on the busiest day.

Reset the attempt counter after a success. A long run that hits one transient failure on page 3 and another on page 17 has not failed twice in a row. Counting attempts per page rather than per run is what stops an otherwise healthy long run from being killed by unrelated blips.

SCRIPT = {
    1: [{"wait": 2}, {"records": ["a", "b"], "next": 2}],
    2: [{"records": ["c"], "next": None}],
}


def make_transport(script):
    """Returns the next scripted response for a page, in order."""
    remaining = {page: list(responses) for page, responses in script.items()}
    def transport(page):
        if not remaining.get(page):
            raise LookupError(f"no scripted response left for page {page}")
        return remaining[page].pop(0)
    return transport


def collect(transport, budget, wait, max_attempts=3):
    """Page through everything, or stop with a reason. Returns (records, reason, used, waited)."""
    records, page, used, waited, attempts = [], 1, 0, 0, 0
    while True:
        if used >= budget:
            return records, "budget spent", used, waited
        used += 1
        response = transport(page)
        if "wait" in response:
            attempts += 1
            if attempts >= max_attempts:
                return records, "gave up", used, waited
            if used >= budget:
                return records, "budget spent", used, waited
            wait(response["wait"])
            waited += response["wait"]
            continue
        records.extend(response["records"])
        if response["next"] is None:
            return records, "complete", used, waited
        page = response["next"]
        attempts = 0


records, reason, used, waited = collect(make_transport(SCRIPT), budget=10, wait=lambda s: None)
print("records:", ",".join(records))
print("reason:", reason)
print("used:", used)
print("waited:", waited)

used += 1 sits before the request rather than after it, so the caller spends its allowance on every attempt. If this scripted transport raises, the exception propagates rather than returning the counter or a reason; production code needs an error policy. A budget that only counts successes is not a budget.

attempts = 0 after a successful page is the per-page reset. continue retries the same page, because page was not reassigned. If a wait response used the final request allowance, the function stops without waiting for a retry it cannot make.

wait is a parameter, and the caller here passes a function that does nothing, so this simulation finishes instantly while showing the duration passed to that function. A production wait adapter must actually delay.

Predict every line.

records: a,b,c
reason: complete
used: 3
waited: 2

Call it with budget=2.

records: a,b
reason: budget spent
used: 2
waited: 2

An assistant was asked to "get all the pages, and handle rate limiting". It produced this.

import time

def get_all(client, max_pages=20):
    records = []
    for page in range(1, max_pages + 1):
        for attempt in range(5):
            response = client.get(page)
            if response.status_code == 429:
                time.sleep(2**attempt)
                continue
            break
        records.extend(response.json()["records"])
        if not response.json()["has_more"]:
            break
    return records

def get_all(client, budget, wait, max_attempts=3):
    records, page, used, attempts = [], 1, 0, 0
    while True:
        if used >= budget:
            return records, "budget spent"
        used += 1
        response = client.get(page)
        if response.rate_limited:
            attempts += 1
            if attempts >= max_attempts:
                return records, "gave up after repeated rate limiting"
            if used >= budget:
                return records, "budget spent"
            wait(response.retry_after)
            continue
        records.extend(response.records)
        if not response.has_more:
            return records, "complete"
        page = response.next_page
        attempts = 0

Write collect, which pages through a scripted service. Pages and retries share one budget, a published wait is passed to the injected wait function before an allowed retry, and each handled stop returns a reason. It is run three times against the same script: with enough budget, with two requests, and with one. The one-request run must not wait when it cannot retry.

Loading this exercise…

Optional, and the transfer task for this lesson: decide when a published wait is too long to honour.

Add a deadline_at to collect and a rule: if the wait the service asks for would take the run past its deadline, stop immediately with a reason saying so, rather than waiting and then failing. Then decide what the job should do next time it runs.

Sign in to track your progress on this exercise.

1. Why should pages and retries share one budget rather than having their own limits?
2. Why does a rate-limited attempt that returns no data still cost a request?
3. Why must a page loop return a reason as well as the records?
4. Why is a fixed doubling backoff worse than the wait the service publishes?

Sign in to track your progress on this exercise.

Give the whole run one request budget, spend it before every attempt including the ones that fail, and honour the wait the service publishes rather than one you calculated. Bound the attempts per page and reset the count after a success. Inject the clock so the waiting path is tested rather than skipped. Return a reason from every handled stop, and specify how a production client reports transport exceptions, because a partial result that looks complete is worse than no result. The next module is about the run itself: what it writes down, how it recovers from stopping halfway, and how anyone notices when it stops running at all.