Retries, Circuit Breakers, and Budgets: Production Python Systems

Outcome

Calculate a bounded retry schedule and explain when a circuit should open.

Why it matters

A dependency returning a transient failure can tempt every caller to retry immediately. That multiplies calls precisely when the dependency may be least able to serve them. A finite schedule limits the waits a caller requests; a real client also needs a deadline for the entire operation and, when failures persist across calls, a separate circuit-breaker policy.

Concept

Retries consume the same capacity as original calls. Retry only failures the dependency contract classifies as transient; do not retry permanent validation or authorization errors. Bound the number of attempts, use backoff with jitter in a real client, and account for waits and request time under one wall-clock deadline. A retry of a write also needs idempotent semantics or an idempotency mechanism.

The function below calculates proposed exponential waits only. Its budget caps the sum of requested waits, not elapsed wall-clock time; it makes no request, sleeps for no time, checks no response and tracks no deadline. Assume base and budget are non-negative and attempts is a positive integer. A production boundary should validate those inputs and use a clock and per-request timeout.

A circuit breaker is a separate stateful policy across calls: closed allows requests while it measures failures; open rejects new calls for a configured cooldown after a failure threshold; half-open admits a limited probe, then closes on recovery or reopens on failure. Its thresholds and probe policy belong to the dependency contract. The delay calculator below does not implement a breaker.

Read the code

def retry_delays(base, attempts, budget):
    delays = []
    elapsed = 0
    for attempt in range(attempts - 1):
        delay = base * (2 ** attempt)
        if elapsed + delay > budget:
            break
        delays.append(delay)
        elapsed += delay
    return delays

print(retry_delays(1, 5, 6))

The first attempt needs no wait, so five allowed attempts produce at most four candidate delays. The loop adds a delay only if the accumulated requested wait stays within budget; it stops before adding a delay that would exceed it. It does not report whether any request succeeded or failed. If this were an actual client, classify the failure before scheduling a retry and account for request runtime under a separate overall deadline.

Predict the output

Predict the exact output before running the example.

Check your prediction

It prints [1, 2]. Those requested waits total 3 seconds. The next candidate is 4 seconds, which would take the requested-wait total to 7, above the 6-second wait budget. No request or sleep was performed, so these are calculated delays rather than observed elapsed time.

A wait budget is one boundary, not the whole operation

Candidate waits for five attempts

First attempt: No wait
Before later attempts: 1 s, 2 s, 4 s, 8 s

These are proposed delays from base × 2^attempt; no request timing was observed.

Six-second requested-wait cap

Accepted waits: 1 s + 2 s = 3 s
Next candidate: 4 s would make 7 s
Returned schedule: [1, 2]

The function stops before adding the 4-second wait.

Separate wall-clock deadline

Request runtime: Not measured here
Actual elapsed time: Not bounded here
Client responsibility: Clock, per-request timeout and overall deadline

A finite wait schedule alone cannot bound a slow or hung request.

Separate circuit-breaker policy

Closed: Allow calls and count failures
Open: Reject calls during cooldown
Half-open: Allow limited recovery probe

State changes need a defined threshold and probe policy; this function has no breaker.

The example calculates requested waits only. A real client separately bounds elapsed time and classifies retries; a breaker decides whether new calls should reach an unhealthy dependency.

Modify the code

Change the example to retry_delays(1, 5, 2). Predict its return value and explain whether that value proves a real operation would finish within two seconds.

Review the change

It returns [1]: the next 2-second wait would raise the requested-wait total from 1 to 3, beyond the wait budget of 2. That does not prove a real operation finishes in two seconds; network calls and processing also consume time, and this function measures neither. A client must enforce its own overall deadline as well as its attempt and wait policy.

Debug the bug

A teammate wraps a write request in except Exception: retry and uses this delay list as proof that the operation has a deadline. Identify two distinct defects.

Diagnose the retry policy

Exception includes permanent failures, so retrying all of them can repeat invalid work. A write may also repeat a side effect unless the operation is idempotent or uses a verified idempotency key. Separately, retry_delays budgets only requested waits; even its finite list cannot bound request runtime. A correct client classifies failures, caps attempts, enforces a wall-clock deadline with per-request timeouts, and handles writes safely. The circuit breaker remains a separate policy across calls.

Try it yourself

Complete delays(base, attempts, budget) so it returns the exponential waits before later attempts while their cumulative requested wait fits the budget. For the supplied inputs, the printed list should contain three waits. This exercise computes a schedule; it does not send requests or enforce a wall-clock deadline.

Loading this exercise…

Practical challenge (optional)

Design retry policy for reads and writes, including idempotency, jitter, deadline, circuit state, and the metric that pages an operator.

AI collaboration

Checkpoint

Why are there at most attempts - 1 delays?
Does the sample budget cap requested waits or total elapsed time? What does a real deadline need to include?
Which failures should not be retried, and what else must be true before retrying a write?
What different overload problem does a circuit breaker address beyond a retry budget?

Answers

The first attempt happens immediately; a delay exists only before a later attempt.
The sample caps only the sum of requested waits. A real deadline includes request execution, waits and local processing, enforced with a clock and per-request timeouts.
Permanent validation or authorization failures should not be retried. A write also needs idempotent semantics or a verified idempotency mechanism.
It temporarily stops new calls to a dependency known to be unhealthy. Closed, open and half-open decisions are stateful across calls; this sample does not implement them.

Summary and next step

The code computes exponential waits under a requested-wait cap. It does not enforce an elapsed-time deadline, classify errors, send requests or implement a circuit breaker. In a client, combine the schedule with bounded attempts, a real deadline, failure classification and a separate breaker policy. Next, apply those boundaries within a wider system.