Failing Halfway, and Starting From There

By the end of this lesson you can decide which failures are worth trying again, resume a run from the point the last one recorded rather than from the beginning, and make sure that a single record which can never be processed is set aside instead of stopping everything behind it.

Retrying is not a virtue. It is the right response to one specific kind of failure and the wrong response to every other kind.

A connection that timed out might succeed in thirty seconds. A record whose date field says not applicable will fail identically in thirty seconds, in thirty minutes, and every night for the next four years. Retrying the first is patience; retrying the second is a job that never finishes and a service that gets hammered on behalf of a record nobody will ever fix.

Resumption has a mirror-image failure. A run that processes 900 of 1000 records and dies has done real work, and a job that starts from zero tomorrow does that work again — which is merely wasteful if processing is idempotent, and is duplicate invoices if it is not. Meanwhile the run that starts from record 901 without checking anything is assuming the first 900 were finished rather than merely attempted.

And there is the case that combines them. One record in the middle cannot be processed, ever. The job fails on it, resumes at it tomorrow, fails on it again, and every record behind it waits forever. The queue is not slow; it is stopped, and the summary says "failed" every morning until somebody reads far enough to find out why.

Transient or permanent. The question is not how bad the failure is, but whether an identical attempt later could succeed.

| Failure | Kind | Because | | ---------------------------------- | ---------------------- | ---------------------------------------------------------- | | Connection timed out | Transient | The service may answer next time | | Service returned 503 | Transient | It is saying so itself | | Rate limited | Transient | With the published wait, yes | | Record is missing a required field | Permanent | The record will not change by being retried | | Output folder does not exist | Permanent for this run | Retrying will not create it; a person or a setup step will | | Authorisation was refused | Permanent for this run | Retrying an unauthorised request just repeats it |

Classify by exception type, not by matching text in a message. Messages change between library versions and translations; types do not. Where a service reports the distinction in a status code, use the code.

The default for anything unclassified is do not retry. A retry loop that treats every unknown failure as transient turns a bug in your own code into an infinite loop, which is how a TypeError becomes a job that runs all night.

Resume from the record, not from a counter. The previous run's record holds what was completed, and that is what the next run subtracts from the batch. Two properties matter:

• It records completion, not attempts. A record that was started and not finished must not appear, or the resumed run skips it.
• Resumption is only safe when processing is idempotent. It is always possible for a record to be completed and the run to die before the record says so, in which case the next run processes it again. That is unavoidable, it is fine, and it is fine only because the first module built the property that makes it fine.

Bound the attempts on one record. Keep a per-record attempt count. After a small number of failed attempts — two or three — move the record to quarantine with the reason, and carry on with the rest. The queue keeps moving, the record is preserved for a person, and the run reports both facts.

This is the difference between a job that says "failed" every morning and one that says "completed, 1 record quarantined: INV-88 has no supplier id". The second is a message somebody can act on in a minute.

Report what actually happened. A resumed run's record carries what it skipped, what it processed, and what it quarantined. "Processed 100 records" from a run that skipped 900 and processed 100 is true and useless; the skipped count is what makes it interpretable.

class RecordRejected(Exception):
    """This record cannot be processed, now or later."""


class ServiceUnavailable(Exception):
    """This attempt failed; a later identical one might not."""


TRANSIENT = (ServiceUnavailable, TimeoutError)


def process_batch(previous, batch, handle, max_attempts=2):
    """Process what the previous run did not, quarantining what cannot be processed."""
    done = list(previous["processed"]) if previous else []
    skipped = sum(1 for item in batch if item in done)
    quarantined = []
    for item in batch:
        if item in done:
            continue
        for attempt in range(1, max_attempts + 1):
            try:
                handle(item)
                done.append(item)
                break
            except TRANSIENT:
                if attempt == max_attempts:
                    return {"status": "partial", "processed": done, "skipped": skipped,
                            "quarantined": quarantined, "reason": f"gave up on {item}"}
            except RecordRejected as rejection:
                quarantined.append({"id": item, "reason": str(rejection)})
                break
    return {"status": "completed", "processed": done, "skipped": skipped,
            "quarantined": quarantined, "reason": ""}


def handle(item):
    if item == "C":
        raise RecordRejected("no supplier id")
    if item == "E":
        raise ServiceUnavailable("upstream down")


record = process_batch(None, ["A", "B", "C", "D", "E"], handle)
print("status:", record["status"])
print("processed:", ",".join(record["processed"]))
print("quarantined:", ",".join(entry["id"] for entry in record["quarantined"]))
print("reason:", record["reason"])

The two exception classes carry the classification, so the decision is made where the failure is raised — by the code that knows what went wrong — rather than by a caller inspecting a message.

break after a successful handle leaves the retry loop; break after a RecordRejected leaves it too, because there is nothing to retry. Only the transient branch loops.

A transient failure that exhausts its attempts returns immediately with partial, because the service is unwell and the remaining records would each spend their attempts discovering the same thing. A permanent failure does not return: it is quarantined and the loop carries on, which is what keeps one bad record from stopping the queue.

Predict all four lines.

status: partial
processed: A,B,D
quarantined: C
reason: gave up on E

Change TRANSIENT to (ServiceUnavailable, TimeoutError, RecordRejected), so every failure is treated as retryable.

status: partial
processed: A,B
quarantined:
reason: gave up on C

An assistant was asked to "make the job resume where it left off if it crashes". It produced this.

def run(batch, state_path):
    last_index = int(state_path.read_text()) if state_path.exists() else 0
    for index, item in enumerate(batch):
        if index < last_index:
            continue
        state_path.write_text(str(index))
        for attempt in range(10):
            try:
                handle(item)
                break
            except Exception:
                time.sleep(1)
    print("done")

def run(previous, batch, handle, max_attempts=2):
    done = set(previous["processed"]) if previous else set()
    processed, quarantined = list(previous["processed"]) if previous else [], []
    for item in batch:
        if item["id"] in done:
            continue
        outcome = attempt_record(item, handle, max_attempts)   # raises nothing
        if outcome["status"] == "processed":
            processed.append(item["id"])                       # after the work, not before
        elif outcome["status"] == "rejected":
            quarantined.append({"id": item["id"], "reason": outcome["reason"]})
        else:
            return partial_record(processed, quarantined, f"gave up on {item['id']}")
    return complete_record(processed, quarantined)

Write run, which processes a batch, skipping what a previous run recorded as done and stopping when a supplied failure point is reached. It is called twice: once with no previous run and an early failure, then again with the record the first call produced.

Loading this exercise…

Optional, and the transfer task for this lesson: decide what happens on the fourth failure of one record.

The concept section says to quarantine a record after two or three failed attempts. Attempts across runs, though, need somewhere to live: the count has to survive the run that failed. Design where that count is stored, and write down what the job does when a record reaches its limit — and what it does if the same record reappears in tomorrow's batch.

Sign in to track your progress on this exercise.

1. What single question separates a transient failure from a permanent one?
2. Why classify by exception type rather than by matching the error message?
3. Why resume from record identities rather than from an index or offset?
4. Why must a permanently failing record be quarantined rather than retried indefinitely?

Sign in to track your progress on this exercise.

Retry only what a later identical attempt could survive, decide that by exception type, and default anything unclassified to not retrying. Resume from the identities the last run recorded as completed, recorded after the work rather than before it, and rely on idempotency for the record that was finished just before the lights went out. Bound the attempts on any single record so a permanent failure is quarantined by name and the queue keeps moving. The last lesson of this module is about the run that never started: choosing a schedule, stopping two runs from overlapping, and alerting on silence.