Capstone Part Two: Interrupting It On Purpose

By the end of this lesson you can interrupt your own workflow at a place you choose, resume it, and demonstrate two things: that the resumed run reaches the same result as one that was never interrupted, and that a third run over the same input preserves the accepted result and digest while reporting zero new work. A durable job may still write a fresh run record or heartbeat.

Recovery code is the least-tested code in any automation, for a simple reason: making it run requires something to go wrong, and nothing goes wrong on demand.

So it gets written, reasoned about, and never executed. Then one night it executes for the first time — unattended, at whatever hour the failure happened, against real data — and whether it works is discovered afterwards, by looking at what it did.

The fix is to stop waiting. Make the failure point a parameter. A run that takes "stop after three records" as an argument can be interrupted anywhere you like, in a test that finishes instantly, as many times as you want. Recovery stops being a thing you believe in and becomes a thing you have watched happen.

Then there are two questions to actually ask, and only one of them is usually asked:

• Does the resumed run finish? Usually asked. Easy to satisfy, and not very interesting.
• Does the resumed run reach the same result as a run that was never interrupted? Rarely asked. This is the one that matters, and the answer is often no — because the resumed run skipped the setup, or double-counted the record that was in flight, or aggregated only what it processed itself rather than everything.

Choose the failure point. Add a parameter that says where to stop — after n records, before the write, during the third page. It costs one argument and it turns every recovery path into something a test can drive. In production it is never set, and there is no branch in the interesting code, only a check in the loop.

Five places worth interrupting, because they fail differently:

1. Before anything is read. The easiest case; nothing has happened.
2. Part-way through processing. First test a stop between records, as the worked fail_after function does. A crash inside a record’s side effect is a separate case: did that effect complete before the stop, and can it be replayed safely?
3. After processing, before the output is written. The work is done and invisible. A resumed run must not conclude that the work is done.
4. Half-way through writing the output. Use a temporary name and a checked atomic replacement on a supported filesystem, so a failed temporary write leaves the previous complete output intact rather than publishing a truncated file.
5. After the output, before the run record. The output exists, but the last durable run record does not include that completed work. The next run must repeat any work absent from that record and safely replace the derived output. In this design, output and run record are separate writes; changing their order alone cannot remove the gap. A shared transactional design would need a different storage contract.

This timeline is a conceptual test map. The worked function below stops only between records in memory, returns in-memory records, and performs no durable output or run-record write. An in-flight partial side effect needs another replay-safe test. The other four boundaries require a separate test setup, including a temporary output file for the mid-write case.

The two properties to check.

resume equivalence:  resumed processed IDs + digest = uninterrupted processed IDs + digest
rerun safety:        third run keeps IDs + digest, reports zero new work

The first says recovery is correct. The second says recovery is safe to trigger when it was not needed — which matters when the previous run might have finished. "Just run it again" is safe for the digest only when its output is derived cumulatively and replacement is idempotent; other effects require their own replay-safe design.

Compare the result, not just the status or totals. A resumed run that reports completed proves little. Compare the cumulative processed IDs and digest; equal aggregate counts and units can hide different records. The per-attempt new counts may differ between a resumed run and an uninterrupted one.

Aggregate over everything processed, not over what this run did. The most common recovery defect in a reporting job: the resumed run reports totals for the three records it handled rather than for all six, because the aggregation stage was handed this run's accepted list. The digest is a statement about the day, not about the attempt.

A rerun with nothing to do is a successful run. It should report completed, with zero new records, and write a record saying so. Reporting failed because there was nothing to do makes every recovery attempt look like a problem, and reporting nothing at all leaves the staleness check without a heartbeat.

ORDERS = {
    "ORD-1": {"units": 4, "status": "confirmed"},
    "ORD-2": {"units": 6, "status": "confirmed"},
    "ORD-3": {"units": 9, "status": "cancelled"},
    "ORD-4": {"units": 2, "status": "confirmed"},
}


def digest_for(orders, processed):
    """The digest is a statement about everything processed, not about this run."""
    accepted = [oid for oid in sorted(processed) if orders[oid]["status"] == "confirmed"]
    return [f"orders: {len(accepted)}", f"units: {sum(orders[oid]['units'] for oid in accepted)}"]


def run(orders, previous, fail_after=None):
    processed = list(previous["processed"]) if previous else []
    done = set(processed)
    new = 0
    for order_id in sorted(orders):
        if order_id in done:
            continue
        if fail_after is not None and new >= fail_after:
            return {"status": "partial", "processed": processed, "new": new,
                    "digest": digest_for(orders, processed)}
        processed.append(order_id)
        done.add(order_id)
        new += 1
    return {"status": "completed", "processed": processed, "new": new,
            "digest": digest_for(orders, processed)}


interrupted = run(ORDERS, None, fail_after=2)
resumed = run(ORDERS, interrupted)
straight = run(ORDERS, None)
print("interrupted:", interrupted["status"], interrupted["new"])
print("resumed:", resumed["status"], resumed["new"])
print("equivalent:", resumed["processed"] == straight["processed"]
      and resumed["digest"] == straight["digest"])
print("digest:", "; ".join(resumed["digest"]))

digest_for takes processed — everything, cumulative — rather than the ids this run handled. That single argument is the difference between a digest about the day and a digest about the attempt.

fail_after is checked before appending, so a run that stops after two has processed exactly two, with no third record half-done.

The equivalence check compares both cumulative processed IDs and digest across independently produced runs. Comparing totals alone could miss a different set of records with the same count and units.

Predict all four lines.

interrupted: partial 2
resumed: completed 2
equivalent: True
digest: orders: 3; units: 12

Change digest_for(orders, processed) to digest_for(orders, processed[len(previous["processed"]) if previous else 0:]), so the digest covers only what this run handled.

interrupted: partial 2
resumed: completed 2
equivalent: False
digest: orders: 1; units: 2

Somebody added recovery to the digest and asked an assistant to review it. The assistant said it looked correct.

def run(orders, state_path):
    state = json.loads(state_path.read_text()) if state_path.exists() else {"processed": []}
    processed = state["processed"]
    accepted = []
    for order_id in sorted(orders):
        if order_id in processed:
            continue
        order = orders[order_id]
        if order["status"] == "confirmed":
            accepted.append(order_id)
        processed.append(order_id)
    write_digest(accepted)
    state_path.write_text(json.dumps({"processed": processed}))
    return len(accepted)

def run(orders, previous, write_digest, fail_after=None):
    processed = list(previous["processed"]) if previous else []
    done, new = set(processed), 0
    for order_id in sorted(orders):
        if order_id in done:
            continue
        if fail_after is not None and new >= fail_after:
            return {"status": "partial", "processed": processed, "new": new}
        processed.append(order_id)
        done.add(order_id)
        new += 1
    write_digest(digest_for(orders, processed))      # every processed order, not just this run's
    return {"status": "completed", "processed": processed, "new": new}

Write run, then let the starter interrupt it, resume it, and run it a third time. The resumed pair must return the same cumulative processed IDs and digest as an uninterrupted run, and the third run must keep both unchanged with zero new work.

Loading this exercise…

Optional, and the transfer task for this lesson: interrupt it in the other four places.

The exercise interrupts part-way through processing. Write a test for each of the remaining four points from the concept list — before reading, after processing but before the write, half-way through the write, and after the write but before the record — and for each one, state in a sentence what the next run must do.

Sign in to track your progress on this exercise.

1. Why make the failure point a parameter rather than testing recovery with a real failure?
2. What are the two properties a recovery test should check?
3. Why must the digest be computed from everything processed rather than from what this run handled?
4. Why is the gap between writing the output and writing the run record unavoidable, and what makes it survivable?

Sign in to track your progress on this exercise.

Make the failure point an argument. The worked code tests the mid-loop stop; extend the test setup to the other four conceptual positions and check two things: that the resumed accepted result and digest equal the uninterrupted ones, and that a further run has zero new work without changing that result. Aggregate over everything processed rather than over one attempt, and treat a rerun with nothing to do as a completed run that writes its record. The last lesson turns the working project into one somebody else can operate: the layout, the settings, the rubric it is measured against, and the runbook that answers the questions nobody is awake to ask.