Recorded Responses and the Marker You Move Too Early

By the end of this lesson you can drive a job entirely from recorded service responses, with no network involved, and you can write the part of an incremental fetch that decides when it is safe to remember where you got to.

There are two problems here, and the second one is the one that loses data.

The first: a job can be tested against a live service, but live tests are slow or variable and rarely produce a particular failure on demand. Recorded or written fixtures let a test repeat exact responses without network access. Refresh real recordings when the service changes; this course uses only hand-written fixtures. Other integration tests can still check the live boundary.

The second problem is the reason incremental fetching exists and the reason it goes wrong. A job that asks for everything, every time, is simple and eventually too slow: an hourly job re-reading four years of records is doing four years of work per hour. So it remembers a marker — the timestamp or token it got to last time — and asks only for what came after.

Now the failure. The run fetches records after 2026-03-01, gets four, writes the marker forward to 2026-03-03, and then fails while writing the report. Tomorrow it asks for records after 2026-03-03. Those four records are gone: nothing will ever ask for them again, no error was raised about them, and the only evidence is a gap that nobody is looking for.

For this one-batch example, the marker is a claim that the fetched records have been handled. Move it before that is true and the next request can skip work.

A transport is the seam. One function, injected: it takes the request and returns the response, and it is the only part of the job that would touch a network. Everything above it — deciding what to ask for, interpreting what comes back, handling an error — is ordinary code that runs anywhere.

def make_recorded_transport(recorded):
    """Replays stored answers. Raises when asked for something not recorded."""
    def transport(request):
        if request not in recorded:
            raise LookupError(f"no recorded response for {request!r}")
        return recorded[request]
    return transport

The raise matters more than it looks. A recorded transport that returns an empty result for an unrecorded request turns "this test is exercising a path nobody recorded" into "this path returns nothing", and the test passes while proving nothing. Missing recordings must be loud.

Recording, and what to take out. A recording is the response a service really gave, saved as a file. It beats a hand-written stub because it has the shape the service actually produces — the field that is sometimes absent, the number that arrives as a string, the extra key nobody documented.

Before storing one, remove anything that would authorise a request or identify a person: whatever your provider's documentation says its authentication mechanism is, plus names, addresses, and identifiers belonging to real people. Store the recording next to the code, and note where and when it came from, so somebody can tell whether it still resembles the service.

This course records nothing, because it makes no requests. Every response in it is written by hand as a deterministic fixture, and no lesson, exercise, or project here requires an account of any kind.

The marker. Sometimes a timestamp, sometimes an opaque token the service hands back. Two rules:

1. Store what the service gave you, not what you calculated. If the response says "next: abc123", store abc123. A marker you derived — the newest timestamp you saw, plus one second — can skip a record that arrives late with an earlier timestamp, which is common and normal.
2. Return the next marker only after this batch has been published. Not after the fetch or parse. The worked function returns an in-memory marker; a caller that persists it also needs an atomic state update and a safe retry policy.

Repeating is safer than skipping when the publisher is idempotent. A failed batch that leaves the marker alone can fetch the same records again. If publishing partly succeeded before it failed, repeating can duplicate effects unless the publisher makes retries safe. The earlier idempotency work is what gives this ordering a recovery path.

RECORDED = {
    "2026-05-01": {"records": ["a", "b"], "next": "2026-05-02"},
    "2026-05-02": {"records": ["c"], "next": "2026-05-04"},
    "2026-05-04": {"records": [], "next": "2026-05-04"},
}


def make_recorded_transport(recorded):
    def transport(since):
        if since not in recorded:
            raise LookupError(f"no recorded response for since={since}")
        return recorded[since]
    return transport


def run_once(transport, marker, publish):
    """Fetch since marker, publish, and only then move the marker."""
    try:
        response = transport(marker)
        records = response["records"]
        next_marker = response["next"]
        publish(records)
    except Exception as error:
        return 0, marker, f"failed: {type(error).__name__}"
    return len(records), next_marker, "ok"


published = []
transport = make_recorded_transport(RECORDED)
marker = "2026-05-01"
for attempt in range(3):
    count, marker, status = run_once(transport, marker, published.extend)
    print(f"run {attempt + 1}: {count} new, marker {marker}, {status}")
print("published:", ",".join(published))

publish is injected for the same reason the transport is: the interesting failure in this lesson happens after the fetch and before the marker moves, and that failure cannot be demonstrated unless the publishing step is something a test controls.

The function reads the required response["next"] key inside the try but returns it only after publish succeeds. The caller then replaces its local marker with that returned value. The final response returns the marker unchanged along with no records, which is how this scripted service says "nothing new".

The except returns the old in-memory marker for a failed fetch, malformed response, or publisher exception. This function does not write a durable checkpoint. A publisher that has already produced some effects can still raise, so replay also requires idempotent publishing.

Predict every line.

run 1: 2 new, marker 2026-05-02, ok
run 2: 1 new, marker 2026-05-04, ok
run 3: 0 new, marker 2026-05-04, ok
published: a,b,c

Make the publish step fail on the second run, by passing a function that raises:

def failing_publish(records):
    raise IOError("report volume unavailable")

Use it for run 2 only, and keep published.extend for runs 1 and 3.

run 1: 2 new, marker 2026-05-02, ok
run 2: 0 new, marker 2026-05-02, failed: OSError
run 3: 1 new, marker 2026-05-04, ok
published: a,b,c

An assistant was asked for "an incremental sync that remembers where it got to". It produced this.

def sync(transport, state_file):
    marker = state_file.read_text() if state_file.exists() else "1970-01-01"
    state_file.write_text(datetime.date.today().isoformat())
    try:
        response = transport(marker)
    except Exception:
        return []
    return response.get("records", [])

def sync(transport, state, publish):
    marker = state.read(default="1970-01-01")
    response = transport(marker)          # a failure here raises, and the marker stays
    records = response["records"]         # a missing key raises, rather than reporting zero
    next_marker = response["next"]        # read required key before publishing
    publish(records)                      # may partially succeed before raising
    state.replace_atomically(next_marker) # required state contract: old or new, never partial
    return len(records), next_marker

Write run_once, which fetches from a recorded transport, publishes, and returns the marker to store. It is exercised against two supplied recordings: one that works for three consecutive runs, and one whose service fails. In these fixtures, the publisher refuses before writing, so failure publishes nothing; the returned marker must remain old. A different publisher may have partial effects before it raises.

Loading this exercise…

Optional, and the transfer task for this lesson: find the gap a marker cannot close.

Suppose the service orders records by the time it received them, and a record can be recorded late — it arrives today carrying yesterday's timestamp. Work out what a marker based on that timestamp does with it, then write down two ways to stop losing it.

Sign in to track your progress on this exercise.

1. Why must a recorded transport raise when asked for a request nobody recorded?
2. Why store the marker the service returned rather than one you computed?
3. Where in a run is it safe to advance the marker?
4. A run fails and repeats the same twenty records tomorrow. What property makes that harmless, and where did this course build it?

Sign in to track your progress on this exercise.

Inject the transport, drive it from recorded responses, and make a missing recording raise. Return the service's marker only after this batch publishes; persist that marker atomically. If a failure leaves the old marker in place, a retry may repeat work, so publishing must be idempotent or transactional. Next: the same transport, asked for many pages, under a budget that retries and pagination have to share.