The Run Record, and the One Line Somebody Reads at Eight

By the end of this lesson you can produce two things at the end of every run: a record, in a form the next run can read, whose numbers can be checked against each other; and one sentence a person can read at eight in the morning and know whether anything needs doing.

A log and a run record are different things, and jobs that have only one of them have usually got the wrong one.

A log is prose for a person, written as the run goes: line after line, in order, useful when you already suspect something and are looking for where it went wrong. It is terrible for answering "did last night's run finish?", because answering that means reading to the end of a file and interpreting sentences.

A run record is data, written once, at the end: a small object with the run's identity, its outcome, its counts, and its position. It answers "did last night's run finish?" in one comparison, and — the part that matters most — the next run reads it. Where to resume from, whether a previous run is still going, how long it has been since anything succeeded: all of that is a lookup in the record and none of it is available from prose.

Then there is the third artefact, which is one sentence. Nobody reads a hundred-line log every morning, and nobody reads a JSON object every morning either. What gets read is a subject line. If that sentence carries the outcome and the numbers, most mornings end there.

The failure this lesson exists to prevent: a run that processed 118 of 120 records and reported success. Not because it lied — because nothing ever compared 118 with 120.

Six fields, in every record.

run_id        unique to this run, so two runs can never be confused
started_at    when it began
finished_at   when it ended, which is how "still running" is detected
status        completed | partial | failed | refused | inconsistent
counts        read, accepted, quarantined, written
reason        why it is not "completed", in words

status deserves more than two values. The first module made "refused" a normal outcome rather than an error; this module adds "partial", for a run that stopped at its budget with correct results that are not all of them. Collapsing those into failed makes a healthy refusal look like a bug, and collapsing them into completed makes a truncated run look finished.

Reconciliation is one line, and it is the point. Every record where records go in and results come out satisfies:

read == accepted + quarantined + rejected

Check it, in the code, at the end of the run. If it does not hold, the correct status is not completed — it is inconsistent, and the reason says which numbers disagreed. A job that cannot account for two of its 120 records has a defect, and the alternative to reporting it is that nobody finds out.

This is worth being blunt about. A summary that reports success without reconciling is not a summary of what happened; it is a summary of what the code believed happened, and those differ exactly when there is a bug.

The summary line. One sentence: what ran, what happened, the numbers, how long. Written from the record, never assembled separately — a summary computed independently of the record is a second implementation of the same claim, and the two will disagree eventually.

nightly-invoices completed: read 120, accepted 118, quarantined 2, in 41s
nightly-invoices inconsistent: read 120 != accepted 100 + quarantined 2

What never goes in. A run record is copied into tickets, pasted into messages, and kept for a long time. It carries counts, identifiers, and outcomes — not the records themselves, not anything that identifies a person, and not any value that authorises anything. The introductory course covers redaction at the point of use; the addition here is to hold a written list of field names that never reach a record, and drop them by name rather than by remembering.

Write it once, at the end, atomically. A record half-written by a run that was killed is worse than no record: the next run reads it and believes something false. Write to a temporary name and rename, the same technique the file module used.

NEVER_RECORD = {"submitted_by", "home_address", "phone"}


def summarise(record):
    """The one line somebody reads. Derived from the record, never assembled separately."""
    counts = record["counts"]
    if record["status"] == "inconsistent":
        return (
            f"{record['job']} inconsistent: read {counts['read']} != "
            f"accepted {counts['accepted']} + quarantined {counts['quarantined']}"
        )
    return (
        f"{record['job']} {record['status']}: read {counts['read']}, "
        f"accepted {counts['accepted']}, quarantined {counts['quarantined']}, "
        f"in {record['seconds']}s"
    )


def finish(job, counts, seconds, status="completed"):
    reconciles = counts["read"] == counts["accepted"] + counts["quarantined"]
    record = {
        "job": job,
        "status": status if reconciles else "inconsistent",
        "counts": counts,
        "seconds": seconds,
    }
    return record, summarise(record)


good, good_line = finish("nightly-invoices", {"read": 120, "accepted": 118, "quarantined": 2}, 41)
bad, bad_line = finish("weekly-suppliers", {"read": 50, "accepted": 40, "quarantined": 5}, 12)
print(good["status"])
print(good_line)
print(bad["status"])
print(bad_line)

finish computes the status rather than accepting it unconditionally: the caller says what it believes, and the reconciliation check can overrule it. That ordering is deliberate — the caller is the code that just ran, and it is the least reliable witness to its own success.

summarise reads only the record. Give it a record and it produces the line; there is no path by which the line and the record can disagree.

Predict all four lines.

completed
nightly-invoices completed: read 120, accepted 118, quarantined 2, in 41s
inconsistent
weekly-suppliers inconsistent: read 50 != accepted 40 + quarantined 5

Move the reconciliation check into summarise, so finish records the status it was given and only the summary line mentions the discrepancy.

An assistant was asked to "log what the job did so we can see if it worked". It produced this.

def process_all(records):
    logging.info("starting")
    accepted = 0
    for record in records:
        try:
            handle(record)
            accepted += 1
        except Exception as error:
            logging.warning(f"skipping {record}: {error}")
    logging.info(f"done, processed {accepted} records")
    return accepted

def process_all(job, run_id, records, handle):
    read = accepted = quarantined = 0
    reasons = []
    for record in records:
        read += 1
        try:
            handle(record)
            accepted += 1
        except RecordRejected as rejection:
            quarantined += 1
            reasons.append({"id": record["id"], "reason": str(rejection)})
    counts = {"read": read, "accepted": accepted, "quarantined": quarantined}
    status = "completed" if read == accepted + quarantined else "inconsistent"
    return {"job": job, "run_id": run_id, "status": status, "counts": counts, "quarantined": reasons}

Write finish, which turns a job's counts into a run record and its summary line. Two supplied runs are passed through it: one whose counts reconcile and one whose counts do not. The record must also drop any field named in NEVER_RECORD.

Loading this exercise…

Optional, and the transfer task for this lesson: work out what a week of records should be able to answer.

Write down five questions somebody might ask about a job — "when did it last succeed", "is one running now", "how many records has it quarantined this week", "is it getting slower", "did anything change on the day the numbers moved" — then check whether the six-field record above can answer each one. Add exactly the fields that are missing, and no more.

Sign in to track your progress on this exercise.

1. What can a run record do that a log cannot?
2. What does the reconciliation check compare, and what status does a failure produce?
3. Why derive the status from the counts rather than from what the calling code believes?
4. Why should the summary line be computed from the record rather than assembled as the run goes?

Sign in to track your progress on this exercise.

End every run with a record — identity, outcome, counts, reason — written once, atomically, in a form the next run can read; check that the counts account for every record read, and let that check overrule the status the code believed; derive the one summary line from the record itself; and keep contents and anything identifying a person out of both. The next lesson is what the next run does with that record: telling a failure worth retrying from one that will fail again, and picking up where the last run stopped.