Reading Records That Disagree With What You Were Told

By the end of this lesson you can take a batch of records arriving at the boundary of an unattended job, check each one against the shape the job was promised, coerce a field that arrived as a plausible other type, and produce either a typed record or a specific reason - never a dictionary that is partly one and partly a guess.

Every automation in this course eventually reads a batch of records from somewhere it does not control: an export a colleague generates, a report a supplier emails, a feed a service publishes. csv.DictReader and json.loads will hand back something for almost anything you point them at, and "something" is exactly the problem. Neither one knows what your job was promised, so neither one can tell you when what arrived does not match it.

The failures are not exotic. A field that was a plain integer for eight months arrives as a quoted string because somebody re-exported it from a spreadsheet. A column gets renamed upstream and every record in the batch is suddenly missing a field your job depends on. A new field gets added, and code that assumes a fixed set of keys either ignores it or trips over it, depending on luck. None of this is malformed CSV or invalid JSON - the parser is satisfied every time - so your code is the first thing in the chain with any chance of noticing.

The instinct to reach for try/except and move on is close to right and wrong in one specific way: catching the exception is correct, and catching it in order to skip the record without saying why is not. A record that fails to parse and vanishes silently is indistinguishable, from the outside, from a record that was never sent. Six months later, "we processed everything" and "we quietly dropped four rows a month" look like the same report.

A schema is the shape the job was promised, written down as a small mapping from field name to declared type: {"id": "str", "quantity": "int", "site": "str"}. It is not a validation library and does not need to be one - a dictionary and a few isinstance checks say everything a boundary read needs to say.

Three questions, asked in order, before any value is trusted:

1. Does this record carry a field the schema does not name? An extra field is often the first sign of a header that changed - somebody added a column upstream, and code that ignores unknown keys will silently carry it nowhere, which is a quieter failure than an error.
2. Does this record lack a field the schema requires? A missing field is the mirror case, and it is usually the same event: a column got renamed rather than added, so the new name shows up as an extra field and the old name shows up as a missing one, on every record in the batch.
3. Does every present field's value coerce to its declared type? A JSON field is sometimes a string and sometimes a number depending on which system produced it this month - a spreadsheet export quotes everything, an API often does not. Coercion means accepting the value the type can honestly become: the string "12" becomes the integer 12. It does not mean accepting anything a program can be made not to crash on.

Coerce, or report - never guess. A field that will not coerce is not evidence about what the value should have been. int("soon") cannot become a quantity, and the honest response is a reason naming the field, the record, and what was found - not a 0, not a None standing in as though it were data, and not a silent skip. A guessed value is worse than a rejected one, because a rejected record shows up in a count and a guessed one does not.

A repeated reason is a different problem than a single one. One record rejected for "missing field quantity" is a bad row. Every record in the batch rejected for the same reason is not forty bad rows - it is one broken export, and the fix is a conversation with whoever produces it, not a loop that tries harder. The per-record reason is what makes that pattern visible; a batch that reports one bare "invalid input" count could not tell you which case you were looking at.

Parse into a typed record or a rejection reason - nothing in between. The function that does this returns one or the other, never a dictionary with some fields converted and others left as raw strings, and never raises past its own boundary. What happens to the rejected ones is the next lesson's question.

A schema, a small batch of raw ticket exports, and the coercion that checks them:

import json

RAW = json.loads(
    '[{"ticket": "T1", "minutes": "45"}, {"ticket": "T2", "minutes": 12}, '
    '{"ticket": "T3", "minutes": "soon"}]'
)

SCHEMA = {"ticket": "str", "minutes": "int"}


def coerce_record(raw, schema):
    """Coerce one raw record to schema, or return (None, reason)."""
    extra = set(raw) - set(schema)
    if extra:
        name = sorted(extra)[0]
        return None, f'unexpected field "{name}"'
    missing = set(schema) - set(raw)
    if missing:
        name = sorted(missing)[0]
        return None, f'missing field "{name}"'
    record = {}
    for name, kind in schema.items():
        value = raw[name]
        if kind == "int":
            if isinstance(value, int):
                record[name] = value
            elif isinstance(value, str):
                try:
                    record[name] = int(value.strip())
                except ValueError:
                    return None, f'field "{name}" is not a valid int: {value!r}'
            else:
                return None, f'field "{name}" is not a valid int: {value!r}'
        else:
            record[name] = value
    return record, None


def parse_records(raw_records, schema):
    records, rejections = [], []
    for raw in raw_records:
        record, reason = coerce_record(raw, schema)
        if reason is None:
            records.append(record)
        else:
            rejections.append((raw["ticket"], reason))
    return records, rejections


records, rejections = parse_records(RAW, SCHEMA)
print("read:", len(RAW))
print("parsed:", len(records))
for record in records:
    print(f"  {record['ticket']}: {record['minutes']} minutes")
for ticket, reason in rejections:
    print(f"  {ticket} rejected: {reason}")

set(raw) - set(schema) and set(schema) - set(raw) are the two shape checks, and they run before any field is inspected for its type - a record with the wrong fields does not get far enough to have the wrong values.

isinstance(value, int) is checked before isinstance(value, str), so T2's minutes of 12 is accepted as-is and never gets a pointless round trip through int(str(12)). T3's minutes of "soon" reaches the try, fails to convert, and the except turns that failure into a returned reason rather than an exception the caller has to catch.

parse_records never raises. Every raw record produces either an appended typed record or an appended (ticket, reason) pair, and the loop always reaches the next one.

Predict every line.

read: 3
parsed: 2
  T1: 45 minutes
  T2: 12 minutes
  T3 rejected: field "minutes" is not a valid int: 'soon'

Inside the except ValueError: branch, replace return None, f'field "{name}" is not a valid int: {value!r}' with record[name] = 0, so a value that will not convert becomes a zero instead of a rejection.

read: 3
parsed: 3
  T1: 45 minutes
  T2: 12 minutes
  T3: 0 minutes

An assistant was asked to "load these ticket exports and give me typed records". It produced this.

def load_tickets(raw_records):
    tickets = []
    for raw in raw_records:
        tickets.append({"ticket": raw["ticket"], "minutes": int(raw["minutes"])})
    return tickets

def parse_tickets(raw_records, schema):
    """Coerce every record against schema. A bad record never stops the batch."""
    records, rejections = [], []
    for raw in raw_records:
        record, reason = coerce_record(raw, schema)
        if reason is None:
            records.append(record)
        else:
            rejections.append({"ticket": raw.get("ticket", "?"), "reason": reason})
    return records, rejections

Five raw records are supplied as JSON in RAW_JSON, checked against the schema id (str), quantity (int), site (str). Write coerce_record and parse_records: coerce what can honestly be coerced, and return a specific reason for what cannot.

Loading this exercise…

Optional, and the transfer task for this lesson: decide what a schema does not yet say.

JSON has two number types where this lesson's schema has one: 45 parses to a Python int, 45.0 parses to a Python float, and both could plausibly mean "forty-five" in a field declared "int". Extend coerce_record's int branch to decide, in a comment, what should happen to a float value - and then decide separately what should happen to 45.5, which cannot become an integer without losing information nobody asked to lose.

Sign in to track your progress on this exercise.

1. Why does an extra field usually get checked before a missing one, and what does it mean if a whole batch reports the same pair of shape problems on every record?
2. Why must a type-coercion failure return a reason rather than letting int()'s exception propagate?
3. quantity is 7 in one record and "7" in another. What should coerce_record do with each, and why should both produce the same typed value?
4. A rewritten function defaults a field that will not coerce to 0 instead of rejecting the record. Why is that worse than rejecting it?

Sign in to track your progress on this exercise.

A schema is the shape a job was promised: field names and declared types, checked before any value is trusted. Reject a record with an unexpected field or a missing one before coercing anything, coerce a field to its declared type only when the value can honestly become it, and return a specific reason - never a guessed value - for the one that cannot. A repeated reason across a batch is one broken export wearing many rows. The next lesson takes the records this one accepts, and the ones it rejects, and answers the harder question: what does a job do with a record that parses cleanly but breaks a rule the parser could never have checked?