Capstone Phase 3 and 4: Normalisation, Validation, and Explainable Rules

By the end of this lesson you can normalise a batch into validated records with errors a reviewer can act on, and apply matching rules read from configuration so that every verdict arrives with the reasons that produced it.

These two phases are what make the assistant worth running. Everything before them moves data around; everything after them stores, reports, or exposes the result.

They are also where the project earns the word "explainable". A shortlist without reasons is a list somebody has to check by hand, which is the work the tool was supposed to remove. A shortlist whose exclusions carry reasons, and whose inclusions can be traced through the rule checks, gives a reviewer evidence to inspect rather than a bare list. Spot-checks help find defects; they do not prove every verdict is correct.

The distinction Module 2 introduced becomes load-bearing here. "Does not match" and "cannot be judged" are different outcomes with different consequences: one is a decision, the other is a data-quality item somebody should fix. Collapsing them is the single most damaging thing this project could do.

Normalisation runs per field, in a fixed order, and every step is one of two kinds.

Repairs are mechanical changes made under a documented field policy to preserve intended comparison meaning. They may discard original formatting: strip surrounding whitespace, fold case on comparison fields, remove thousands separators before converting, and map documented missing-value spellings to one representation.

Reports are everything requiring a judgement: an amount that will not parse, a date in an unrecognised format, a missing identifier, an agency code that matches nothing, a response date before its posted date, a duplicate identifier within one batch. Each produces an error naming the record and the field, and the record does not enter the clean set.

Start with the accounting check clean count plus error count equals input count. Also test that each input record enters exactly one bucket: equal counts alone could hide one dropped record and one duplicated record.

Matching is driven from configuration, not from branches. A rule names the field it reads, the comparison, the value, and the reason text it contributes when it fails:

{
  "id": "minimum-value",
  "field": "estimated_value",
  "operator": "at-least",
  "value": 100000,
  "failureReason": "estimated value is below the configured minimum"
}

The engine loops over the rules. That gives three properties worth having: adding a rule with a supported operator needs no code change, comparison and reason text stay together in configuration, and the whole rule set can be swapped in a test. Their proximity reduces drift; tests and review still have to check that the reason actually describes the comparison.

Three verdicts, not two:

• match: every rule passed.
• no match: at least one rule failed, and the reasons list says which.
• incomplete: a rule's field is absent or null, so the record cannot be judged at all.

Order matters within a record too. Check for absence before comparing, exactly as in Module 2, so a missing field never produces a confident mismatch.

One design decision to make deliberately and write down: does one missing field make the whole record incomplete, or only the rules that read it? Either is defensible. The stricter reading is easier to explain and is what the worked example uses.

MARKERS_TO_STRIP = ",$"


def normalise(raw):
    """Return (record, error). Exactly one is None."""
    notice_id = (raw.get("noticeId") or "").strip()
    if not notice_id:
        return None, "record has no noticeId and cannot be identified"

    amount_text = raw.get("estimatedValue")
    if amount_text is None:
        return None, f"{notice_id}: estimated value is not published"
    cleaned = str(amount_text)
    for marker in MARKERS_TO_STRIP:
        cleaned = cleaned.replace(marker, "")
    try:
        amount = int(cleaned)
    except ValueError:
        return None, f"{notice_id}: estimated value is not a number: {amount_text!r}"

    set_aside = raw.get("setAside")
    return {
        "notice_id": notice_id,
        "amount": amount,
        "set_aside": set_aside.strip().upper() if set_aside else None,
        "state": (raw.get("placeOfPerformance") or {}).get("state"),
    }, None


BATCH = [
    {"noticeId": "SYN-0001", "estimatedValue": "310000",
     "setAside": "  total small business  ", "placeOfPerformance": {"state": "OR"}},
    {"noticeId": "SYN-9002", "estimatedValue": "to be determined",
     "setAside": "Total Small Business", "placeOfPerformance": {"state": "OR"}},
    {"noticeId": "SYN-0007", "estimatedValue": "96,500", "setAside": "Total Small Business"},
]

clean, errors = [], []
for raw in BATCH:
    record, error = normalise(raw)
    (errors if error else clean).append(error if error else record)

print(len(clean), len(errors), len(clean) + len(errors) == len(BATCH))
print(clean[0])
print(clean[1]["amount"], clean[1]["state"])
print(errors[0])

The mechanical repairs run before the semantic checks; an unparseable amount returns an error with the notice id, so a reviewer knows which record to open. This short listing demonstrates only some of the field checks required for the full capstone.

set_aside.strip().upper() if set_aside else None folds an empty string and an explicit null to the same None, which is the "one representation for nothing" rule from Module 6. Note it also folds an empty string, which is deliberate here and is a decision the data dictionary must record.

(raw.get("placeOfPerformance") or {}).get("state") is Module 6's nested-default idiom, so a record with no location yields None for the state rather than raising.

The len(clean) + len(errors) == len(BATCH) check is the accounting result printed here. The one-append-per-input loop supports it, but a separate test of record identity is needed to prove each input landed in exactly one bucket.

Predict all four printed lines.

2 1 True
{'notice_id': 'SYN-0001', 'amount': 310000, 'set_aside': 'TOTAL SMALL BUSINESS', 'state': 'OR'}
96500 None
SYN-9002: estimated value is not a number: 'to be determined'

Change the amount handling so an unparseable value becomes 0 and the record joins the clean set. Predict the four lines, then say what the shortlist would report.

3 0 True
{'notice_id': 'SYN-0001', 'amount': 310000, 'set_aside': 'TOTAL SMALL BUSINESS', 'state': 'OR'}
0 OR

An assistant was asked to add the matching rules to the pipeline. It produced this.

def matches(record, rules):
    for rule in rules:
        actual = record.get(rule["field"])
        if rule["operator"] == "at-least" and actual < rule["value"]:
            return False
        if rule["operator"] == "equals" and actual != rule["value"]:
            return False
    return True


shortlist = [r for r in clean if matches(r, RULES)]
print(len(shortlist))

def evaluate(record, rules):
    """Return (verdict, reasons)."""
    for rule in rules:
        if rule["operator"] not in ("at-least", "equals", "in"):
            raise ValueError(f"unknown rule operator: {rule['operator']!r}")
    reasons = []
    for rule in rules:
        operator = rule["operator"]
        actual = record.get(rule["field"])
        if actual is None:
            return "incomplete", []
        if operator == "at-least":
            passed = actual >= rule["value"]
        elif operator == "equals":
            passed = actual == rule["value"]
        else:
            passed = actual in rule["value"]
        if not passed:
            reasons.append(rule["failureReason"])
    return ("no match", reasons) if reasons else ("match", [])

Write the rule engine. Three rules from configuration, three records, and all three verdicts.

Loading this exercise…

Optional, and it completes phases 3 and 4. Run your normalisation over the real fixtures, including capstone/fixtures/opportunities-malformed.json, and confirm all six of its defects appear as errors naming the offending field: the non-ISO date, the unparseable amount, the unrecognised agency code, the response date preceding its posted date, the record with no notice id, and the duplicate identifier. Then apply capstone/fixtures/matching-rules.json to the twelve-record batch and confirm exactly five records match. Both counts are acceptance criteria for the capstone.

Sign in to track your progress on this exercise.

1. Which normalisation steps are safe to apply silently, and which must be reported?
2. Which count and identity checks establish that each input entered exactly one bucket?
3. Why must a rule engine produce three verdicts rather than two?
4. Why should an unknown rule operator raise rather than be skipped?

Sign in to track your progress on this exercise.

Repair the mechanical, report the semantic, assert that clean plus errors equals input, drive the rules from configuration, and produce three verdicts with reasons drawn from the rule that failed. Next: storing the result, reporting it, and covering the whole thing with the four categories of test.