Analysis That Is Correct and Still Wrong: Python Foundations

Outcome

By the end of this lesson you can read a data summary somebody else produced, including one an AI assistant wrote, and name the claims the underlying data does not support.

Why it matters

Every aggregate in the previous lesson can be computed perfectly and still support a false conclusion. The arithmetic is rarely the problem. What goes wrong is the sentence written underneath it.

This matters more, not less, with an assistant in the loop. Ask for "insights from this data" and you will receive fluent, confident prose containing claims the data cannot support, because generating a plausible sentence and verifying a claim are different activities and only one of them is happening. Reviewing that prose is a skill, and it is mostly a small number of recurring patterns.

Concept

Four patterns account for most of it.

The missing denominator. "Eighty per cent of matched opportunities are in Oregon" means one thing over four hundred records and nothing at all over five. Any proportion without its base is unreadable. Insist on "four of five", not "eighty per cent".

Survivor bias. The batch you are analysing is not the population; it is what your filters, your API query, and your parsing let through. If unparseable rows were dropped, and rows are unparseable because a particular agency publishes dates in an unusual format, then that agency is now absent from your conclusions about agencies. The rows you dropped are not random.

The date that sorts as text. Dates in YYYY-MM-DD form happen to compare correctly as strings, because the most significant part comes first. This works until one row arrives as 04/15/2026, at which point the comparison silently produces a wrong ordering rather than an error, because comparing two strings is always legal. Parse dates into real date values, then compare.

The uncontrolled comparison. "Opportunities are larger this quarter" can be a genuine trend, or it can be one very large award, a change in what the publisher includes, or a different number of days in the window. Ask what else changed before accepting that the thing you measured is the thing that moved.

A short review checklist you can apply to any summary, including your own:

What is the denominator of every number here?
What was excluded to produce this table, and how many rows was that?
Is any comparison between two periods, categories, or sources that are not otherwise alike?
Does any sentence claim a cause where the data only shows an association?
Would this conclusion survive one more row of data?

Read the code

Here is a summary an assistant produced from a batch, followed by the code that generated the numbers.

March summary. Oregon dominates our pipeline, accounting for 75% of matched opportunities. Average contract value has risen sharply to $612,500, indicating a shift towards larger engagements. All matched notices were posted in March.

from datetime import date

rows = [
    {"notice_id": "A-1", "state": "OR", "amount": 300000, "posted": "2026-03-04"},
    {"notice_id": "A-2", "state": "OR", "amount": 1800000, "posted": "2026-03-27"},
    {"notice_id": "A-3", "state": "OR", "amount": 150000, "posted": "2026-03-30"},
    {"notice_id": "A-4", "state": "WA", "amount": 200000, "posted": "2026-03-11"},
    {"notice_id": "A-5", "state": "OR", "amount": None, "posted": "04/02/2026"},
]

parsed = []
for row in rows:
    try:
        row["posted_date"] = date.fromisoformat(row["posted"])
    except ValueError:
        continue
    parsed.append(row)

oregon = [row for row in parsed if row["state"] == "OR"]
amounts = [row["amount"] for row in oregon if row["amount"] is not None]

print(len(oregon), "of", len(parsed))
print(sum(amounts) // len(amounts))

Every number in the summary is arithmetically correct and every sentence in it is misleading.

Three of four parsed rows are in Oregon, which is 75%, from a sample of four. The average of 300,000, 1,800,000, and 150,000 is 750,000, and the summary's 612,500 came from an earlier run over a different batch, which nobody checked. The claim that all matched notices were posted in March is true only because the one April row failed to parse and was silently discarded by that continue.

Predict the output

Predict the two printed lines.

Check your prediction

3 of 4
750000

Four rows survived parsing, not five, and the fifth was dropped without a word. The average is 750,000, not the 612,500 the summary asserts.

Notice that the code itself printed the denominator, 3 of 4, and the summary dropped it in favour of a percentage. The information was available and was discarded on the way to the prose.

Modify the code

Replace the continue in the except block with code that appends the row to an unparsed list, and print len(unparsed) at the end. Predict what changes about the summary you could honestly write.

What changes, and why

The output gains a line reading 1, and the third sentence of the summary becomes impossible to write.

"All matched notices were posted in March" was only ever an artefact of the discard. With the unparsed row visible, the honest sentence is "four of five notices were parsed; the fifth publishes its date in an unsupported format and has not been assessed."

The code change is three lines. The change in what the report can claim is the entire point of the module: the difference between an analysis that is trustworthy and one that is merely confident is usually a count of what was thrown away.

Debug the bug

An assistant was asked to count notices posted in a date window. It produced this and said it counts March postings.

rows = [
    {"notice_id": "A-1", "posted": "2026-03-04"},
    {"notice_id": "A-2", "posted": "2026-11-27"},
    {"notice_id": "A-3", "posted": "04/15/2026"},
]

in_window = [row for row in rows if "2026-03-01" <= row["posted"] <= "2026-03-31"]

print(len(in_window), [row["notice_id"] for row in in_window])

What's actually wrong

It prints 1 ['A-1'], which happens to be right, and the method is not.

The comparison is between strings. For YYYY-MM-DD values that works, because the most significant component comes first, so text order and date order agree. A-2 in November is correctly excluded.

A-3 is the problem. "04/15/2026" is compared as text against "2026-03-01", and "0" sorts before "2", so it falls below the start of the window and is excluded. The right answer for a March window is to exclude it, so the code appears correct.

Change the window to April, "2026-04-01" to "2026-04-30", and A-3 is still excluded, because "04/15/2026" is below every string beginning with "2026". A row that genuinely belongs in April is invisible in every month, forever, and nothing raises.

The repair is to parse before comparing, and to count what will not parse:

from datetime import date

in_window = []
unparsed = 0

for row in rows:
    try:
        posted = date.fromisoformat(row["posted"])
    except ValueError:
        unparsed += 1
        continue
    if date(2026, 3, 1) <= posted <= date(2026, 3, 31):
        in_window.append(row)

Now A-3 raises ValueError, is counted, and is reported. A row your program cannot understand should be visible, not merely absent.

Try it yourself

Four rows, one of them in a date format fromisoformat does not accept. Count the notices posted inside the window, and report how many could not be parsed at all.

Loading this exercise…

Practical challenge (optional)

Optional: take the misleading summary from Read the Code and rewrite it as three sentences that the data does support. State every denominator, name the excluded row, and either drop the claim about a shift towards larger engagements or say what would be needed to justify it. Keep both versions side by side; the contrast is the most useful artefact this module produces, and the capstone's retrospective asks for something very like it.

AI collaboration

Checkpoint

A report says: "Small-business set-asides made up 90% of qualifying notices last month, up from 60%, indicating a policy shift."

What single number would you ask for first?
What would you check about how "qualifying" was determined?
What could explain the change other than a policy shift?
Why is "up from 60%" harder to evaluate than it looks?

Answers

The denominator. Nine of ten and ninety of a hundred support very different conclusions, and the percentage hides which one this is.
Whether the filters, the query, or the parsing changed between the two periods. A rule that started excluding rows it used to accept produces exactly this shape of movement.
A different number of notices published, one large agency's batch arriving late, a change in how the publisher labels categories, or rows dropped because of an unparseable field.
The two percentages may have different denominators and may have been produced by different code. Comparing them assumes both were measured the same way, which is precisely the thing that usually changed.

Summary and next step

Every proportion needs its base, dropped rows are not random, dates must be parsed before they are compared, and a fluent sentence is not evidence. Run the five-question checklist over any summary before you circulate it, including your own. Module 9 gives the assistant somewhere to keep its records, and a query language that asks these questions directly.