Looping Over Records: Filtering, Counting, and the First Shortlist: Python Foundations

Outcome

By the end of this lesson you can read a for loop and say what each name holds after every pass, and you can turn a batch of opportunity records into a shortlist that survives a record with a missing field.

Why it matters

This is the lesson where the review assistant becomes worth running. One record judged by hand is a demonstration; forty records filtered into six is a tool.

Loops are also where two specific, frequent bugs live: counting one too many or one too few, and changing a collection while walking through it. Both produce plausible output, so neither announces itself.

Concept

for item in collection: runs its indented block once per item, binding item to each in turn. It stops when the collection is exhausted; there is no counter to get wrong.

When you do need positions, range(n) produces 0 up to but not including n, which is the same half-open rule as slicing. range(1, 4) gives 1, 2, 3. Almost always, looping over the collection directly is clearer than looping over indices and looking each item up.

The accumulator pattern is the backbone of nearly every useful loop. Create the result before the loop, add to it inside, read it after:

matches = []
for record in batch:
    if qualifies:
        matches.append(record["notice_id"])
print(len(matches))

Creating the accumulator inside the loop is the single most common structural mistake; it resets on every pass, so the result only ever reflects the last item.

Two traps worth naming:

Modifying a list while looping over it. Removing items during iteration makes the loop skip entries, because the positions shift underneath it. Build a new list instead.
Reading a field that is not always there. One record without the key raises KeyError and stops the whole run partway through, leaving a half-built result. Use get with a default for optional fields.

A comprehension is a compact way to write a filtering loop: [r["notice_id"] for r in batch if r["estimated_value"] > 100000]. It is exactly the accumulator pattern with the boilerplate removed. Use it when the loop body is a single expression and a filter; use a full loop when the body does more than one thing, because a comprehension that needs a comment is a loop in disguise.

Read the code

batch = [
    {"notice_id": "A-1", "estimated_value": 120000},
    {"notice_id": "A-2", "estimated_value": 45000},
    {"notice_id": "A-3", "estimated_value": 300000},
]

shortlist = []
total_value = 0

for record in batch:
    total_value += record["estimated_value"]
    if record["estimated_value"] >= 100000:
        shortlist.append(record["notice_id"])

print(shortlist)
print(len(shortlist), "of", len(batch))
print(total_value)

Trace it pass by pass:

Before the loop, shortlist is empty and total_value is 0.
Pass one: total_value becomes 120000; 120000 >= 100000 so "A-1" is appended.
Pass two: total_value becomes 165000; 45000 fails the test, nothing is appended.
Pass three: total_value becomes 465000; "A-3" is appended.

Both accumulators live outside the loop, which is why they survive it. total_value counts every record while shortlist collects only some, and the two coexist happily in one pass over the data.

Inspect the accumulator trace

One batch, two accumulators

Initialise once before the loop: total_value = 0; shortlist = []

Each row starts with the previous row's After state. A record qualifies when estimated_value >= 100000.

State before and after each record
Record	total_value	shortlist
A-1 Value 120000 Rule: True	Before `0` After `120000`	Before `[]` After `['A-1']`
A-2 Value 45000 Rule: False	Before `120000` After `165000`	Before `['A-1']` After `['A-1']`
A-3 Value 300000 Rule: True	Before `165000` After `465000`	Before `['A-1']` After `['A-1', 'A-3']`

The position of each update matters

Before the if: total_value adds every record's value, including A-2.
Inside the if: shortlist appends an ID only when the record qualifies.
Before the loop: Initialising once lets both accumulators carry state into the next pass.

Initialising an accumulator again on every pass would discard its earlier work.

Read down each accumulator column to follow the same state through all three records. A-2 changes the total but leaves the shortlist unchanged. These are the original worked loop's states, before the later indentation change or in-place removal example.

Predict the output

Predict the three printed lines.

Check your prediction

['A-1', 'A-3']
2 of 3
465000

The second line uses print with several arguments, which joins them with single spaces. The third is the sum of all three values, not only the shortlisted ones, because that accumulator was updated before the filter, not inside it. Where a line sits relative to the if is what decides which records it sees.

Modify the code

Move the line total_value += record["estimated_value"] so it sits inside the if, indented one level further. Predict all three printed lines.

What changes, and why

Only the third line changes, to 420000: the total now counts only the two shortlisted records, because it is reached only when the condition holds.

Neither total is wrong in itself; they answer different questions. "What is this batch worth?" and "what is our shortlist worth?" are both reasonable, and a single indentation level is the entire difference between them. When a number looks surprising, checking which side of the if it is computed on is a fast and frequently correct first guess.

Debug the bug

An assistant was asked to drop the low-value records from a batch in place. It produced this and said two records would remain.

batch = [
    {"notice_id": "A-1", "estimated_value": 20000},
    {"notice_id": "A-2", "estimated_value": 30000},
    {"notice_id": "A-3", "estimated_value": 400000},
]

for record in batch:
    if record["estimated_value"] < 100000:
        batch.remove(record)

print(len(batch))

What's actually wrong

It prints 2, and one of the two survivors should have been removed.

The loop walks by position. It examines position 0, A-1, removes it, and every remaining record shifts down one place. A-2 is now at position 0, but the loop has moved on to position 1, which now holds A-3. A-2 is never examined at all, and stays in the batch.

The output is a plausible-looking 2, which is why this bug survives casual testing: with three records and one removal, the count that a reader half-expects is exactly what appears.

Build a new list instead of mutating the one you are walking:

kept = []
for record in batch:
    if record["estimated_value"] >= 100000:
        kept.append(record)
batch = kept

Or, since the body is a single filter, a comprehension says the same thing in one line: batch = [r for r in batch if r["estimated_value"] >= 100000].

Try it yourself

Five fixture records. Collect the identifiers worth at least 100,000 whose set-aside is exactly "Total Small Business". One record does not publish a set-aside at all, and must not stop the run.

Loading this exercise…

Practical challenge (optional)

Optional: extend your loop so it also collects the records it could not judge, the ones with no published set-aside, into a second list, and print that list under a heading such as "needs review". A reviewer who sees only the shortlist has no idea anything was skipped, and "records this run could not assess" is a report line the capstone eventually requires.

AI collaboration

Checkpoint

How many times does the body of for i in range(3): run, and what values does i take?
Why does creating an accumulator inside the loop instead of before it lose data?
What goes wrong when you remove items from a list while looping over it?
When is a comprehension a better choice than a full loop?

Answers

Three times, with i taking 0, 1, and 2. range stops before its argument, the same half-open rule as slicing.
It is recreated empty on every pass, so only the last item's contribution survives.
Removing an item shifts the remaining positions down while the loop advances, so the item immediately after each removal is skipped entirely.
When the body is a single expression plus an optional filter. Once the body does two things, or needs a comment, a full loop is clearer.

Summary and next step

Loops repeat a block per item, accumulators live outside the loop, range and slices both stop early, mutating a collection while iterating skips items, and optional fields need a defaulted lookup. The assistant now turns a batch into a shortlist. Module 4 breaks that script into small named functions you can test one at a time.