Text Files, Encodings, and the Lines You Did Not Expect

By the end of this lesson you can open a file so that it always closes, state its encoding rather than inherit one, and read it line by line without being caught by the header row, a stray blank line, or the newline still attached to each line.

Everything the review assistant has processed so far was typed into the program. Real batches arrive as files, and files bring three problems that have nothing to do with your logic: they can be left open, they can be decoded wrongly, and they contain lines that are not records.

None of these is intellectually difficult. All three produce failures that look like bugs in your rules, so they are worth recognising quickly. A count that is one too high is far more often a trailing newline than a mistake in a filter.

Open a file with with, always:

with open("batch.txt", "r", encoding="utf-8") as handle:
    for line in handle:
        ...

with closes the file when the block ends, including when an exception is raised inside it. Without it, a file left open can hold a lock, and on some systems written data may not reach disk. There is no situation in this course where the manual open and close pair is preferable.

Name the encoding. Text on disk is bytes, and an encoding is the agreement about which bytes mean which characters. utf-8 is the correct default for new work and for nearly all published data. Leaving it out means Python picks a platform default, so the same program can read a file correctly on your machine and raise UnicodeDecodeError on a colleague's. Passing encoding="utf-8" explicitly is one argument that removes an entire class of "works on my machine".

The modes you need: "r" reads, "w" writes and truncates the file to empty first, "a" appends. Confusing "w" with "a" destroys data silently and instantly, which is why the mode deserves a second look every time you write one.

Iterating over a file yields one line at a time and never loads the whole file into memory, which matters once files are large. Each line keeps its trailing newline character, so line == "GSA-2026-0731" is false for a line that reads GSA-2026-0731\n. Call .strip() before comparing.

Two kinds of line exist in real files and are not records: the header row at the top, and blank lines anywhere in the body. Skip both deliberately.

Watch how you split, too. Iterating a file object yields only real lines, so a file ending in a newline does not produce a phantom final entry. Splitting the whole text on the newline character does:

import io

text = "a\nb\n"

print(len(list(io.StringIO(text))))  # 2, one entry per real line
print(len(text.split("\n")))         # 3, the last one empty
print(len(text.splitlines()))        # 2

Most off-by-one record counts trace back to that difference. Prefer iterating the file, or splitlines(), over splitting on the newline character by hand.

The in-page runner has no persistent filesystem, so exercises here use io.StringIO, which behaves like an opened text file. The loop body is identical to one over a real file; only the two lines that obtain the handle differ.

import io

raw = "notice_id,agency\nGSA-2026-0731,General Services Administration\n\nDLA-2026-0088,Defense Logistics Agency\n"

records = []
skipped_blank = 0

with io.StringIO(raw) as handle:
    header = handle.readline().strip()
    for line in handle:
        cleaned = line.strip()
        if not cleaned:
            skipped_blank += 1
            continue
        notice_id, agency = cleaned.split(",", 1)
        records.append({"notice_id": notice_id, "agency": agency})

print(header)
print(len(records), "records,", skipped_blank, "blank lines skipped")
print(records[0]["agency"])

The \n sequences in raw are newline characters, so this text is four lines followed by a blank one. On your own machine the first two lines of the with block would read with open("batch.txt", encoding="utf-8") as handle: and nothing else would change.

handle.readline() consumes exactly one line, so the for loop that follows starts at the second. Inside, each line is stripped once and then tested: an empty result means a blank line, which is counted and skipped rather than silently dropped, because "how many blank lines did this file have" is a data-quality signal worth reporting.

split(",", 1) splits on the first comma only, so an agency name containing a comma stays intact. This is a deliberate half-measure: it handles one common case and still breaks on quoted fields, which is exactly why the csv module exists and why lesson 3 uses it.

Predict the three printed lines.

notice_id,agency
2 records, 1 blank lines skipped
General Services Administration

Delete the header = handle.readline().strip() line, leaving the loop unchanged. Predict what the program prints.

An assistant was asked to append a shortlisted identifier to a running log file. It produced this and said the log accumulates across runs.

def log_shortlisted(path, notice_id):
    handle = open(path, "w")
    handle.write(notice_id + "\n")
    handle.close()


log_shortlisted("shortlist.log", "GSA-2026-0731")
log_shortlisted("shortlist.log", "DLA-2026-0088")

def log_shortlisted(path, notice_id):
    with open(path, "a", encoding="utf-8") as handle:
        handle.write(notice_id + "\n")

Read a small text stream line by line, skipping the header and any blank line, then report what you found.

Loading this exercise…

Optional: change the stream so one line has trailing spaces after the agency name and another has a comma inside the agency name, then run your loop again. Report what each one does to your parsed records. Then write one sentence on why split(",") is not sufficient for real published data. That sentence is the argument for the csv module, and having made it yourself is better than being told.

Sign in to track your progress on this exercise.

1. What does with guarantee that a plain open does not?
2. Why pass encoding="utf-8" explicitly?
3. What is the difference between mode "w" and mode "a"?
4. Why can splitting a file's text on the newline character report one more line than the file actually has?

Sign in to track your progress on this exercise.

Use with, name the encoding, read the mode out loud before writing it, strip lines before comparing, and skip the header and the trailing blank line on purpose. Next: JSON, the format most public data actually arrives in, and the nested shapes that come with it.