Reading a Folder Without Being Fooled By It

By the end of this lesson you can point a Python program at a folder and get back an accurate, repeatable description of what is in it: which entries are files and which are folders, what kind each file is, and in an order that does not change between runs.

Every automation in this course starts by looking at something. For most of them, that something is a folder, and a folder is a surprisingly unreliable narrator.

A directory listing arrives in whatever order the filesystem felt like. It contains folders alongside files. It contains Report.CSV and report.csv, which are the same name on Windows and two different names on Linux. It contains notes.tar.gz, whose "extension" is either .gz or .tar.gz depending on what you meant. It contains README, which has no extension at all and will break any code that assumes one.

None of that is exotic. All of it is in the first real folder you point a script at, and each one produces a different flavour of the same bug: a job that processed 41 of 42 files and reported success.

Getting the survey right is also what makes everything after it reviewable. If the listing is deterministic, you can diff today's run against yesterday's and the difference is real. If it is not, every run looks different and you cannot tell a change from a shuffle.

pathlib.Path is the standard-library type for filesystem paths. A Path is not a string, and the difference is the point: it knows how to join, split, and inspect a path without any string surgery.

folder / "inbox" / "report.csv"   join, correct separator on any platform
path.name                          "report.csv"
path.stem                          "report"
path.suffix                        ".csv"      (the last one only)
path.suffixes                      [".tar", ".gz"] for notes.tar.gz
path.parent                        the containing folder
path.is_file() / path.is_dir()     what kind of thing this is
path.iterdir()                     the entries directly inside a folder
path.rglob("*.csv")                matching paths at any depth; filter file/link types

Four rules turn that into a survey you can trust.

1. Ask what each entry is; do not assume. iterdir() yields folders as well as files. A job that copies "every entry" into a processing loop will eventually try to read a directory as text and produce an IsADirectoryError — or worse, silently skip it inside a broad except.

2. Normalise the suffix before you compare it. path.suffix preserves case, so .CSV and .csv are different strings. Lowercase it once, at the point of comparison, and the rest of the program stops caring. path.suffix is also '' for a name with no dot, which is a real case — decide what it means rather than letting it fall through a chain of elifs into nothing.

3. suffix is the last extension only. For notes.tar.gz, suffix is .gz and suffixes is ['.tar', '.gz']. Neither is wrong; they answer different questions. Choose deliberately.

4. Sort. Every time. iterdir() makes no ordering promise, and the order it happens to give you today may change after a file is deleted and rewritten. Sorting is one call and makes repeated runs over the same folder comparable. Path ordering differs between Windows and POSIX when names mix case, so choose an explicit sort key if output must match across platforms.

There is one more habit worth building now, before the next lesson makes it matter: decide what to do with each entry before doing anything to any of them. A survey is a read-only pass, and the plan it produces is the artefact the previous module taught you to review.

A practice folder, built and then surveyed:

import pathlib
import tempfile

root = pathlib.Path(tempfile.mkdtemp()) / "downloads"
root.mkdir(parents=True)
(root / "keep").mkdir()
for name in ["march.csv", "APRIL.CSV", "notes.tar.gz", "LICENCE"]:
    (root / name).write_text("placeholder", encoding="utf-8")


def survey(folder):
    """Read-only description of a folder: its files by kind, and its subfolders."""
    files, folders = [], []
    for entry in sorted(folder.iterdir()):
        if entry.is_dir():
            folders.append(entry.name)
        else:
            files.append((entry.name, entry.suffix.lower() or "(none)"))
    return files, folders


files, folders = survey(root)
print("files:", len(files))
print("folders:", len(folders))
for name, kind in files:
    print(f"  {name} -> {kind}")

sorted(folder.iterdir()) sorts Path objects, so a second run over this unchanged folder on the same platform has a stable order. Windows path comparisons fold case; POSIX path comparisons do not. Choose an explicit name key if the order must match across platforms.

entry.suffix.lower() or "(none)" does two jobs in one expression. The .lower() makes .CSV and .csv the same kind. The or "(none)" catches the empty string a name with no dot produces, turning a value that would silently group with nothing into one that appears in the output where somebody can see it.

is_dir() is asked first, because the interesting question about keep is not what its extension is. This controlled practice folder contains only ordinary files and one ordinary directory, so the else branch means file here. For a real folder, decide how to handle links before is_dir() or is_file() follows their targets, and report anything unclassified.

Predict every line, including the order of the indented ones.

files: 4
folders: 1
  APRIL.CSV -> .csv
  LICENCE -> (none)
  march.csv -> .csv
  notes.tar.gz -> .gz

Replace entry.is_dir() with entry.suffix == "" as the test for "this is a folder".

files: 3
folders: 2
  APRIL.CSV -> .csv
  march.csv -> .csv
  notes.tar.gz -> .gz

An assistant was asked to "count the spreadsheets in the downloads folder". It produced this.

def count_spreadsheets(folder):
    total = 0
    for entry in folder.iterdir():
        if str(entry).endswith(".csv"):
            total += 1
    return total

def find_spreadsheets(folder):
    """Every .csv file directly inside folder, by name, in a stable order."""
    return sorted(
        entry.name for entry in folder.iterdir() if entry.is_file() and entry.suffix.lower() == ".csv"
    )

The starter builds a practice folder in the browser's in-memory filesystem from the supplied FILES_JSON, and creates one subfolder inside it. Write survey, which returns the file count, the folder count, and a count of files by normalised suffix.

Loading this exercise…

Optional, and the transfer task for this lesson: survey a folder you did not design.

Extend survey to also report the largest file by entry.stat().st_size, and the number of entries it could not classify. Then run it against a folder with something awkward in it — a name with a trailing space, a file with two dots, an empty subfolder, a shortcut or symbolic link.

Sign in to track your progress on this exercise.

1. Why must a survey sort the entries it lists?
2. What does path.suffix return for notes.tar.gz, and what does path.suffixes return?
3. Why is entry.suffix == "" a poor test for whether something is a folder?
4. A job reports "processed 41 files" over a folder containing 42. Name two things in the folder that could explain it.

Sign in to track your progress on this exercise.

Use pathlib, ask the filesystem what each entry is rather than inferring it from the name, set a policy for links, lowercase a suffix before comparing it, decide what an empty suffix means, and sort so that repeated runs over the same folder produce the same answer. That survey is a read-only pass, which is exactly what the next lesson needs: it turns a survey into a plan of renames, and checks that no two of them are aiming at the same name before anything moves.