Capstone Phase 1 and 2: Scope, Data Dictionary, and Loading the Batch

By the end of this lesson you can state who the Opportunity Review Assistant is for and what it will not do, and you can write a data dictionary from a survey of the actual batch rather than from an assumption about it.

The capstone is one project built across eleven phases, and this lesson covers the first two. Everything downstream depends on the two documents produced here, and neither is code.

The scope document matters most for what it excludes. A tool that shortlists government contracting opportunities is one careless sentence away from sounding like it tells you whether to bid, whether you qualify, or whether you would win. It does none of those things, and saying so plainly, in the project's own documentation, is part of building it responsibly.

The data dictionary matters because every later phase makes a decision that belongs in it. What does an absent set-aside mean? Is a zero amount real? Which fields are required? Answer those once, in writing, and the normalisation, the matching rules, and the tests all agree with each other. Answer them ad hoc in three files and they will not.

Everything runs against the fixtures in this course's capstone/fixtures/ directory. No network, no key, no account.

The user story, in one paragraph, names the reader and the decision. For example: a small-business owner reviewing published federal contracting opportunities each week, who wants a shortlist of notices matching their categories and size, with the reason for each verdict, so they can spend their reading time on the ones worth reading.

The non-goals are the more valuable half. At minimum:

• It does not give procurement, legal, eligibility, or award advice.
• It does not decide whether you should bid, or predict any outcome.
• It does not submit anything anywhere.
• It does not ingest restricted, proprietary, or client data. Every record it reads is a synthetic fixture committed alongside the project.
• It does not run unattended against a live service as part of this course.

The data dictionary has one entry per field, with four parts: the source name, one sentence of meaning, the type you store it as, and what your program does when the field is absent or unusable. That fourth part is the one people skip and the one every later phase needs.

Write it after surveying the batch. A field you assumed was always present, and is not, becomes a KeyError in production and a decision in a document.

The acceptance criteria for the whole project are written now, while it is easy to be honest about them. Module 13's rule applies: two people should agree whether each is met without discussing it.

Phase 2, loading the batch, reuses Module 7's design unchanged. The client takes a transport; the fixture transport returns the stored page for the offset it is asked for; the paging loop stops on a short page and is bounded by a hard page cap. Building it this way now is what lets phase 8 test the client's failure paths by handing it a different stored response rather than by rewriting anything.

import json

PAGE_1 = json.loads(
    '{"totalRecords": 3, "limit": 2, "offset": 0, "results": ['
    '{"noticeId": "SYN-2026-0001", "setAside": "Total Small Business",'
    ' "placeOfPerformance": {"state": "OR"}},'
    '{"noticeId": "SYN-2026-0005", "setAside": null,'
    ' "placeOfPerformance": {"state": "OR"}}]}'
)
PAGE_2 = json.loads(
    '{"totalRecords": 3, "limit": 2, "offset": 2, "results": ['
    '{"noticeId": "SYN-2026-0007", "setAside": "Total Small Business"}]}'
)

PAGES = {0: PAGE_1, 2: PAGE_2}


def fixture_transport(params):
    """Returns a stored page. Deterministic, offline, no credential."""
    page = PAGES.get(params["offset"])
    if page is None:
        return {"status": 404, "body": None}
    return {"status": 200, "body": page}


def load_all(transport, page_size=2, max_pages=5):
    """Return (records, requests, error). Bounded by max_pages."""
    records, offset = [], 0
    for attempt in range(max_pages):
        response = transport({"offset": offset, "limit": page_size})
        if response["status"] != 200:
            return records, attempt + 1, f"status {response['status']}"
        page = response["body"]["results"]
        records.extend(page)
        if len(page) < page_size:
            return records, attempt + 1, None
        offset += page_size
    return records, max_pages, "hit the page limit"


records, requests, error = load_all(fixture_transport)
print(len(records), requests, error)
print(sorted({key for record in records for key in record}))
print(sum(1 for record in records if "placeOfPerformance" not in record))

The transport returns a 404 for an offset it does not know about, so a paging bug shows up as a failure rather than as a quietly short result.

The last two lines are the survey. The first collects the union of every key seen anywhere, which is the list your data dictionary must cover. The second counts records missing a specific field, which is how "is this optional" gets answered with a number rather than an impression.

Predict the three printed lines.

3 2 None
['noticeId', 'placeOfPerformance', 'setAside']
1

Change load_all's page_size to 3 and predict all three lines.

2 1 None
['noticeId', 'placeOfPerformance', 'setAside']
0

A first draft of the project scope reads:

The Opportunity Review Assistant scans SAM.gov for contract opportunities and tells you which ones you are eligible for and most likely to win, so you can focus your bidding effort. It automatically ingests your company profile and past award history to improve its recommendations over time.

Survey a batch the way phase 1 requires: find which fields are present in every record and which are only sometimes there.

Loading this exercise…

Optional but strongly recommended, because phase 3 depends on it. Run the survey over the real capstone fixtures, capstone/fixtures/opportunities-page-1.json and opportunities-page-2.json, and write the full data dictionary: every field, its meaning, its stored type, and its behaviour when absent or unusable. Then write the project's user story and at least three non-goals, including the one excluding procurement, legal, eligibility, and award advice. These two documents are the deliverable for phase 1.

Sign in to track your progress on this exercise.

1. Which four things does a data-dictionary entry record?
2. Why must the data dictionary be written after surveying the batch rather than before?
3. Name three claims the project's scope must explicitly exclude.
4. Why does the fixture transport return 404 for an unknown offset rather than an empty page?

Sign in to track your progress on this exercise.

Phase 1 produces a user story, explicit non-goals, a surveyed data dictionary, and acceptance criteria; phase 2 loads the full batch through a transport-injected client bounded by a page cap, and confirms the count against totalRecords before anything is surveyed. Next: turning those raw records into validated ones, and applying matching rules that explain themselves.