Module 7: APIs and Web Data
A Client Boundary You Can Test Without a Network
Reading an API contract, then separating the logic of a client from the act of making a request, so every test runs against stored fixtures at full speed with nothing to configure.
Lesson 24 of 46 in the recommended order · About 30 min (estimate)
On this page
Outcome
By the end of this lesson you can read an API contract and say what it commits to, and build a client whose logic is completely separable from the act of making a request, so every test runs from stored fixtures.
Why it matters
A client that calls the network directly is harder to isolate in tests. Live-service tests can be slow, need credentials, depend on someone else's uptime, and produce different results on different days; patching the HTTP call is possible, but an explicit transport boundary makes that substitution easier. Teams in that position usually stop writing tests for the client at all, which is unfortunate, because the client is where most of the interesting failures live.
The fix is a structural one, and it is the same idea as Module 4's separation of decision from presentation: the part that interprets a response and the part that obtains one are different jobs. Split them and the interpreting logic becomes easier to test independently of a live service. A client that calls an injected transport is not automatically a pure function; its behavior still depends on that transport and its responses.
For this course the split is not just good practice, it is the whole arrangement. Every exercise, every test, and the entire capstone run against stored fixture responses, with no connection, no account, and nothing to configure. That is not a reduced version of the real thing. A stored response can be short, malformed, rate-limited, or truncated on purpose, which is exactly what a real service will not produce on demand, and the failure paths are where most of a client's real work lives.
Concept
Give the client its transport as an argument. A transport is anything callable that takes request details and returns a response: a real HTTP call in production, a function returning fixture data in a test. The client never chooses which one it gets.
def search(transport, params):
response = transport(params)
...
Everything interesting stays in the client: interpreting the status, walking pages, applying retries, normalising records. All of it is exercised from fixtures, at full speed, with no credentials and no network.
Design fixtures for the cases that actually break clients, not only the happy one. A success with records. A success with zero records. A refusal. A rate-limit response. A malformed body. A short final page. Each is a few lines and each pins down a behaviour that is otherwise discovered in production.
Reading the contract first. Before any of that, read what the provider actually published. A contract is a page of documentation, and five things in it decide the shape of your client:
- Inputs. Which parameters exist, which are required, what their types are, what happens to values out of range, and what happens to a parameter the service does not recognise. A service that ignores unknown parameters will silently widen your filtered result set when you misspell one.
- Response shape. The envelope around the records, the fields on each record, and, most importantly, which fields can be absent, null, or a different type than you assumed. An amount published as a string is a normalisation decision, not a surprise, if you read it here first.
- Pagination model. How you ask for the next page and how you know there is not one. Offset and limit, a cursor, a link to follow: each ends differently.
- Errors. Which statuses the service returns, what each means, and which of them are worth retrying. A
429and a500are usually worth retrying; a400or a401never is, because repeating a request the service already refused does not change its mind. - Authentication. Whether a credential is required at all, and if so, the mechanism the provider expects.
That last one deserves a plain statement, because it is the thing most often gotten wrong by assumption.
How a credential is transmitted is decided by the provider and stated in its documentation. Some read a request header. Some read a query parameter. Some require a signed request. There is no universal rule, so there is nothing to infer: you look it up in that provider's current official documentation and you do what it says. Code written from a guess either fails outright or sends the credential somewhere the provider never intended.
What is general is how a credential is looked after, whatever the mechanism:
- It is read from configuration the operator controls, not written into source.
- It is never committed, never logged, never printed, never included in an error message, and never pasted into an AI tool or a support ticket.
- It never reaches browser code, where anything the page can read, a visitor can read.
- Its path is as short as possible: from wherever it is configured to the one request that needs it, and nothing along that path widens.
This course requires none of this in practice. You are not asked to obtain, configure, or hold a service credential. The worked client and exercise use fixture transports, not live services. The deliberately flawed HTTP example below is for analysis, not a request to run it against a service. The rules above are worth knowing because you will meet them; the client you build here is proven against fixtures.
Finally, and this belongs in the client's documentation rather than its code: the output of this project is a reading aid. It is not procurement, legal, eligibility, or award advice, and nothing it produces should be presented as such.
Read the code
FIXTURE_PAGES = {
0: {"status": 200, "body": {"totalRecords": 2, "results": [{"noticeId": "SYN-1"}, {"noticeId": "SYN-2"}]}},
2: {"status": 200, "body": {"totalRecords": 2, "results": []}},
}
def fixture_transport(params):
"""Deterministic stand-in. Returns a stored page. No network."""
return FIXTURE_PAGES.get(params.get("offset", 0), {"status": 404, "body": None})
def search(transport, params, page_size=2, max_pages=5):
"""Return (records, error). All interpretation lives here; obtaining a
response is entirely the transport's job."""
records = []
offset = 0
for _ in range(max_pages):
response = transport({**params, "offset": offset, "limit": page_size})
if response["status"] != 200:
return None, f"request failed with status {response['status']}"
page = response["body"]["results"]
records.extend(page)
if len(page) < page_size:
return records, None
offset += page_size
return records, "stopped at the page limit"
def describe_policy(page_size, max_pages, timeout_seconds, max_attempts):
"""Policy values to display, not enforcement. Each is a number
somebody chose, and every one belongs in a startup log line."""
return (
f"page_size={page_size} max_pages={max_pages} "
f"timeout={timeout_seconds}s max_attempts={max_attempts}"
)
records, error = search(fixture_transport, {"state": "OR"})
print(len(records), error)
print(describe_policy(2, 5, 10, 3))
search contains every decision and makes no request. It is handed a callable, calls it, and interprets the result. Swapping fixture_transport for a transport that really does reach a service would change one argument at one call site and no line inside search. That substitution is not part of this course. This worked client demonstrates pagination and status handling against the expected fixture shape; it does not yet validate malformed bodies, implement retries, or enforce a request timeout.
fixture_transport returns a 404 for any offset it does not know about, which is deliberate: a fixture that silently returns an empty success for unexpected input hides mistakes in the very code it is meant to be testing.
describe_policy exists to make a point about bounded work. The page size and page cap are parameters that search actually uses. In this example, the timeout and attempt limit are only values formatted by describe_policy; printing them does not enforce either limit. A live transport would need to enforce its timeout, and retry logic would need to enforce an attempt limit. Tests should check behavior as well as displayed configuration.
Trace the fixture transport boundary
An injected callable; no live service in this run
Executed path: search calls fixture_transport, which looks up a page in FIXTURE_PAGES.
These are in-process function calls and dictionary reads. The state parameter is passed through; this fixture chooses by offset only and does not filter by state or enforce limit.
search
Build offset/limit parameters; interpret status and results.
fixture_transport
Accept params; read offset with default 0.
FIXTURE_PAGES.get(...)
Look up the stored response, or supply the fallback 404 response.
Actual execution boundary
- Local only
- Two fixture lookups; no HTTP request, credentials or network wait.
- Caller output
- search returns records and error; outer print displays the result.
Call 1 · offset 0
Passed parameters- state="OR", offset=0, limit=2
Stored response- status=200; results=[SYN-1, SYN-2]
Client decision- Two items fill the page, so offset advances to 2.
totalRecords is present in the fixture body but search does not consult it.
Call 2 · offset 2
Passed parameters- state="OR", offset=2, limit=2
Stored response- status=200; results=[]
Client decision- An empty page is shorter than 2. Return both collected records and error=None.
The second lookup is how this implementation discovers completion after a full first page.
What the example enforces
Used by search- page_size=2 and max_pages=5
Only displayed- timeout=10s and max_attempts=3 are text from describe_policy.
Failure boundary- Non-200 returns an error. Malformed bodies and transport exceptions can still raise.
There is no timeout or retry implementation hidden behind the printed policy line.
Conceptual live adapter · not executed
Same callable boundary- A separately implemented adapter could accept request details and return the agreed response shape.
Different responsibilities- It would perform HTTP, enforce a timeout and handle provider-specific credentials.
Approval and contract- No adapter or service is configured or called in this lesson.
This alternative is outside the executed path above; injection alone does not prove live-service compatibility.
Predict the output
Predict both printed lines.
Check your prediction
2 None
page_size=2 max_pages=5 timeout=10s max_attempts=3
Two records collected in two local transport calls: the first fixture page has two results, which is a full page, so a second transport call returns an empty and therefore short page, ending the loop.
Notice what did not happen on the way to that answer. Nothing was configured, nothing was fetched, and the result would be identical on any machine, on any day, with the network unplugged. That is the property the whole capstone is built on.
Modify the code
Change the final line of search from return records, "stopped at the page limit" to return records, None. Say what breaks, given that the printed output above does not change at all.
What changes, and why
The output is identical, because these fixtures end on a short page long before the cap is reached. The defect is invisible here and serious elsewhere.
The cap exists to bound the work: without it, a source that keeps returning full pages loops until something else gives out. The error exists to say the bound was reached. Remove the error and a caller that hits the cap receives a truncated result labelled complete.
That is worse than a failure. A shortlist missing a third of its rows looks exactly like a shortlist that had nothing more to add, so nobody investigates. The rule is worth stating in general: when a limit stops you early, the partial result and the fact that it is partial must travel together, or the limit has quietly become a data-loss bug.
Debug the bug
An assistant was asked for "a client that fetches all the pages". It produced this, and it does technically fetch all the pages.
import requests
BASE = "https://example-procurement.test/v1/search"
def search(params):
records = []
offset = 0
while True:
response = requests.get(BASE, params={**params, "offset": offset})
page = response.json()["results"]
records.extend(page)
if not page:
return records
offset += len(page)
What's actually wrong
Six defects, and the first is the one that makes the other five hard to find.
- There is no transport boundary. The function reaches for
requestsitself. A test can patch that call to avoid the network, but an injected callable makes the substitution explicit and lets fixture tests expose these faults without a live service. - The loop has no cap.
while Trueis bounded only by the service's willingness to stop returning results. A source that keeps returning full pages runs until memory or patience gives out. - No timeout. A request with no timeout waits indefinitely. The default in most HTTP libraries is to wait forever, so this has to be set deliberately every time.
- The status is never checked. A
429or a500reachesresponse.json()as though it were data. If the error body is JSON, theKeyErroron"results"is confusing; if a gateway returned HTML, the parse failure is worse. - It stops on an empty page rather than a short one. Under a contract where a short page means completion, a nonempty short final page causes an unnecessary next request. An exactly full final page may still need another request to discover completion. A source that keeps returning nonempty pages never ends this loop.
- A partial result is indistinguishable from a complete one. The function returns a bare list only after an empty page. A request or parse failure propagates an exception instead of returning the collected records; the caller receives no structured partial-result report. A future cap or recovery path must make incompleteness explicit rather than silently returning a partial list.
The corrected shape puts the boundary back and makes each limit explicit:
def search(transport, params, page_size=8, max_pages=5):
records = []
offset = 0
for _ in range(max_pages):
response = transport({**params, "offset": offset, "limit": page_size})
if response["status"] != 200:
return None, f"request failed with status {response['status']}"
body = response["body"]
if not isinstance(body.get("results"), list):
return None, f"malformed response at offset {offset}: no results list"
page = body["results"]
records.extend(page)
if len(page) < page_size:
return records, None
offset += page_size
return None, f"stopped at the {max_pages}-page cap before reaching a short page"
The timeout now belongs to whichever transport is passed in, which is the right place for it: it is a property of making a request, not of interpreting one. The cap, short-page rule and status check are client decisions testable from stored responses. The shown shape check only validates results after assuming body is a dictionary; a None or other non-dictionary body still raises at body.get. It is a limited example, not complete response validation.
Try it yourself
Complete the client. It receives its transport as an argument and must keep an empty result distinguishable from a refusal.
Loading this exercise…
Practical challenge (optional)
Optional: add two more stand-in transports, one returning status 429 and one returning a 200 whose body has no results key at all, and decide what your client should do with each. The rate-limit case should be retryable; the malformed body is a contract violation and deserves a distinct error. Then write down which of your fixtures corresponds to which real-world situation. That list becomes the capstone's client test suite, and it covers cases a live service would not produce for you on request.
Sign in to track your progress on this exercise.
AI collaboration
Checkpoint
- Why does taking the transport as an argument make a client testable?
- What five things does reading an API contract tell you before you write any code?
- Why should a search that matched nothing and a refused request return different values?
- Why can nobody tell you, in general, where a credential goes in a request?
Answers
- The client contains only interpretation, which is ordinary logic. Tests hand it a stand-in that returns stored responses, so they need no network, no account, and no external uptime.
- Its inputs, its response shape including which fields may be absent, its pagination model, its error statuses and which are worth retrying, and its authentication requirement.
- Because they mean opposite things. An empty result is an answer; a refusal is the absence of one. Collapsing them hides permission and configuration failures behind a plausible empty shortlist.
- Because it is provider-specific. Each provider decides the mechanism and states it in its own documentation; a header, a query parameter, and a signed request are all real answers for different services, so there is nothing to infer and nothing to guess.
Sign in to track your progress on this exercise.
Summary and next step
Read the contract before writing the client, inject the transport, keep interpretation and pagination limits in the client while the transport enforces request timeouts, and build fixtures for the failure cases rather than only the happy one. The assistant can now ingest data shaped by a published contract, and prove it does so correctly, without a connection. Module 8 asks what the data actually says.