JSON: Nested Data and the Keys That Are Not There

By the end of this lesson you can turn a JSON payload into Python objects, reach a value buried two or three levels down without crashing when a branch is missing, and write down what each field means so the next person does not have to guess.

JSON is how nearly all public data arrives, including the procurement notices this course's project reviews. It maps almost exactly onto the Python types from Module 1 and Module 3, so the parsing itself is one line.

The work is everything after that line. Real payloads nest, and they omit. A search response wraps its records in an envelope; a record's location is an object that some records simply do not carry; a field is a number in one record and a string in the next. None of that is malformed JSON, so the parser is perfectly happy, and your code is the first thing that notices.

json.loads(text) parses a JSON string into Python objects. json.dumps(obj) goes the other way, and json.dumps(obj, indent=2) produces something a person can read. There are also json.load(file) and json.dump(obj, file) for file handles; the names without the s take a file, the ones with it take a string.

The type mapping is direct: JSON object becomes dict, array becomes list, string becomes str, number becomes int or float, true/false become True/False, and null becomes None. Note the last one: an explicit null in the payload gives you a key whose value is None, which is not the same as the key being absent. Module 3's rule applies exactly here, get's default fires only for a missing key.

Malformed text raises json.JSONDecodeError, which is a subclass of ValueError, so except ValueError: catches it. The message includes a line and column, which is usually enough to find the problem in a payload you can see.

For nested access, chaining square brackets is brittle: record["placeOfPerformance"]["state"] raises KeyError the first time a record omits the location. When the location is either absent or an object, defaulting the intermediate step to an empty dictionary handles the absent branch:

state = record.get("placeOfPerformance", {}).get("state", "unknown")

The outer get yields an empty dictionary when the branch is absent, and the inner get then finds nothing and returns the default. An explicit null is different: it becomes None, which has no .get method. Validate or normalise a present branch of the wrong type before using this chain; a present state whose value is None also keeps that value rather than using the default.

Finally, a data dictionary. For each field the project relies on, record four things: its name in the source, what it means in one sentence, its type, and what your program does when it is absent or unusable. This is a document, not code, and it is the artefact that stops a team from arguing about whether an empty set-aside means "none" or "not yet published". The capstone requires one.

import json

payload = """
{
  "totalRecords": 2,
  "results": [
    {"noticeId": "A-1", "estimatedValue": 310000, "setAside": null},
    {"noticeId": "A-2", "estimatedValue": "155000"}
  ]
}
"""

data = json.loads(payload)
results = data.get("results", [])

print(data["totalRecords"], len(results))
print(results[0].get("setAside", "absent"))
print(results[1].get("setAside", "absent"))
print(type(results[0]["estimatedValue"]).__name__, type(results[1]["estimatedValue"]).__name__)

The payload is an envelope: a count plus an array of records, which is how most search responses are shaped. Reading results with a default of [] supplies an empty list only when that key is absent from the parsed dictionary. It does not validate the envelope or replace an explicit null or a value of the wrong type; the later totalRecords read and record indexing still assume this worked payload’s structure.

The two setAside reads are the lesson in one pair of lines. The first record carries an explicit null; the second omits the key entirely. Neither state alone establishes the business meaning of the missing information. The data dictionary must define that meaning; the two states also behave differently in code.

The last line asks what type each estimated value actually arrived as. One is a number, the other is text, in the same array, in a well-formed payload. Discovering that at parse time is much cheaper than discovering it when a comparison raises three modules later.

Predict all four printed lines.

2 2
None
absent
int str

Change the first record's "setAside": null to "setAside": "", and predict lines 2 and 3.

An assistant was asked to extract the state from each record. It produced this and said missing locations default to "unknown".

import json

payload = '{"results": [{"noticeId": "A-1", "placeOfPerformance": {"state": "OR"}}, {"noticeId": "A-2"}]}'

data = json.loads(payload)

for record in data["results"]:
    state = record["placeOfPerformance"].get("state", "unknown")
    print(record["noticeId"], state)

KeyError: 'placeOfPerformance'

state = record.get("placeOfPerformance", {}).get("state", "unknown")

Parse a payload shaped like a real search response and flatten it, handling the record that omits its location entirely.

Loading this exercise…

Optional: write data-dictionary entries for four fields, noticeId, title, estimatedValue, and placeOfPerformance.state. For each, record the source name, one sentence of meaning, the type you will store it as, and what your program does when it is absent, null, or empty. Keep the result; the capstone's first phase asks for exactly this document, and writing it now while the examples are in front of you is far easier than reconstructing it later.

Sign in to track your progress on this exercise.

1. What Python value does JSON null become, and how does that differ from an absent key?
2. Which exception does malformed JSON raise, and which broader type can catch it?
3. Why does record.get("a", {}).get("b", "x") survive a missing a, while record["a"].get("b", "x") does not?
4. Name the four things a data-dictionary entry should record.

Sign in to track your progress on this exercise.

JSON maps onto Python types directly, null and absent are different states, nested access needs an explicit policy for absent keys and present values of the wrong type, and the data dictionary is where "what does empty mean here" gets settled once. Next: CSV, the format reviewers actually want back, and the normalisation that turns a mixed batch into validated records.