Structured Observability and Service Objectives: Production Python Systems

Outcome

Define a service objective and emit structured diagnostic fields without secrets or personal data.

Why it matters

At this level, code is judged by how safely it changes under load, failure, and team ownership. The technique in this lesson makes an important boundary visible enough to test and review.

Concept

Observability is evidence for decisions. Logs explain individual events, metrics show trends, and traces connect work across boundaries. Start from a user-visible objective and collect only fields needed to act.

Read the code

import json

def event(route, status, duration_ms, request_id):
    return json.dumps({
        "event": "request_complete",
        "route": route,
        "status": status,
        "duration_ms": duration_ms,
        "request_id": request_id,
    }, sort_keys=True)

print(event("/jobs/:id", 200, 18, "req-demo"))

Read from the public behavior inward: identify the input boundary, the decision, and the observable result before studying syntax.

Compare an event with metrics and traces

One request, three kinds of evidence

Actual output versus illustration: The listing prints one JSON event. It does not issue an HTTP request, measure a duration, emit a metric or create a trace.

req-demo is a synthetic input. The other requests and trace spans below are invented teaching fixtures, not observations of a running service.

Log · one event from the listing

event("/jobs/:id", 200, 18, "req-demo")

Serialize the supplied fields; print one JSON line

event: request_complete
route: /jobs/:id
status: 200
duration_ms: 18
request_id: req-demo

A log describes this supplied event. The function does not verify that the request happened or succeeded.

Metric · aggregate a synthetic set

Four illustrative completed requests

Group by route template; never by request ID

Synthetic requests for /jobs/:id
Request	Status	ms
req-demo	200	18
req-demo-b	200	22
req-demo-c	503	40
req-demo-d	200	20

Defined success rule: For this illustration, status 200 counts as success.
Aggregate: 3 successful / 4 total = 75%; sum duration = 100 ms; mean = 25 ms.
Scope: The request column explains the fixture; it is not a metric label. This tiny set is not a production service-level result.

Metrics summarize a defined population and window. An aggregate does not identify which request failed.

Trace · connect synthetic spans

Request req-demo ↔ trace-demo

Explicit illustrative correlation; request_id alone is not a trace

Invented spans in trace-demo
Span	Parent	Duration
s-root	none	18 ms
s-db	s-root	7 ms

Timing assumption: s-root covers 0–18 ms; its child s-db covers 5–12 ms within that interval.
Relationship: The parent identifier connects database work to the request span. Both use the same trace identifier.
Limit: These spans are not emitted by the example. Do not add child time to parent time; the intervals overlap.

A trace records instrumented relationships and timings. It does not by itself explain why a dependency was slow.

Decide what the evidence supports

Service objective: Define successful user behavior, eligible requests, target proportion and measurement window before judging availability.
Correlation: Join only records with an explicit correlation link. The example supplies a request ID but no trace ID.
Minimize data: Use route templates and approved values. Avoid secrets and personal data in all fields, including identifiers.
Serializer limit: JSON field selection is not validation or redaction. These functions serialize caller-supplied values.

Use logs for particular events, metrics for aggregates and traces for connected work. None makes missing instrumentation or unsafe data safe.

Only the log panel is output from the runnable listing. The metric and trace examples are explicitly synthetic extensions showing different questions and limits; no live telemetry or service objective is verified.

Predict the output

Predict the exact output before running the example.

Check your prediction

{"duration_ms": 18, "event": "request_complete", "request_id": "req-demo", "route": "/jobs/:id", "status": 200}

It prints one structured JSON event with a route template rather than a user-supplied URL. No body, cookie, or credential is recorded.

Modify the code

Change one valid input into the nearest invalid or overloaded case. Write down which layer should reject it and what the caller should observe.

Review the change

Keep the failure at the narrowest boundary that owns the rule. Preserve a stable return value or exception contract so callers do not need to inspect implementation details.

Debug the bug

High-cardinality identifiers in metric labels make costs and queries explode. Keep them in sampled logs or traces, not metric dimensions.

Try it yourself

Complete the focused implementation and run its deterministic checks.

Loading this exercise…

Practical challenge (optional)

Define an availability objective, its error-budget calculation, and one alert tied to user harm rather than raw CPU.

AI collaboration

Checkpoint

What distinct questions do logs, metrics, and traces answer?
Why is a request identifier useful in logs or traces but dangerous as a metric label?
What should an availability objective name before you choose an alert threshold?

Answers

Logs describe individual events, metrics aggregate trends, and traces connect one operation across boundaries.
It correlates one operation, but its near-unique values create unbounded metric cardinality and cost.
The user-visible successful behavior, the measured indicator, the target proportion, and the time window.

Summary and next step

You made the boundary explicit, predicted its behavior, tested a deterministic implementation, and examined its failure mode. Continue to the next lesson to combine this technique with a wider application or production constraint.