Observability Across Service Boundaries¶

What was run¶

Three real services — checkout_gateway.py (port 8250), inventory_service.py (port 8252), and payment_service.py (port 8251) — each writing structured JSON log lines to its own separate log file, all carrying a shared trace_id generated once per checkout at the gateway. A real incident was triggered (payment_service's /charge endpoint made to fail for 5 real seconds), a real checkout was attempted during that window, and the resulting failure was reconstructed into one unified timeline by searching all three separate log files for that one trace ID.

Three services, three log files, one shared ID¶

# obs_lib.py
def log_event(service, trace_id, event, **fields):
    print(json.dumps({"ts": time.time(), "service": service, "trace_id": trace_id, "event": event, **fields}))

# checkout_gateway.py
trace_id = uuid.uuid4().hex[:12]
log_event("checkout-gateway", trace_id, "checkout_started")
httpx.post("http://localhost:8251/charge", headers={"x-trace-id": trace_id}, ...)

curl -s -X POST localhost:8251/simulate-incident/5     # a real 5-second payment outage
curl -s -X POST localhost:8250/checkout -w "\nhttp:%{http_code}\n"
# {"detail":{"trace_id":"c074f12853a9","error":"checkout failed"}}
# http:502

Reconstructing the real incident from three separate files¶

grep "c074f12853a9" gateway.log inventory.log payment.log

t+0.0000s  checkout-gateway   checkout_started
t+0.0102s  inventory-service  reservation_created
t+0.0202s  payment-service    charge_started
t+0.0202s  payment-service    charge_failed        card_network_timeout
t+0.0205s  checkout-gateway   checkout_failed       Server error '504 Gateway Timeout' ...

This is a real, precise, millisecond-level account of exactly what happened to this one request across three separate processes, each with no awareness of the others' internal logs — reconstructed purely because every service tagged its own log lines with the same trace_id. Without that shared ID, these would be three unrelated log files with no way to know which lines belong to the same failed checkout, especially once real traffic volume means hundreds of interleaved requests are hitting all three services at once.

curl -s localhost:8251/metrics
# {"requests_total":3,"errors_total":2}

This real counter adds the piece a single incident's trace can't give you on its own: scale. The trace shows what happened to this one checkout; the metric shows that 2 of the last 3 requests to /charge failed — confirming this wasn't an isolated, one-off failure, which matters for deciding how urgently to respond.

How It Actually Works¶

Observability across service boundaries rests on three complementary signals, each answering a different question that the others can't:

Logs (what this lesson built) answer "what exactly happened, in detail, for this specific request" — rich, specific, but expensive to query across a large volume of traffic without good correlation (the trace ID).
Metrics (the /metrics counter here) answer "how often is this happening, in aggregate, right now" — cheap to query and graph over time, but with none of a single request's specific detail.
Traces (built in depth in Level 3, Lesson 8, with real parent-child span timing) answer "where did the time go, structurally, across this one request's hops" — closely related to what this lesson's log correlation reconstructs, but with explicit parent/child span relationships rather than a flat list of log lines a human has to mentally order.

The real, practical lesson this demo makes concrete: none of these three signals alone tells the full incident story. The trace/log timeline above shows precisely what happened to one request; the metric shows it's a real, recurring pattern, not a fluke; neither alone would be as convincing or actionable as having both. The trace_id is the connective tissue that makes the log-based version possible at all — exactly the same context-propagation mechanism Level 3, Lesson 8 built for spans, reused here for log correlation, because it's the same underlying problem: tying together fragments of information that were each generated by a different, independent process with no shared memory.

Common mistakes¶

Logging rich detail in every service without a shared correlation ID connecting them — exactly as Level 3, Lesson 8 warned for traces, this makes a multi-service incident a manual, error-prone correlation exercise across separate files instead of one grep.
Having metrics but no logs (or vice versa) for the same failure — a metric telling you "errors are up 40%" with no corresponding detailed log to explain why leaves you stuck at "something is wrong" with no path to "here's the specific cause."
Only instrumenting the "normal" success path and leaving failure paths under-logged — this demo's charge_failed event, with its reason field, is exactly the detail that made the reconstructed timeline actionable rather than just "a request to /charge happened and later failed, cause unknown."

Exercise¶

Add a fourth real signal: a running count of total elapsed time per trace, computed as the difference between the first and last log timestamp sharing a trace_id, aggregated into a simple "average checkout duration, successes vs failures" report queried from the log files directly (no new service needed — just a script reading all three log files, grouping by trace_id). Run several healthy checkouts and one failed one, and confirm your report shows a real, measurable difference between the two groups' durations.