Distributed Tracing

In a Nutshell

Distributed tracing follows a single request as it travels through many services, recording the timing and outcome of each step, so you can see the request's entire journey as one connected picture. In a microservices system, one user action might touch a dozen services — an API gateway, auth, several backend services, databases, caches, message queues. When that request is slow or fails, metrics tell you something is wrong and logs show scattered fragments, but only a trace shows you the whole path: which service was the bottleneck, where the time went, and where it broke. Tracing is the observability signal purpose-built for understanding distributed request flows.

How It Actually Works

Traces and Spans

The two core concepts:

• Trace — the complete journey of one request through the system, identified by a unique trace ID.
• Span — a single unit of work within that trace (one service call, one DB query), with a start time, duration, and metadata. Spans nest to form a tree.

Trace (trace_id: abc123) — total 340ms
├─ span: gateway            [0ms ────────────────────────── 340ms]
   ├─ span: auth-service    [5ms ──── 25ms]
   ├─ span: order-service   [30ms ─────────────────── 330ms]
   │  ├─ span: db query     [40ms ── 60ms]
   │  └─ span: payment-svc  [70ms ──────────────── 320ms]  ← 250ms! bottleneck
   │     └─ span: stripe API[75ms ─────────────── 315ms]  ← the real culprit
   └─ (returns)
→ Instantly visible: 250ms of the 340ms was the payment call to Stripe.

Each span records parent-child relationships, so the trace reconstructs the exact call tree and timing.

Context Propagation: The Core Mechanism

Tracing works by propagating context (the trace ID + parent span ID) across every service boundary — passed in request headers so each service knows which trace it's part of and who called it:

Request → Service A (starts trace, creates root span)
  → adds trace headers (traceparent: abc123-span1) to its call to B
    Service B (reads headers, creates child span under span1)
      → adds headers to its call to C
        Service C (child span under B's span)...

The trace context flows through HTTP headers / message metadata,
stitching independent service spans into ONE coherent trace.

This is why tracing requires instrumentation at every service (or an auto-instrumenting agent) — each service must read incoming context and propagate it onward. The standard is W3C Trace Context headers (traceparent), and OpenTelemetry is the vendor-neutral instrumentation standard.

The Three Pillars Working Together

Tracing completes the observability picture alongside metrics and logs:

Signal	Answers	Scope
Metrics	Is something wrong? How bad?	Aggregate, whole system
Traces	Where in the request flow?	One request, across services
Logs	What exactly happened?	Individual events

Typical debugging flow:
  1. METRICS alert: p99 latency spiked.
  2. TRACES: pull slow traces → the payment span is the bottleneck.
  3. LOGS: filter that span's trace_id → the exact error/query detail.
Metrics detect → traces localize → logs explain.

The shared trace ID links all three (see Logging) — a trace ID in your logs lets you jump from a log line to the full trace and vice versa.

Sampling: You Can't Trace Everything

Tracing every request at high volume is expensive (storage + overhead), so systems sample:

Strategy	How	Trade-off
Head-based	Decide to sample at the start (e.g., 1% of requests)	Simple; may miss rare errors
Tail-based	Decide after seeing the whole trace (keep all errors/slow ones)	Smarter; more complex/costly to buffer

Tail-based sampling is powerful because it keeps the traces you actually care about (errors, slow requests) while discarding the boring majority.

Seeing It in Action

Scenario: Diagnosing intermittent slow checkouts with distributed tracing.

Symptom: metrics show checkout p99 latency spiked to 3s, but only sometimes.
Logs alone are unhelpful — scattered across 8 services, hard to connect.

With distributed tracing (OpenTelemetry + tail-based sampling keeping slow traces):

  Pull a slow checkout trace (trace_id: xyz789, total 3,100ms):
  ├─ gateway            [═══════════════════════════════ 3100ms]
  │  ├─ auth            [══ 40ms]
  │  ├─ cart-service    [═══ 60ms]
  │  ├─ inventory-svc   [════ 90ms]
  │  └─ order-service   [══════════════════════════ 2900ms]
  │     ├─ db: insert   [═ 15ms]
  │     ├─ payment-svc  [═════════════════ 1800ms]
  │     │  └─ stripe    [════════════════ 1750ms]   ← slow downstream
  │     └─ email-svc    [═══════════ 1050ms]         ← BLOCKING the response!

  Two problems instantly visible that metrics/logs alone wouldn't reveal:
    1. The Stripe call is slow (1750ms) — a downstream issue.
    2. WORSE: order-service calls email-svc SYNCHRONOUSLY (1050ms) and
       waits for it before responding — email should be ASYNC (queue it!).
       The trace exposed an architectural flaw: a non-critical step is on
       the critical path.

  Fixes the trace revealed:
    - Move email to a message queue (async) → removes 1s from checkout.
    - Add a circuit breaker + timeout to the payment call.
  Confirm via metrics afterward: p99 drops back to normal.

Without tracing:
  ✗ You'd see high latency (metrics) and disconnected log lines, but not
    that email-svc was synchronously on the critical path or that 1750ms
    was inside the Stripe call specifically. You'd guess for hours.

Why tracing is indispensable for microservices: in a monolith, a profiler shows you where time goes within one process; in a distributed system, that time is spread across many independent services, and no single service can see the whole picture. Distributed tracing reconstructs the complete request journey as one connected waterfall, making it immediately obvious which service is the bottleneck, how the time breaks down across the call tree, and where a failure originated — the payment span took 1750ms, the email call was needlessly synchronous on the critical path. This is information that metrics (too aggregate) and logs (too fragmented) simply cannot provide on their own. The mechanism that makes it work — propagating trace context through every service boundary via standard headers — is also why tracing requires instrumentation everywhere, and why standards like OpenTelemetry and W3C Trace Context matter. Together with metrics (which detect and quantify) and logs (which explain the specifics), tracing (which localizes within the request flow) forms the three pillars of observability, all tied together by a shared trace ID, giving you a fast path from "something is slow" to "here's exactly which call, in which service, and why."

Interview Questions

1. Q: What is distributed tracing and what problem does it solve? Hint: It follows a single request through all the services it touches, recording each step's timing and outcome as connected spans, so you can see the request's entire journey. It solves the problem that in microservices one action spans many services — metrics are too aggregate and logs too fragmented to show the whole path, so only a trace reveals which service is the bottleneck and where a request slowed or failed.

2. Q: Explain traces and spans. Hint: A trace is the complete journey of one request through the system, identified by a unique trace ID. A span is a single unit of work within that trace (a service call, a DB query) with a start time, duration, and metadata. Spans nest via parent-child relationships to form a tree, reconstructing the exact call structure and timing so you can see where time went.

3. Q: How does trace context propagation work, and why does tracing need instrumentation everywhere? Hint: Each service passes the trace context (trace ID + parent span ID) to downstream services via request headers (W3C traceparent), so each creates a child span under its caller, stitching independent service spans into one trace. Every service must read incoming context and propagate it onward — hence instrumentation (or an auto-instrumenting agent like OpenTelemetry) is required at each service, or the trace breaks.

4. Q: How do metrics, traces, and logs work together? Hint: Metrics detect and quantify a problem across the whole system ("p99 latency spiked"). Traces localize it within the request flow ("the payment span is the bottleneck"). Logs explain the specifics ("this exact query/error"). A shared trace ID links all three, so you can jump between them: metrics alert → traces pinpoint the slow service → logs reveal the root cause → metrics confirm the fix.

5. Q: Why is sampling necessary in tracing, and what's the difference between head- and tail-based sampling? Hint: Tracing every request at high volume is expensive (storage + overhead), so systems sample. Head-based decides at the request's start (e.g., keep 1%) — simple but may miss rare errors. Tail-based decides after seeing the full trace, keeping the interesting ones (errors, slow requests) and discarding the boring majority — smarter and more useful, but requires buffering traces and is more complex/costly.

References

• OpenTelemetry documentation — the standard for tracing instrumentation
• Google Dapper paper — the foundational distributed tracing system
• W3C Trace Context — the standard propagation headers

Dive Deeper

• Jaeger / Zipkin — open-source distributed tracing backends
• Honeycomb: distributed tracing and observability — high-cardinality tracing
• Distributed Systems Observability by Cindy Sridharan — the three pillars in depth