Logging

8 min read

In a Nutshell

Logging is the practice of recording discrete, timestamped events as your system runs — "user 42 logged in," "payment failed with error X," "request took 340ms." Logs are the most detailed record of what actually happened, and they're your primary tool for debugging: when something goes wrong, logs tell you the specific sequence of events that led there. But logs are only useful if they're structured, searchable, centralized, and at the right level of detail. In a distributed system with hundreds of services, raw text logs scattered across machines are useless; the real work of logging is turning that firehose of events into something you can query, correlate, and act on.

2D minimalistic diagram showing multiple services each emitting streams of timestamped log events, all flowing into a central log aggregation system where they're indexed and made searchable, with a magnifying glass querying across all of them, illustrating centralized structured logging

How It Actually Works

Structured vs Unstructured Logging

The single most impactful logging decision: emit structured logs (machine-parseable key-value/JSON) rather than free-text strings.

❌ Unstructured:  "User 42 failed login from 1.2.3.4 at 10:03"
   → hard to query: "show all failed logins for user 42 last hour" = regex hell

✅ Structured (JSON):
   { "ts": "2024-06-01T10:03:00Z", "level": "warn", "event": "login_failed",
     "user_id": 42, "ip": "1.2.3.4", "reason": "bad_password",
     "trace_id": "abc123" }
   → trivially queryable: filter event=login_failed AND user_id=42

Structured logs are the difference between "grep and pray" and running real queries over your event history.

Log Levels

Levels let you control verbosity and filter by severity:

Level	Use	Example
DEBUG	Detailed diagnostic info (dev/troubleshooting)	Variable values, flow tracing
INFO	Normal operational events	"Order created", "Service started"
WARN	Something unexpected but handled	"Retrying after timeout"
ERROR	A failure needing attention	"Payment failed", "DB unreachable"
FATAL/CRITICAL	System is unusable	"Out of memory, shutting down"

Production typically runs at INFO or WARN; DEBUG is too noisy/expensive at scale. Levels should be adjustable without redeploying.

Centralized Log Aggregation

In a distributed system, logs must be shipped off individual machines to a central store — otherwise they vanish when ephemeral instances are replaced, and you can't correlate across services:

Services → log shipper (Fluentd/Vector) → central store (ELK/Loki/Splunk/
           Datadog) → indexed, searchable, dashboards + alerts

The classic ELK stack:
  Elasticsearch (store + search) + Logstash (process) + Kibana (visualize)

Never rely on logs living only on the instance that produced them.

Correlation IDs: Tracing a Request Across Services

The key technique for distributed debugging: attach a unique correlation/trace ID to each request and propagate it through every service it touches, so you can reconstruct the full journey:

Request enters at the gateway → assigned trace_id "abc123"
  gateway   logs {trace_id: abc123, ...}
  → order-svc   logs {trace_id: abc123, ...}
    → payment-svc logs {trace_id: abc123, ...}
Query trace_id=abc123 → the ENTIRE request flow across all services,
in order. Without it, you're grepping disconnected logs by timestamp.

This is the bridge to distributed tracing.

What (and What Not) to Log

Do Log	Never Log
Events, errors, state transitions	Passwords, tokens, secrets
Request/trace IDs, user/tenant IDs	Full credit-card numbers, SSNs
Timings, status codes, key parameters	Raw PII beyond what's needed
Enough context to debug	Sensitive data (see Data Privacy)

Logging secrets or PII is a real breach vector — logs are widely accessible and long-lived. Scrub/redact sensitive fields.

Cost and Volume Management

Logs are expensive at scale (storage + indexing + ingestion). Manage the firehose:

Sampling — log a fraction of high-volume, low-value events.
Retention tiers — recent logs hot/searchable, older logs archived cheaply (see Backup & Retention).
Right level — don't run DEBUG in production.
Rate limiting — cap noisy log sources to avoid "log storms."

2D minimalistic diagram showing correlation-ID propagation: a single request assigned a trace ID at the gateway, flowing through three services that each log the same trace ID, then a query on that trace ID assembling all the scattered log lines into one ordered timeline of the request's journey

Seeing It in Action

Scenario: Debugging a failed checkout with structured, correlated logs.

A customer reports their checkout failed at 10:03. With good logging:

1. Find their request by user_id or order_id (structured fields):
     event=checkout_failed AND user_id=42
   → returns one log line with trace_id="abc123".

2. Pull the FULL request flow by trace_id:
     trace_id=abc123  (across ALL services, time-ordered)
   → { ts:10:03:00 svc:gateway   event:request_received  path:/checkout }
     { ts:10:03:00 svc:order-svc  event:order_validated   order_id:9001 }
     { ts:10:03:01 svc:payment    event:charge_attempt    amount:59.99 }
     { ts:10:03:03 svc:payment    event:charge_failed
       reason:"gateway_timeout" downstream:"stripe" latency_ms:2000 }
     { ts:10:03:03 svc:order-svc  event:checkout_failed   trace_id:abc123 }

3. Root cause is immediately visible: the payment provider timed out after
   2s. Not a bug in our code — a downstream dependency issue. Cross-check:
     event=charge_failed AND reason=gateway_timeout  (last 10 min)
   → 47 occurrences → it's systemic, not one unlucky user → page on-call,
     consider tripping the circuit breaker to the payment provider.

Contrast without structured/correlated logs:
  ✗ Free-text logs on 5 different machines, no shared ID → you'd grep by
    rough timestamp across services, guessing which lines belong together,
    for hours — and might never connect them.

Why structured, centralized, correlated logging is transformative: the difference between the two scenarios is the difference between a five-minute diagnosis and a multi-hour outage investigation. Structured fields turn logs into a queryable database of events — you can ask precise questions ("all failed charges with a gateway timeout in the last 10 minutes") instead of crafting fragile regexes. Centralization means the logs survive their ephemeral instances and live in one searchable place. And the correlation ID is the linchpin: it stitches a single request's scattered log lines across many services into one coherent, ordered story, which is the only practical way to debug a distributed system where one user action touches a dozen services. Logging isn't just "print statements to a file" — done well, it's an investigative tool that lets you reconstruct exactly what happened, distinguish your bugs from downstream failures, and tell whether a problem is isolated or systemic, all in minutes.

Interview Questions

Q: Why are structured logs better than free-text logs? Hint: Structured logs (JSON/key-value) are machine-parseable, so you can run precise queries and aggregations over specific fields (event=login_failed AND user_id=42) rather than fragile regex over free text. They enable filtering, dashboards, and alerting, and make correlation across services practical. Free text is "grep and pray"; structured logs turn your event history into a queryable database.
Q: What is a correlation/trace ID and why is it essential in distributed systems? Hint: A unique ID assigned to each request and propagated through every service it touches, tagged onto every log line. It lets you reconstruct a single request's entire journey across many services in order by querying that one ID — the only practical way to debug distributed flows. Without it, you're guessing which scattered log lines across machines belong to the same request by timestamp.
Q: Why must logs be centralized, and what's the risk of not doing so? Hint: In distributed systems, logs on individual instances vanish when ephemeral instances are replaced (deploys, scaling, crashes) and can't be correlated across services. Centralized aggregation (ELK, Loki, Datadog) ships logs off-host into one indexed, searchable store surviving instance churn, enabling cross-service queries, dashboards, and alerts. Relying on per-instance logs means losing them exactly when you need them.
Q: How do you manage log volume and cost at scale? Hint: Sample high-volume/low-value events, use appropriate log levels (don't run DEBUG in production), tier retention (recent logs hot/searchable, older archived cheaply), rate-limit noisy sources to prevent log storms, and avoid over-logging. Logs are expensive (ingestion + storage + indexing), so you balance debuggability against cost deliberately rather than logging everything forever.
Q: What should you never log, and why? Hint: Passwords, tokens, secrets, full card numbers, SSNs, and unnecessary raw PII. Logs are widely accessible, long-lived, replicated to aggregation systems, and often less protected than production databases — so logging sensitive data is a real breach vector and a privacy/compliance violation. Redact/scrub sensitive fields; log identifiers (user_id) not secrets.

References

Google SRE Book — Monitoring & logging — logging in production
The Twelve-Factor App: Logs — treat logs as event streams
Elastic (ELK) Stack docs — centralized log aggregation

Dive Deeper

OpenTelemetry Logs — standardized, correlated logging
Honeycomb: Structured logging and observability — events as the unit of observability
Charity Majors on logs vs events — the philosophy of good logging