Observability & Monitoring
Knowing What Your System Is Actually Doing
You cannot operate what you cannot see. Once a system is running in production — especially a distributed one spread across many services and machines — the hardest question is often simply "what is happening right now, and why?" Monitoring is watching known signals to detect when something goes wrong; observability is the broader property of being able to understand a system's internal state from its outputs, including asking questions you didn't anticipate. Together they're what turn a black box into something you can debug, tune, and trust. Without them, every incident is a guessing game and every performance problem is a mystery.
The reason this topic is essential is that failures in production are inevitable, and the difference between a five-minute incident and a five-hour outage is almost entirely about visibility. The three pillars — logs (individual events), metrics (aggregate numbers), and traces (request journeys) — each answer a different question, and used together they let you detect a problem, localize it, and explain it. Health checks make automated self-healing and load balancing possible, and SLIs/SLOs turn the fuzzy goal of "be reliable" into concrete numbers that guide engineering decisions. Observability isn't an afterthought bolted on after launch; it's what makes running a system in production sustainable.
When This Comes Up
- System design interviews: After you've designed a system, "how do you know if it's healthy? how would you debug a slowdown?" is a natural follow-up. Strong candidates mention the three pillars, correlation/trace IDs, the four golden signals, health checks for load balancing and self-healing, and SLOs/error budgets — showing they think about operating the system, not just building it.
- Real architecture: Instrumentation is a first-class design concern. Where you emit metrics, how you structure logs, whether you propagate trace context, and how you define SLIs shape how operable the system is. These decisions determine whether you can diagnose problems in minutes or spend hours grepping disconnected logs.
- Production operations: This topic is the on-call experience. Every incident is detected by monitoring, diagnosed with logs/metrics/traces, and judged against SLOs. Health checks drive the automated routing and recovery that keep systems up, and error budgets decide when to slow down and stabilize.
How the Sub-Topics Connect
The sub-topics build the observability toolkit: the three pillars in turn — events (logging) → aggregate numbers (metrics) → request journeys (distributed tracing) — then the mechanism that drives self-healing and load balancing (health checks), and finally the framework that turns all this measurement into reliability targets and decisions (SLI/SLO/SLA):
1. Logging
The most detailed record of what happened: discrete, timestamped events. The real work of logging is turning a firehose into something queryable — structured logs (JSON, not free text) you can filter and aggregate, centralized aggregation so logs survive ephemeral instances and can be searched in one place, and above all correlation/trace IDs that stitch a single request's scattered log lines across many services into one coherent story. Logs are your primary debugging tool, but only if you log the right things (events, IDs, timings — never secrets or PII) and manage volume and cost deliberately.
2. Metrics
Numeric measurements sampled over time — cheap, efficient, and ideal for answering "how is the system doing right now versus normal?" Metrics power dashboards and alerting: they tell you that something is wrong and how bad. The four golden signals (latency, traffic, errors, saturation) are the principled minimal set to watch, and percentiles (p95, p99) are essential because averages hide the tail latency users actually feel. The discipline that keeps metrics useful is alerting on symptoms users experience (high error rate, high latency) rather than every internal fluctuation — because alert fatigue makes people ignore the alarm that matters.
3. Distributed Tracing
The signal purpose-built for microservices: following one request through every service it touches, so you see its entire journey as one connected waterfall of spans. It reveals which service is the bottleneck and where a request slowed or failed — information metrics (too aggregate) and logs (too fragmented) can't provide alone. The mechanism is context propagation — passing the trace ID through every service boundary via standard headers (W3C Trace Context, OpenTelemetry) — which is why tracing needs instrumentation everywhere. Together with metrics and logs, tracing completes the three pillars: metrics detect, traces localize, logs explain — all tied together by a shared trace ID.
4. Health Checks
The small, unglamorous mechanism that makes automated self-healing and load balancing work: endpoints that report whether an instance is okay. The crucial distinction is liveness (is the process alive? — failure triggers a restart) versus readiness (can it serve traffic right now? — failure removes it from the load-balancer pool without a restart). Conflating them causes real outages — restarting healthy instances during a dependency blip, or routing traffic to instances still warming up. Add the deep-check danger (coupling all instances to a shared dependency can pull the whole service from rotation) and startup handling, and health checks become the reliable foundation for intelligent routing and recovery — without the automation itself becoming the outage.
5. SLI, SLO, SLA
How teams turn "be reliable" into measurable targets and decisions. An SLI measures performance (good events / total), an SLO is the internal target (stricter than the SLA), and an SLA is the contractual promise with penalties. The most powerful idea they produce is the error budget — the reliability you've chosen to allow — which becomes a shared currency: when budget is healthy, ship features and take risks; when it's exhausted, freeze and stabilize. This resolves the dev-vs-ops tension with one objective number, and embraces that 100% reliability is the wrong goal (exponentially costly, imperceptible to users, and incompatible with shipping). Reliability is a means to happy users, not an end.
Sub-Topics
| # | Sub-Topic | What You'll Learn |
|---|---|---|
| 1 | Logging | Structured, centralized, correlated event records for debugging |
| 2 | Metrics | Numeric measurements, golden signals, and alerting |
| 3 | Distributed Tracing | Following a request across services to find bottlenecks |
| 4 | Health Checks | Liveness vs readiness for self-healing and load balancing |
| 5 | SLI, SLO, SLA | Measuring, targeting, and promising reliability with error budgets |