Metrics

8 min read

In a Nutshell

Metrics are numeric measurements of your system sampled over time — request rate, error rate, latency, CPU usage, queue depth. Where logs record individual events, metrics record aggregate numbers you can chart, alert on, and reason about at a glance. They're cheap to store (just numbers over time), efficient to query, and ideal for answering "how is the system doing right now, and how does that compare to normal?" Metrics are the backbone of monitoring dashboards and alerting: they tell you that something is wrong and how bad, so you know when to act — even if you then need logs and traces to find out why.

2D minimalistic diagram showing several time-series line graphs on a dashboard: request rate rising, error rate spiking above a threshold line (highlighted red), and latency climbing, with an alert bell triggered by the error-rate threshold breach, illustrating metrics as numeric measurements over time driving monitoring and alerting

How It Actually Works

Metrics vs Logs

	Metrics	Logs
Record	Aggregate numbers over time	Individual events
Answers	"How much / how many / how fast?"	"What exactly happened?"
Cost	Cheap (numbers)	Expensive (full events)
Cardinality	Low (bounded label sets)	High (unique details)
Best for	Dashboards, alerting, trends	Debugging specific incidents

Metrics tell you that error rate jumped to 5%; logs tell you which requests failed and why. You need both.

The Metric Types

Type	What It Measures	Example
Counter	A value that only increases	Total requests, total errors
Gauge	A value that goes up and down	Current memory, active connections, queue depth
Histogram	Distribution of values (buckets)	Request latency distribution
Summary	Like histogram, with quantiles	p50/p95/p99 latency

Histograms/summaries matter enormously for latency — an average latency hides the tail. p99 latency (the slowest 1%) is often what users actually feel.

The Four Golden Signals

Google's SRE guidance distills what to monitor for any user-facing system:

Signal	Question	Metric
Latency	How long do requests take?	Response time (p50/p95/p99)
Traffic	How much demand?	Requests/sec, throughput
Errors	How many requests fail?	Error rate / error count
Saturation	How full is the system?	CPU, memory, queue depth, connections

Monitor these four and you catch most problems. A related framework, RED (Rate, Errors, Duration), focuses on request-driven services; USE (Utilization, Saturation, Errors) focuses on resources.

Why Averages Lie: Percentiles

Average latency = 50ms sounds great. But:
  p50 = 20ms, p95 = 80ms, p99 = 2000ms
→ 1% of requests take 2 SECONDS. On a page with 100 requests, nearly
  EVERY page load hits at least one slow request. The average hid it.

Always monitor percentiles (p95, p99, p99.9), not just the mean.
Tail latency is what users experience and what compounds at scale.

Push vs Pull Collection

Model	How	Example
Pull	Monitoring system scrapes metrics endpoints	Prometheus scrapes `/metrics`
Push	Services push metrics to a collector	StatsD, some cloud agents

Pull (Prometheus) is common for its simplicity in service discovery and health (if scraping fails, the target may be down); push suits short-lived jobs that don't stick around to be scraped.

From Metrics to Alerts

Metrics power alerting — but alert on symptoms users feel, not every fluctuation:

✅ Alert: error rate > 2% for 5 min   (users are seeing failures)
✅ Alert: p99 latency > 1s for 5 min  (users are waiting)
❌ Alert: CPU > 80% (may be fine — alert on the USER impact, not the cause)

Good alerts are: actionable, symptom-based, and not noisy.
Alert fatigue (too many false alarms) is dangerous — people start
ignoring alerts, including the real one.

2D minimalistic diagram showing a latency distribution as a histogram with most requests clustered at low latency but a long tail stretching right; markers show p50, p95, and p99, with p99 far out in the tail highlighted and labeled "what users actually feel", contrasted with a misleadingly low average marker near p50

Seeing It in Action

Scenario: Monitoring a web API with the four golden signals.

Dashboard (the four golden signals):
  Latency:    p50=25ms  p95=90ms  p99=180ms   (chart over time)
  Traffic:    4,200 req/sec
  Errors:     0.3% error rate
  Saturation: CPU 45%, memory 60%, DB connections 30/100

Instrumentation (Prometheus-style counters/histograms):
  http_requests_total{method,status}      → counter (traffic + errors)
  http_request_duration_seconds{...}      → histogram (latency percentiles)
  db_connections_active                   → gauge (saturation)

An incident unfolds — metrics catch it first:
  10:00  p99 latency: 180ms → 1,400ms  (climbing)
  10:01  error rate:  0.3% → 4%          (breaching 2% threshold)
  10:01  DB connections: 30 → 100/100   (SATURATED)
  → Alert fires: "error rate > 2% for 2 min" → on-call paged.

  The metrics tell the STORY: latency and errors spiked exactly as DB
  connections saturated at 100/100 → the connection pool is exhausted.
  → Now use LOGS/TRACES to confirm which queries are holding connections,
    and METRICS confirm the fix worked (connections drop, latency recovers).

Why alert on error rate, not CPU:
  CPU alone might spike harmlessly. The error-rate + latency alerts fire
  only when USERS are actually affected — actionable and symptom-based.

Why metrics are the first line of monitoring: metrics are what tell you a problem exists and how severe it is, fast and cheaply, across the whole system at once. Because they're just numbers over time, you can chart every service's health on a dashboard, spot anomalies at a glance, and trigger alerts the moment user-facing symptoms (errors, latency) cross a threshold — often before customers complain. The four golden signals give you a principled, minimal set to watch, and percentiles ensure you see the tail latency that averages hide and that users actually feel. Crucially, metrics answer "is something wrong and how bad?" but not "why?" — so they work in concert with logs and traces: metrics detect and quantify the incident, then you drill into logs and traces to find the root cause, and finally watch the metrics recover to confirm the fix. And the discipline of alerting on symptoms rather than every internal fluctuation is what keeps alerts trustworthy — because an on-call engineer drowning in false alarms will eventually miss the one that matters.

Interview Questions

Q: What's the difference between metrics and logs, and when do you use each? Hint: Metrics are aggregate numbers sampled over time (rate, latency, errors, saturation) — cheap, efficient, low-cardinality, ideal for dashboards, trends, and alerting ("is something wrong, how bad?"). Logs are individual timestamped events — richer but expensive and high-cardinality, ideal for debugging the specifics ("what exactly happened, why?"). Metrics detect/quantify problems; logs (and traces) diagnose them. You need both.
Q: What are the four golden signals? Hint: Latency (how long requests take — track percentiles), Traffic (demand — requests/sec), Errors (rate of failed requests), and Saturation (how full the system is — CPU, memory, queue depth, connections). Monitoring these four catches most user-facing problems. Related frameworks: RED (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for resources.
Q: Why is average latency misleading, and what should you use instead? Hint: Averages hide the tail — a mean of 50ms can conceal a p99 of 2 seconds, meaning 1% of requests are very slow. On pages making many requests, nearly every user hits at least one slow request, so tail latency dominates the actual experience. Monitor percentiles (p95, p99, p99.9), which reflect what users feel and compound at scale.
Q: Name the metric types and what each is for. Hint: Counter (monotonically increasing — total requests/errors), Gauge (goes up and down — current memory, active connections, queue depth), Histogram (distribution across buckets — latency distribution), and Summary (like histogram with computed quantiles — p95/p99). Histograms/summaries are essential for latency because they capture the distribution and tail, not just an average.
Q: What makes a good alert, and what is alert fatigue? Hint: Good alerts are actionable, symptom-based (fire when users are actually affected — high error rate or latency — not on internal causes like CPU that may be harmless), and not noisy. Alert fatigue is when too many false/low-value alarms train people to ignore alerts, so they miss the real one. Alert on user-facing symptoms with sensible thresholds/durations to keep alerts trustworthy.

References

Google SRE Book — Monitoring Distributed Systems (Four Golden Signals) — the canonical guidance
Prometheus documentation — metric types and collection
Brendan Gregg: USE Method — resource-focused monitoring

Dive Deeper

The RED Method (Weaveworks) — request-focused metrics
Prometheus & Grafana in practice — dashboards and alerting
Latency: the tail at scale (Dean & Barroso) — why percentiles matter at scale