Health Checks

In a Nutshell

A health check is an endpoint or probe that reports whether a service instance is working correctly — a simple, automated way for the rest of your infrastructure to ask "are you okay?" Load balancers use them to route traffic only to healthy instances; orchestrators use them to restart or replace broken ones; monitoring uses them to alert. The crucial subtlety is distinguishing liveness (is the process alive?) from readiness (can it serve traffic right now?) — conflating them causes real outages, like killing a healthy service that's just warming up, or sending traffic to an instance that isn't ready. Health checks are the small, unglamorous mechanism that makes automated self-healing and load balancing actually work.

How It Actually Works

Liveness vs Readiness: The Critical Distinction

These answer different questions and trigger different actions — conflating them is a classic source of outages:

	Liveness	Readiness
Question	Is the process alive/not hung?	Can it serve traffic right now?
Failure action	Restart the instance	Remove from load-balancer pool (don't kill)
Fails when	Deadlock, unrecoverable crash	Starting up, warming cache, overloaded, dependency down
Recovery	A restart fixes it	Wait — it may recover on its own

Liveness fails  → the process is broken → RESTART it.
Readiness fails → the process is fine but not ready → STOP sending traffic,
                  but DON'T restart (restarting won't help and loses progress).

The classic bug: using a liveness check that also fails when a DEPENDENCY
is down → the orchestrator restarts a perfectly healthy service in a loop
because its database is briefly unavailable. Use readiness for dependencies.

Shallow vs Deep Health Checks

Type	Checks	Trade-off
Shallow	Just "is the process responding?" (returns 200)	Fast, cheap; misses deeper problems
Deep	Dependencies too (DB, cache, downstream reachable)	Thorough; risky — can cause cascading failures

The deep health-check danger: if every instance's health check verifies the database, and the database has a brief blip, every instance fails its check simultaneously → the load balancer marks them all unhealthy → the whole service is pulled out of rotation → total outage from a minor DB hiccup. This is why deep checks must be used carefully, often with separate handling for critical vs non-critical dependencies.

The Startup Problem

A newly-started instance often needs time before it can serve — loading config, warming caches, establishing connections, JIT compilation. Sending traffic too early causes errors:

Instance starts → NOT ready yet (cold caches, no DB pool)
  ❌ LB sends traffic immediately → requests fail/timeout
  ✅ Readiness probe fails until warm → LB waits → traffic only when ready

Kubernetes adds a startup probe for slow-starting apps, so liveness
checks don't kill an instance that's simply taking a while to boot.

What Makes a Good Health Check

• Fast and lightweight — checks run frequently; a heavy check adds load and can time out.
• Meaningful — a check that only confirms the web server is up (not the app logic) gives false confidence.
• Right dependencies — check critical dependencies for readiness, but avoid coupling all instances' health to a shared dependency (deep-check danger).
• Distinct endpoints — separate /livez, /readyz, /startupz (or equivalent) for their distinct purposes.
• Appropriate thresholds — require N consecutive failures before acting (hysteresis) so a single blip doesn't cause flapping.

Where Health Checks Are Used

Load balancer   → route only to instances passing READINESS (see load-balancing.md)
Orchestrator    → restart instances failing LIVENESS; hold traffic on readiness fail
Service mesh    → route around unhealthy instances
Monitoring      → alert when instances go unhealthy
Auto-scaling    → replace instances that stay unhealthy

Seeing It in Action

Scenario: Kubernetes probes for a web service, done right.

containers:
- name: web
  # STARTUP probe: give a slow-starting app time before liveness kicks in
  startupProbe:
    httpGet: { path: /startupz, port: 8080 }
    failureThreshold: 30       # allow up to 30×2s = 60s to start
    periodSeconds: 2

  # LIVENESS: is the process itself healthy? (NOT dependencies)
  livenessProbe:
    httpGet: { path: /livez, port: 8080 }   # returns 200 if the app loop is alive
    periodSeconds: 10
    failureThreshold: 3        # 3 consecutive fails → RESTART
    # /livez does NOT check the database — a DB blip must not restart us

  # READINESS: can we serve traffic right now? (checks critical deps)
  readinessProbe:
    httpGet: { path: /readyz, port: 8080 }   # 200 only if warm + DB reachable
    periodSeconds: 5
    failureThreshold: 2        # 2 fails → REMOVE from Service endpoints (no restart)

# /livez — shallow: is the process responsive? (never checks dependencies)
@app.route("/livez")
def livez(): return "ok", 200

# /readyz — checks readiness, including critical dependencies
@app.route("/readyz")
def readyz():
    if not cache_warmed:           return "warming", 503   # still starting
    if not db.can_connect():       return "db down", 503   # can't serve → pull from LB
    return "ready", 200            # warm + deps ok → receive traffic

What each probe correctly handles:

• Startup: the app gets up to 60s to boot without liveness prematurely killing it — no more "restart loop because the app is slow to start."
• Liveness on a hung process: if the app deadlocks, /livez stops responding → 3 fails → Kubernetes restarts it → recovered automatically.
• Readiness during warmup: while caches are cold, /readyz returns 503 → the instance is kept out of the load-balancer pool → no traffic hits a cold instance → no user-facing errors during rollout.
• Readiness on a DB blip: if the database briefly drops, /readyz fails and the instance stops receiving traffic — but /livez still passes, so it is not restarted. When the DB recovers, /readyz passes again and traffic resumes. The instance was never needlessly killed.

Why the liveness/readiness split is the whole point: the two most damaging health-check mistakes are opposite failures of this distinction. If you make liveness too deep (checking dependencies), a transient database blip makes every instance fail liveness and get restarted in a loop — turning a minor hiccup into a self-inflicted outage. If you make readiness too shallow (just "process up"), you route traffic to instances that are still warming or whose dependencies are down, causing user-facing errors during every deploy. The correct model keeps liveness shallow and about the process itself (restart only truly-broken instances) and readiness meaningful about the ability to serve right now (pull, don't kill, instances that are warming or temporarily can't reach a dependency). Add a startup probe for slow boots and hysteresis to prevent flapping, and health checks become the quiet, reliable foundation that lets load balancers route intelligently and orchestrators self-heal — without the automation itself becoming the cause of outages.

Interview Questions

1. Q: What's the difference between a liveness and a readiness check? Hint: Liveness asks "is the process alive/not hung?" — failure triggers a restart (the instance is broken). Readiness asks "can it serve traffic right now?" — failure triggers removal from the load-balancer pool without a restart (the instance is fine but starting up, warming, overloaded, or has a temporarily-unreachable dependency). Conflating them causes outages: restarting healthy instances or routing to unready ones.

2. Q: Why can a liveness check that verifies dependencies cause an outage? Hint: If liveness fails when a dependency (e.g., the database) is down, a brief DB blip makes every instance fail liveness simultaneously → the orchestrator restarts them all in a loop, even though the instances themselves are healthy. Restarting doesn't fix a downstream dependency, so you turn a minor hiccup into a self-inflicted outage. Dependency health belongs in readiness (pull from traffic), not liveness (restart).

3. Q: What is the danger of deep health checks, and how do you mitigate it? Hint: If every instance's health check verifies a shared dependency (DB), a brief dependency blip makes all instances fail at once → the LB marks them all unhealthy → the whole service is pulled from rotation → total outage from a minor issue. Mitigate by keeping liveness shallow, being careful which dependencies gate readiness, distinguishing critical vs non-critical dependencies, and using hysteresis so a single blip doesn't flap everything.

4. Q: Why do newly-started instances need special handling, and how? Hint: A fresh instance often needs time to load config, warm caches, establish connection pools, and JIT-compile before it can serve — sending traffic too early causes errors. Use a readiness probe that fails until the instance is warm (so the LB withholds traffic) and a startup probe for slow-booting apps so the liveness check doesn't prematurely kill an instance that's simply still starting.

5. Q: What makes a health check well-designed? Hint: Fast/lightweight (runs frequently without adding load or timing out), meaningful (verifies actual app health, not just that the web server responds), correct dependency scope (readiness checks critical deps but avoids coupling all instances to a shared dependency), separate endpoints for liveness/readiness/startup, and appropriate thresholds (N consecutive failures / hysteresis) to prevent flapping.

References

• Kubernetes: Liveness, Readiness, and Startup Probes — the canonical model
• Google SRE Book — Health checking — health checks in reliable systems
• AWS: Health checks for load balancers — LB-driven health

Dive Deeper

• Google SRE: Load balancing and health checking pitfalls — deep-check dangers at scale
• Kubernetes probe best practices — common mistakes and fixes
• Health check patterns (microservices.io) — the health check API pattern