Idempotency

In a Nutshell

An operation is idempotent if performing it multiple times has the same effect as performing it once. This sounds academic but is one of the most practically important properties in distributed systems, because networks are unreliable: requests time out, retries happen, and messages get delivered more than once. If charging a credit card is idempotent, then a client that retries after a timeout won't double-charge the customer. Idempotency is what makes safe retries possible — and since at-least-once delivery (the common default in messaging) guarantees occasional duplicates, idempotent consumers aren't a nice-to-have, they're a requirement for correctness.

How It Actually Works

Why Duplicates Are Inevitable

Client → charge $50 → Server processes it → response LOST in network
Client sees a timeout → retries → charge $50 AGAIN → 💥 double charge

The server DID the work; the client just never heard back.
The client can't tell "it failed" from "the ack was lost."
So it retries — and without idempotency, that's a duplicate charge.

This is unavoidable in any networked system. At-least-once messaging makes it explicit: the ack mechanism redelivers a message if a consumer crashes before acknowledging, so the same message will sometimes be processed twice.

Naturally Idempotent vs Not

Some operations are idempotent by nature; others need to be made idempotent:

Operation	Idempotent?	Why
SET balance = 100	Yes	Same result no matter how many times
balance = balance + 50	No	Each execution adds again
HTTP GET, PUT, DELETE	Yes	By spec (see HTTP & HTTPS)
HTTP POST (create)	No	Each call creates a new resource
"Set status to shipped"	Yes	Terminal state; re-setting is a no-op
"Send an email"	No	Each call sends another email

The general test: does re-running it change the state further, or produce another side effect? If yes, it's not idempotent and needs protection.

Techniques to Achieve Idempotency

1. Idempotency keys. The client generates a unique key per logical operation and sends it with every retry. The server records processed keys and ignores duplicates.

POST /charges
Idempotency-Key: a1b2c3-unique-per-operation

Server logic:
  if key in processed_keys:
      return stored_response(key)      # duplicate → return original result
  result = do_charge()
  processed_keys.store(key, result)    # atomically with the work
  return result

This is how Stripe, PayPal, and most payment APIs make charges safe to retry.

2. Deduplication by message ID. Consumers track processed message IDs (in a DB/cache) and skip any they've already handled.

3. Natural idempotency via state design. Design operations to be absorbing: "set state = X" instead of "increment" or "append." Upserts (INSERT ... ON CONFLICT DO NOTHING/UPDATE) instead of blind inserts.

4. Conditional updates / optimistic concurrency. Use a version or expected-state check so a repeated operation is a no-op:

UPDATE orders SET status = 'shipped'
WHERE id = 42 AND status = 'pending';   -- second run affects 0 rows (no-op)

The Atomicity Catch

Idempotency keys only work if recording the key and doing the work are atomic — otherwise a crash between them reopens the duplicate window:

❌ do_work(); then store_key();   → crash in between → retry re-does work
✅ Store the key and the work's effect in ONE transaction, OR use the
   key as a unique constraint so the duplicate insert fails cleanly.

Exactly-Once = At-Least-Once + Idempotency

True "exactly-once delivery" is famously hard/impossible in general. The practical equivalent is at-least-once delivery + idempotent processing: the system may deliver a message multiple times, but because processing is idempotent, the effect is exactly once. This is the standard, achievable pattern.

Seeing It in Action

Scenario: Idempotent payment processing from a message queue.

def handle_payment_message(msg):
    # msg may be delivered MORE THAN ONCE (at-least-once queue).
    idem_key = msg["payment_id"]        # stable per logical payment

    with db.transaction():              # atomicity is essential
        # Try to claim this payment_id via a UNIQUE constraint.
        try:
            db.execute(
                "INSERT INTO processed_payments (payment_id, status) "
                "VALUES (%s, 'processing')", idem_key)
        except UniqueViolation:
            # Already processed (or in-flight) → skip. No double charge.
            return ack(msg)

        # First time we've seen this payment → do the real work.
        charge_result = payment_gateway.charge(
            amount=msg["amount"], token=msg["token"],
            idempotency_key=idem_key)   # gateway is ALSO idempotent (defense in depth)

        db.execute("UPDATE processed_payments SET status='done', "
                   "result=%s WHERE payment_id=%s", charge_result, idem_key)

    ack(msg)                            # ack only after the tx commits

Why every layer matters here: the queue redelivers on consumer crashes (at-least-once), so the same payment_id can arrive twice. The UNIQUE constraint on payment_id makes claiming it atomic — the second delivery's insert fails cleanly and is skipped, so the customer is charged once. The payment gateway also takes an idempotency key (defense in depth: even if our dedup somehow failed, the gateway wouldn't double-charge). And we ack only after the transaction commits, so a crash mid-processing safely redelivers. This layered idempotency is the difference between a payment system you can trust and one that occasionally double-charges customers.

Interview Questions

1. Q: What does idempotent mean, and why is it critical in distributed systems? Hint: An operation is idempotent if doing it multiple times has the same effect as doing it once. It's critical because networks are unreliable — timeouts and retries are unavoidable, and at-least-once messaging guarantees occasional duplicate deliveries. Idempotency makes retries safe (no double charges, no duplicate side effects), which is the only way to get correctness with retries and duplicates.

2. Q: Why can't a client just avoid retrying to prevent duplicates? Hint: Because the client can't distinguish "the request failed" from "the request succeeded but the response was lost." On a timeout, not retrying risks losing a legitimately-failed operation; retrying risks a duplicate. Since you must retry for reliability, the correct fix is to make the operation idempotent so retries are safe — rather than trying to guarantee exactly-one delivery, which is impractical.

3. Q: How does an idempotency key work? Hint: The client generates a unique key per logical operation and includes it on every retry. The server checks whether it has already processed that key: if yes, it returns the stored original result without redoing the work; if no, it performs the work and records the key+result atomically. Repeated requests with the same key produce one effect. Used by Stripe/PayPal for safe payment retries.

4. Q: Why must recording the idempotency key and doing the work be atomic? Hint: If they're separate, a crash between doing the work and recording the key (or vice versa) reopens the duplicate window — a retry re-does the work, or a done operation isn't recorded. Make them one transaction, or use the key as a unique constraint so a duplicate attempt fails cleanly. Also ack the message only after the transaction commits.

5. Q: How do you achieve "exactly-once" processing in practice? Hint: True exactly-once delivery is generally impractical. The achievable pattern is at-least-once delivery + idempotent processing: the system may deliver a message multiple times, but idempotent handling (dedup by ID/key, upserts, conditional updates) ensures the effect is exactly once. So you design consumers to tolerate duplicates rather than trying to prevent them at the transport layer.

References

• Stripe: Idempotent Requests — the canonical idempotency-key API design
• AWS Builders' Library: Making retries safe with idempotent APIs — patterns and pitfalls
• Designing Data-Intensive Applications by Martin Kleppmann — exactly-once semantics

Dive Deeper

• You Cannot Have Exactly-Once Delivery (Brave New Geek) — why idempotency is the real answer
• Exactly-once semantics in Kafka — how a system approximates it
• Idempotency patterns (Microsoft) — practical implementation guidance