Distributed Systems Correctness

Sign in to save your progress across devices. Until then it is kept in this browser.

Why this matters

Concurrency bugs do not show up in demos. They show up as a customer paid twice, a voucher redeemed three times, a seat sold to two people, or a balance that does not add up at month end. In fintech, these are regulatory events, not just bugs.

Learning outcomes

1. Recognise race conditions and fix them with atomic operations, constraints and locks.
2. Understand replication, CAP and eventual consistency well enough to design UX around them.
3. Know when distributed locks and leader election are needed, and their pitfalls.
4. Avoid deadlocks.
5. Handle time correctly despite clock skew.
6. Diagnose memory leaks and garbage-collection pauses.

Lessons

10.1 Race conditions (39, 43)

The classic shape is check-then-act: read a value, decide, write. Between read and write, someone else acts. Fixes, in order of preference:

1. Atomic statement: UPDATE accounts SET balance = balance - 100 WHERE id = $1 AND balance >= 100 and check rows affected.
2. Database constraint: unique index makes the second insert fail.
3. Optimistic locking (version column) for low contention.
4. Pessimistic lock (SELECT ... FOR UPDATE) for high contention. In app code: shared mutable state in a long-lived Node process is shared across requests; module-level variables are not per-user. Thread safety (43) matters in Go, Java, Python async and workers too.

10.2 Deadlocks (40)

Two transactions each hold a lock the other needs. Prevent by always locking resources in a consistent order (e.g. by ascending ID), keeping transactions short, and setting lock timeouts. Postgres detects deadlocks and aborts one; your code must retry.

10.3 Replication, CAP and eventual consistency (32, 34, 35, 95)

• Replication (32): copies on multiple nodes for durability and read scale. Async replicas lag.
• Network partitions (95) happen. During one, CAP says a system must choose: refuse some requests (consistency) or answer with possibly stale data (availability).
• Eventual consistency (35): replicas converge over time. Design implications: read-your-own-writes (route a user's reads to the primary right after they write), show "processing" states, never decide money on a lagging replica.
• Ledgers and balances: strongly consistent, single source of truth, append-only double-entry records.

10.4 Distributed locks and leader election (33, 38)

• Needed when multiple instances must not do the same thing at once (one cron runner, one consumer of a partition).
• Prefer DB-native options first: Postgres advisory locks, SKIP LOCKED job queues.
• Redis locks need expiry (TTL) and fencing tokens: a paused process may wake up after its lock expired and still write. The storage layer must reject stale tokens.
• Leader election (Raft, etcd, ZooKeeper, Kubernetes leases) is mostly something you consume from infrastructure, not build.

10.5 Time and clock skew (96)

• Machine clocks drift; never use wall-clock time from different servers to order events.
• Use DB-generated timestamps or sequence numbers for ordering; monotonic clocks for measuring durations.
• Token expiry and webhook timestamp checks need a small skew tolerance.

10.6 Memory leaks and garbage collection (41, 42)

• Leaks in managed languages are usually references you keep: growing caches without limits, event listeners never removed, global arrays, unclosed streams.
• Symptoms: memory climbs steadily, then OOM restarts; latency spikes from long GC pauses.
• Tools: heap snapshots (Chrome DevTools for Node), --inspect, memory metrics per instance, bounded LRU caches.

Lab 10: The double-spend challenge

1. Build a wallet: top-up, transfer, withdraw. Write a test firing 200 concurrent transfers. Watch the naive AI version lose or create money.
2. Fix with atomic updates and constraints; implement as a double-entry ledger where the sum of all entries is always zero.
3. Induce a deadlock with two opposite-order transfers; fix with lock ordering.
4. Read from a lagging replica (simulated delay) and show the read-your-writes bug; fix.
5. Find a planted memory leak with heap snapshots.

Definition of Done

• Concurrency test suite (200 parallel ops) passes 100 consecutive runsEvidence
• Ledger invariant (sum = 0) checked by a scheduled reconciliation jobEvidence
• Lock ordering documented; deadlock retry implementedEvidence
• Replica-lag strategy written (which reads may be stale, which may not)Evidence
• Memory dashboard per instance; no unbounded cachesEvidence

AI prompt pack

• "Find every check-then-act sequence in this codebase. For each, show the race and the atomic fix."
• "Rewrite this balance logic as a double-entry ledger with an invariant test."