Performance and Scaling

Sign in to save your progress across devices. Until then it is kept in this browser.

Why this matters

Two opposite mistakes kill products: shipping something that falls over at 100 users, and over-building for a million users you do not have. This module teaches you to measure first, then apply the cheapest fix, in a known order.

Learning outcomes

1. Find bottlenecks with load tests and profiling instead of guessing.
2. Apply the scaling ladder in the right order.
3. Cache correctly, including invalidation.
4. Understand load balancers, reverse proxies and API gateways.
5. Know serverless limits and cold starts.
6. Control cloud and AI cost as a first-class metric.

The scaling ladder (climb only when measured)

1. Fix the code: indexes, N+1, pagination, payload size (03 · Data Foundations).
2. Cache: CDN for static, edge cache for public dynamic, app cache for expensive computations.
3. Move work async (08 · Async, Events and Real-Time).
4. Vertical scale (47): bigger database/server. Cheap in engineering time; goes further than people think.
5. Horizontal scale stateless app tier (46) behind a load balancer, with autoscaling (45).
6. Read replicas (29) for read-heavy load (accept replication lag).
7. Partition (31) big tables (e.g. by month for events/logs).
8. Shard (30) only when one primary database truly cannot cope. Most companies never need this.

Lessons

9.1 Caching (2, 48, 49, 50)

• Layers: browser (Cache-Control), CDN for static assets (48), edge caching of dynamic responses (49), application cache (Redis), DB query cache.
• Patterns: cache-aside (read through app), write-through, stale-while-revalidate.
• Invalidation (50): TTLs (simple, eventually fresh), explicit purge on write (tag-based purges), versioned keys (user:42:v7).
• Never cache per-user private data in a shared cache under a shared key (data leak). Include user/tenant in the key or mark Cache-Control: private.
• Prevent stampedes: request coalescing, jittered TTLs, early refresh.
• LLM-specific: use provider prompt caching for long static prefixes; cache deterministic results (embeddings, classifications).

9.2 Front-door components (3, 4, 5)

• Reverse proxy (4): terminates TLS, compresses, buffers slow clients, routes (Nginx, Caddy, Cloudflare, platform edge).
• Load balancer (3): spreads traffic across instances using health checks; L4 (TCP) or L7 (HTTP). Sticky sessions are a smell; keep app servers stateless.
• API gateway (5): auth, quotas, rate limits, routing, API keys for many services. Useful once you have several services or sell API access.

9.3 Serverless realities (89, 90)

• Limits: max execution time, memory, payload size, concurrent executions, and DB connection pressure. Know your platform's numbers.
• Cold starts: first request after idle is slow. Mitigate with smaller bundles, lighter runtimes (edge), provisioned/min instances for latency-critical paths.
• Long-running agent work does not belong in a request-bound function; use durable jobs.

9.4 Cost optimisation (88)

• Treat cost per active user and cost per AI task as product metrics on the dashboard.
• Biggest wins: right-sized instances, killing idle resources, caching LLM results, routing simple tasks to smaller/cheaper models, batching, storage lifecycle rules, egress awareness.
• Hard spend caps and anomaly alerts on every provider. Tag resources by product/venture for attribution.

9.5 Load testing method

Define the scenario (realistic user journey, think time, data volume) → ramp (smoke, load, stress, soak) → watch p95/p99, error rate, saturation (CPU, memory, DB connections, queue depth) → find the first thing that breaks → fix → repeat.

Lab 9: Find the cliff

1. Write a k6 scenario for your main journey; ramp until error rate exceeds 1% or p95 breaks budget. Record the "cliff."
2. Profile and fix the top bottleneck. Repeat twice. Chart capacity gain per fix.
3. Add CDN + edge caching for one public dynamic page with correct invalidation on update.
4. Build a cost dashboard: monthly infra + AI cost per active user.

Definition of Done

• Documented capacity: "handles N concurrent users at p95 under X ms"Evidence
• Three fixes with before/after measurementsEvidence
• Caching with explicit invalidation strategy; no private data in shared cachesEvidence
• Serverless limits documented for your platform; long work moved off-requestEvidence
• Cost per user and per AI task visible; caps and anomaly alerts setEvidence
• ADR: "what we will do at 10x and 100x, and what we deliberately will not build yet"Evidence

AI prompt pack

• "Given this k6 output and these DB stats, rank the likely bottlenecks and propose the cheapest fix for each."
• "Add caching to this endpoint. Include the cache key design (with tenant), TTL, invalidation on write, and stampede protection."