System design

Building blocks, numbers and trade-offs for designing distributed systems, plus a step-by-step framework and a worked design. Protocol details live in Communication networks; code-level structure in Software architecture.

Approach framework

Availability targets and their yearly downtime budget:

Serial dependencies multiply: two 99.9% services in a chain give about 99.8%.

Back-of-envelope numbers

Order-of-magnitude latencies on current hardware (Jeff Dean's list, updated). See Latency and bandwidth for network detail.

Estimation formulas

Example: 10M DAU × 20 reads = 200M reads/day ≈ 2,300 QPS avg, ~7,000 peak. At 2 KB per response that is ~14 MB/s egress. With 5M writes/day × 1 KB × 365 × 3 replicas ≈ 5.5 TB/year.

Little's law for concurrency: in-flight requests = arrival rate × latency. 1,000 req/s at 200 ms means 200 concurrent requests (and connections, threads or DB sessions).

Scaling & load balancing

Stateless services keep session, uploads and caches out of process memory (Redis, DB, object store, signed tokens) so any instance can serve any request, instances can be killed at will, and autoscaling works.

Health checks: active (LB probes /healthz) plus passive (eject on errors). Separate liveness (restart me) from readiness (send me traffic).

Caching & CDNs

interface Cache {
  get(key: string): Promise<string | null>;
  set(key: string, v: string, ttlS: number): Promise<void>;
}
 
const inFlight = new Map<string, Promise<string>>();
 
export async function cached(
  cache: Cache,
  key: string,
  ttlS: number,
  load: () => Promise<string>,
): Promise<string> {
  const hit = await cache.get(key);
  if (hit !== null) return hit;
  const pending = inFlight.get(key);
  if (pending) return pending;            // coalesce
  const p = load()
    .then(async (v) => {
      const jitter = 0.9 + Math.random() * 0.2;
      await cache.set(key, v, Math.round(ttlS * jitter));
      return v;
    })
    .finally(() => inFlight.delete(key));
  inFlight.set(key, p);
  return p;
}

CDNs

Databases

Default to Postgres until a measured need says otherwise. See Postgres and Database design.

Replication

Read-your-writes with async replicas: route a user's reads to the leader for a few seconds after they write, or pass the commit LSN and wait for the replica to catch up.

Sharding (partitioning)

Pick a shard key that is in almost every query, has high cardinality, and spreads writes (tenant_id, user_id). Cross-shard joins and transactions become application work.

Indexes

Consistency

CAP: during a network partition a replicated store must choose consistency (refuse some requests) or availability (serve possibly stale data). PACELC adds: else (no partition), choose latency or consistency.

Distributed transactions: avoid 2PC across services; use a saga (local transactions plus compensating actions) orchestrated or choreographed by events.

Queues & streams

Idempotent consumers: dedupe by message ID in a processed-messages table inside the same transaction as the side effect, or make the write naturally idempotent (upsert, set not increment). Failed messages go to a dead-letter queue after N attempts.

Transactional outbox

Write the business row and an outbox row in one DB transaction; a relay (poller or CDC like Debezium) publishes outbox rows and marks them sent. No dual-write gap.

BEGIN;
INSERT INTO orders (id, user_id, total)
  VALUES ($1, $2, $3);
INSERT INTO outbox (id, topic, payload)
  VALUES (gen_random_uuid(), 'order.created',
          jsonb_build_object('orderId', $1));
COMMIT;
 
-- relay, run in a loop (multiple relays safe)
SELECT id, topic, payload FROM outbox
WHERE sent_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;

Rate limiting

Where: at the edge/gateway per IP and per API key, again per user in the service. Return 429 Too Many Requests with Retry-After and RateLimit headers. Distributed limits need a shared store (Redis with a Lua script) or accept per-node approximations.

export class TokenBucket {
  private tokens: number;
  private last = performance.now();
 
  constructor(
    private readonly capacity: number,
    private readonly perSecond: number,
  ) {
    this.tokens = capacity;
  }
 
  take(cost = 1): boolean {
    const now = performance.now();
    const elapsed = (now - this.last) / 1000;
    this.tokens = Math.min(
      this.capacity,
      this.tokens + elapsed * this.perSecond,
    );
    this.last = now;
    if (this.tokens < cost) return false;
    this.tokens -= cost;
    return true;
  }
}

Reliability

const sleep = (ms: number) =>
  new Promise<void>((r) => setTimeout(r, ms));
 
export async function retry<T>(
  fn: (signal: AbortSignal) => Promise<T>,
  { attempts = 5, baseMs = 100, capMs = 5_000,
    timeoutMs = 2_000 } = {},
): Promise<T> {
  for (let i = 0; ; i++) {
    try {
      return await fn(AbortSignal.timeout(timeoutMs));
    } catch (err) {
      if (i + 1 >= attempts) throw err;
      const max = Math.min(capMs, baseMs * 2 ** i);
      await sleep(Math.random() * max);   // full jitter
    }
  }
}

Observability

Alert on symptoms (SLO burn rate), not causes; track p50/p95/p99 not averages. Setup details in Grafana.

IDs

Prefer time-ordered IDs for primary keys (index locality) and don't expose sequential IDs where enumeration is a risk.

Search & blob storage

Keep the database as the source of truth and feed the index via CDC or the outbox; the index is a rebuildable projection with eventual consistency.

Trade-off cheat table

Worked design: URL shortener

1. Requirements

Functional: create a short link for a long URL (optional custom alias, expiry), redirect, basic click counts. Non-functional: redirects p99 under 50 ms, 99.99% available for reads, links never collide, created links durable.

2. Estimates

3. API

POST /api/links
Idempotency-Key: 5f3c...
Content-Type: application/json
 
{"url": "https://example.com/very/long", "alias": null}
 
201 Created
{"code": "aZ3kP9q", "shortUrl": "https://sho.rt/aZ3kP9q"}
 
GET /aZ3kP9q
302 Found
Location: https://example.com/very/long

4. Data model

CREATE TABLE links (
  code        text PRIMARY KEY,       -- base62, 7 chars
  url         text NOT NULL,
  owner_id    uuid,
  created_at  timestamptz NOT NULL DEFAULT now(),
  expires_at  timestamptz
);
CREATE INDEX ON links (owner_id, created_at DESC);

Access pattern is a point lookup by code: any store works; Postgres (with read replicas) or a key-value store like DynamoDB both fit. Click events go elsewhere.

5. High-level design

client ─► CDN/edge ─► L7 LB ─► redirect service ─► Redis ─► DB replicas
                                   │ (miss)
                                   └─► click event ─► queue ─► analytics store
client ─► L7 LB ─► link API ─► ID generator ─► DB primary

6. Deep dives

7. Trade-offs

Pre-allocated counter blocks leave gaps and need coordination on startup, but avoid a hot central counter; random codes avoid coordination but cost a uniqueness check. Eventual consistency between primary and replicas means a just-created link may 404 for a moment on a replica: read from the primary on cache miss for links younger than a few seconds.

References

• Designing Data-Intensive Applications (2nd ed.) (opens in a new tab): replication, partitioning, transactions, streams, in depth
• The System Design Primer (opens in a new tab): broad topic index with diagrams
• Google SRE book (opens in a new tab): SLOs, error budgets, load balancing, overload handling
• Google SRE workbook: alerting on SLOs (opens in a new tab): burn-rate alerts
• Amazon Builders' Library (opens in a new tab): timeouts, retries and backoff with jitter, load shedding, caching
• AWS: exponential backoff and jitter (opens in a new tab): why full jitter
• Latency numbers every programmer should know (opens in a new tab): Jeff Dean's numbers by year
• RFC 9562: UUIDs (opens in a new tab): UUIDv7 layout
• Jepsen consistency models (opens in a new tab): precise definitions and hierarchy
• microservices.io patterns (opens in a new tab): outbox, saga, CQRS