Skip to content
Shubham JangleResources
Scalability & reliability

Explore topics

CDN vs Load Balancer, Vertical vs Horizontal Scaling, Rate Limiting vs Throttling

CDN vs Load Balancer, Vertical vs Horizontal Scaling, Rate Limiting vs Throttling

  • Three pairs of concepts that come up in almost every system design interview, all about handling more traffic.
User → CDN (edge cache) → Rate limiter / API gateway → Load balancer → App servers (scaled horizontally)

CDN vs Load Balancer

CDN (Content Delivery Network)

  • A CDN is a network of servers distributed across many geographic regions, called edge locations or Points of Presence (PoPs).
  • Imagine watching Netflix in India. Instead of streaming every movie from a server in the US, Netflix delivers cached content from a nearby server.
  • How it works:
    1. The user's request goes to the nearest edge server.
    2. Cache hit: if the edge already has the file, it responds immediately.
    3. Cache miss: the edge fetches it from your origin server, returns it, and stores a copy for the next user.
  • How long content stays cached is controlled by headers like Cache-Control: max-age=86400. When content changes, you invalidate the cache or use versioned file names like app.3f9a1c.js.
  • Benefits:
    • Lower latency: content travels a shorter distance.
    • Faster page loads, which improves user experience and Core Web Vitals, and can indirectly help SEO.
    • Less load on the origin, since most requests never reach it.
    • Absorbs traffic spikes and helps protect against DDoS attacks.
  • Best for static content: images, videos, CSS, JavaScript, and fonts. Many CDNs can also speed up dynamic requests through optimized routes and run code at the edge.
  • Examples: Cloudflare, Amazon CloudFront, Akamai, Fastly.

Load Balancer

  • A load balancer distributes incoming requests across multiple backend servers so no single server gets overloaded.
  • For example, if your application has three servers, the load balancer spreads requests among them.
  • Health checks: the load balancer regularly checks each server and stops sending traffic to unhealthy ones. This is what improves availability.
  • Common algorithms:
    • Round Robin: sends requests to servers in turn: 1, 2, 3, 1, 2, 3. Weighted round robin gives bigger servers more requests.
    • Least Connections: sends each request to the server with the fewest active connections. Good when requests take varying amounts of time.
    • IP Hashing: hashes the client's IP so the same client keeps reaching the same server (sticky sessions).
    • Consistent Hashing: maps clients or keys onto a hash ring, so adding or removing a server only moves a small fraction of keys. Common for caches and stateful services.
  • Layer 4 vs Layer 7: an L4 load balancer routes by IP address and port (fast, protocol-agnostic). An L7 load balancer reads HTTP data and can route by path, host, headers, or cookies.
  • Examples: AWS Application Load Balancer and Network Load Balancer, NGINX, HAProxy.
  • The load balancer itself must not become a single point of failure, so it's typically run as a redundant pair or managed service.

CDN vs Load Balancer comparison

CDNLoad Balancer
Main jobServe cached content close to usersSpread requests across servers
LocationEdge servers worldwideIn front of your servers, in your region
ImprovesLatency, origin loadAvailability, scalability, utilization
Typical contentStatic files, mediaDynamic application requests
  • They're complementary: a large system often uses a CDN at the edge and load balancers in each region.

Vertical vs Horizontal Scaling

  • Imagine traffic grows and your server can no longer handle the load.

Vertical scaling (scale up)

  • Increase the capacity of the existing server: more CPU, RAM, or faster storage.
  • Pros: simple, with no code or architecture changes.
  • Cons:
    • There's a hardware ceiling; you can't upgrade forever.
    • Large machines get disproportionately expensive.
    • It remains a single point of failure.
    • Upgrading often requires a restart and downtime.

Horizontal scaling (scale out)

  • Add more servers and distribute the workload across them with a load balancer.
  • Pros:
    • Practically no upper limit.
    • Fault tolerant: if one server fails, others keep serving.
    • Supports auto-scaling: add servers at peak and remove them when traffic drops.
  • Cons:
    • More moving parts to deploy and monitor.
    • Servers should be stateless: store sessions in a shared store like Redis instead of server memory, so any server can handle any request.
    • Scaling the database horizontally is much harder, and needs read replicas, caching, or sharding.
VerticalHorizontal
HowBigger machineMore machines
LimitHardware ceilingPractically unlimited
Failure impactSingle point of failureSurvives individual failures
ComplexityLowHigher: load balancing, stateless design
  • In practice, teams often scale vertically first because it's simple, then scale horizontally as traffic and availability needs grow.

Rate Limiting vs Throttling

  • Imagine your API is receiving thousands of requests, some from a buggy client stuck in a loop.

Rate limiting

  • Rate limiting sets the maximum number of requests a client can make in a time window, identified by user ID, API key, or IP address.
  • For example, with a limit of 100 requests per minute, extra requests are rejected with 429 Too Many Requests, often with a Retry-After header telling the client when to try again.
  • It protects against abuse, brute-force login attempts, scraping, and one client starving others, and it's often used to enforce pricing tiers.
  • Common algorithms:
    • Fixed window: count requests per calendar minute. Simple, but allows bursts at window edges.
    • Sliding window: counts over the last 60 seconds at any moment, smoothing those edges.
    • Token bucket: tokens refill at a steady rate, and each request spends one. Allows short bursts up to the bucket size.
    • Leaky bucket: requests drain from a queue at a fixed rate, producing a smooth output.
  • With many servers, counters are kept in a shared store like Redis so limits apply across the whole fleet.

Throttling

  • Throttling controls the rate at which requests are processed, rather than only counting requests per client.
  • When the system is under heavy load, requests can be delayed, queued, or rejected to keep processing at a manageable rate. For example, a payment processor might handle at most 500 transactions per second and queue the rest.
  • The terms overlap in practice: some platforms, like AWS API Gateway, call per-client request limits "throttling".
  • Related: JavaScript throttling applies the same idea on the client, limiting how often a function runs.
Rate LimitingThrottling
ControlsHow many requests a client may sendHow fast requests are processed
Typical scopePer user, API key, or IPSystem or service capacity
Excess requestsRejected with 429Delayed, queued, or rejected
GoalFairness, abuse preventionProtect capacity under load

Takeaway

  • CDN: caches content near users. Load Balancer: spreads requests across servers.
  • Vertical scaling: a bigger server. Horizontal scaling: more servers.
  • Rate limiting: caps requests per client. Throttling: controls the processing rate under load.

References