CDN vs Load Balancer, Vertical vs Horizontal Scaling, Rate Limiting vs Throttling
CDN vs Load Balancer, Vertical vs Horizontal Scaling, Rate Limiting vs Throttling
- Three pairs of concepts that come up in almost every system design interview, all about handling more traffic.
User → CDN (edge cache) → Rate limiter / API gateway → Load balancer → App servers (scaled horizontally)
CDN vs Load Balancer
CDN (Content Delivery Network)
- A CDN is a network of servers distributed across many geographic regions, called edge locations or Points of Presence (PoPs).
- Imagine watching Netflix in India. Instead of streaming every movie from a server in the US, Netflix delivers cached content from a nearby server.
- How it works:
- The user's request goes to the nearest edge server.
- Cache hit: if the edge already has the file, it responds immediately.
- Cache miss: the edge fetches it from your origin server, returns it, and stores a copy for the next user.
- How long content stays cached is controlled by headers like
Cache-Control: max-age=86400. When content changes, you invalidate the cache or use versioned file names like app.3f9a1c.js.
- Benefits:
- Lower latency: content travels a shorter distance.
- Faster page loads, which improves user experience and Core Web Vitals, and can indirectly help SEO.
- Less load on the origin, since most requests never reach it.
- Absorbs traffic spikes and helps protect against DDoS attacks.
- Best for static content: images, videos, CSS, JavaScript, and fonts. Many CDNs can also speed up dynamic requests through optimized routes and run code at the edge.
- Examples: Cloudflare, Amazon CloudFront, Akamai, Fastly.
Load Balancer
- A load balancer distributes incoming requests across multiple backend servers so no single server gets overloaded.
- For example, if your application has three servers, the load balancer spreads requests among them.
- Health checks: the load balancer regularly checks each server and stops sending traffic to unhealthy ones. This is what improves availability.
- Common algorithms:
- Round Robin: sends requests to servers in turn: 1, 2, 3, 1, 2, 3. Weighted round robin gives bigger servers more requests.
- Least Connections: sends each request to the server with the fewest active connections. Good when requests take varying amounts of time.
- IP Hashing: hashes the client's IP so the same client keeps reaching the same server (sticky sessions).
- Consistent Hashing: maps clients or keys onto a hash ring, so adding or removing a server only moves a small fraction of keys. Common for caches and stateful services.
- Layer 4 vs Layer 7: an L4 load balancer routes by IP address and port (fast, protocol-agnostic). An L7 load balancer reads HTTP data and can route by path, host, headers, or cookies.
- Examples: AWS Application Load Balancer and Network Load Balancer, NGINX, HAProxy.
- The load balancer itself must not become a single point of failure, so it's typically run as a redundant pair or managed service.
CDN vs Load Balancer comparison
- They're complementary: a large system often uses a CDN at the edge and load balancers in each region.
Vertical vs Horizontal Scaling
- Imagine traffic grows and your server can no longer handle the load.
Vertical scaling (scale up)
- Increase the capacity of the existing server: more CPU, RAM, or faster storage.
- Pros: simple, with no code or architecture changes.
- Cons:
- There's a hardware ceiling; you can't upgrade forever.
- Large machines get disproportionately expensive.
- It remains a single point of failure.
- Upgrading often requires a restart and downtime.
Horizontal scaling (scale out)
- Add more servers and distribute the workload across them with a load balancer.
- Pros:
- Practically no upper limit.
- Fault tolerant: if one server fails, others keep serving.
- Supports auto-scaling: add servers at peak and remove them when traffic drops.
- Cons:
- More moving parts to deploy and monitor.
- Servers should be stateless: store sessions in a shared store like Redis instead of server memory, so any server can handle any request.
- Scaling the database horizontally is much harder, and needs read replicas, caching, or sharding.
- In practice, teams often scale vertically first because it's simple, then scale horizontally as traffic and availability needs grow.
Rate Limiting vs Throttling
- Imagine your API is receiving thousands of requests, some from a buggy client stuck in a loop.
Rate limiting
- Rate limiting sets the maximum number of requests a client can make in a time window, identified by user ID, API key, or IP address.
- For example, with a limit of 100 requests per minute, extra requests are rejected with
429 Too Many Requests, often with a Retry-After header telling the client when to try again.
- It protects against abuse, brute-force login attempts, scraping, and one client starving others, and it's often used to enforce pricing tiers.
- Common algorithms:
- Fixed window: count requests per calendar minute. Simple, but allows bursts at window edges.
- Sliding window: counts over the last 60 seconds at any moment, smoothing those edges.
- Token bucket: tokens refill at a steady rate, and each request spends one. Allows short bursts up to the bucket size.
- Leaky bucket: requests drain from a queue at a fixed rate, producing a smooth output.
- With many servers, counters are kept in a shared store like Redis so limits apply across the whole fleet.
Throttling
- Throttling controls the rate at which requests are processed, rather than only counting requests per client.
- When the system is under heavy load, requests can be delayed, queued, or rejected to keep processing at a manageable rate. For example, a payment processor might handle at most 500 transactions per second and queue the rest.
- The terms overlap in practice: some platforms, like AWS API Gateway, call per-client request limits "throttling".
- Related: JavaScript throttling applies the same idea on the client, limiting how often a function runs.
Takeaway
- CDN: caches content near users. Load Balancer: spreads requests across servers.
- Vertical scaling: a bigger server. Horizontal scaling: more servers.
- Rate limiting: caps requests per client. Throttling: controls the processing rate under load.
References