Estimate Servers for an API Load

Problem

A consumer product with 100 million daily active users exposes a JSON API from its mobile apps. Each user makes on the order of 20 API calls per day. A typical call does a couple of database reads, some business logic, and serializes a response, taking about 50 ms end to end on an unloaded server, of which roughly 20 ms is CPU on the API server and the rest is waiting on downstream calls.

Estimate how many API servers you need to run. Show the request rate, the per-server capacity, and the headroom you would add before you would sign off on the number.

• Assume each API server is a 32-core machine.
• Plan for peak traffic, not average.
• The service runs in three zones and must survive losing one at peak.

Examples

Defaults to state; each can be swapped for whatever the interviewer gives you:

• Requests per day: 100M users × 20 calls = 2 × 10^9 requests/day.
• Peak-to-average ratio: 3× is standard for a consumer app concentrated in a few time zones; 2× for a global audience.
• CPU per request: 20 ms of CPU on the API server. If the interviewer says "it's an I/O-bound service, CPU is 5 ms," recompute; the method is the same.
• Target utilization: run servers at 50-70% CPU at peak, not 100%. Latency degrades sharply above ~70-80% because queuing delay grows non-linearly with utilization.
• Zone failure: with three zones, losing one leaves two-thirds of capacity, so provision 1.5× the peak need.
• Latency budget: the 50 ms is service time; you are asked for capacity, not tail latency, but note that they interact.

Hints

1. Start with QPS at peak. Average QPS is a trap; nobody provisions for 3 a.m.
2. A server's throughput is cores ÷ CPU-seconds per request, scaled by target utilization. Wall-clock latency including I/O waits does not limit throughput if the server is asynchronous or has enough threads; CPU does.
3. Headroom is multiplicative: utilization target, zone failure, and growth each add a factor. Apply them in sequence and state each one.