Break Down a Latency Budget

Problem

A mobile app on a cellular connection calls an API that returns a personalized feed. The product requirement is that the response arrives within 300 ms at the median and within 1 second at the 99th percentile. Break the 300 ms down: estimate how much time each stage of the request consumes, from the tap to the rendered response, and show whether the budget is achievable. Then identify which stages you would attack first if it is not.

• Assume the user is in one region and the service runs in a different region on the same continent unless you decide to add an edge.
• Assume the connection may be cold (no existing TCP/TLS session) for the median case; treat a warm connection as a follow-up.
• Use the standard reference latencies; you can quote them from memory but show the composition.

Examples

Reference numbers to quote from memory (the well-known "latency numbers every engineer should know," rounded, with a route factor applied to the long-haul ones):

• Speed of light in fiber: ~200,000 km/s, so ~5 ms per 1,000 km one way in a straight line. Real fiber paths are 1.5-2× the great-circle distance and add router and regenerator delay, so budget ~7-8 ms per 1,000 km. Cross-continent (US coast to coast, ~4,000 km) ≈ 30 ms one way, ~60 ms round trip as measured, not the 40 ms the straight-line figure suggests. Transatlantic ≈ 35-40 ms one way, 70-80 ms RTT.
• Cellular last-mile RTT: 30-60 ms on LTE, 10-30 ms on 5G, 100+ ms on 3G. Use 40 ms.
• Datacenter round trip (same building): ~0.5 ms. Cross-zone within a region: 1-2 ms.
• Memory access: ~100 ns. SSD random read: ~100 µs. Spinning disk seek: ~10 ms.
• Cache hit (Redis/Memcached over the network): ~0.5-1 ms, dominated by the network round trip.
• Indexed database read: 1-10 ms depending on whether the page is in the buffer pool.
• Response size: 50 KB of JSON for a feed page.
• Cellular download bandwidth: 10-20 Mbps effective. Use 10 Mbps.
• DNS lookup: 0 ms if cached on device; 20-50 ms if it requires a resolver round trip.

Hints

1. Count round trips before you count anything else. A cold connection needs DNS, a TCP handshake, and a TLS handshake before the first byte of the request is sent; each is at least one RTT.
2. The server's own processing is usually not where the time goes for a well-built service. Sequential downstream calls are.
3. Transfer time is bytes × 8 ÷ bandwidth, and it can rival the RTT for responses in the tens of kilobytes on mobile.