Latency vs Throughput
A web service has a latency of 500 ms per request and can handle 1,000 requests per second. An engineer proposes adding a cache in front of the database, which reduces latency to 20 ms for 80% of requests.
- How does caching affect throughput?
- What is the risk introduced by caching, and how would you mitigate it?
1. Effect on throughput: If latency drops from 500 ms to 20 ms for 80% of requests, those cached requests are processed ~25× faster. With the same number of workers, the system can now handle significantly more requests per second because workers spend less time blocked per request. Throughput generally improves together with latency when the bottleneck is computation or I/O time.
2. Risk — stale data: A cache serves data from a previous point in time. If the underlying database is updated, the cache continues serving the old value until the TTL expires or the cache is explicitly invalidated. Mitigation strategies:
- Set an appropriate TTL based on how stale data is acceptable.
- Use write-through caching: update both the cache and the database on writes.
- Explicitly invalidate (delete) the cache key when the underlying data changes.