Latency is the time required to perform an action or produce a result. Throughput is the number of actions or results completed per unit of time.
A good design aims for the highest throughput that keeps latency within the product's limit. Maximizing only one is usually a mistake. A batch processor may accept higher latency to increase throughput. An interactive request needs a tight latency target even when its throughput is modest.
- Below capacity
The service has spare capacity and little queueing.
- Knee
As utilization rises, queueing delays become significant.
- Saturation
The maximum sustainable throughput is bounded; more offered load can increase latency and errors.
- Measure
The instrument is qualitative. Real capacity and latency curves require workload measurements.
Beyond the illustrated knee, added demand mainly increases waiting rather than useful throughput.
How to read this model
A work-capacity instrument with an admission dial and waiting magazine. The object is a teaching analogy. Inspecting a part highlights its role; it does not execute or simulate the software. Cabinet trays are records, workpieces are data, and latches are checks. Unless an explanation states a quantity, compartments and sample workpieces do not denote data counts. Part dimensions are not a quantitative scale.
As utilization approaches capacity, requests wait for CPU, connections, disk, or another finite resource. That queueing delay creates the sharp rise often called the knee of the curve. Leave headroom for bursts and failures instead of operating at the knee.
Measure latency with percentiles. An average can hide the slow requests that users experience. Track a target such as p95 or p99 alongside throughput, error rate, and resource utilization.
The cache and communication protocols pages describe ways to reduce work and network delay. See latency numbers for rough baseline comparisons.