ANALYSIS / PERFORMANCE
A serving system is more than a tokens/s number.
Capacity and user experience belong on the same chart.
The mechanism
MLPerf Endpoints evaluates systems across concurrency levels. Each point records total output throughput, per-user token rate, first-token latency, and load. Its frontier exposes the trade-off that a single peak-throughput number conceals.
Read the conditions
A result is tied to a model, workload, hardware configuration, and serving setup. The current rules use P90 first-token latency. The official frontier is stepped: it does not imply that unmeasured operating points have been validated.
How much useful capacity remains once the system must meet the latency and quality requirements of the actual product?
From measurement to a decision
Choose the service objectives first. Sweep concurrency with representative prompt and response lengths, then identify the configurations that meet them. Our proposed extension is to account for the full provisioned cost at those feasible points, including idle capacity and transfer overhead.
For technical leaders
Ask for the workload and the curve behind the headline. A capacity plan should explain both the expected operating point and what happens during a burst.
Go to the source
- MLCommons: What MLPerf Endpoints measures ↗
- MLCommons: Endpoints measurement rules ↗
- MLCommons: Published Pareto explorer ↗
A synthesis of the linked primary sources, reviewed 16 November 2026. The research questions and proposed experiments are turingfold’s interpretation; these notes do not report original benchmark results.