ANALYSIS / SERVING ARCHITECTURE
Prefill and decode want different things.
Independent scaling is useful. Moving the state between stages is the price of admission.
The mechanism
Prefill processes the input and builds the KV cache. Decode uses that state to generate subsequent tokens. NVIDIA Dynamo separates these stages into worker pools, allowing their capacity and placement to be chosen independently.
The engineering implication
Long prompts can interfere with ongoing generation in a shared serving pool. Separating the stages can help tune first-token latency and inter-token latency independently. But the transfer fabric, cache size, and request mix determine whether the extra coordination is worthwhile. Dynamo’s documentation explicitly notes that aggregated serving can be simpler and faster for smaller workloads.
At what input length and concurrency does separate prefill/decode serving improve latency enough to justify KV-transfer overhead?
A useful experiment
Hold the model, precision, and total GPU budget constant. Compare aggregated and separated pools across a prompt-length and concurrency sweep. Record first-token latency, inter-token latency, transfer time, throughput, and the fraction of requests meeting the same service objective. Keep warmup and cache conditions explicit.
Go to the source
A synthesis of the linked primary sources, reviewed 16 November 2026. The research questions and proposed experiments are turingfold’s interpretation; these notes do not report original benchmark results.