ANALYSIS / SERVING ARCHITECTURE

Prefill and decode want different things.

Published Source: project documentation

Independent scaling is useful. Moving the state between stages is the price of admission.

The mechanism

Prefill processes the input and builds the KV cache. Decode uses that state to generate subsequent tokens. NVIDIA Dynamo separates these stages into worker pools, allowing their capacity and placement to be chosen independently.

The engineering implication

Long prompts can interfere with ongoing generation in a shared serving pool. Separating the stages can help tune first-token latency and inter-token latency independently. But the transfer fabric, cache size, and request mix determine whether the extra coordination is worthwhile. Dynamo’s documentation explicitly notes that aggregated serving can be simpler and faster for smaller workloads.

THE QUESTION WE WOULD TEST

At what input length and concurrency does separate prefill/decode serving improve latency enough to justify KV-transfer overhead?

A useful experiment

Hold the model, precision, and total GPU budget constant. Compare aggregated and separated pools across a prompt-length and concurrency sweep. Record first-token latency, inter-token latency, transfer time, throughput, and the fraction of requests meeting the same service objective. Keep warmup and cache conditions explicit.

Go to the source

A synthesis of the linked primary sources, reviewed 16 November 2026. The research questions and proposed experiments are turingfold’s interpretation; these notes do not report original benchmark results.

← Back to Daily Brief