Consider an AI agent reviewing a collection of supplier documents. It reads the evidence, generates a response, calls a tool and returns to the model with new information. Each turn changes the balance between processing input and producing output. Several agents running at once add another variable: requests arrive with different context lengths, response lengths and deadlines.
For the infrastructure operator, this is a production problem. How much useful inference can the system deliver, at an agreed quality and speed, for the money it consumes?
Heterogeneous inference offers one way to improve that equation. Different accelerators have different strengths. Separating the stages of inference creates an opportunity to assign work according to those strengths and adjust capacity as demand changes.
The principle applies across hardware generations and suppliers. Its value comes from the performance of the complete serving system.
One request, two different workloads
For a typical autoregressive language model, inference has two main stages.
Prefill processes the input tokens and builds the attention state needed for generation. It can perform substantial work across those tokens in parallel, making compute throughput especially valuable for many prompt workloads.
Decode generates subsequent tokens sequentially. At small batch sizes, repeatedly accessing model weights and attention state often makes memory bandwidth a major constraint. Memory capacity also limits how many active sequences the service can retain.
These are tendencies to measure. Batch size, context length, model architecture and numerical precision can change the bottleneck. Microsoft's Splitwise research uses the differing characteristics of prompt processing and generation to explore separate hardware allocations, including heterogeneous clusters.
The two stages also shape different parts of the user experience:
Scroll horizontally to compare the measures.
| Measure | What the user experiences | What operators must account for |
|---|---|---|
| Time to first token, or TTFT | The wait before a response starts | Queueing, prompt processing and any handoff on the critical path |
| Time per output token, or TPOT | The average pace of generation after it starts | Decode scheduling, memory access and competing work |
| End-to-end latency | The wait for the complete response | Both stages, response length and service overhead |
Average token speed can hide uncomfortable pauses. Streaming services should also inspect the distribution of gaps between tokens, especially during bursts.
Separate the stages, then choose the hardware
When prefill and decode share workers, a large incoming prompt can compete with requests already generating responses. Separating them into dedicated pools allows each stage to use its own resource allocation and parallelism strategy.
The DistServe paper, published at OSDI 2024, studies this approach around goodput: the request rate a system can sustain while satisfying specified latency requirements. It also treats network bandwidth and worker placement as part of the design.
A heterogeneous system adds hardware choice to that separation. One accelerator type might be economical for a particular prefill workload; another might deliver the required generation speed with a better balance of memory and cost.
There are two distinct opportunities. Between stages, prefill and decode can use different hardware types. Within a stage, a pool can contain different worker types, with requests routed to suitable instances. That second arrangement requires a scheduler that understands their different capacities; equal request counts do not imply equal work.
Router: workload, cache and load
Transfer attention state to the selected decode worker
Conceptual architecture. Each request uses a compatible prefill/decode pair. Types A, B and C illustrate possible worker classes, not a claim that arbitrary devices interoperate.
This pool design routes requests among instances. It does not require every accelerator in a pool to jointly execute one model operation. The chosen instances still need compatible model state, kernels and transfer paths.
The handoff is part of the computation
Prefill produces a key-value cache, or KV cache: stored attention state that decode reuses as it generates tokens. When the stages run on different workers, that state must reach the destination in a usable form.
Moving bytes is only part of the job. The receiving worker needs the correct model version and a compatible cache representation. Different tensor layouts, data types or parallelism configurations can require supported conversion or redistribution.
The vLLM NIXL connector guide documents asynchronous cache transfer, configuration requirements and an experimental path for particular heterogeneous cache layouts. These are concrete integration mechanisms, with compatibility limits to verify for each deployment.
Transfer performance also depends on context length, cache size, effective link bandwidth and contention. A transfer-time figure is useful only alongside its payload and test conditions. Operators need to measure the delay added to the request, including transfer setup and waiting for a decode slot, and test what happens when the handoff fails.
The economic test is whether better stage placement saves more than the handoff and additional infrastructure cost.
Schedule for the workload that actually arrives
A useful hardware profile records more than peak arithmetic performance. It captures measured prefill and decode behaviour for the intended model: throughput at different batch sizes, memory use, context limits and latency under load.
Consider two illustrative services. A document-extraction service reads long inputs and returns short fields. A report-writing service produces much longer outputs. Their resource needs differ, and the balance can change again when requests reuse cached context.
The serving layer should use those measurements alongside live queue depth, available memory and service-level objectives. It can then route requests and adjust the balance between prefill and decode capacity. Reassigning hardware still has costs, including loading models and warming caches; adaptation needs to account for that delay.
NVIDIA Dynamo's routing documentation illustrates the trade-off between cache reuse and worker load. Sending a request to a worker holding useful state can avoid computation, but overloading that worker can erase the benefit through queueing.
Reduce repeated work and the state it consumes
Prefix caching lets a request reuse attention state for an identical leading sequence of tokens. Shared instructions, repeated documents and continuing conversations can benefit when the relevant state remains available. As the vLLM documentation explains, this avoids repeated prefill computation; it does not remove the work of generating new output tokens.
Cache hit rate therefore needs a definition. A request with a small matching prefix and one with almost its entire input cached should not be treated as equivalent savings. Report reused tokens or blocks, the resulting latency improvement and the memory cost of retention. Reuse must also respect tenant and data-access boundaries.
Reducing the state itself can help. KV-cache quantisation stores attention state at lower precision, potentially freeing memory for more concurrent sequences. Its benefit depends on backend support and calibration, with output quality checked on the intended tasks. Smaller state does not automatically produce a proportional speed increase.
These optimisations interact. Less prefill recomputation can shift demand towards decode. A smaller cache can change both memory pressure and transfer requirements. Re-profile the whole system after a material model or runtime change.
Establish when separation pays
Disaggregation adds workers, model replicas, networking and operational responsibilities. Those costs can be difficult to justify for small deployments, light traffic or workloads with little interference between stages.
There are effective alternatives to compare. Sarathi-Serve, also published at OSDI 2024, uses chunked prefill and scheduling to introduce prompt work without stalling ongoing decode. A well-tuned colocated service is an important baseline.
The vLLM disaggregated-prefill documentation emphasises separate latency tuning and control of tail inter-token latency, and explicitly warns that its disaggregation feature does not itself improve throughput. This is consistent with a broader systems lesson: gains in sustainable service capacity depend on the constraints, allocation and workload being tested.
Start with a representative request trace. Compare the baseline, homogeneous disaggregation and a compatible heterogeneous configuration under the same quality and latency requirements. Include cold caches, warm caches, traffic bursts and failures. That establishes which part of the design earns its cost.
Measure the economics of dependable output
For Cooma's token-factory perspective, a useful infrastructure measure is the total serving cost per million output tokens from requests that complete within the agreed service targets.
Define the model, quality threshold, input/output length distribution and latency targets first. Report prompt tokens, cached input and output tokens separately. Count the costs of all attempts, including failures and retries, even when their output does not qualify for the denominator.
The cost boundary should include rented or owned capacity, idle headroom, host resources, networking, cache storage and operations. For owned infrastructure, account for capital and energy costs consistently. A hardware-only comparison can miss the expense introduced by the serving architecture.
At the application layer, carry the measurement through to cost per accepted result. An agent that needs fewer retries or produces a better-supported answer may deliver more business value even when its individual model calls cost more. This connects inference efficiency to the approach described in The Token Factory Flywheel.
Heterogeneous inference expands the design space for turning compute capacity into dependable output. The opportunity is to match each stage to suitable hardware, preserve useful state and keep the system balanced as demand changes. The result earns its place in a token factory when it delivers accepted work at a demonstrably better cost and service level.