Multi-turn agentic workloads build on accumulated context as an agent reasons, invokes tools, and incorporates earlier results. With each new turn, the client resends the conversation history, including system instructions, prior prompts, tool results, and model responses, causing the growing context to be processed repeatedly. In Prefill-decode (P-D) disaggregation, each request is split across specialized nodes: the prefill node processes the prompt and builds its key-value (KV) cache, then the decode node retrieves those KV blocks and generates the response.
Conventional disaggregated serving, however, only transfers KV cache in one direction: from prefill to decode node. This one-directional flow creates two inefficiencies for multi-turn workloads. First, after generating a response, the decode node holds KV cache entries for both the prompt and the newly generated tokens. When processing the next turn, the prefill node cannot retrieve those entries, so it must recompute the KV projections for the previous response tokens. Second, the decode node maintains the session state while the prefill nodes don’t, increasing the probability of cache eviction for active sessions on the prefill nodes. On a full cache miss, the prefill node must recompute KV projections for the entire conversation history, including system instructions, prior user requests, tool results, and model-generated responses. As the conversation grows, this redundant work consumes more accelerator capacity and increases time to first token (TTFT).
When you build conversational assistants and agentic applications, longer interaction histories can increase latency and consume accelerator capacity through repeated prefill computation. In this post, we show how bi-directional KV cache transfer lets prefill and decode nodes reuse previously computed context across turns, which improves accelerator utilization and helps applications sustain responsive TTFT as conversations grow.
In our multi-turn benchmarks on a 1P1D deployment using Amazon Elastic Compute Cloud (Amazon EC2) P5en instances, bi-directional KV transfer reduced TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8 (see the Performance data section). Results vary with workload and configuration. The instances are connected with AWS Elastic Fabric Adapter (AWS EFA), a low-latency remote direct memory access (RDMA) network used across servers in AWS. On supported instance types, AWS EFA supports GPUDirect RDMA and Neuron Direct RDMA to move KV blocks directly between GPU or AWS Trainium device memory on the prefill and decode nodes.
We originally built and validated the feature on AWS Trainium instance clusters connected with EFA and contributed it to vLLM as an accelerator-agnostic capability. The results presented here are from an Amazon EC2 P5en GPU cluster with EFA, showing that the benefits also carry over to GPUs. This post explains the bi-directional KV transfer architecture, router metadata flow, KV-block retention and recompute controls, performance results, and how to reproduce the benchmark.
Bi-directional KV transfer
Bi-directional KV transfer is a mechanism in which a decode node returns previously computed KV to a prefill node on subsequent turns of a conversation. This avoids recomputing shared context on the prefill node and is governed by a configurable KV recompute threshold. It mitigates both computational-redundancy scenarios described earlier. To make it work, the KV must survive past the turn that produced it and travel in the reverse direction, coordinated by a KV-cache-aware router.
NIXL KV connector enhancements for bi-directional transfer
We extended the NVIDIA Inference Xfer Library (NIXL) KV connector to let prefill nodes consume remote KV blocks, similar to how decode nodes already consume KV blocks that the prefill node provides. This is not a trivial change, because the two node types are architecturally different: prefill nodes compute KV representations for new prompt tokens and can provide complete prompt KV blocks. Decode nodes lack the capability (or efficiency) to generate complete KV representations and can only serve previously cached KV blocks. As a result, prefill and decode nodes still carry their own roles and logic for computing the metadata of remote and local KV blocks. The actual KV transfer mechanism (a NIXL READ), however, is common to both the P->D and D->P directions.
The following diagram first shows the existing flow in blue. The router sends the request to prefill node, prefill node computes the prompt KV and returns its block metadata, and the router forwards the request and metadata to decode node for the standard P→D NIXL READ. The numbered red path illustrates the bi-directional KV transfer additions: (1) decode node retains eligible KV blocks after generation, (2) returns their metadata in the streaming response, and (3) the router caches that metadata for the next prefill request. Prefill node then (4) checks the remotely reusable KV after it applies local cache matches, (5) queues an asynchronous receive when the reusable token count meets the threshold, and (6) performs a D→P NIXL READ before computing the remaining suffix.
KV-cache-aware router
vLLM includes connector-side support for bi-directional KV transfer, while coordinating KV reuse across conversation turns requires a KV-cache-aware router. vLLM GitHub provides an example proxy that implements the flow used in this post which can be adapted for your environment.
The KV-cache-aware router is a lightweight, stateful control-plane component between the client, prefill, and decode nodes. It maintains an in-memory mapping from the client-provided conversation_id to decode’s kv_transfer_params. The router stores only block-location metadata, not KV tensors, and obtains that metadata from the response stream rather than separately querying decode node.
The following diagram shows how the router manages KV metadata across conversation turns. On the first turn, the cache lookup misses and the request follows the standard P→D flow. The router then saves decode’s metadata from the response stream. On later turns, the router retrieves that metadata using the conversation_id so prefill can reuse KV from decode. It then refreshes the metadata after generation for the next turn.
Figure 2: KV-cache-aware router metadata flow across conversation turns
Block retention TTL
Normally a decode instance frees a request’s blocks as soon as generation finishes. Under bi-directional KV transfer mode, it holds them for a configurable lifetime (decoder_kv_blocks_ttl, default 480 seconds) and publishes their location and expiry. The conversation’s KV, both prompt prefix and generated response, then stays available for the next turn. For a turn that arrives before the time to live (TTL) expires, prefill node reuses the KV. One that arrives later falls back to normal prefill. Because this TTL is fixed and not renewed, tune it to the expected time between turns to balance KV reuse against accelerator-memory usage. A natural next improvement is to extend the heartbeat-renewed lease to the decode side. A decode instance could then reclaim blocks as soon as a conversation goes idle rather than at the end of a fixed timer.
KV recompute threshold
KV reuse isn’t always faster than local recomputation. For a short reusable prefix, interconnect and coordination overhead can exceed the time needed to recompute the tokens. The kv_recompute_threshold setting, which defaults to 64 tokens, controls this decision. Prefill node recomputes when the token count to compute is below the threshold and pulls KV from decode node when it is at or above the threshold. Deployments can tune this value based on model computation time, KV size, and interconnect performance.
Performance data
For evaluation, we used Qwen3-32B and GLM-5-FP8 models on a 1P1D deployment using Amazon EC2 P5en instances connected with EFA. To emulate scenarios in which the prefill node has evicted its local cache or the router selects a different prefill node that does not have the conversation’s KV blocks, we disabled prefix caching on prefill node with --no-enable-prefix-caching.
Across both the models, turning on bi-directional KV transfer reduces prefill computation on multi-turn conversations, and the benefit typically grows with prompt length. With the feature OFF, the prefill instance recomputes the entire grown context each turn, so TTFT rises steeply. Qwen3-32B climbs from 158 ms at 2k to 778 ms at 24k, and GLM-5-FP8 from 500 ms at 2k to 2,334 ms at 20k. With it ON, the decode instance’s retained KV is reused instead of recomputed, keeping TTFT far flatter. Qwen3-32B rises only from 122 ms to 460 ms over the same range (about a 1.6x reduction at the longest prompt), while GLM-5-FP8 rises from 358 ms to 702 ms and runs up to 3.3x below the recompute path at long prompts. The prefill-duration breakdown explains why.
Bi-directional transfer adds a small KV-transfer time (about 30 ms to 130 ms for Qwen3-32B; about 29 ms to 110 ms for GLM-5-FP8), but the prefill compute it removes far outweighs that KV-transfer time. As a result, the total prefill duration with transfer ON (Qwen3-32B: 128 ms to 414 ms; GLM-5-FP8: 316 ms to 541 ms) stays well below the recompute-only path (Qwen3-32B: 153 ms to 744 ms; GLM-5-FP8: 500 ms to 2,331 ms). The larger GLM-5-FP8 model shows a bigger absolute and relative gap, since its higher per-token recompute time makes KV reuse pay off even more as models and contexts scale.
Qwen3-32B
The following chart plots TTFT against prompt length for Qwen3-32B, comparing bi-directional KV transfer ON versus OFF.
Figure 3: Qwen3-32B TTFT versus prompt length with bi-directional KV transfer on and off
The next chart breaks the prefill duration for the same runs into its recompute and KV-transfer components.
Figure 4: Qwen3-32B prefill duration split into recompute and KV-transfer components
Together the charts show TTFT staying nearly flat with the feature ON while the OFF path climbs steeply as context grows, because the small KV-transfer time is far outweighed by the recompute it removes.
zai-org/GLM-5-FP8
The same two views follow for GLM-5-FP8. The first plots TTFT against prompt length with the feature ON versus OFF.
Figure 5: zai-org/GLM-5-FP8 TTFT with KV reuse on and off
The second breaks the prefill duration into its recompute and KV-transfer components.
Figure 6: zai-org/GLM-5-FP8 prefill duration breakdown
GLM-5-FP8 shows a much larger gap than Qwen3-32B, up to about 3.3x lower TTFT at long prompts, because its higher per-token recompute time makes KV reuse pay off even more as context length grows.
How to run benchmark
To reproduce our 1P1D benchmark configuration, one prefill node and one decode node, using Amazon EC2 P5en instances with EFA, follow the step-by-step instructions in the Reproducing our results guide. The guide covers environment setup, launching the vLLM prefill and decode servers, running the benchmark, and collecting results. The feature can run on other supported configurations, but performance results will vary depending on the hardware, model, workload, and network configuration.
Conclusion
In this post, we showed how bi-directional KV cache transfer can reduce TTFT as an agent’s conversation history grows. In P-D disaggregated deployments, the feature lets prefill node reuse KV blocks retained by decode node across turns, reducing repeated computation. Configurable block-retention and recompute thresholds determine when reuse occurs. On a 1P1D deployment using Amazon EC2 P5en instances connected with EFA, at the longest prompt lengths tested, we observed TTFT reductions of 1.6x for Qwen3-32B and 3.3x for zai-org/GLM-5-FP8. Results depend on the workload and deployment configuration. To implement this in your environment, use vLLM v0.21.0 or later with NixlConnector, enable bidirectional_kv_xfer on both prefill and decode servers, use a kv-cache-aware router, and follow the GitHub reproduction guide.
Related resources
- AWS EFA
- Prefill-Decode (P-D) disaggregated serving
- Benchmark KV Cache Offloading with Multi-Turn Conversations
Acknowledgments
We thank the vLLM community for their support in maintaining this feature.