SyncAI.news, a Varaisys broadcasting
Amazon SageMaker AI real-time endpoints now support KV cache aware routing
DS

Dmitry Soldatkin

· 20 min read

EngineeringAWS Machine Learning Blog

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set RoutingStrategy to KV_AWARE in the endpoint configuration. Your clients call the endpoint with the same APIs as before.

Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation (RAG), and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory, the KV cache, and skips that work when the same start reaches the same instance again. In part 1 of this series, we introduced prefix-aware routing for Amazon SageMaker AI, which sends requests that start the same way to the same instance. Prefix-aware routing infers where a prefix is cached from the request alone, so it can’t tell when an instance has evicted that cache.

Prefix-aware routing picks an instance from the request alone. It hashes the start of the request, up to the PrefixLength you configure (bytes of the InvokeEndpoint payload, or characters of the message text for the OpenAI-compatible Chat Completions API), and sends requests that start the same way to the same instance. It never sees what an instance actually has cached. This works well when every request starts with the same shared text, and less well in two cases:

The cache on an instance changes over time. An instance evicts older KV cache entries when its GPU memory fills up, but the router keeps sending matching requests there as if the cache were still warm.

A conversation starts out short. In a multi-turn chat, each turn adds to the history. While the request is still shorter than PrefixLength, every new turn changes the text the router hashes, so the next turn can land on a different instance than the one that cached the earlier turns. The turns of a conversation stay on one instance only after the request grows past PrefixLength.

In this post, we explain how KV-aware routing works, when to use it, and how to enable, invoke, and verify it on single model endpoints and inference component endpoints.

How KV-aware routing works

KV-aware routing has three parts: the model container reports what it has cached, SageMaker AI keeps a managed cache index of that state for the endpoint, and the endpoint uses the index to choose an instance for each request.

When you turn events on with SM_VLLM_KV_EVENTS_CONFIG, the container publishes a KV cache event each time it stores or evicts cache entries, and SageMaker AI uses these events to keep the index current. The index holds only pointers: one-way, truncated hashes of cached content, each mapped to the instance or inference component copy that holds it. It never stores prompts, token IDs, request payloads, or KV tensors. When a request arrives, the endpoint tokenizes the prompt and looks up the tokens in the index. The index returns the instances or inference component copies that hold part of the prompt in their cache, and how much of it each one holds. For example, if instance A holds the cached start of a conversation, the next turn of that conversation goes to A. Because the index matches on tokens rather than a fixed number of bytes, a multi-turn history can match in full as it grows. A request that shares only part of its prompt still goes to the instance with the longest match.

The endpoint then picks among these candidates by weighing how much of the prompt each one has cached against how busy it is. A candidate that is too busy is skipped, so if the instance with the longest match is busy, the request goes to the next-best one that is available. If no candidate has a useful match or all of them are busy, the request falls back to load-aware routing, as described in Routing overhead and fallbacks. You don’t configure a concurrency threshold for this, as you do with prefix-aware routing. The index also follows the fleet. When you scale in, entries for removed instances are dropped. When you scale out or an update replaces instances, the new instances start with empty caches, so latency can be higher for a short time.

Figure 1: KV-aware routing on a SageMaker AI endpoint. The router looks up the managed cache index and forwards the request to the instance that holds the longest cached match. Each instance reports KV cache events back to the index

In Figure 1, the endpoint has three instances, A, B, and C. Each instance runs the same model in its own model container (the AWS vLLM Deep Learning Container listed in Requirements and limits) and has its own KV cache in GPU memory, so each instance holds different cached content: A holds turns 1 to 3 of conversation 1, B holds conversations 2 and 3, and C holds document 7. When turn 4 of conversation 1 arrives (step 1), the router tokenizes it and looks it up in the managed cache index (step 2). The index shows that A holds the first 6,000 of the request’s 6,200 tokens, so the router sends the request to A (step 3), and A only has to compute the remaining tokens. If A is busy, the router sends the request to the least busy instance instead (step 4). KV-aware routing needs a choice of destinations, so it is active only with at least two instances, or at least two copies of an inference component. With a single instance, every request goes to that instance and the setting has no effect.

Routing strategies on SageMaker AI

With this launch, SageMaker AI offers four routing strategies for real-time endpoints. You set the strategy per production variant in the RoutingConfig of your endpoint configuration.

Strategy How it routes Recommended for
RANDOM (default) Sends each request to a randomly chosen instance. General-purpose workloads, non-LLM models, and requests that are interchangeable.
LEAST_OUTSTANDING_REQUESTS Sends each request to the instance with the fewest in-flight requests. Workloads where processing time varies and you want to keep all instances equally busy.
PREFIX_AWARE Sends requests that share the same prompt prefix, up to PrefixLength, to the same instance. Overflows to another instance at ConcurrencyThreshold. LLM workloads with shared text at a fixed position at the start of the request, on any serving framework that supports prefix caching.
KV_AWARE (new) Sends each request to an instance that holds a matching cached prefix, based on cache state reported by the container, weighing match length against load. Falls back to load-aware routing. LLM workloads on a supported container with multi-turn conversations, frequent cache eviction, or requests that share only part of their start.

Prefix-aware or KV-aware

Both strategies send requests to where their prefix is likely to be cached. They differ in how they know where that is, and in what they need from your container.

Prefix-aware routing KV-aware routing
Source of cache location Inferred from the request prefix Cache state reported by the container
Added latency per request 1.3–1.9 milliseconds in part 1 A few milliseconds for short prompts, more for long ones, mostly for tokenization
Matching First PrefixLength bytes (Invoke API) or characters (OpenAI-compatible API) Tokens, longest cached match
Parameters PrefixLength, ConcurrencyThreshold None needed
Container Any container with prefix caching on A supported container, with prefix caching and KV cache events on (see Requirements and limits)
Request payload Any OpenAI-compatible Completions or Chat Completions format

Performance

We measured KV-aware routing against random routing in two ways: a large-scale benchmark on production-sized instances, and a smaller, reproducible comparison you can follow in Verify that KV-aware routing is working.

Benchmark results

In our benchmarks with Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances, we ran 30-minute tests with 20 or 100 simulated users holding multi-turn chats and document Q&A sessions, with prompts of up to about 64,000 tokens. Compared with random routing, and with long prompts under load, KV-aware routing reduced p50 time to first token (TTFT) by up to 97 percent and increased throughput by up to 234 percent. It raised the KV cache hit rate from 3–48 percent to 80–93 percent. Long prompts gained even at lower load. The gain was smallest with short prompts, where there is less prefill to save. Results depend on your workload. To measure them for your own traffic, use the checks in Verify that KV-aware routing is working.

Routing overhead and fallbacks

Before it routes a request, the endpoint tokenizes the prompt and looks it up in the managed cache index. The lookup takes a few milliseconds for short prompts and longer for long ones, mostly for tokenization, but it stays small next to the time a cache hit saves. For text-only requests that use standard options, the tokens it computes are passed to the model, so the prompt isn’t tokenized twice.

If the lookup is slow, finds no useful match, or all matching instances are busy, the endpoint routes the request with load-aware routing, the same behavior as the LEAST_OUTSTANDING_REQUESTS strategy. The request is still served, without the cache benefit. The same happens when the endpoint can’t tokenize a request, except in the two error cases described in Invoking the endpoint. These fallbacks aren’t reported in the SageMaker AI console or metrics at launch, so use the checks in Verify that KV-aware routing is working to confirm that routing is effective for your traffic.

With very short prompts there is less prefill to save, so prefix-aware routing can be all you need.

When to use KV-aware routing

Prefix-aware routing needs no changes to your container, but it can’t see what an instance actually holds. KV-aware routing asks the container what it holds, which makes it more precise. It pays off when there is a lot of cached context to reuse and that cache is under pressure: long prompts, high load, and instances that run short of GPU memory for the KV cache. When that happens, the container evicts the least recently used parts of the cache, including parts of shared prompts and conversation history.

For example, a support assistant runs on three instances, and each conversation begins with the same 6,000-token knowledge-base excerpt. Conversation 1 starts on instance A, which caches the excerpt. Traffic rises, A runs short of memory, and the container evicts most of that excerpt. Instance B recently served another conversation over the same excerpt and still holds its first 4,000 tokens. When the next turn of conversation 1 arrives, prefix-aware routing sends it to A again, and A recomputes the excerpt. With KV-aware routing, A’s container has already reported the eviction, so the index shows that B holds the longest match, and the request goes to B, which recomputes only the remaining tokens. At low load, A wouldn’t have evicted anything, and both strategies would have sent the request to A with the cache intact.

KV-aware routing helps most when requests reuse cached context in ways a fixed-length prefix can’t capture:

Multi-turn conversations: Each turn sends the full conversation history, so the cached prefix grows with every turn. KV-aware routing matches the whole cached history and keeps each conversation on the instance that holds it. This matters most for long conversations. After the managed cache index reflects an eviction, the router doesn’t keep sending the conversation to an instance that has already dropped its cache.

High-throughput inference with shared context: RAG applications prepend retrieved documents, and assistants send long system prompts with every request. KV-aware routing fits when these prompts are long and instances are under memory pressure. With a stable shared prefix and moderate load, prefix-aware routing also fits. Put the shared content first and in the same order: if the same documents appear in a different order, the prompts no longer share a start, so they don’t match.

Workloads where prefix length is hard to tune: With prefix-aware routing, you choose a PrefixLength that covers your shared text without scattering requests that should stay together. KV-aware routing has no prefix length to tune, because it matches on what is actually cached.

Prefix-aware routing is a good fit, with no container changes, if your requests share a stable system prompt or document at the start, your prompts are short, or load is moderate. In these cases, instances have enough GPU memory to keep the shared text cached, so both strategies find it. Prefix-aware routing is also the choice if your container isn’t supported by KV-aware routing, because it works with any container that has prefix caching turned on. If your requests are short and interchangeable, or your model isn’t an LLM, there is little cached context to reuse, so use RANDOM or LEAST_OUTSTANDING_REQUESTS.

If you aren’t sure which fits, start with prefix-aware routing, run your own traffic against a KV_AWARE endpoint, and compare time to first token, throughput, and hit rate as described in Verify that KV-aware routing is working.

Requirements and limits

Before you enable KV-aware routing, check that your container, endpoint, and requests meet these requirements:

Container: At launch, KV-aware routing supports the AWS vLLM Deep Learning Container for SageMaker AI: images with vLLM 0.33.0 or later, or the Amazon Linux build server-sagemaker-cuda-v2.8.0 or later. Prefix caching must be on, which is the default, and KV cache events must be turned on with the SM_VLLM_KV_EVENTS_CONFIG environment variable, as shown in How to enable it. SageMaker AI doesn’t set it for you.

Endpoints and instances: Real-time single model endpoints and inference component endpoints, including dynamic Low-Rank Adaptation (LoRA) adapters, on GPU instance types. You can also configure KV-aware routing with AWS CloudFormation. Routing is active only with at least two instances for a single model endpoint, or at least two copies of an inference component, which can be on the same instance. With fewer, you can still set KV_AWARE and the endpoint isn’t rejected, but every request goes to the single instance or copy.

Requests: OpenAI-compatible Completions or Chat Completions payloads, sent with InvokeEndpoint, InvokeEndpointWithResponseStream, or the SageMaker AI OpenAI-compatible Chat Completions API. InvokeEndpointWithBidirectionalStream isn’t supported. If you set the model field, it must match the value of SM_VLLM_MODEL.

SDK versions: Detailed observability, which you use to track the KV cache hit rate, needs AWS Command Line Interface (AWS CLI) version 2.35.6 or later, or Boto3 version 1.43.31 or later.

SageMaker AI checks these requirements before your endpoint serves traffic. If a check fails, the request or deployment fails with a message that describes what is missing. For the full list of settings, validation behavior, and quotas, see the SageMaker AI documentation.

How to enable it

You turn on KV-aware routing in two places: the model and the endpoint configuration. The example notebook runs every step end to end, for both a single model endpoint and an inference component endpoint.

1. Turn on KV cache events in the model – When you create the model, add SM_VLLM_KV_EVENTS_CONFIG to the container environment. Without it, the endpoint has no cache state to route on. The ports in this setting are used only inside the instance, so you don’t open them to your network.

"Environment": {
    "SM_VLLM_MODEL": "<model-id-or-/opt/ml/model>",
    "SM_VLLM_KV_EVENTS_CONFIG": "{\"enable_kv_cache_events\": true, \"publisher\": \"zmq\", \"endpoint\": \"tcp://*:5557\", \"topic\": \"kv-events\", \"replay_endpoint\": \"tcp://*:5658\"}"
}

2. Choose KV-aware routing in the endpoint configuration – Set RoutingStrategy to KV_AWARE in the RoutingConfig of the production variant:

"RoutingConfig": {
    "RoutingStrategy": "KV_AWARE"
}

If your container image is built for CUDA 13, also set InferenceAmiVersion in the production variant to an AMI that supports it, such as al2023-ami-sagemaker-inference-gpu-4-1. Otherwise the container can fail to start. See Inference GPU drivers.

Check that your account has enough endpoint quota for the instance type and count you choose, and request an increase in Service Quotas if needed.

The same RoutingConfig setting works for inference component endpoints, including dynamic LoRA adapters. For adapters, KV-aware routing chooses among the instances that already have the adapter loaded.

Invoking the endpoint

Nothing changes in how you call the endpoint. Send a Chat Completions payload with InvokeEndpoint, or with InvokeEndpointWithResponseStream and "stream": true in the request body:

{"messages": [{"role": "user", "content": "What is your return policy?"}]}

You can also call the endpoint with the OpenAI Python client through the SageMaker AI OpenAI-compatible Chat Completions API. The Use the OpenAI-compatible API with SageMaker AI notebook on GitHub shows how, for both single model endpoints and inference components.

KV-aware routing doesn’t apply the multi-tenant prefix isolation from part 1: it ignores the prefix-aware ID header and the prompt_cache_key field. A request that sets a top-level cache_salt field to a non-null value is rejected with a 400 error.

KV-aware routing adds two error cases: a request whose model field names a model that the container doesn’t serve fails with a ModelError, and a request with a non-null cache_salt is rejected with a 400 error. Errors from inference itself are returned as before, and your model container’s own logs in Amazon CloudWatch can help you find the cause. The endpoint doesn’t deliver logs about its KV-aware routing decisions to your account, so use the checks in Verify that KV-aware routing is working, such as cached_tokens per request and the KV cache hit rate, to see how routing behaves.

Verify that KV-aware routing is working

KV-aware routing doesn’t add metrics of its own, so you check its effect on your model. There are three ways to do this.

Check cache hits per request – The container doesn’t report prompt token details by default, so set SM_VLLM_ENABLE_PROMPT_TOKENS_DETAILS to true in the model environment. Each response then includes usage.prompt_tokens_details.cached_tokens, the number of prompt tokens that were served from the KV cache. When you send a follow-up turn, or another request that starts the same way as an earlier one, a non-zero value shows that the request reached an instance that held its prefix.

Check model metrics in Amazon CloudWatch – Turn on detailed observability by adding --metrics-config '{"EnableDetailedObservability": true}' when you create the endpoint configuration, using the SDK versions in Requirements and limits. SageMaker AI then publishes the model container’s metrics, including the KV cache hit rate and TTFT. Query them as described in Detailed observability with PromQL.

Figure 2: Verifying KV-aware routing. (a) A follow-up request shows most prompt tokens served from the KV cache. (b) The KV cache hit rate for the same endpoint in the SageMaker AI Insights dashboard

Compare with the strategy you would otherwise use – For most LLM workloads the real choice is between KV_AWARE and PREFIX_AWARE, so compare those two, and add a RANDOM endpoint as a baseline. Use the same model, instance type, and instance count for each endpoint, and send the same prompts at the same concurrency for the same duration, so that the only difference is the routing strategy. Make sure your test reuses context the way your production traffic does, for example with multi-turn conversations or shared documents. If KV_AWARE and PREFIX_AWARE perform about the same for your traffic, prefix-aware routing is the simpler choice.

As an example of this comparison, we compared RANDOM and KV_AWARE on a small endpoint. We sent the same multi-turn workload to two endpoints that differ only in routing strategy. Each endpoint had two ml.g5.xlarge instances serving Qwen3-4B-Instruct-2507 on a supported container. Each run sent 128 requests: 32 simulated users, each asking 4 questions about a document chosen from 12 documents of about 7,400 tokens. Each question resends the conversation so far, so every question after the first can reuse cached context. We ran two pairs of runs, with RANDOM first in pair 1 and KV_AWARE first in pair 2, and used new documents for each pair so that both endpoints started with empty caches. Table 1 shows the results as pair 1 / pair 2.

Metric RANDOM KV_AWARE
CloudWatch ModelLatency average (ms) 4,124 / 5,100 1,235 / 1,314
CloudWatch ModelLatency p50 (ms) 3,153 / 4,083 656 / 700
CloudWatch ModelLatency p90 (ms) 8,334 / 10,746 3,213 / 3,535
CloudWatch ModelLatency p99 (ms) 17,779 / 16,249 6,876 / 7,238
Time to first token p50, questions 2–4 (ms) 1,284 / 1,632 237 / 245
Time to first token p90, questions 2–4 (ms) 5,039 / 6,101 456 / 367
Time to first token p90, all requests (ms) 5,003 / 5,683 1,465 / 1,759
Share of prompt tokens served from cache 0.761 / 0.701 0.903 / 0.903
Run duration (s) 31.1 / 36.9 17.9 / 18.0
CloudWatch OverheadLatency average (ms) 13.1 / 8.1 27.1 / 27.5

Table 1: RANDOM and KV_AWARE routing under the same workload. Each cell shows pair 1 / pair 2. ModelLatency and OverheadLatency are Amazon CloudWatch metrics for the endpoint. The client measured time to first token and run duration. The cache share is the sum of cached_tokens divided by the sum of prompt_tokens over a run

In both pairs, KV_AWARE had lower CloudWatch ModelLatency (average 1,235 and 1,314 ms against 4,124 and 5,100 ms) and lower time to first token for questions 2–4 (p50 237 and 245 ms against 1,284 and 1,632 ms). Its cache share was higher, at 90 percent in both pairs against 76 and 70 percent. OverheadLatency, the time SageMaker AI spends outside the model container, was 14 to 19 ms higher on average with KV_AWARE. This is one model, one instance type, and one workload with long shared documents, so measure your own traffic as described earlier in this section. Figure 3 shows the model latency charts that the SageMaker AI console displays for the two endpoints.

Figure 3: One-minute average model latency in Amazon CloudWatch for the RANDOM endpoint (blue) and the KV_AWARE endpoint (orange), on a shared 0–7 second axis. Each endpoint received seven consecutive runs of the same workload on new documents, RANDOM first. Average model latency was 4.9–6.2 seconds for RANDOM and 1.2–1.3 seconds for KV_AWARE. The last RANDOM point is lower because that minute holds only the last 20 requests of the run

If neither does better than RANDOM, check that prefix caching and KV cache events are turned on in the container, as described in Requirements and limits.

Example notebook

The example notebook applies the steps in this post to Qwen/Qwen3-4B-Instruct-2507 on two ml.g5.xlarge instances, the minimum for a single model endpoint, and on an inference component endpoint with two copies. It calls the endpoint with both APIs, shows in the container logs that follow-up requests are served from the KV cache, and doesn’t compare routing strategies. Run its cleanup cells at the end to avoid ongoing charges.

Clean up

To avoid ongoing charges, delete the resources you created. For an inference component endpoint, delete the inference components first (delete-inference-component), then the endpoint, endpoint configuration, and model (delete-endpoint, delete-endpoint-config, delete-model).

Practical guidance

These three practices help you keep KV-aware routing effective after you turn it on.

Monitor the KV cache hit rate – Keep detailed observability on after launch and watch the KV cache hit rate over time in the SageMaker AI Insights dashboard (Figure 2b), or query it with PromQL, as described in Verify that KV-aware routing is working.

Plan for cold starts – The first request with new context, such as a new document or the first turn of a conversation, has nothing cached on any instance. KV-aware routing speeds up the requests that follow, not the first one.

Plan for cache rebuilds during scaling and updates – As described in How KV-aware routing works, new instances start with empty caches. To soften the latency increase, use a rolling or blue/green update for single model endpoints, or a RollingUpdatePolicy for inference components.

Conclusion

KV-aware routing gives SageMaker AI real-time endpoints a second way to send requests to where their prompts are cached. It uses the cache state that your model container reports, so it follows the cache when instances evict it under memory pressure. Prefix-aware routing from part 1 of this series remains the simpler choice for stable shared prompts, short contexts, and moderate load. SageMaker AI handles routing inside the endpoint for both strategies, so you don’t build or run a routing layer yourself. You turn it on with one environment variable in the model and "RoutingStrategy": "KV_AWARE" in the endpoint configuration.

KV-aware routing is available today in all commercial AWS Regions that support SageMaker AI real-time inference, for endpoints that use a supported container. It is available at no additional charge. SageMaker AI runs and manages the cache index for you, and you pay only for the instances your endpoint uses. KV_AWARE is a new value for the existing RoutingStrategy field, so you don’t need a new configuration object, and older versions of the AWS CLI and Boto3 accept it. To get started, read the documentation, run the example notebook on a small endpoint, and then compare KV_AWARE with PREFIX_AWARE on a copy of your own traffic.

About the authors

Dmitry Soldatkin

Dmitry is a Worldwide Leader for Specialist Solutions Architecture, Amazon SageMaker AI Inference at AWS. He helps customers design, build, and optimize generative AI and machine learning solutions, with a focus on inference performance optimization, deep learning, and scaling ML across the enterprise. In previous roles, he led teams at financial services, insurance, and telecommunications companies.

Shashi Kiran

Shashi is the Worldwide Leader for Specialist Solutions Architecture, Distributed Inference at AWS, where he drives solutions to complex customer challenges in scaling and optimizing AI/ML inference. He helps customers design and build distributed serving platforms for generative AI and deep learning, translating their needs and real-world insights into product direction for Amazon SageMaker.

Apoorva Chandra

Apoorva is a Product Manager on the Amazon SageMaker AI Inference team, where she focuses on the endpoint experience and hosting experience. She works with customers deploying generative AI models in production to make accelerated compute easier to obtain, more cost-efficient to run, and more predictable to operate. Outside of work, Apoorva enjoys hiking, exploring coffee shops, and spending time with friends.

Shijie Lyu

Shijie is a Software Development Engineer on the Amazon SageMaker AI Inference team, working on large language model inference and training at scale. His work spans KV-aware routing, LoRA adapter serving, speculative decoding, rapid scaling, and heterogeneous clusters. He also builds reinforcement learning post-training infrastructure, optimizes GPU kernels with CUDA and Triton, and contributes to the vLLM and SGLang open-source projects.

Original source

This story was published by AWS Machine Learning Blog and written by Dmitry Soldatkin. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on aws.amazon.com

Similar News