SageMaker Inference now includes prefix-aware routing. This routing strategy directs requests sharing the same prompt prefix to the same instance. The KV cache remains warm for subsequent requests with the same prefix. Benchmarks on Llama 3.1 70B demonstrated a reduction in P50 time-to-first-token of up to 77%. KV cache hit rates increased from approximately 25% to over 80%.
Prefix-aware routing improves inference performance by minimizing the need for the model to recompute KV cache contents. This is particularly beneficial for applications with high request volumes and repetitive prompts. The strategy reduces latency and improves throughput.
This feature is available for models deployed on Amazon SageMaker Inference. It is designed to be integrated with existing deployment workflows. The routing strategy is configurable through the SageMaker Inference API.
