Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference
· By Antonio Sedino, CTRO · Published by Reinventy Solutions Corp.
Amazon SageMaker Inference introduces prefix-aware routing to keep KV cache warm for shared prompt prefixes.
Amazon SageMaker Inference introduces prefix-aware routing to keep the KV cache warm for requests sharing the same prompt prefix.