Amazon SageMaker AI real-time endpoints now support KV cache aware routing
AWS · By Dmitry Soldatkin ·
Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt…