International
The Morning Dev

Only the news worth knowing

‹
 

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

AWS · By Dmitry Soldatkin ·

Amazon SageMaker AI real-time endpoints now support KV cache aware routing

Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt…

How do you feel about this?

0 Comments

    Amazon SageMaker AI real-time endpoints now support KV cache aware routing – The Morning Dev