Change
e29f4a9
e29f4a9740573397cd5b2eec43ef30e5c1b9ed52 · commit on GitHub
aws-blog-feed: changed (571431 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +4,313
- Lines removed
- -4,706
- Stored bytes at this commit
- 571,431
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-09-11T04:41:17.791Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Thu, 10 Sep 2026 21:58:36 GMT |
| date | Fri, 11 Sep 2026 04:41:17 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Wed, 09 Sep 2026 22:26:29 +0000</lastBuildDate>+
<lastBuildDate>Thu, 10 Sep 2026 21:58:09 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,577 +13,564 @@ 1 </sy:updateFrequency> <item>-
<title>Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM</title>-
<link>https://aws.amazon.com/blogs/machine-learning/deploying-qwen3-8-2-4t-a95b-on-amazon-sagemaker-hyperpod-with-vllm/</link>+
<title>Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference</title>+
<link>https://aws.amazon.com/blogs/machine-learning/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference/</link> -
<dc:creator><![CDATA[Dmitry Soldatkin]]></dc:creator>-
<pubDate>Wed, 09 Sep 2026 22:26:29 +0000</pubDate>+
<dc:creator><![CDATA[Kareem Syed-Mohammed]]></dc:creator>+
<pubDate>Thu, 10 Sep 2026 21:58:09 +0000</pubDate> <category><![CDATA[Advanced (300)]]></category>-
<category><![CDATA[Amazon SageMaker HyperPod]]></category>-
<category><![CDATA[Technical How-to]]></category>-
<guid isPermaLink="false">6973e0078540c7d789cc946cb3c7e1fd791a8c2f</guid>-
-
<description>Learn how to deploy Qwen3.8-2.4T-A95B, a 2.4-trillion-parameter open-weight model, on Amazon SageMaker HyperPod with vLLM. This walkthrough covers cluster provisioning, NVFP4 quantization, and an OpenAI-compatible endpoint with built-in reasoning, tool calling, and native MTP specu…-
<content:encoded><p>On August 12, 2026, Alibaba’s Qwen team released <a href="https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B" target="_blank" rel="noopener">Qwen3.8-2.4T-A95B</a>. This is the first time a Qwen-Max-class model has been made available as open weights. With 2…-
<p>Open weights models give you full control. Data stays within your infrastructure, inference behavior can be customized, and there are no per-token API fees at scale. The trade-off is operational: hosting a 2.4T-parameter model requires purpose-built GPU infrastructure and an optimized servi…-
<p>In this post we show how to deploy Qwen3.8-2.4T-A95B on <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod.html" target="_blank" rel="noopener">Amazon SageMaker HyperPod</a> using <a href="https://docs.vllm.ai/" target="_blank" rel="noopener">vLLM&…-
<p>This is the second post in our series on deploying open trillion-parameter models on Amazon SageMaker HyperPod. For the first post covering Kimi K3, see <a href="https://aws.amazon.com/blogs/machine-learning/deploying-kimi-k3-on-amazon-sagemaker-hyperpod-and-amazon-eks/" target="_blank" …-
<h2 id="qwen3.8-2.4t-a95b-at-a-glance">Qwen3.8-2.4T-A95B at a glance</h2> -
<p>Qwen3.8-2.4T-A95B (the open-weight release of Qwen3.8-Max) is the largest and most capable model in the Qwen family. The following is a summary of the key architectural details relevant to deployment.</p> -
<h3 id="architecture">Architecture</h3> -
<table border="1px" width="100%" cellpadding="10px"> -
<tbody> -
<tr> -
<td><strong>Attribute</strong></td> -
<td><strong>Value</strong></td> -
</tr> -
<tr> -
<td>Total parameters</td> -
<td>2.4 T</td> -
</tr> -
<tr> -
<td>Activated parameters per token</td> -
<td>95 B</td> -
</tr> -
<tr> -
<td>Architecture</td> -
<td>Fine-grained Mixture of Experts (MoE)</td> -
</tr> -
<tr> -
<td>Expert count</td> -
<td>512 routed + 1 shared (10 routed experts activated per token)</td> -
</tr> -
<tr> -
<td>Layers</td> -
<td>92</td> -
</tr> -
<tr> -
<td>Layer layout</td> -
<td>3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE), repeated</td> -
</tr> -
<tr> -
<td>Context window</td> -
<td>262,144 tokens native. Extensible to 1,010,000</td> -
</tr> -
<tr> -
<td>Max output length</td> -
<td>128K tokens</td> -
</tr> -
<tr> -
<td>Multi-Token Prediction</td> -
<td>Native MTP draft heads (enables speculative decoding without a separate model)</td> -
</tr> -
</tbody> -
</table> -
<p>The hybrid attention design is key to efficient long-context inference. <strong>Gated DeltaNet</strong> layers (69 of 92) use linear attention with a bounded recurrent state, replacing the growing KV-cache with a fixed-size memory. <strong>Gated Attention</strong> la…-
<p>The fine-grained MoE distributes capacity across 512 small experts rather than a few large ones, improving routing efficiency and specialization. Only approximately 95B parameters are active per forward pass, so serving costs track activated parameters, not the full 2.4T.</p> -
<h3 id="capabilities-and-reasoning-control">Capabilities and reasoning control</h3> -
<p>Qwen3.8 is designed for agentic execution: multi-step coding, autonomous tool use, long-horizon planning, and complex research workflows. It includes built-in reasoning controls through the <code>reasoning_effort</code> parameter (<code>low</code>, <code>medium…-
<h3 id="model-weights-and-quantization">Model weights and quantization</h3> -
<p>The open weights are published on <a href="https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B" target="_blank" rel="noopener">Hugging Face</a> in the standard Transformers format. Community quantizations include MXFP4 and NVFP4 (W4A4), which compress the model to approximately 1.2 TB…-
<h3 id="benchmark-highlights">Benchmark highlights</h3> -
<p>According to the vendor’s benchmarking results, Qwen3.8-2.4T-A95 shows particular strength in research workflows (PaperBench 93.0), instruction following (IFBench 82.8), and terminal-based coding (86.6). It performs comparably with leading frontier models across most categories, with remain…-
<h2 id="why-amazon-sagemaker-hyperpod-for-large-moe-inference">Why Amazon SageMaker HyperPod for large MoE inference</h2> -
<p>Deploying a 2.4T-parameter model is not only a GPU problem. It requires orchestration that handles model download, container scheduling, health monitoring, autoscaling, and node failures without manual intervention. <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyper…-
<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/09/ML-21725-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/09/ML-21725-1.png" alt="Architecture diagram of…-
<p class="wp-caption-text">Figure 1: High-level architecture of Amazon SageMaker HyperPod</p>-
</div> -
<p><strong>EKS-orchestrated clusters.</strong> HyperPod clusters use Amazon Elastic Kubernetes Service (Amazon EKS) as the control plane. You get the full Kubernetes landscape (<code>kubectl</code>, Helm charts, custom resource definitions), while AWS manages the underl…-
<p><strong>Inference Operator.</strong> The HyperPod Inference Operator (installed automatically or as an EKS Add-on) provides a single custom resource definition (CRD), <code>InferenceEndpointConfig</code>, that declaratively specifies your model, container image, GPU …+
<category><![CDATA[Amazon SageMaker AI]]></category>+
<category><![CDATA[Announcements]]></category>+
<guid isPermaLink="false">32206d53293b6830e987ee4659338e8d02374647</guid>+
+
<description>Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit…+
<content:encoded><p>When you build an application on top of a large language model (LLM), the prompt you send to the model typically has two parts. There’s a fixed part that sets up context (instructions, reference documents, conversation history) and a variable part that contains th…+
<p>This means that across hundreds or thousands of requests, your model is processing that same 3,000-token beginning over and over again.</p> +
<p>LLM serving frameworks like vLLM and TensorRT-LLM have a solution for this. They cache the computed key-value (KV) pairs for prompt prefixes that have been seen before. When the same beginning shows up in a new request, the model reuses the cached computation and only processes the new toke…+
<p>But there’s a problem when you scale beyond a single instance. If you have a fleet of machines behind an endpoint, requests get distributed across all of them. That same 3,000-token prefix lands on instance A for one request, instance B for the next, instance C after that. Each instance com…+
<p>Today, Amazon SageMaker Inference introduces prefix-aware routing. It is a new routing strategy that looks at the beginning of each request and consistently sends requests with the same beginning to the same instance. The KV cache on that instance actually builds up and gets reused. In our …+
<h2 id="what-prefix-aware-routing-does">What prefix-aware routing does</h2> +
<p>When a request arrives at your endpoint, Amazon SageMaker looks at the beginning of the payload and uses it to decide which instance should handle it. The same beginning goes to the same instance. Different beginnings spread across different instances. If 10 requests share a prefix or begin…+
<p>You don’t need to tag requests or manage affinity yourself. The endpoint handles it based on the content of the request.</p> +
<p>There are two built-in safeguards:</p> +
<p>Overload protection. If one prefix is extremely popular and the target instance is already at capacity, the endpoint routes the request to a less busy instance instead. You configure the concurrency limit, and the endpoint respects it. You might miss a cache hit on that one request, but you…+
<p>Stable behavior during scaling. When you add or remove instances, most requests continue going to the same instance they were going to before. Only a small fraction of traffic shifts to account for the changed fleet. Your caches don’t get invalidated every time you scale.</p> +
<h2 id="performance-benchmarks">Performance benchmarks</h2> +
<p>We benchmarked prefix-aware routing against the default random routing baseline using Llama 3.1 70B Instruct on 7 ml.p5.48xlarge instances with vLLM (prefix caching enabled). We ran 16 test configurations covering single model endpoints, inference component endpoints, the native Invoke API,…+
<h3 id="long-context-workloads">Long context workloads</h3> +
<p>8,000-token shared prefixes, sustained over 1 hour:</p> <ul> -
<li>Model weight download (from Hugging Face Hub, Amazon Simple Storage Service (Amazon S3), or Amazon FSx).</li> -
<li>Container scheduling and GPU allocation.</li> -
<li>Health checks and readiness gates.</li> -
<li>Rolling updates and endpoint lifecycle management.</li> -
<li>Autoscaling through KEDA with Amazon CloudWatch or Prometheus metrics.</li> +
<li>P90 TTFT: reduced by 33–37 percent.</li> +
<li>P50 TTFT: reduced by 71–77 percent.</li> +
<li>KV cache hit rate: from approximately 25–82 percent.</li> +
<li>Throughput: increased 15–16 percent.</li> </ul> -
<p><strong>Reserved capacity with Flexible Training Plans.</strong> The <code>ml.p6-b300.48xlarge</code> instance type requires reserved capacity. Flexible Training Plans provide committed GPU reservations that can be allocated directly to your HyperPod cluster. There’s…-
<p><strong>Resilience.</strong> HyperPod continuously monitors node health and automatically replaces degraded nodes. For sustained inference workloads running 24/7, this alleviates the operational overhead of manually detecting and recovering from hardware failures.</p> -
<p><strong>Additional inference features</strong> (Inference Operator v3.x):</p> +
<h3 id="short-context-workloads">Short context workloads</h3> +
<p>Variable-length ShareGPT-style conversations, 30 minutes:</p> <ul> -
<li><a href="https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-dpd.html" target="_blank" rel="noopener">Disaggregated Prefill and Decode</a> (DPD) – separates prefill and decode onto distinct GPU pools for predictable per-token latency under concu…-
<li><a href="https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-data-capture.html" target="_blank" rel="noopener">Inference data capture</a> – log inputs/outputs at the endpoint, load balancer, or pod level.</li> -
<li>Local NVMe model deployment – load weights from node-local storage to reduce cold-start latency.</li> -
<li><a href="https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-custom-certs.html" target="_blank" rel="noopener">Amazon Route 53 DNS management</a> – automatic custom domain records for your endpoints.</li> +
<li>P90 TTFT: reduced by 24–37 percent.</li> +
<li>P50 TTFT: reduced by 13–16 percent.</li> +
<li>KV cache hit rate: from approximately 30–80 percent.</li> +
<li>Throughput: increased 1.7–2.0 percent.</li> </ul> -
<p>In short: You write a YAML manifest describing <em>what</em> to deploy. HyperPod handles <em>how</em> to run it reliably at scale.</p> -
<h2 id="infrastructure-sizing-matching-hardware-to-the-model">Infrastructure sizing: Matching hardware to the model</h2> -
<h3 id="the-p6-b300-instance">The p6-b300 instance</h3> -
<p>The <code>ml.p6-b300.48xlarge</code> provides the compute density required for single-node serving of Qwen3.8:</p> +
<p>The longer your shared prefix, the bigger the win. Long context workloads benefit the most because there is more computation to skip on each cache hit. Short context workloads still benefit, but the shared prefixes are smaller so the savings per request are proportionally smaller.</p>…+
<h3 id="routing-overhead">Routing overhead</h3> +
<p>The prefix-aware routing logic adds 1.3–1.9 milliseconds per request. Model TTFT in these tests ranged 63–280 milliseconds. The routing cost is negligible.</p> +
<p>Traffic distribution remained balanced across all scenarios. Each of the 7 instances received 13.3–15.4 percent of requests, within 1 percent of an ideal even split. No hot spots.</p> +
<h2 id="routing-strategies-on-sagemaker-inference">Routing strategies on SageMaker Inference</h2> +
<p>With this launch, Amazon SageMaker Inference offers three routing strategies for real-time endpoints:</p> +
<p>RANDOM (default): Distributes requests uniformly across instances. Recommended for general-purpose workloads, non-LLM models, or a situation where requests are interchangeable and there’s no benefit to sending specific requests to specific instances.</p> +
<p>LEAST_OUTSTANDING_REQUESTS: Sends each request to the instance with the fewest in-flight requests. Recommended when request processing times vary and you want to keep all instances equally busy. Helps prevent slow requests from piling up on one machine while others sit idle.</p> +
<p>PREFIX_AWARE (new): Sends requests sharing the same prompt prefix to the same instance. Recommended for LLM workloads where many requests share common text at the beginning and your serving framework has prefix caching enabled.</p> +
<p>You set the strategy per production variant in your endpoint configuration. You can switch between them by updating the endpoint configuration without redeploying your model.</p> +
<h2 id="when-to-use-prefix-aware-routing">When to use prefix-aware routing</h2> +
<p>The feature delivers value when your requests share text at the beginning. Here are the patterns where it has the most impact:</p> +
<p>Retrieval Augmented Generation (RAG) applications. You retrieve a document and prepend it before the user’s question. When multiple users ask questions about the same document, they all share that document as a prefix. Prefix-aware routing sends them to the same instance, where the KV cache…+
<p>Multi-turn conversations. Each turn in a conversation includes the full history of previous turns. As the conversation grows, that shared history becomes a longer and more expensive prefix. Routing on that prefix keeps the conversation’s cache on one instance across turns.</p> +
<p>Templated bots and assistants. Bots with long, structured instructions (policies, formatting rules, persona definitions) send those same instructions with every request. Only the user message at the end changes. Prefix-aware routing means that expensive instruction block gets processed once…+
<p>Code completion. Coding assistants include file contents as context. While a developer works in the same file, every completion request shares that file content as a prefix.</p> +
<h2 id="how-to-enable-it">How to enable it</h2> +
<p>You configure prefix-aware routing when you create your endpoint configuration. Two parameters control the behavior:</p> +
<p>PrefixLength (1024–65536): How much of the request to use for routing. For the native Amazon SageMaker Invoke API, this is bytes from the beginning of the request body. For the OpenAI-compatible API, this is characters from the extracted message text. Set this to cover your shared prefix pl…+
<p>ConcurrencyThreshold (1–1024): The maximum in-flight requests on the target instance before overflow kicks in. If the target instance is at this limit, the request goes to a less loaded instance instead.</p> +
<p>Here is an example:</p> +
<div class="hide-language"> +
<pre><code class="language-bash">aws sagemaker create-endpoint-config \+
--endpoint-config-name example-llm-config \+
--production-variants '[{+
"VariantName": "AllTraffic",+
"ModelName": "example-llm-model",+
"InitialInstanceCount": 3,+
"InstanceType": "ml.p5.48xlarge",+
"RoutingConfig": {+
"RoutingStrategy": "PREFIX_AWARE",+
"PrefixAwareRoutingConfig": {+
"PrefixLength": 4096,+
"ConcurrencyThreshold": 10+
}+
}+
}]'</code></pre> +
</div> +
<p>Then create your endpoint as usual:</p> +
<div class="hide-language"> +
<pre><code class="language-bash">aws sagemaker create-endpoint \+
--endpoint-name example-llm-endpoint \+
--endpoint-config-name example-llm-config</code></pre> +
</div> +
<p>No changes to your model container or serving framework are needed. Prefix-aware routing operates entirely at the endpoint routing layer.</p> +
<h2 id="invoking-the-endpoint">Invoking the endpoint</h2> +
<p>Nothing changes about how you call the endpoint. The same InvokeEndpoint and InvokeEndpointWithResponseStream APIs work exactly as before:</p> +
<div class="hide-language"> +
<pre><code class="language-bash">aws sagemaker-runtime invoke-endpoint \+
--endpoint-name example-llm-endpoint \+
--content-type application/json \+
--body fileb://request.json \+
output.json</code></pre> +
</div> +
<p>Same for the OpenAI-compatible Chat Completion API:</p> +
<div class="hide-language"> +
<pre><code class="language-python">from openai import OpenAI+
from sagemaker.core.token_generator import generate_token+
+
client = OpenAI(+
base_url=f"https://runtime.sagemaker.us-west-2.amazonaws.com"+
f"/endpoints/example-llm-endpoint/openai/v1",+
api_key=generate_token(region="us-west-2")+
)+
response = client.chat.completions.create(+
model="example-model",+
messages=[+
{"role": "user", "content": "What is your return policy?"},+
],+
)</code></pre> +
</div> +
<h2 id="multi-tenant-prefix-isolation">Multi-tenant prefix isolation</h2> +
<p>If different tenants share the same prompt instructions but you want them routed separately (to keep cache contexts independent), pass an optional ID:</p> +
<p>Native Invoke API: set the <code>X-Amzn-SageMaker-Prefix-Aware-Id</code> header (up to 64 ASCII characters).</p> +
<p>OpenAI API: include the <code>prompt_cache_key</code> field in the request body.</p> +
<p>This ID combines with the prefix so that requests with identical prefixes but different IDs land on different instances.</p> +
<h2 id="inference-components-and-lora-adapters">Inference components and LoRA adapters</h2> +
<p>Prefix-aware routing works with inference component endpoints and dynamic Low-Rank Adaptation (LoRA) adapters. For inference components, it behaves the same as single model endpoints. For LoRA adapters, it operates within the adapter’s sticky instance set, using prefix-based selection among…+
<h2 id="practical-guidance">Practical guidance</h2> +
<p>Enable prefix caching in your serving framework. Prefix-aware routing gets repeated prefixes to the same instance, but your container needs prefix caching turned on to actually store and reuse those KV pairs. In vLLM, this is enabled by default in recent versions. Other frameworks might req…+
<p>Keep request serialization consistent. For the native Invoke API, PrefixLength operates on raw bytes. JSON whitespace, key ordering, and formatting all affect routing. If you serialize the same prompt differently across requests, they might end up on different instances. Use consistent seri…+
<p>Size PrefixLength carefully. Too short and all requests with the same short prefix get funneled to one instance, triggering overflow. Too long and small payload differences (like temperature values) scatter requests that should stay together. Start with the length of your shared prefix plus…+
<p>You need at least two instances. With one instance, all requests go to the same place regardless of strategy.</p> +
<p>Monitor cache hit rates. Enable SageMaker detailed observability to track KV cache hit rates at the model level. This confirms whether prefix-aware routing is working for your specific workload.</p> +
<h2 id="conclusion">Conclusion</h2> +
<p>Prefix-aware routing is available today on SageMaker real-time inference endpoints. Update your AWS SDK or CLI to the latest version to access the new RoutingStrategy and PrefixAwareRoutingConfig parameters. Refer to this notebook for examples of how to enable it during endpoint creation.&l…+
<p style="clear: both"></p> +
<hr style="width: 100%"> +
<h2>About the authors</h2> +
<footer> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/10/ML-21885-1.jpg" alt="Kareem Syed-Mohammed" width="100" height="133">+
</div> +
<h3 class="lb-h4">Kareem Syed-Mohammed</h3> +
<p style="overflow: hidden">Kareem is a Principal Product Manager at AWS. He focuses on enabling generative AI model development and governance on Amazon SageMaker HyperPod. Prior to this, at Amazon Quick Sight, he led embedded analytics and developer experience. In addition to Quick Sight, …+
</div> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/10/ML-21885-2.jpg" alt="Vivek Gangasani" width="100" height="133">+
</div> +
<h3 class="lb-h4">Vivek Gangasani</h3> +
<p style="overflow: hidden">Vivek is a Worldwide Leader for Solutions Architecture, SageMaker Inference. He leads Solution Architecture, Technical Go-to-Market (GTM) and Outbound Product strategy for SageMaker Inference. He also helps enterprises and startups deploy and optimize generative A…+
</div> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/10/ML-21885-3.jpg" alt="Dmitry Soldatkin" width="100" height="133">+
</div> +
<h3 class="lb-h4">Dmitry Soldatkin</h3> +
<p style="overflow: hidden">Dmitry is a Worldwide Leader for Specialist Solutions Architecture, SageMaker Inference at AWS. He leads efforts to help customers design, build, and optimize generative AI and AI/ML solutions across the enterprise. His work spans a wide range of ML use cases, wit…+
</div> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/10/ML-21885-4.jpg" alt="Vamsi Goparaju" width="100" height="133">+
</div> +
<h3 class="lb-h4">Vamsi Goparaju</h3> +
<p style="overflow: hidden">Vamsi leads Go-to-Market (GTM) and Revenue Acceleration initiatives at AWS, enabling Inference and ModelOps workloads for our customers, and driving OSS Use Cases and Product features for Amazon SageMaker AI and Generative AI. With over 16 years of experience span…+
</div> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full wp-image-139112" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/10/xu.jpg" alt="" width="100" height="100">+
</div> +
<h3 class="lb-h4">Xu Deng</h3> +
<p style="overflow: hidden">Xu Deng is a Software Engineer Manager with the SageMaker team. He focuses on helping customers build and optimize their AI/ML inference experience on Amazon SageMaker. In his spare time, he loves traveling and snowboarding.</p> +
</div> +
</footer></content:encoded>+
+
+
+
</item>+
<item>+
<title>Reduce inference cold starts on Amazon SageMaker HyperPod with model caching</title>+
<link>https://aws.amazon.com/blogs/machine-learning/reduce-inference-cold-starts-on-amazon-sagemaker-hyperpod-with-model-caching/</link>+
+
<dc:creator><![CDATA[Kareem Syed-Mohammed]]></dc:creator>+
<pubDate>Thu, 10 Sep 2026 21:37:49 +0000</pubDate>+
<category><![CDATA[Amazon SageMaker HyperPod]]></category>+
<category><![CDATA[Announcements]]></category>+
<category><![CDATA[Expert (400)]]></category>+
<guid isPermaLink="false">3b91429cf30acb250cd56430f226e5855e1f213f</guid>+
+
<description>Amazon SageMaker HyperPod now supports model caching for inference, which pre-loads model weights and container images onto cluster nodes so pods read from local NVMe storage instead of downloading over the network. Learn how model caching cuts cold starts from tens of minutes to s…+
<content:encoded><p>When you deploy a large language model (LLM) for inference on <a href="https://aws.amazon.com/sagemaker/ai/hyperpod/" target="_blank" rel="noopener">Amazon SageMaker HyperPod</a>, there’s a gap between when you request a pod and when it’s ready to serv…+
<p>Today we’re launching model caching for Amazon SageMaker Inference on HyperPod. Model caching pre-loads model weights and container images onto cluster nodes before pods need them. When you start your pod, it can read from local NVMe storage at approximately 7 GB/s instead of downloading ov…+
<h2 id="the-cold-start-problem-in-detail">The cold start problem in detail</h2> +
<p>To understand why model caching matters, consider what happens when an inference pod starts without it. The Kubernetes scheduler places the pod on a node. Kubelet begins pulling the container image from ECR. For inference server images like vLLM or LMI, these are multi-gigabyte images that …+
<p>During scale-out the same sequence repeats for every new pod. If traffic spikes and your HorizontalPodAutoscaler requests five new pods, all five go through this download sequence independently. The autoscaling policy may react in seconds. However, the actual time to serve additional traffi…+
<h2 id="how-model-caching-works">How model caching works</h2> +
<p>Model caching eliminates both of these latency sources by pre-loading data onto nodes before pods are scheduled. It introduces two independent capabilities that you can enable together or separately.</p> +
<h3 id="weights-cache">Weights cache</h3> +
<p>The weights cache downloads model weights to local NVMe storage on each node ahead of time. Here’s what happens when you enable it:</p> +
<ol type="1"> +
<li>You add modelCacheConfig with weightsCache enabled to your InferenceEndpointConfig or JumpStartModel resource and apply it.</li> +
<li>The HyperPod Inference Operator automatically creates a ModelDataCacheConfig resource and begins downloading model weights from your configured source (Amazon S3, Amazon FSx for Lustre, HuggingFace Hub, or JumpStart) to local NVMe on all target nodes.</li> +
<li>After the node completes the download, the operator labels that node as cache-ready.</li> +
<li>The operator waits until all target nodes become cache-ready before creating the inference deployment, so that your pods can always access local data.</li> +
<li>When you start your pod, it reads from local NVMe storage at typical speeds of approximately 7 GB/s instead of downloading over the network.</li> +
</ol> +
<p>The cache that you configure remains available across pod restarts on the same node. During scale-out, if new pods land on nodes that already have the weights cached, they start immediately.</p> +
<h3 id="image-cache">Image cache</h3> +
<p>The image cache pre-pulls the inference server container image onto nodes so pods do not wait for ECR downloads. Here’s what happens when you enable it:</p> +
<ol type="1"> +
<li>You add modelCacheConfig with imageCache enabled to your resource and apply it.</li> +
<li>The operator creates a DaemonSet that pulls the container image onto all target nodes.</li> +
<li>The inference deployment is created immediately by the operator. Unlike the weights cache, the image cache does not block deployment creation.</li> +
<li>When you start a pod with the image already cached, it skips the ECR pull entirely, saving 5–7 minutes.</li> +
<li>When you start a pod before the image cache is complete on that node, it pulls from ECR normally.</li> +
</ol> +
<p>Multiple deployments that use the same container image share a single image cache resource. The operator tracks references and only cleans up the cached image when no deployments reference it.</p> +
<h3 id="fallback-behavior">Fallback behavior</h3> +
<p>Both caching capabilities use preferred scheduling rather than required scheduling. Pods prefer nodes with cached data, but they’re never blocked from starting. When your scheduler places your pod on a node without a warm cache (for example, during rapid scale-out that exceeds the number of…+
<h2 id="architecture-and-crds">Architecture and CRDs</h2> +
<p>The operator introduces two Custom Resource Definitions (CRDs) to manage caching lifecycles. The operator creates and manages these automatically when you enable caching. You don’t need to create them directly.</p> +
<p><strong>ModelDataCacheConfig</strong> manages the full lifecycle of model weights caching. The operator creates one per InferenceEndpointConfig or JumpStartModel that has weights caching enabled. It controls downloading weights from the source to local NVMe on target nodes, labe…+
<p>You can inspect the state of the weights cache at any time:</p> +
<div class="hide-language"> +
<pre><code class="language-bash">kubectl get modeldatacacheconfig -n &lt;namespace&gt;</code></pre> +
</div> +
<div class="hide-language"> +
<pre><code class="language-plaintext">NAME STATE TARGET READY AGE+
example-model-cache Ready 10 10 5m</code></pre> +
</div> +
<p><strong>ModelImageCache</strong> manages the lifecycle of container image caching. It controls pre-pulling the inference server image onto all target nodes, labeling nodes as image-ready once the pull completes, reporting per-node pull status, and cleaning up when no deployments…+
<div class="hide-language"> +
<pre><code class="language-bash">kubectl get inferenceimagecache -n hyperpod-inference-system</code></pre> +
</div> +
<div class="hide-language"> +
<pre><code class="language-plaintext">NAME PHASE CACHED TARGET AGE+
iic-vllm-openai-ml-g5-24xlarge-a1b2 Complete 10 10 3m</code></pre> +
</div> +
<p>When you change the model source (for example, pointing to a new Amazon S3 path with updated weights), the operator creates a new cache, rolls out the updated deployment, and then cleans up the old cache. The same applies to image changes, ensuring zero-downtime transitions with no stale da…+
<h2 id="how-to-enable-model-caching">How to enable model caching</h2> +
<p>You enable model caching by adding a modelCacheConfig section to your existing InferenceEndpointConfig or JumpStartModel resource. No additional infrastructure setup is needed.</p> +
<h3 id="inferenceendpointconfig-example">InferenceEndpointConfig example</h3> +
<div class="hide-language"> +
<pre><code class="language-yaml">apiVersion: inference.sagemaker.aws.amazon.com/v1+
kind: InferenceEndpointConfig+
metadata:+
name: example-model+
namespace: default+
spec:+
modelName: example-model+
modelSourceConfig:+
modelSourceType: s3+
s3Storage:+
bucketName: example-bucket+
region: us-west-2+
modelLocation: "models/example-model"+
modelCacheConfig:+
weightsCache:+
enabled: true+
imageCache:+
enabled: true+
instanceType: ml.g5.24xlarge+
worker:+
image: vllm/vllm-openai:latest+
modelInvocationPort:+
containerPort: 8000+
modelVolumeMount:+
name: model-weights+
mountPath: /opt/ml/model+
resources:+
limits:+
nvidia.com/gpu: "4"</code></pre> +
</div> +
<h3 id="jumpstartmodel-example">JumpStartModel example</h3> +
<div class="hide-language"> +
<pre><code class="language-yaml">apiVersion: inference.sagemaker.aws.amazon.com/v1+
kind: JumpStartModel+
metadata:+
name: example-jumpstart-model+
namespace: default+
spec:+
model:+
modelId: "meta-textgeneration-llama-3-1-8b-instruct"+
acceptEula: true+
server:+
instanceType: ml.g5.24xlarge+
modelCacheConfig:+
weightsCache:+
enabled: true+
imageCache:+
enabled: true</code></pre> +
</div> +
<p>You can enable either capability independently. If you only want to cache the image, omit weightsCache or set it to false. The weights cache also supports an optional hostPath override if you want to use a non-default NVMe mount path (default is /opt/dlami/nvme). If configured on Amazon Sag…+
<h2 id="supported-model-sources">Supported model sources</h2> +
<p>Model caching works across all model sources supported by HyperPod Inference:</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> -
<td><strong>Resource</strong></td> -
<td><strong>Specification</strong></td> -
</tr> -
<tr> -
<td>GPUs</td> Diff display stops at 400 lines. The line counts above are from the whole diff. 46 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.