Change
33a8ca1
33a8ca138104afce84156bbde92310d51db98d75 · commit on GitHub
aws-blog-feed: changed (667015 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +4,339
- Lines removed
- -4,787
- Stored bytes at this commit
- 667,015
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-10-28T05:42:11.890Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Tue, 27 Oct 2026 23:07:22 GMT |
| date | Wed, 28 Oct 2026 05:42:11 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Mon, 26 Oct 2026 17:56:58 +0000</lastBuildDate>+
<lastBuildDate>Tue, 27 Oct 2026 18:03:12 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,555 +13,314 @@ 1 </sy:updateFrequency> <item>-
<title>Fleet-wide computer vision lifecycle at the edge with SageMaker SDK v3</title>-
<link>https://aws.amazon.com/blogs/machine-learning/fleet-wide-computer-vision-lifecycle-at-the-edge-with-sagemaker-sdk-v3/</link>+
<title>AWS named a Leader in the 2026 Gartner Magic Quadrant for Contact Center as a Service</title>+
<link>https://aws.amazon.com/blogs/machine-learning/aws-named-a-leader-in-the-2026-gartner-magic-quadrant-for-contact-center-as-a-service/</link>+
-
<dc:creator><![CDATA[Kiowa Jackson]]></dc:creator>-
<pubDate>Mon, 26 Oct 2026 17:56:58 +0000</pubDate>-
<category><![CDATA[Advanced (300)]]></category>-
<category><![CDATA[Amazon SageMaker]]></category>-
<category><![CDATA[Technical How-to]]></category>-
<guid isPermaLink="false">24d8d5b36e78caf18670ef6d83892be19ec25356</guid>+
<dc:creator><![CDATA[Pasquale DeMaio]]></dc:creator>+
<pubDate>Tue, 27 Oct 2026 18:03:12 +0000</pubDate>+
<category><![CDATA[Amazon Connect]]></category>+
<category><![CDATA[Announcements]]></category>+
<category><![CDATA[Artificial Intelligence]]></category>+
<guid isPermaLink="false">ccf2074f88c67d02606251e40a31cdb4a40bc3ad</guid>-
<description>Fleet-wide computer vision needs a model per camera, not one model for the whole fleet. This post presents a working Amazon SageMaker SDK v3 reference architecture that trains, quality-gates, compiles per-camera models with SageMaker Neo, and versions thousands of them, with an AWS…-
<content:encoded><p>Fleet-wide computer vision model lifecycle management is a problem that shows up the moment a quality inspection proof of concept succeeds. One camera and one model work fine. Then the plant asks for the same result on 40 lines, each with several cameras, and ever…-
<p>This post updates the pattern introduced in the 2023 series <a href="https://aws.amazon.com/blogs/machine-learning/build-an-end-to-end-mlops-pipeline-for-visual-quality-inspection-at-the-edge-part-1/" target="_blank" rel="noopener">Build an end-to-end MLOps pipeline for visual quality…-
<p>The complete sample is available in the <a href="https://github.com/aws-samples/sample-sagemaker-fleet-cv-pipeline" target="_blank" rel="noopener">aws-samples GitHub repository</a>.</p> -
<h2 id="solution-overview">Solution overview</h2> -
<p>The following diagram shows the sample’s fleet-wide model lifecycle. An orchestration layer queues re-training requests and limits concurrent work. A per-camera training pipeline prepares data, trains and evaluates an anomaly detector and a classifier, and produces versioned model artifacts…-
<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/19/ML-20483-1-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/19/ML-20483-1-1.png" alt="Solution architectu…-
<p class="wp-caption-text">Figure 1: Solution architecture</p>-
</div> -
<p>The training half is a single SageMaker Pipeline, parameterized by camera, that runs 13 steps per execution. Three entry steps run in parallel: Preprocess, Fetch Latest Anomaly, and Fetch Latest Classifier. The pipeline preprocesses that camera’s images and fetches the camera’s most recent …-
<p>The orchestration half paces the fleet. Re-training requests land on an Amazon Simple Queue Service (Amazon SQS) FIFO queue, and an AWS Lambda function starts a Step Functions execution per request. Each execution acquires a slot through a DynamoDB semaphore before starting its pipeline, so…-
<p>The output of the system is versioned, quality-gated, per-camera model artifacts. The Model Registry holds the portable ONNX, and S3 holds the Neo-compiled build of that model, which is a device-optimized shared library loaded by the DLR runtime rather than ONNX. Both are ready for whatever…-
<h2 id="prerequisites">Prerequisites</h2> -
<p>To deploy the sample you need the following:</p> -
<ul> -
<li>An AWS account with permissions to create AWS Identity and Access Management (IAM) roles.</li> -
<li>An Amazon Simple Storage Service (Amazon S3) bucket.</li> -
<li>An Amazon Elastic Container Registry (Amazon ECR) repository.</li> -
<li>The SageMaker AI, AWS Step Functions, AWS Lambda, Amazon Simple Queue Service (Amazon SQS), Amazon Simple Notification Service (Amazon SNS), Amazon EventBridge, and Amazon DynamoDB resources in the stacks.</li> -
<li>Python 3.10, 3.11, or 3.12, Docker.</li> -
<li>The AWS CDK CLI version 2.1131.0 or later.</li> -
</ul> -
<p>Previous CLI versions fail with a cloud assembly schema mismatch against recent versions of <code>aws-cdk-lib</code>. No service quota increase is needed. The orchestration layer admits at most five pipeline executions at a time by default, which keeps concurrent SageMaker train…-
<p>If you deploy to an AWS Region other than your CLI default, set all three of <code>AWS_REGION</code>, <code>AWS_DEFAULT_REGION</code>, and <code>CDK_DEFAULT_REGION</code> on the deploy command itself rather than in your shell profile, for example, <cod…-
<h2 id="defining-the-training-pipeline-with-sdk-v3">Defining the training pipeline with SDK v3</h2> -
<p>The code in this section is excerpted from the CDK application in the repository rather than a standalone script, and it runs during cdk deploy, when the stack synthesizes the pipeline definition. Environment setup and the deploy commands come later, under Deploying and running the sample. …-
<div class="hide-language"> -
<pre><code class="language-python">from sagemaker.train import ModelTrainer-
from sagemaker.train.configs import InputData, Compute, OutputDataConfig-
from sagemaker.core.training.configs import MetricDefinition-
from sagemaker.serve.model_builder import ModelBuilder-
from sagemaker.mlops.workflow.pipeline import Pipeline-
from sagemaker.mlops.workflow.steps import ProcessingStep, TrainingStep-
from sagemaker.mlops.workflow.model_step import ModelStep-
from sagemaker.mlops.workflow.condition_step import ConditionStep-
from sagemaker.core.workflow.conditions import ConditionGreaterThanOrEqualTo-
from sagemaker.core.workflow.functions import JsonGet</code></pre> -
</div> -
<p>Installing <code>sagemaker&gt;=3.0.0,&lt;4</code> pulls in <code>sagemaker-core</code>, <code>sagemaker-train</code>, <code>sagemaker-serve</code>, and <code>sagemaker-mlops</code> as separate packages. Training configurati…-
<div class="hide-language"> -
<pre><code class="language-python">anomalib_trainer = ModelTrainer(-
training_image=custom_training_image_uri,-
compute=Compute(-
instance_type=training_instance_type.to_string(),-
instance_count=1,-
),-
base_job_name="cv-pipeline-anomalib-train",-
sagemaker_session=pipeline_session,-
role=role_arn,-
hyperparameters={-
"program": "train_anomalib.py",-
"model_name": "autoencoder",-
"image_size": "256",-
"num_epochs": "3",-
"batch_size": "16",-
"learning_rate": "0.001",-
},-
input_data_config=anomalib_input_data,-
output_data_config=OutputDataConfig(-
s3_output_path=run_prefix("anomalib-training")-
),-
).with_metric_definitions(-
[-
MetricDefinition(-
name="image_auroc",-
regex=r"validation:image_auroc=([0-9]*\.?[0-9]+)",-
),-
]-
)</code></pre> -
</div> -
<p><code>anomalib_input_data</code> wires four channels (<code>train</code>, <code>validation</code>, <code>test</code>, <code>pretrained_model</code>) from upstream step properties, and <code>run_prefix</code> scopes ever…-
<p>Two v3 behaviors matter when the pipeline lives inside a CDK application, and neither appears in the changelog as a breaking change. First, SDK v3 validates the pipeline’s execution role against live IAM while building the pipeline definition. This validation happens at synthesis time rathe…-
<p>Second, the v3 role validator simulates ECR pull permissions without a resource ARN, so an execution role scoped to a single ECR repository fails validation even though it would work at runtime. The sample’s role carries a broader ECR grant for this reason.</p> -
<p>Lastly, one v3 habit the sample uses: constructing the pipeline with <code>PipelineSession(default_bucket=your_bucket)</code> routes SDK-managed uploads, such as step code, into the pipeline’s own bucket, so every artifact lives in one place and the execution role’s permissions …-
<h2 id="quality-gates-that-match-how-defect-data-behaves">Quality gates that match how defect data behaves</h2> -
<p>Every trained model is evaluated before registration, and the evaluation runs against the exported ONNX artifact rather than the in-memory PyTorch model. That distinction matters: the artifact you gate is the artifact you ship, so an export problem can never slip a broken model past a healt…-
<p>The anomaly branch gates on AUROC of at least 0.8. The classifier branch gates on two metrics at once, and the choice of metrics is shaped by the defining property of defect data: heavy class imbalance. In this sample roughly 90 percent of images are good, which is realistic for a productio…-
<div class="hide-language"> -
<pre><code class="language-python">step_check_classifier = ConditionStep(-
name="CheckClassifierQuality",-
conditions=[-
ConditionGreaterThanOrEqualTo(-
left=JsonGet(-
step_name=step_eval_classifier.name,-
property_file="evaluation",-
json_path="balanced_accuracy",-
),-
right=min_accuracy,-
),-
ConditionGreaterThanOrEqualTo(-
left=JsonGet(-
step_name=step_eval_classifier.name,-
property_file="evaluation",-
json_path="defect_recall",-
),-
right=min_defect_recall,-
),-
],-
if_steps=[step_compile_classifier, step_register_classifier],-
else_steps=[step_fail_classifier],-
depends_on=[step_eval_classifier],-
)</code></pre> -
</div> -
<p>A <code>ConditionStep</code> ANDs its conditions, so both must pass. The thresholds are pipeline parameters, adjustable per execution without redeploying. If a gate fails, a <code>FailStep</code> marks the execution failed and that branch’s model never reaches the re…-
<p>The training code addresses the same imbalance from the other side. The classifier is a fine-tuned <code>resnet18</code> with ImageNet weights and a two-class head. At per-camera dataset sizes, fine-tuning pretrained weights is what keeps the model from collapsing onto the major…-
<h2 id="compiling-for-the-edge-with-sagemaker-neo-inside-the-pipeline">Compiling for the edge with SageMaker Neo, inside the pipeline</h2> -
<p>Notice the gate’s <code>if_steps</code> in the preceding code: past the gate, compilation runs first and registration depends on it. Both training scripts export ONNX with dynamic batch axes. The quality gates score those exported artifacts under ONNX Runtime, and then a <cod…-
<p>The compile step encapsulates three error-prone details. Neo requires the input tarball to contain only the model file, but training jobs emit <code>model.tar.gz</code> with the checkpoint and metrics alongside the ONNX model. The step repackages an ONNX-only tarball first. Neo …-
<p>The target is a pipeline parameter: <code>NeoTargetDevice</code> selects a named device such as <code>jetson_xavier</code> or <code>rasp4b</code>, and its default compiles for the <code>NeoTargetOs</code>/<code>NeoTargetArch</code>…-
<h2 id="per-camera-models-in-the-model-registry">Per-camera models in the Model Registry</h2> -
<p>Each branch registers into its own model package group, and every package carries the camera ID and its evaluation metrics. From the sample, with the metrics wiring elided:</p> -
<div class="hide-language"> -
<pre><code class="language-python">classifier_model_builder = ModelBuilder(-
s3_model_data_url=classifier_model_data,-
image_uri=custom_training_image_uri,-
sagemaker_session=pipeline_session,-
role_arn=role_arn,-
)-
step_register_classifier = ModelStep(-
name="RegisterClassifierModel",-
depends_on=[step_compile_classifier],-
step_args=classifier_model_builder.register(-
model_package_group_name="sm1k-classifier-model-group",-
approval_status="Approved",-
model_metrics=classifier_metrics,-
customer_metadata_properties={"CameraId": camera_id},-
),-
)</code></pre> -
</div> -
<p>Camera identity in <code>customer_metadata_properties</code> is what turns two shared package groups into a fleet catalog. Packages from all cameras live in the same groups, and per-camera history is recovered by filtering on the metadata. Each branch has a fetch step that pages…-
<p>Note what these registry entries are and are not: they catalog versioned, quality-gated artifacts, but their container reference is the training image, which serves no inference endpoints. They exist for a fleet workflow where a downstream system pulls artifacts and ships them to devices, n…-
<h2 id="orchestrating-fleet-training-requests">Orchestrating fleet training requests</h2> -
<p>Every AWS account has service quotas on concurrent SageMaker training jobs, and each pipeline execution here launches several jobs, so fleet re-training needs pacing. Something has to keep a burst of 500 requests from becoming 500 simultaneous pipeline launches. The sample uses a semaphore,…-
<div class="hide-language"> -
<pre><code class="language-python">acquire_semaphore = tasks.DynamoUpdateItem(-
self, "AcquireSemaphore",-
table=semaphore_table,-
key={"LockName": tasks.DynamoAttributeValue.from_string("pipeline-concurrency")},-
update_expression="SET ActiveCount = if_not_exists(ActiveCount, :zero) + :one",-
condition_expression="attribute_not_exists(ActiveCount) OR ActiveCount &lt; :limit",-
)</code></pre> -
</div> -
<p>If all slots are taken, the execution waits and retries. When its pipeline finishes, it decrements the counter to release the slot. One detail is worth carrying into your own designs. The DynamoDB condition expression enforces the cap in a single atomic operation, rather than the state mach…-
<p>The cap of five is a stack parameter, set low so the sample deploys and runs in any standard account without requesting quota increases. Adjust it to whatever your account’s training-job quotas support. Past a single account’s quotas, scaling fleet training is its own topic. The standard le…-
<h2 id="deploying-and-running-the-sample">Deploying and running the sample</h2> -
<p>The walkthrough in the repository README is the authoritative version. The following short form starts from a clean checkout and sets up the environment before the first deploy:</p> -
<div class="hide-language"> -
<pre><code class="language-bash">git clone https://github.com/aws-samples/sample-sagemaker-fleet-cv-pipeline-
cd sample-sagemaker-fleet-cv-pipeline &amp;&amp; python3 -m venv .venv &amp;&amp; source .venv/bin/activate-
pip install -r requirements.txt-
cdk bootstrap # once per account/region-
cdk deploy SageMakerCVPipelineStack -c bootstrap=true # roles, bucket, ECR-
cdk deploy --all # pipeline + orchestration-
./scripts/push_ecr_image.sh # build and push training image</code></pre> -
</div> -
<p>By default the pipeline generates synthetic production line images, so you can validate the whole system end to end with no dataset and no data approvals. A full run executes all 13 steps, including both Neo compilations, in about 15 minutes and costs roughly $0.07 in billed <code>ml.…-
<p>To exercise the pipeline on real images, the repository includes a downloader for the <a href="https://github.com/Shenggan/BCCD_Dataset" target="_blank" rel="noopener">BCCD dataset</a>. The dataset is a collection of 364 annotated blood smear microscopy images originally open so…-
<h2 id="cleaning-up">Cleaning up</h2> -
<p>Model packages are created at pipeline runtime, so AWS CloudFormation cannot delete them. Remove them with the AWS CLI first, then destroy the stacks with <code>cdk destroy --all -c bootstrap=true --force</code>. The <a href="https://github.com/aws-samples/sample-sagemaker-fl…-
<h2 id="conclusion">Conclusion</h2> -
<p>In this post, I showed a working SageMaker SDK v3 reference architecture for the fleet-scale computer vision lifecycle. The architecture includes a dual-branch training pipeline where every model is quality-gated on its exported ONNX artifact, compiled with SageMaker Neo, and only then regi…-
<p>The sample deploys in minutes and a full validation run costs about seven cents of compute, so a low-cost next step is to clone the <a href="https://github.com/aws-samples/sample-sagemaker-fleet-cv-pipeline" target="_blank" rel="noopener">repository</a> and run it. From there, p…-
<p>If you want help adapting this pattern to your production environment, reach out to your AWS account team.</p> -
<p>&nbsp;</p> -
<p style="clear: both"></p> +
<description>AWS has been named a Leader in the 2026 Gartner Magic Quadrant for Contact Center as a Service (CCaaS) for AWS Connect Customer. This recognition marks an important milestone for our team, and it reflects years of work pushing on what's possible in customer experience.&nbsp;</d…+
<content:encoded><p><img class="alignnone size-full wp-image-142251" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/26/22275-Gartner.png" alt="" width="2893" height="3200"></p> +
<p>I’m proud to share that Amazon Web Services (AWS) has been named a Leader in the 2026 Gartner Magic Quadrant for Contact Center as a Service (CCaaS) for Amazon Connect for the fourth consecutive year. We believe this recognition reflects years of work pushing on what’s possible in customer …+
<p>When we launched Amazon Connect Customer, we set out to evolve how businesses serve their customers: to make personalized, intelligent experiences possible without giving up efficiency, flexibility, or control. That belief still drives how we build today.</p> +
<p>Customer expectations have shifted. People expect you to recognize them, understand what they need, and stay consistent across every channel. They don’t want to repeat themselves. They don’t want to navigate disconnected systems to solve a simple problem. They want interactions that feel na…+
<p><strong>Customer experience without compromise</strong></p> +
<p>What matters most to your customers hasn’t changed: they want to be known, they want their problems fully resolved, and they want to trust you in the moments that matter. Read the <a href="https://aws.amazon.com/resources/analyst-reports/gartner/global-mq-ardm-26-contact-center-as-a-serv…+
<p><strong>Experiences that start with knowing the customer. </strong>&nbsp;Connect Customer builds a living picture of every customer: who they are, what they’ve been through, what they need next. Every touchpoint draws on that intelligence. Your customer feels known before a …+
<p><strong>Resolution that carries all the way through. </strong></p> +
<p>Most of the industry is still optimizing for deflection: getting customers away from humans as fast as possible. We took a different path. Connect Customer is built for the hard things: the disputed claim, the renegotiated mortgage, the disrupted itinerary that needs real action and can’t g…+
<p>Generative and agentic AI push this further: understanding what a customer needs and acting on it. Every action is auditable. Humans step in when the moment calls for it. You get enterprise scale and the confidence that nothing slips through the cracks.</p> +
<p><strong>Building for what’s next</strong></p> +
<p>The companies that win the next decade of customer experience won’t be the ones with the most channels or the fastest bot. They’ll be the ones whose customers feel known, whose problems actually get resolved, and whose trust is earned in the moments that matter most. Connect Customer exists…+
<p>I’m proud of the Connect Customer team for receiving this recognition for the fourth year in a row. Our engineers, product teams, partners, and customers shaped what we’ve built. This is a moment to celebrate how far we’ve come, and a signal of what’s ahead.</p> +
<p><b><span data-contrast="auto">Every customer is unique. Every interaction should reflect that.</span></b></p> +
<p>Read the 2026 Gartner Magic Quadrant for Contact Center as a Service:&nbsp;<a href="https://aws.amazon.com/resources/analyst-reports/gartner/global-mq-ardm-26-contact-center-as-a-service/?trk=21a9418a-9a81-4797-8372-7df087f349c5&amp;sc_channel=el" target="_blank" rel="noopener"&g…+
<p><em>Gartner does not endorse any company, vendor, product or service depicted in its publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner publications consist of the opinions of Gartner’s business and te…+
<p><em>Gartner and Magic Quadrant are a trademark of Gartner, Inc., and/or its affiliates.</em></p> +
<p><em>This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from AWS.</em></p> <hr style="width: 100%"> -
<h2>About the author</h2> +
<h2>About the authors</h2> <footer> <div class="blog-author-box" style="padding-top: 2.0em"> <div class="blog-author-image" style="margin-right: 1.0em">-
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/19/ML-20483-2-1.jpg" alt="Kiowa Jackson" width="100" height="133">+
<img loading="lazy" class="alignnone wp-image-142255 size-thumbnail" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/26/22275-author-100x133.png" alt="" width="100" height="133"> </div> -
<h3 class="lb-h4">Kiowa Jackson</h3> -
<p style="overflow: hidden">Kiowa is a Senior Machine Learning Engineer at AWS ProServe. He specializes in visual intelligence systems that combine computer vision with agentic AI, and he brings these to production in the enterprise industrial sector.</p> +
<h3 class="lb-h4">Pasquale DeMaio</h3> +
<p style="overflow: hidden">Pasquale is the Vice President of AWS Connect Customer. Founded in 2015, AWS Connect Customer is a purpose-built agentic AI solution for customer experience powering the full lifecycle of customer interactions, from proactive outreach to self-service resolution to… </div> </footer></content:encoded> +
+
</item> <item>-
<title>Groundhog Day for AI agents: Prompt caching patterns for using OpenAI models on Amazon Bedrock</title>-
<link>https://aws.amazon.com/blogs/machine-learning/groundhog-day-for-ai-agents-prompt-caching-patterns-for-using-openai-models-on-amazon-bedrock/</link>+
<title>Announcing SOCI-indexed images for faster SageMaker AI jobs</title>+
<link>https://aws.amazon.com/blogs/machine-learning/announcing-soci-indexed-images-for-faster-sagemaker-ai-jobs/</link> -
<dc:creator><![CDATA[Claudio Mazzoni]]></dc:creator>-
<pubDate>Mon, 26 Oct 2026 17:27:20 +0000</pubDate>-
<category><![CDATA[Amazon Bedrock]]></category>-
<category><![CDATA[Best Practices]]></category>-
<category><![CDATA[Expert (400)]]></category>-
<guid isPermaLink="false">de80c503415138bb46de169401d016e12857bb15</guid>+
<dc:creator><![CDATA[Kanwaljit Khurmi]]></dc:creator>+
<pubDate>Tue, 27 Oct 2026 17:00:39 +0000</pubDate>+
<category><![CDATA[Advanced (300)]]></category>+
<category><![CDATA[Amazon SageMaker AI]]></category>+
<category><![CDATA[Announcements]]></category>+
<guid isPermaLink="false">4fef4e2052ceafe5be77506f74414eb6efbd5cd7</guid>-
<description>Every time an agent takes a step, it re-reads its entire context before acting. Here is how prompt caching for OpenAI models on Amazon Bedrock lets your agent remember, cutting input cost and latency without resending the same tools, instructions, and history every turn.</descripti…-
<content:encoded><p><em>Every time you spin off your agent, it has to re-read its whole life story before performing an action. Here’s how to make it remember without having to feed it the same context again.</em></p> -
<p>In the movie <em>Groundhog Day</em>, Bill Murray’s character re-lives the same February 2 every day. He wakes up to the same song playing on the radio, in the same inn, and the same small town, again and again for what some movie fans estimate to be 34 years.</p> -
<p>Nothing he accomplished the day before carries over unless he works it out again from scratch. AI agents live in a version of Groundhog Day. Every time an agent takes a step (calls a tool, reads the result, decides what to do next), it has to send the model its entire story so far to mainta…-
<p>Prompt caching breaks the cycle. The service stores your input prompt (instructions, tools, and content) and reuses it on the next request, at a fraction of the input price.</p> -
<p>In this post, I walk you through an agent’s life, from its first turn to when it gets new tools and spawns helpers. Along the way, I show you how prompt caching keeps costs down and keeps agents responsive and consistent.</p> -
<h2 id="what-an-agent-sends-to-its-llm">What an agent sends to its LLM</h2> -
<p>Before talking about caching, it helps to look at what a typical agent’s first invocation to the large language model (LLM) contains:</p> +
<description>Seekable OCI (SOCI) indexing for Amazon SageMaker AI training and processing jobs takes full container image download off the critical path. SageMaker AI lazy-loads your image to start containers in seconds instead of minutes, with no changes to your job definition. See the benchma…+
<content:encoded><p>Every minute your deep learning container image spends downloading is a minute you pay for and a minute your team waits. Container image download is pure overhead: billable job time that produces no model progress. For modern deep learning images, that overhead is…+
<p>With SOCI indexing, SageMaker AI lazy-loads your container image. It fetches only the files needed to start the container and streams the rest in the background as your code runs, so your job begins in seconds instead of minutes. You adopt it by pointing to a pre-indexed AWS Deep Learning C…+
<p>SOCI-indexed images start Amazon SageMaker AI training containers in seconds, with no impact on training performance. As we show in the <em>What we measured</em> section, this reduced GPU idle time during image pulls by up to 90 percent in our benchmarks.</p> +
<p>This post shows how to adopt SOCI indexing: what it is, the results we measured, and how to turn it on for your own jobs.</p> +
<h1>The challenge</h1> +
<p>SageMaker AI training and processing jobs run inside container images stored in Amazon Elastic Container Registry (Amazon ECR). Before a job can execute your code, the service downloads the entire image to the instance. It decompresses and writes every layer to disk.</p> +
<p>For small images this takes seconds. But deep learning images routinely exceed 10 GB or more. That download adds minutes to every job, and you pay for it as job time. The impact is largest exactly where iteration speed matters most: hyperparameter tuning sweeps for hundreds of jobs, quick e…+
<h1>How SOCI indexing works</h1> +
<p><a href="https://github.com/awslabs/soci-snapshotter" target="_blank" rel="noopener">Seekable OCI (SOCI)</a> is an open source technology that lazy loads container images. Instead of downloading the whole image before starting, the runtime uses a SOCI index to fetch only the fil…+
<p>A SOCI index is metadata that maps every file inside your image. You store it in Amazon ECR as an OCI-compliant artifact linked to your original image. Because the index is a separate artifact, your image digest and any existing signatures remain valid.</p> +
<h2 id="sagemaker-ai-checks-for-an-index-automatically">SageMaker AI checks for an index automatically</h2> +
<p>When you create a training or processing job, SageMaker AI checks whether the image you specified has a SOCI index and behaves accordingly:</p> +
<table border="1px" width="100%" cellpadding="10px"> +
<tbody> +
<tr> +
<td><strong>Scenario</strong></td> +
<td><strong>What happens</strong></td> +
</tr> +
<tr> +
<td><strong>The image includes a SOCI index</strong></td> +
<td>SageMaker AI lazy-loads the image and starts your container before the full download completes</td> +
</tr> +
<tr> +
<td><strong>The image has no SOCI index</strong></td> +
<td>Standard download, no change in behavior</td> +
</tr> +
<tr> +
<td><strong>The index can’t be read, or lazy loading doesn’t succeed</strong></td> +
<td>The job falls back to a standard download and continues running</td> +
</tr> +
</tbody> +
</table> +
<p>Because jobs always fall back to a standard download when needed, adding a SOCI index never prevents your job from running. You only gain faster startup. You never risk breaking an existing workflow.</p> +
<h1>What we measured</h1> +
<p>We pulled the same container images two ways on real SageMaker AI training jobs: standard download and SOCI-indexed lazy loading. We then ran full training jobs from both to measure the effect on training speed.</p> +
<h2 id="setup">Setup</h2> <ul> -
<li><strong>Tool list</strong> – The definitions of every function the agent can call. Changes when you ship a new release.</li> -
<li><strong>Instructions</strong> – The developer prompt: rules, persona, policies, examples. Also changes per release.</li> -
<li><strong>History</strong> – Everything that happened in this session so far: the task, the model’s tool calls, and the tool results (which grows every turn).</li> -
<li><strong>New turn</strong> – The latest tool result or user message (which might be different every time).</li> +
<li><strong>Eight images</strong>, 1.8–35.4 GB compressed.</li> +
<li><strong>GPU and CPU instances</strong> ml.g4dn, ml.g5, ml.g6, ml.g6e, ml.g7, ml.g7e, ml.p4d, ml.p5, and ml.m7i, from xlarge to 48xlarge, in the US West (Oregon) Region (us-west-2) and the US East (N. Virginia) Region (us-east-1).</li> +
<li><strong>Download time</strong> measured from the start of the image pull until the container is ready to run. All figures are medians of repeated runs.</li> </ul> -
<p><strong>What an agent sends to the model, turn after turn if you don’t set up caching:</strong></p> +
<h2 id="result-1-download-time-drops-from-minutes-to-seconds-at-every-image-size">Result 1: Download time drops from minutes to seconds at every image size</h2> +
<table border="1px" width="100%" cellpadding="10px"> +
<tbody> +
<tr> +
<td><strong>Image size (compressed)</strong></td> +
<td><strong>Standard download</strong></td> +
<td><strong>Lazy loading</strong></td> +
<td><strong>Improvement</strong></td> +
</tr> +
<tr> +
<td><strong>1.8 GB</strong></td> +
<td>48 s</td> +
<td>3-11 s</td> +
<td>~8x</td> +
</tr> +
<tr> +
<td><strong>13.2 GB</strong></td> +
<td>3 min 42 s</td> +
<td>~6 s</td> +
<td>~34x</td> +
</tr> +
<tr> +
<td><strong>35.4 GB</strong></td> +
<td>7 min 17 s</td> +
<td>3-11 s</td> +
<td>~38x</td> +
</tr> +
</tbody> +
</table> +
<p>The following chart compares container image download time for a standard Docker pull against SOCI lazy loading across eight image sizes, from 1.8 GB to 35.4 GB.</p> +
<p>Each image size shows two bars. The Docker download time bar (blue) grows steadily with image size, from 48 seconds at 1.8 GB to 7 minutes 17 seconds at 35.4 GB.</p> +
<p>A speedup label above each pair shows the ratio of Docker time to SOCI time. Speedups range from approximately 10x for the smallest image to over 35x for the largest.</p> <div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/23/ML-22246-1-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/23/ML-22246-1-1.png" alt="Five growing agent …-
<p class="wp-caption-text">Figure 1: What an agent sends to the model on every turn</p>+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/23/ML-21882-1-2.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/23/ML-21882-1-2.png" alt="Bar chart comparing…+
<p class="wp-caption-text">Figure 1: Container image download time, standard Docker pull compared to SOCI lazy loading, across eight image sizes</p></div> -
<h2 id="how-prompt-caching-works-under-the-hood">How prompt caching works under the hood</h2> -
<p>When a model reads a prompt, it first does a write called <em>prefill</em>. During prefill, it processes every input token and builds its key and value vectors (also known as key-value, or KV), which form the model’s working state. This happens before the model writes its first …-
<p>Prompt caching saves that state, which contains the system prompt, tools, content, history, and so on. When a later request starts with the same input tokens (that is, the same text), the prompt caching service loads the saved state instead of computing it again. It then processes only what…-
<p><strong>Turn 1 compared to turn 2</strong></p> +
<p>Standard download time grows with image size because every file is decompressed and written. Lazy loading stayed at 3–11 seconds regardless of size, an 8x–46x improvement. The bigger the image, the more minutes you save.</p> +
<h2 id="result-2-the-saving-holds-on-every-instance-type-and-size">Result 2: The saving holds on every instance type and size</h2> +
<p>Running the 13.2 GB image across 13 instance type/size combinations, standard download ranged from 3 min 16 s to 4 min 20 s, while lazy loading held at 6-8 s. That is about 34x faster on every configuration. You don’t need to change instance type or size to benefit.</p> <div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/23/ML-22246-2-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/23/ML-22246-2-1.png" alt="Turn 1 writes the t…-
<p class="wp-caption-text">Figure 2: Turn 1 saves the prompt prefix up to the breakpoint, and turn 2 reuses it and pays full price only for the new question</p>+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/23/ML-21882-2-2.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/23/ML-21882-2-2.png" alt="Chart comparing sta…+
<p class="wp-caption-text">Figure 2: Download time for a 13.2 GB image across 13 instance type and size combinations</p></div> -
<p><strong>Implicit or explicit, and why do we need both?</strong></p> -
<p>For agentic workloads, implicit caching lets Bedrock choose what to cache as the conversation grows. With explicit caching, you can mark shared content, such as tools and instructions, for reuse across sessions and when spinning up helper agents. You don’t always need both forms of caching,…-
<p>New sessions can also reuse that shared starting point while each conversation builds on its own cached history. In this post, we start with the first turn, then show how to carry cached content through later turns, new tools, and across subagents.</p> -
<h2 id="hello-cached-world">Hello cached world</h2> -
<p>The first win is to cache the static part of the prompt: the tools and instructions that stay the same across requests. These examples use your existing Amazon Bedrock client, <code>TOOLS</code>, and <code>task</code>. Replace the shortened <code>INSTRUCTIONS&l…-
<p>When caching static content, use explicit mode: choose where the saved prefix ends.</p> -
<div class="hide-language"> -
<pre><code class="language-python">KEY = "support-agent:v7:gpt-6-astra"-
INSTRUCTIONS = "Your full support policies and examples..."-
-
# Rendered context: [provider context][TOOLS][INSTRUCTIONS] | [task]-
# Cache starts at the beginning; the marker below chooses its end.-
resp = client.responses.create(-
model="us.openai.gpt-6-astra",-
store=False, # Response storage; prompt caching still works.-
include=["reasoning.encrypted_content"],-
prompt_cache_key=KEY,-
prompt_cache_options={"mode": "explicit", "ttl": "30m"},-
tools=TOOLS, # Rendered before instructions; keep order stable.-
input=[-
{-
"role": "developer",-
"content": [{-
"type": "input_text",-
"text": INSTRUCTIONS,-
# Save through this whole block, including TOOLS.-
"prompt_cache_breakpoint": {"mode": "explicit"},-
}],-
},-
# User message is processed but not cached.-
{"role": "user", "content": task},-
],-
)</code></pre> -
</div> -
<p>The <code>prompt_cache_key</code> tells the service which requests are meant to share a cache. We come back to how to use it when we get to reading the cache in new sessions. In one of our tests using explicit caching, the roughly 4,000-token tools-and-instructions prefix wrote …-
<p>Keep in mind that prompt caching works only when you pass exactly the same text. Make sure that the agent’s instructions and tool lists are serialized the same way every time. Model Context Protocol (MCP) servers and other forms of tool delivery need to be indexed to make sure they stay con…-
<p>As the content grows and varies (different tools and content), use implicit mode to let the service follow the conversation. Here, the first request has no explicit marker. Its latest eligible message is the user’s task, so the automatic breakpoint comes after that task and covers the tools…Diff display stops at 400 lines. The line counts above are from the whole diff. 60 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.