Change
a595fbf
a595fbfd2ddc4e0340733523e8f2b9244ee4932f · commit on GitHub
aws-blog-feed: changed (734012 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +2,971
- Lines removed
- -3,118
- Stored bytes at this commit
- 734,012
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-09-19T04:38:08.708Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Sat, 19 Sep 2026 00:23:02 GMT |
| date | Sat, 19 Sep 2026 04:38:08 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Thu, 17 Sep 2026 18:02:01 +0000</lastBuildDate>+
<lastBuildDate>Fri, 18 Sep 2026 21:17:31 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,597 +13,1065 @@ 1 </sy:updateFrequency> <item>-
<title>Reduce time-to-hire for quality candidates with AI-powered Amazon Connect Talent</title>-
<link>https://aws.amazon.com/blogs/machine-learning/reduce-time-to-hire-for-quality-candidates-with-ai-powered-amazon-connect-talent/</link>-
+
<title>Amazon SageMaker Inference: 2026 year-to-date launches in review</title>+
<link>https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-inference-2026-year-to-date-launches-in-review/</link> -
<dc:creator><![CDATA[Ayesha Borker]]></dc:creator>-
<pubDate>Thu, 17 Sep 2026 17:55:20 +0000</pubDate>-
<category><![CDATA[Amazon Connect]]></category>+
<dc:creator><![CDATA[Kareem Syed-Mohammed]]></dc:creator>+
<pubDate>Fri, 18 Sep 2026 20:52:14 +0000</pubDate>+
<category><![CDATA[Amazon SageMaker AI]]></category> <category><![CDATA[Announcements]]></category>-
<guid isPermaLink="false">a6559b315a35bac095e4d57dabbff35e09f15496</guid>-
-
<description>Amazon Connect Talent is an AI hiring solution built for talent acquisition leaders managing scaled hiring. It delivers AI-led interviews, data-driven assessments, and consistent evaluation, helping recruiters identify strong candidates more efficiently while providing applicants w…-
<content:encoded><p>Hiring at scale in industries such as retail, logistics, hospitality, and others has its fair share of challenges. Recruiting teams are expected to fill hundreds of roles within tight timelines, often with limited capacity and with tools that weren’t designed to s…-
<p>Today, we’re launching <a href="https://aws.amazon.com/products/connect/talent/" target="_blank" rel="noopener">Amazon Connect Talent,</a> an AI hiring solution built for talent acquisition leaders managing scaled hiring. It delivers AI-led interviews, data-driven assessments, a…-
<p>Recruiters configure the <a href="https://docs.aws.amazon.com/talent/latest/userguide/evaluations.html" target="_blank" rel="noopener">evaluation criteria</a>, assessments, and interview questions based on job requirements. <a href="https://docs.aws.amazon.com/talent/latest/u…-
<p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/16/ConnectTalent-1.png" target="_blank" rel="noopener"><img class="alignnone wp-image-139627 size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b5…-
<p><em>Amazon Connect Talent lets recruiters configure AI-led interviews and candidate assessments in minutes, tailored to each role.</em></p> -
<p><strong> <a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/16/ConnectTalent-2-1.png" target="_blank" rel="noopener"><img loading="lazy" class="alignnone wp-image-139629 size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836c…-
<p><em>Candidates experience a consistent, structured AI-led interview — available whenever they’re ready, day or night.</em></p> -
<h2><strong>Informed by decades of Amazon’s hiring science</strong></h2> -
<p>Amazon is one of the world’s largest employers, and Amazon Connect Talent puts decades of Amazon’s hiring science to work for organizations, with a highly configurable solution that adapts to specific hiring requirements. With consistent, evidence-based assessments applied to every candidat…-
<h3><strong>Reducing human preconceptions in hiring</strong></h3> -
<p>Connect Talent’s AI focuses exclusively on measuring a candidate’s job-related competencies, testing abilities including problem-solving, logic, listening, and role-specific capabilities. All candidate data is anonymized during AI evaluation, removing factors that can introduce unconscious …-
<h3><strong>Transparency and consistent evaluation standards</strong></h3> -
<p>Connect Talent communicates clearly to both recruiters and candidates about what data is collected, how it is used, and what is not collected. Candidates are informed of what to expect before they proceed to take the evaluation. Each competency is scored against a rubric that defines what a…-
<h3><strong>Human in the loop: recruiter control at every step</strong></h3> -
<p>Recruiters maintain final decision authority over every hire. Connect Talent gives them scored candidate summaries with competency breakdowns, complete interview transcripts, comparative analytics, and clear reasoning behind every score, all in a single view instead of pieced together from …-
<p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/16/ConnectTalent-3.png" target="_blank" rel="noopener"><img loading="lazy" class="alignnone wp-image-139630 size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f…-
<p><em>Amazon Connect Talent lets recruiters configure AI-led interviews and candidate assessments in minutes, tailored to each role.</em></p> -
<p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/16/ConnectTalent-4.png" target="_blank" rel="noopener"><img loading="lazy" class="alignnone wp-image-139631 size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f…-
<p><a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/16/ConnectTalent-5.png" target="_blank" rel="noopener"><img loading="lazy" class="alignnone wp-image-139632 size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f…-
<p><em>Amazon Connect Talent lets recruiters review candidate responses and evaluation scores with details and full transparency</em></p> -
<h2><strong>Enterprise-grade security built on AWS infrastructure</strong></h2> -
<p>Hiring data is sensitive. Candidate information deserves the same level of protection&nbsp;you’d&nbsp;expect for financial records or health information. Amazon Connect Talent delivers enterprise-grade security built on AWS infrastructure:&nbsp;the same foundation trusted by ban…-
<p>Built-in security controls help protect candidate data, with rigorous measures to meet your requirements. Access is controlled, auditable, and configurable. When your candidates share their information, they can trust&nbsp;it’s&nbsp;protected.</p> -
<ul> -
<li><strong>Integrity monitoring and fraud protection:&nbsp;</strong>Amazon Connect Talent uses&nbsp;text-based&nbsp;analysis to flag unnatural cadence, filler words, pauses, and response latency.&nbsp;Human review is mandatory for every flag. No candidate is disqu…-
<li><strong>Audit trails and explainability:&nbsp;</strong>Every&nbsp;candidate&nbsp;interaction is logged with a complete audit trail and clear job-related evaluation reasoning. The system includes ongoing monitoring and tracking to help your organization&nbsp;dem…-
</ul> -
<h2><strong>Improved business outcomes</strong></h2> -
<p>Amazon Connect Talent improves business outcomes by getting recruiters out of the workflow management cycle and empowering them with faster decision-making. Candidates get a faster and more flexible hiring experience. Organizations fill roles before lost revenue, rising costs, and competiti…-
<p>Your business moves fast and now your hiring can too.</p> -
<p><iframe loading="lazy" title="Amazon Connect Talent Introduction" width="500" height="281" src="https://www.youtube-nocookie.com/embed/0X1PH7ZwvRo?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" r…-
<p><a href="https://aws.amazon.com/products/connect/talent/" target="_blank" rel="noopener noreferrer">Learn more about Amazon Connect Talent</a> and discover how AI-powered hiring can transform your talent acquisition strategy.</p> -
<h2>About the authors</h2> -
<footer> -
<div class="blog-author-box" style="padding-top: 2.0em"> -
<div class="blog-author-image" style="margin-right: 1.0em">-
<img loading="lazy" class="alignnone wp-image-139633 size-thumbnail" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/16/Ayesha-100x133.jpg" alt="" width="100" height="133">-
</div> -
<h3 class="lb-h4">Ayesha Borker</h3> -
<p style="overflow: hidden">Ayesha is a Principal Solutions Architect, Applied AI at AWS. With over a decade of experience at the intersection of human and AI collaboration specializing in customer experience. She simplifies the complexity of AI, helping organizations stay focused on the out…-
</div> -
<div class="blog-author-box" style="padding-top: 2.0em"> -
<div class="blog-author-image" style="margin-right: 1.0em">-
<img loading="lazy" class="alignnone size-thumbnail wp-image-139635" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/16/Kate-100x133.jpg" alt="" width="100" height="133">-
</div> -
<h3 class="lb-h4">Kate Totaro</h3> -
<p style="overflow: hidden">Kate is the Principal Product Manager for Amazon Connect Talent, AWS’s AI-powered hiring service. She brings 20 years of experience building global businesses, including more than 15 years at Amazon and AWS across product, business, and operations. At Amazon, she …-
</div> -
</footer></content:encoded>-
-
-
-
-
-
</item>-
<item>-
<title>Selecting a vector store for Amazon Bedrock Knowledge Bases</title>-
<link>https://aws.amazon.com/blogs/machine-learning/selecting-a-vector-store-for-amazon-bedrock-knowledge-bases/</link>-
-
<dc:creator><![CDATA[Deepak Dalakoti]]></dc:creator>-
<pubDate>Thu, 17 Sep 2026 15:53:13 +0000</pubDate>-
<category><![CDATA[Amazon Bedrock Knowledge Bases]]></category>-
<category><![CDATA[Best Practices]]></category> <category><![CDATA[Intermediate (200)]]></category>-
<guid isPermaLink="false">e129d8e2814c4c6f57fdc5feec60ab4d06f5e04c</guid>+
<guid isPermaLink="false">2ee3ed6a4b11eb69260daaa48214c0e6564f2a28</guid>-
<description>Choosing the right vector store for your Amazon Bedrock Knowledge Bases RAG application affects performance and cost. This post compares Amazon OpenSearch Service, Amazon Aurora PostgreSQL with pgvector, and Amazon S3 Vectors across three RAG use cases, with benchmarks and a practi…-
<content:encoded><p>When building a Retrieval Augmented Generation (RAG) solution with <a href="https://aws.amazon.com/bedrock/knowledge-bases/" target="_blank" rel="noopener">Amazon Bedrock Knowledge Bases</a>, selecting the right vector store impacts performance and cos…-
<p>For broader guidance across all AWS vector solutions, see <a href="https://aws.amazon.com/blogs/machine-learning/aws-vector-solutions-build-agentic-ai-where-your-data-lives/" target="_blank" rel="noopener">AWS vector solutions: Build agentic AI where your data lives</a>. For the…-
<h2 id="how-vector-databases-fit-into-rag-solutions">How vector databases fit into RAG solutions</h2> -
<p>A RAG architecture combines the capabilities of large language models (LLMs) with information retrieval systems to generate more accurate, up-to-date, and contextually relevant responses. It is based on the mathematical concept of a <em>vector</em>, where the text is translated …-
<p>When a user submits a query, it is converted into a vector embedding using an embedding model. The vector database, where document content has been pre-processed, chunked, and stored as vector embeddings, performs a similarity search to find the chunks whose embeddings are most similar to t…-
<p>The vector database serves as the key bridge between raw information and contextual understanding. It transforms unstructured data into a searchable, semantically meaningful knowledge space that helps large language models deliver more precise and relevant responses. Vector databases achiev…-
<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/09/ML-19758-1.jpeg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/09/ML-19758-1.jpeg" alt="Diagram of a RAG arch…-
<p class="wp-caption-text">Figure 1: Retrieval Augmented Generation (RAG) architecture, where documents are chunked, embedded, and stored in a vector database during ingestion, and at query time the query is embedded, similar chunks are retrieved, and passed to the LLM as context for response…-
</div> -
<h2 id="vector-store-backends-for-amazon-bedrock-knowledge-bases">Vector store backends for Amazon Bedrock Knowledge Bases</h2> -
<p>Amazon Bedrock Knowledge Bases with a customer-managed (unmanaged) configuration supports three vector store backends. For the full AWS vector portfolio covering six services, see <a href="https://aws.amazon.com/blogs/machine-learning/aws-vector-solutions-build-agentic-ai-where-your-data…-
<p><a href="https://aws.amazon.com/opensearch-service/" target="_blank" rel="noopener">Amazon OpenSearch Service</a> provides high-speed results from data held in memory. It supports high-dimensional vector embeddings with both managed cluster and serverless deployment options, and…-
<p><a href="https://aws.amazon.com/rds/aurora/" target="_blank" rel="noopener">Amazon Aurora PostgreSQL with pgvector</a> combines the high-performance relational database capabilities of Amazon Aurora with pgvector’s vector similarity search functionality. It supports multiple ind…-
<p><a href="https://aws.amazon.com/s3/features/vectors/" target="_blank" rel="noopener">Amazon S3 Vectors</a> is the AWS cloud object storage service with native vector support, designed for cost-effective storage and querying of vector embeddings at scale. It provides sub-second q…-
<p>To understand how these options perform in practice, let’s examine three distinct RAG use cases, each with different latency, cost, and search requirements, and see which vector database is the best fit for each.</p> -
<h2 id="use-case-1-product-catalog-search">Use case 1: Product catalog search</h2> -
<p>Ecommerce platforms face the challenge of helping customers find exactly what they’re looking for among thousands of products. An effective product search tool must understand natural language queries and scale to handle thousands of concurrent queries during peak shopping periods while mai…-
<h3 id="why-amazon-opensearch-is-the-best-fit-for-this-use-case">Why Amazon OpenSearch is the best fit for this use case</h3> -
<p>Amazon OpenSearch Serverless is well suited for product catalog search because it supports combining semantic understanding with traditional keyword matching through hybrid search capabilities. When dealing with large product catalogs, performance matters. Amazon OpenSearch Serverless handl…-
<p>What makes it particularly valuable for ecommerce is the built-in support for complex filtering and aggregations that power faceted navigation (think filtering by price, brand, or color). You can also choose from multiple distance metrics like cosine similarity or Euclidean distance to fine…-
<p>Amazon OpenSearch Serverless Classic collections offer several optimization options to balance cost and search quality, as detailed in the following section.</p> -
<p><strong>Note:</strong> Amazon Bedrock Knowledge Bases supports both Amazon OpenSearch Serverless and Managed Clusters. The following benchmarks were run on Serverless Classic collections. Amazon OpenSearch Serverless NextGen collections (generally available May 2026) aren’t yet …-
<h3 id="performance-analysis-and-optimizations-for-amazon-opensearch-serverless-vector-search">Performance analysis and optimizations for Amazon OpenSearch serverless vector search</h3> -
<p>Amazon OpenSearch is highly configurable and provides several configuration options. Be careful when selecting these options because they can significantly affect the performance of the vector index. We consider some of these options targeted at optimizing cost and database size and quantit…-
<p>Some of the common optimization options are:</p> -
<ol type="1"> -
<li><strong>Size of vector embeddings</strong>: A larger vector can generally contain more semantic information about the embedded text. However, it also leads to higher memory consumption, which increases vector index size and cost. Modern embedding models like Amazon Titan Text …-
<li><strong>Data type of embeddings</strong>: We can also reduce vector index size (and thus cost) by storing embeddings in lower precision data types, such as binary embeddings. This can significantly reduce the size of the vector index.</li> -
<li><strong>Disk optimized storage</strong>: Amazon OpenSearch Serverless Classic collections offer disk-based vector search (on_disk mode) that applies 32× binary quantization internally while rescoring against full-precision vectors from disk. This preserves quality while reduci…-
</ol> -
<p>Depending on the indexing algorithm used, users may also configure HNSW parameters (ef_construction, m) to tune the trade-off between index build time, memory usage, and search accuracy (for practical guidance, see <a href="https://opensearch.org/blog/a-practical-guide-to-selecting-hnsw-…-
<h3 id="dataset">Dataset</h3> -
<p>We use the “Shopping Queries Data Set” (ESCI), a large dataset of difficult search queries provided by Amazon. The dataset contains 1,215,851 unique US products (title, description, bullets, and brand; approximately 1,140 characters median) and 97,345 judged queries. For each query, the dat…-
<p>Query:</p> -
<p><code>self-seal envelopes without window</code></p> -
<p>Relevant product title:</p> -
<p><code>BAZIC Security Self Seal Envelope 4 1/8" x 9 1/2" #10, No Window Tint Pattern Mailing Envelopes, Peel &amp; Seal, Office Checks Invoices (30/Pack), 1-Pack</code></p> -
<p>Irrelevant product title:</p> -
<p><code>ValBox 200 Count #8 Double Window Envelopes 3 5/8" x 8 11/16" Flip and Seal Double Window Security Check Envelopes- Security Tint Pattern Designed for Home Office Secure Mailing</code></p> -
<p>We sample 5,000 queries (approximately 19 judged products per query, approximately 17 relevant) and index all 1,215,851 product descriptions for benchmarking. We measure retrieval quality (NDCG@10), latency (p50/p95/p99 at concurrency 1 and 10), and index size (ANN in-memory footprint).<…-
<h3 id="vector-index-construction">Vector index construction</h3> -
<p>We test all combinations of embedding dimension (1024, 512, 256) and data type (float, binary), totaling six configurations, plus 1024-float in on_disk mode at the default compression_level: 32x, compared against the 1024-float in-memory baseline (seven configurations total). All indexes us…+
<description>Amazon SageMaker AI shipped 13 inference launches in year-to-date across two deployment paths: fully managed endpoints and Amazon SageMaker HyperPod Inference. This post reviews each launch, from inference recommendations and capacity-aware instance pools to tiered KV caching and d…+
<content:encoded><p>Generative AI inference is uniquely hard: models are tens to hundreds of gigabytes, latency requirements are measured in tokens per second, cold starts can span multiple minutes as containers and weights transfer, GPU capacity is constrained, and traditional monit…+
<p>Amazon SageMaker AI offers customers the ability to deploy AI models and consume them by the instance (instead of by the token), using two paths: managed endpoints for teams that want AWS to handle infrastructure and operations, and Amazon SageMaker HyperPod Inference for teams that need Ku…+
<h2 id="choose-the-deployment-that-fits-your-workload">Choose the deployment that fits your workload</h2> +
<p>The table below compares the two deployment paths across seven dimensions.</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> -
<td><strong>Configuration</strong></td> -
<td><strong>Embedding size</strong></td> -
<td><strong>Embedding type</strong></td> +
<td><strong>Dimension</strong></td> +
<td><strong>Endpoints</strong></td> +
<td><strong>HyperPod</strong></td> </tr> <tr> -
<td>In memory</td> -
<td>1024</td> -
<td>float (baseline)</td> +
<td><strong>Infrastructure</strong></td> +
<td>Fully managed by AWS</td> +
<td>Managed Kubernetes stack</td> </tr> <tr> -
<td>In memory</td> -
<td>512</td> -
<td>float</td> +
<td><strong>Deploy target</strong></td> +
<td>Console, SDK, CLI</td> +
<td>kubectl, Terraform, Console, CLI, SDK</td> </tr> <tr> -
<td>In memory</td> -
<td>256</td> -
<td>float</td> +
<td><strong>Scaling</strong></td> +
<td>Managed auto scaling with Amazon CloudWatch</td> +
<td>Auto scaling with Karpenter, KEDA, CloudWatch</td> </tr> <tr> -
<td>In memory</td> -
<td>1024</td> -
<td>binary</td> +
<td><strong>Customization and Control</strong></td> +
<td>Customizable at the container and model layers</td> +
<td>More customizability with Node level access, frameworks and AMI.</td> </tr> <tr> -
<td>In memory</td> -
<td>512</td> -
<td>binary</td> +
<td><strong>API protocol</strong></td> +
<td>OpenAI compatible with SageMaker endpoint</td> +
<td>HTTP, gRPC and custom load balancer capability</td> </tr> <tr> -
<td>In memory</td> -
<td>256</td> -
<td>binary</td> +
<td><strong>Best for</strong></td> +
<td>Fast and fully managed deployment with minimal ops overhead</td> +
<td>Kubernetes-based, train-to-serve multi-cloud/hybrid-cloud deployments</td> </tr> <tr> -
<td>On disk (32×)</td> -
<td>1024</td> -
<td>float</td> +
<td><strong>Corresponding Launches:</strong></td> +
</tr> +
<tr> +
<td><strong>Launches</strong></td> +
<td>Inference recommendations, Capacity Aware Inference, OpenAI API, Container Caching, Observability, Async Inference Inline Payloads, Prefix-Aware Routing</td> +
<td>Simplified Operator, Tiered KV Cache, Data Capture, Performance Features, Disaggregated Prefill and Decode for HyperPod Inference, Model Caching</td> </tr> </tbody> </table> -
<h3 id="evaluation-methodology">Evaluation methodology</h3> -
<p>For each configuration, we create a vector index in Amazon OpenSearch Serverless (Classic collection), ingest all 1,215,851 products, wait for merges to settle, then run an adaptive warm-up until latency stabilizes before measuring. We measure 1,000 queries × 3 repetitions at concurrency 1 …-
<ol type="1"> -
<li>Retrieval latency: Latency is measured as the time taken to retrieve relevant matches from the vector index as reported by the Amazon OpenSearch results. This doesn’t include the time to convert text to embeddings.</li> -
<li>Retrieval performance: We use the <a href="https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG" target="_blank" rel="noopener">Normalized Discounted Cumulative Gain (NDCG)</a> metric to score the retrievals for each query. This metric measures the quality o…-
<li>Index size: We report the ANN (Approximate Nearest Neighbor) index size, which is the in-memory structure that drives search compute cost and determines capacity requirements. This differs from total store size, which includes the _source JSON copy of each document and varies with documen…-
</ol> -
<h3 id="results">Results</h3> -
<p>Table 1: Semantic search (k-NN only) performance across seven Amazon OpenSearch Serverless configurations (1,215,851 indexed vectors, 5,000 queries, k=10). Latency is server-side at concurrency 1 unless noted. Deltas are paired bootstrap against the 1024-float in-memory baseline. See Table …-
<ol type="1"> -
<li>Reducing dimensions doesn’t always reduce quality. On this dataset, 512-float was statistically indistinguishable from the 1024-float baseline (NDCG 0.3628 vs 0.3627, p = 0.87) at half the index size (2.79 vs 5.34 GiB) and lower latency (25 vs 31 ms p50). Dropping to 256 dimensions showed…-
<li>Binarization offers large index size reductions, but the quality cost depends on the number of dimensions. At 1024 dimensions, binary embeddings reduced index size by 13.4× (0.40 vs 5.34 GiB) with a 5.2 percent NDCG loss and comparable latency (22 vs 31 ms p50). At 256 dimensions the qual…-
<li>Disk mode (on_disk 32×) preserves quality at the cost of latency. At 1024 dimensions, disk mode achieved NDCG 0.3610 (−0.5 percent vs baseline) with the same 0.40 GiB index size as 1024-binary, but at approximately 3× higher latency (99 ms p50 vs 31 ms in-memory). Both 1024-binary and on_…-
</ol> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="images/image1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/18/ML-21809-1-1.png" alt="Two SageMaker AI inference deployment paths delivered in 2026: managed endpoints and HyperPod Inference" wid…+
<p class="wp-caption-text">Figure 1: Two inference paths delivered in 2026</p>+
</div> +
<h2 id="sagemaker-ai-endpoints-from-model-to-production-in-hours">SageMaker AI endpoints: From model to production in hours</h2> +
<p>Managed SageMaker Inference endpoints are the faster path for teams that want AWS to handle GPU provisioning, scaling, and operational monitoring. You bring the model and define the performance target. SageMaker handles the rest. The seven launches year-to-date in 2026 below address deploym…+
<h3 id="inference-recommendations-and-benchmarking-april-2026">Inference recommendations and benchmarking (April 2026)</h3> +
<p>Blog: <a href="https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-now-supports-optimized-generative-ai-inference-recommendations/" target="_blank" rel="noopener">Amazon SageMaker AI now supports optimized generative AI inference recommendations</a></p> +
<p>Choosing the right instance type, serving container, and optimization settings for a generative AI model typically takes two to three weeks of manual benchmarking against 1000+ combinations, requiring expertise most teams do not have in-house. Inference recommendations automate this end-to-…+
<p>Customers specify a model and performance goal (cost, latency, or throughput). SageMaker then runs a three-step process:</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="images/image2.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/18/ML-21809-2-1.png" alt="The three-step inference recommendations process: narrow, optimize, and benchmark" width="800"></a>+
<p class="wp-caption-text">Figure 2: Inference recommendations 3-step process</p>+
</div> +
<ul> +
<li><strong>Narrow.</strong> Filter the instance type space by analyzing model architecture, size, and memory requirements.</li> +
<li><strong>Optimize.</strong> Apply goal-aligned techniques: EAGLE 3.0 speculative decoding for throughput, kernel tuning for latency, tensor parallelism based on model size.</li> +
<li><strong>Benchmark.</strong> Run NVIDIA AIPerf on real GPU infrastructure with statistically rigorous multi-run confidence reporting.</li> +
</ul> +
<p>The output is a SageMaker Model Package with deployment-ready configurations and validated metrics: time to first token (TTFT), inter-token latency (ITL), P50/P90/P99 latency percentiles, throughput, and cost projection. In a demonstrated example, throughput optimization on <code>GPT-…+
<h3 id="capacity-aware-instance-pools-may-2026">Capacity-aware instance pools (May 2026)</h3> +
<p>Blog: <a href="https://aws.amazon.com/blogs/machine-learning/capacity-aware-inference-automatic-instance-fallback-for-sagemaker-ai-endpoints/" target="_blank" rel="noopener">Capacity-aware inference: automatic instance fallback for SageMaker AI endpoints</a></p> +
<p>When a SageMaker endpoint required a single instance type, a capacity shortage meant the endpoint failed before serving a single request. Instance pools address that single point of failure.</p> +
<p>Customers define a prioritized list of up to five instance types. SageMaker automatically works through the list at endpoint creation, during scale-out, and during scale-in. At creation, SageMaker tries the first-choice type and falls back immediately if capacity is unavailable. During scal…+
<p>Per-instance-type CloudWatch metric dimensions enable weighted scaling policies for heterogeneous fleets. Each pool entry can reference a separate optimized model configuration (tensor parallelism on high-memory instances, speculative decoding on mid-tier, quantization on smaller fallbacks)…+
<h3 id="openai-compatible-apis-may-2026">OpenAI-compatible APIs (May 2026)</h3> +
<p>Blog: <a href="https://aws.amazon.com/blogs/machine-learning/announcing-openai-compatible-api-support-for-amazon-sagemaker-ai-endpoints/" target="_blank" rel="noopener">Announcing OpenAI-compatible API support for Amazon SageMaker AI endpoints</a></p> +
<p>Applications built on the OpenAI SDK, LangChain, or Strands Agents previously required custom client adapters and authentication rewrites to work with SageMaker-hosted models. That migration cost was a real barrier.</p> +
<p>SageMaker endpoints now expose an /openai/v1 path supporting Chat Completions with streaming. Migration requires changing only the endpoint URL. SDK calls, streaming logic, and prompt formatting remain identical. Authentication uses bearer tokens generated from existing AWS credentials, val…+
<p>Multi-model endpoints allow hosting multiple models, each callable through the same OpenAI SDK with independent resource allocation. For agentic workloads, AI agents can run entirely on customer-owned GPU infrastructure using the same OpenAI-compatible interface they were built on. Availabl…+
<h3 id="container-caching-june-2026">Container caching (June 2026)</h3> +
<p>Blog: <a href="https://aws.amazon.com/blogs/machine-learning/introducing-container-caching-in-amazon-sagemaker-ai-for-faster-model-scaling/" target="_blank" rel="noopener">Introducing container caching in Amazon SageMaker AI for faster model scaling</a></p> +
<p>During inference auto scaling events, new instances responding to traffic spikes previously had to pull the full container image from Amazon Elastic Container Registry (Amazon ECR) before serving requests. For large serving containers exceeding 10 GB, that pull alone added several minutes o…+
<p>Container caching pre-pulls images automatically, so new instances launch with the container already available locally. Zero configuration, no code changes, no container modifications. It activates automatically on supported accelerator instance types. With Qwen3-8B on ml.g6.2xlarge using t…+
<p>Container caching is the third layer in a three-part scaling optimization suite:</p> <table border="1px" width="100%" cellpadding="10px"> <tbody> <tr> -
<td><strong>Configuration</strong></td> -
<td><strong>NDCG@10</strong></td> -
<td><strong>Δ vs baseline</strong></td> -
<td><strong>p50 (ms)</strong></td> -
<td><strong>p95 (ms)</strong></td> -
<td><strong>p99 (ms)</strong></td> -
<td><strong>Index size (ANN)</strong></td> -
<td><strong>p50 @ conc 10</strong></td> -
</tr> -
<tr> -
<td>1024 float</td> -
<td>0.3627</td> -
<td>baseline</td> -
<td>31</td> -
<td>44</td> -
<td>52</td> -
<td>5.34 GiB</td> -
<td>161 ms</td> -
</tr> -
<tr> -
<td>512 float</td> -
<td>0.3628</td> -
<td>+0.0002</td> -
<td>25</td> -
<td>37</td> -
<td>62</td> -
<td>2.79 GiB</td> -
<td>149 ms</td> -
</tr> -
<tr> -
<td>256 float</td> -
<td>0.3468</td> -
<td>−4.4%</td> -
<td>22</td> -
<td>35</td> -
<td>47</td> -
<td>1.51 GiB</td> -
<td>85 ms</td> -
</tr> -
<tr> -
<td>1024 binary</td> -
<td>0.3438</td> -
<td>−5.2%</td> -
<td>22</td> -
<td>37</td> -
<td>55</td> -
<td>0.40 GiB</td> -
<td>70 ms</td> +
<td><strong>Layer</strong></td> +
<td><strong>Optimization</strong></td> +
<td><strong>Impact</strong></td> </tr> <tr> -
<td>512 binary</td> -
<td>0.3200</td> -
<td>−11.8%</td> -
<td>19</td> -
<td>30</td> -
<td>47</td> -
<td>0.32 GiB</td> -
<td>57 ms</td> +
<td><strong>Detection</strong></td> +
<td>Sub-minute CloudWatch metrics</td> +
<td>Triggers scale-up 6x faster than standard 1-minute metrics</td> </tr> <tr> -
<td>256 binary</td> -
<td>0.2600</td> -
<td>−28.3%</td> -
<td>17</td> -
<td>24</td> -
<td>32</td> -
<td>0.28 GiB</td> -
<td>52 ms</td> +
<td><strong>Existing instances</strong></td> +
<td>Instance-store data caching</td> +
<td>Removes image pull and model download for instances already running</td> </tr> <tr> -
<td>1024 float, on_disk 32×</td> -
<td>0.3610</td> -
<td>−0.5%</td> -
<td>99</td> -
<td>139</td> -
<td>176</td> -
<td>0.40 GiB</td> -
<td>255 ms</td> +
<td><strong>New instances</strong></td> +
<td>Container image caching</td> +
<td>Avoids image pull time; 51% startup latency reduction demonstrated</td> </tr> </tbody> </table> -
<h3 id="hybrid-search-results">Hybrid search results</h3> -
<p>Table 2: Hybrid search comparison at 1024 dimensions (1,215,851 vectors, 5,000 queries, k=10). Hybrid uses normalization-processor with 0.7 semantic / 0.3 lexical weighting.</p> -
<table border="1px" width="100%" cellpadding="10px"> -
<tbody> -
<tr> -
<td><strong>Method</strong></td> -
<td><strong>1024-float NDCG</strong></td> -
<td><strong>float p50</strong></td> -
<td><strong>1024-binary NDCG</strong></td> -
<td><strong>binary p50</strong></td> -
</tr> -
<tr> -
<td>Keyword (BM25)</td> -
<td>0.3141</td> -
<td>11 ms</td> -
<td>0.3177</td> -
<td>17 ms</td> -
</tr> -
<tr> -
<td>Semantic (k-NN)</td> -
<td>0.3633</td> -
<td>30 ms</td> -
<td>0.3451</td> -
<td>17 ms</td> -
</tr> -
<tr> -
<td>Hybrid</td> -
<td>0.3850</td> -
<td>38 ms</td> -
<td>0.3658</td> -
<td>35 ms</td> -
</tr> -
</tbody> -
</table> -
<p>Hybrid search provides a consistent quality lift over semantic-only search. On this dataset, hybrid improved NDCG by +6.0 percent over semantic-only for both float (0.3633 → 0.3850) and binary (0.3451 → 0.3658). In particular, 1024-binary with hybrid (0.3658) exceeded 1024-float with semant…-
<p>Hybrid search adds latency. The fusion step roughly doubles sequential latency (30 → 38 ms for float, 17 → 35 ms for binary) and the gap widens under concurrency.</p> -
<p>Note: This dataset (product search) favors keyword matching: BM25 alone reached NDCG 0.314, only 13 percent behind semantic. On natural-language RAG queries the absolute hybrid lift may differ. The fusion weight (0.7/0.3) was set for demonstration, not tuned.</p> -
<h2 id="use-case-2-deep-research-agent">Use case 2: Deep research agent</h2> -
<p>Deep research agents represent a significant evolution beyond traditional RAG systems. While standard RAG performs a single retrieval-generation cycle, deep research agents tackle complex, multi-turn research tasks through dynamic reasoning, adaptive planning, and iterative information retr…Diff display stops at 400 lines. The line counts above are from the whole diff. 65 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.