Change
2518b4d
2518b4de17c8723ffe1993190d1a6e9e5b391eab · commit on GitHub
aws-blog-feed: changed (693848 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +4,543
- Lines removed
- -4,015
- Stored bytes at this commit
- 693,848
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-11-05T06:46:17.953Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Thu, 05 Nov 2026 00:10:16 GMT |
| date | Thu, 05 Nov 2026 06:46:17 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Tue, 03 Nov 2026 16:49:29 +0000</lastBuildDate>+
<lastBuildDate>Wed, 04 Nov 2026 18:46:36 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,309 +13,674 @@ 1 </sy:updateFrequency> <item>-
<title>Bring structured and unstructured data together with Amazon Quick</title>-
<link>https://aws.amazon.com/blogs/machine-learning/bring-structured-and-unstructured-data-together-with-amazon-quick/</link>+
<title>Use decision models with Amazon Bedrock AgentCore Evaluations</title>+
<link>https://aws.amazon.com/blogs/machine-learning/use-decision-models-with-amazon-bedrock-agentcore-evaluations/</link> -
<dc:creator><![CDATA[Puneeth Komaragiri]]></dc:creator>-
<pubDate>Tue, 03 Nov 2026 16:49:29 +0000</pubDate>-
<category><![CDATA[Amazon Quick Suite]]></category>-
<category><![CDATA[Intermediate (200)]]></category>+
<dc:creator><![CDATA[Sungyeon Kim]]></dc:creator>+
<pubDate>Wed, 04 Nov 2026 18:46:36 +0000</pubDate>+
<category><![CDATA[Advanced (300)]]></category>+
<category><![CDATA[Amazon Bedrock AgentCore]]></category> <category><![CDATA[Technical How-to]]></category>-
<guid isPermaLink="false">a3db7b2c3fdf8a48e563f027e982fa25a50c5a02</guid>+
<guid isPermaLink="false">af187d4dcc140f40d628b5b7a009df4d46179afa</guid>-
<description>Amazon Quick brings structured and unstructured data together in one experience. In this post, you connect Amazon S3 as a knowledge base, enrich a dataset, build a Space, and create a custom chat agent that answers business questions across both data types through a single conversa…-
<content:encoded><p>Organizations collect data in many forms. Structured data lives in databases and spreadsheets, whereas unstructured data sits in documents, reports, and files stored across file systems. Analyzing both together typically requires switching between multiple tools, …-
<p><a href="https://docs.aws.amazon.com/quick/latest/userguide/welcome.html" target="_blank" rel="noopener">Amazon Quick</a> addresses this with a unified experience. You can connect structured data through <a href="https://docs.aws.amazon.com/quick/latest/userguide/working-with…-
<p>In this post, you learn how to:</p> -
<ul> -
<li>Connect Amazon Simple Storage Service (Amazon S3) as a data source and create a knowledge base.</li> -
<li>Build a Space that combines structured and unstructured data.</li> -
<li>Create and configure a custom chat agent that queries across both data types.</li> -
<li>Ask questions that span structured and unstructured sources in a single conversation.</li> -
</ul> -
<h2 id="solution-architecture">Solution architecture</h2> -
<p>The following diagram illustrates how each component connects to deliver a unified data experience through Amazon Quick.</p> +
<description>Some agent-evaluation questions are narrow and repeatable, and decision models answer exactly that kind of question. This post shows how to connect two decision models, Jev and Strands Decider, to Amazon Bedrock AgentCore Evaluations through code-based evaluators, and how to valida…+
<content:encoded><p>Some of the questions you ask when evaluating an agent are narrow and repeatable. Did this response stay on policy? Is this tool call safe to approve? Does this answer match the retrieved context? Decision models answer exactly that kind of question. They take a s…+
<p><a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html" target="_blank" rel="noopener">Amazon Bedrock AgentCore Evaluations</a>, a capability of Amazon Bedrock AgentCore, scores an agent through its entire lifecycle, on demand during development,…+
<p>Decision models suit questions with a fixed set of answers that an informed person could decide at a glance, such as whether the agent refused a request or whether a response stayed on topic. After you validate a question on your own traffic, a decision model can be sufficient for that deci…+
<p>This post shows how to connect two decision model backends to AgentCore Evaluations through code-based evaluators. The first is <a href="https://typesafe.ai/blog/introducing-system-one-models-and-jev" target="_blank" rel="noopener">Jev</a>, a hosted decision model from TypeSafe.…+
<h2 id="decision-models">Decision models</h2> +
<p>A decision model is a small, specialized model trained to select from a fixed set of options and return calibrated confidence scores rather than generating free-form text. A general-purpose large language model (LLM) evaluator produces a score with a rationale. A decision model instead prod…+
<p>Decision models are most useful when you want a well-calibrated confidence score alongside the label, not just a binary judgment. They are also appropriate when you want to enforce a specific rubric consistently without relying on a general-purpose LLM to interpret it.</p> +
<p>Jev, the hosted decision model from TypeSafe, and <a href="https://strandsagents.com/blog/introducing-strands-decider/" target="_blank" rel="noopener">Strands Decider</a>, an open-source, 2-billion-parameter decision model from the Strands Agents team, both support three types o…+
<h2 id="decision-models-versus-llm-as-a-judge">Decision models versus LLM-as-a-judge</h2> +
<p>The built-in evaluators in AgentCore today use a general-purpose LLM to assess the agent response against a rubric and explain its judgment. This is valuable when the evaluation requires connecting evidence, identifying missing information, or working through a response in depth in tasks wh…+
<h2 id="solution-overview">Solution overview</h2> +
<p>To show how these evaluator types work together, this post builds an end-to-end example of an HR Assistant agent evaluated by deterministic code-based checks, built-in LLM-as-a-judge evaluators, and decision-model evaluators backed by Jev or Strands Decider 2B.</p> +
<p>The agent is built with Strands Agents and deployed to AgentCore runtime, a capability of Amazon Bedrock AgentCore. It answers employee questions using five tools that return fixed mock data such as paid time off (PTO) balances, PTO requests, HR policies, benefits, and pay stubs. We use fix…<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-1.png" alt="Architecture diagram sh…-
<p class="wp-caption-text">Figure 1: Solution architecture for combining structured and unstructured data in Amazon Quick</p>+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-1.png" alt="Architecture of the HR …+
<p class="wp-caption-text">Figure 1: End-to-end architecture of the HR Assistant evaluated by code-based checks, built-in evaluators, and decision-model evaluators backed by Jev or Strands Decider</p></div> -
<p>The architecture has three layers:</p> -
<ul> -
<li><strong>Data layer:</strong> Your data exists in two forms. Unstructured content (documents, PDFs, reports) is stored in Amazon S3 and indexed through a knowledge base. Structured data (metrics, sales figures, and operational key performance indicators) is loaded as enriched d…-
<li><strong>Workspace layer:</strong> A Space brings both data types together into a single workspace. The Space references your knowledge base for unstructured queries and your datasets for structured queries. It can also include dashboards, file uploads, and action connectors. T…-
<li><strong>Agent layer:</strong> The custom chat agent links to the Space and uses it as its knowledge source. When a user asks a question, the agent determines whether to query the knowledge base, the dataset, or both, and returns a combined response with citations. Amazon Quick…-
</ul> -
<p>With this layered approach, you can expand the solution over time. When you add a new data source to the Space, the agent linked to that Space can use it, and responses respect the permissions of the user who asks.</p> -
<h2 id="prerequisites">Prerequisites</h2> -
<p>To follow along with this walkthrough, you need:</p> -
<ul> -
<li>An Amazon Quick account with an Enterprise subscription for the user who follows this walkthrough (creating datasets requires Enterprise). For details, see <a href="https://aws.amazon.com/quick/pricing/" target="_blank" rel="noopener">Amazon Quick pricing</a>.</li> -
<li>An Amazon S3 bucket for your business documents. To create one, see the <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html" target="_blank" rel="noopener">Amazon S3 User Guide</a>.</li> -
<li><a href="https://docs.aws.amazon.com/quick/latest/userguide/s3-admin-setup.html#s3-grant-bucket-access" target="_blank" rel="noopener">Amazon Quick access to your S3 bucket</a>, granted by an Amazon Quick administrator. In the admin console, the administrator chooses Permissio…-
<li>Permissions to create integrations, knowledge bases, datasets, spaces, and chat agents in Amazon Quick.</li> -
</ul> -
<h2 id="prepare-your-sample-data">Prepare your sample data</h2> -
<p>This walkthrough uses two types of data:</p> -
<p><strong>Unstructured documents: </strong>Upload these to your Amazon S3 bucket for the knowledge base:</p> -
<ul> -
<li><a href="https://aws-blogs-artifacts-public.s3.us-east-1.amazonaws.com/BI-7992/AnyCompany_Business_Review.pdf" target="_blank" rel="noopener">AnyCompany_Business_Review.pdf</a></li> -
<li><a href="https://aws-blogs-artifacts-public.s3.us-east-1.amazonaws.com/BI-7992/AnyCompany_Market_Analysis_Report.pdf" target="_blank" rel="noopener">AnyCompany_Market_Analysis_Report.pdf</a></li> -
<li><a href="https://aws-blogs-artifacts-public.s3.us-east-1.amazonaws.com/BI-7992/AnyCompany_Product_Strategy_Memo.pdf" target="_blank" rel="noopener">AnyCompany_Product_Strategy_Memo.pdf</a></li> -
</ul> -
<p><strong>Structured data: </strong>Download this file to your computer. You upload it directly to Amazon Quick in Step 3:</p> -
<ul> -
<li><a href="https://aws-blogs-artifacts-public.s3.us-east-1.amazonaws.com/BI-7992/Product_Sales.csv" target="_blank" rel="noopener">Product_Sales.csv</a></li> -
</ul> -
<h2 id="step-1-create-an-amazon-s3-integration">Step 1: Create an Amazon S3 integration</h2> -
<p>Start by connecting Amazon Quick to your S3 bucket. From the Amazon Quick console, select <strong>Knowledge</strong> in the left navigation panel.</p> -
<p>The knowledge page shows available connector types, including Amazon Q Business, Web Crawler, Amazon S3, and third-party options like Google Drive, OneDrive, and Microsoft SharePoint.</p> -
<p>Under Amazon S3, choose <strong>Add</strong>. In the Create S3 knowledge base dialog, on the Connect S3 bucket page, provide the following details:</p> +
<h3 id="prerequisites">Prerequisites</h3> +
<p>To deploy the sample, you need the following:</p> <ul> -
<li><strong>Name:</strong> A descriptive name for your integration (for example, “My Data Access Integration”).</li> -
<li><strong>S3 bucket location:</strong> Choose <strong>Quick Suite instance account</strong> to use a bucket in the same AWS account as Amazon Quick, or <strong>Other AWS account</strong> for a bucket in a different account.</li> -
<li><strong>S3 bucket URL:</strong> The S3 URI of the bucket containing your data. The bucket must be in the same Region as Amazon Quick.</li> +
<li>An AWS account in a <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-regions.html" target="_blank" rel="noopener">Region</a> where AgentCore Evaluations is available. The sample defaults to US West (Oregon).</li> +
<li><a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-get-started.html#enabling-transaction-search" target="_blank" rel="noopener">Amazon CloudWatch Transaction Search enabled</a> in the account.</li> +
<li><a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html" target="_blank" rel="noopener">Access to a foundation model in Amazon Bedrock</a>.</li> +
<li>Python 3.12 or later, the AWS Command Line Interface (AWS CLI), and Docker.</li> +
<li>Jev requires a TypeSafe API key. Visit <a href="https://typesafe.ai/" target="_blank" rel="noopener">typesafe.ai</a> for access details.</li> +
<li>Strands Decider requires a host running the Strands Decider server. You can run it locally for testing, or on Amazon Elastic Compute Cloud (Amazon EC2) inside your virtual private cloud (VPC) so the Lambda function can reach it.</li> </ul> -
<p>Choose <strong>Next</strong>.</p> -
<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-2.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-2.png" alt="Create S3 knowledge bas…-
<p class="wp-caption-text">Figure 2: Create integration dialog showing the S3 connection fields</p>-
</div> -
<h2 id="step-2-create-a-knowledge-base">Step 2: Create a knowledge base</h2> -
<p>After the S3 integration is created, Amazon Quick prompts you to create a knowledge base. The knowledge base indexes your S3 content for AI-powered search.</p> -
<p>Configure the knowledge base:</p> +
<h3 id="seeing-it-in-action">Seeing it in action</h3> +
<p>You can find the full code sample in the <a href="https://github.com/awslabs/agentcore-samples/tree/main/01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/custom-code-based-evaluation" target="_blank" rel="noopener">AgentCore samples GitHub repository</a>. Two Pyth…<ul> -
<li><strong>Name:</strong> “Business Knowledge Base”.</li> -
<li><strong>Description (optional):</strong> Notes about how the knowledge base will be used.</li> -
<li><strong>Content:</strong> Select “Add all content” to index everything in the connected S3 bucket, or “Add specific content” to specify the S3 prefixes for the folders and files to include (prefix filters are case-sensitive).</li> +
<li><code>../utils/deploy.py</code> deploys the HR Assistant to AgentCore runtime.</li> +
<li><code>evaluate.py</code> deploys evaluators, invokes the agent, and runs the evaluation modes.</li> </ul> -
<p><strong>Next:</strong> Additional settings opens optional settings for document-level access control lists (ACLs) and a metadata files folder. Decide whether to use ACLs before you create the knowledge base, because you can’t change that choice later. This walkthrough uses sampl…-
<p>Choose <strong>Create</strong>. A success banner confirms the S3 integration, and the knowledge base begins indexing in the background.</p> -
<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-3.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-3.png" alt="Create knowledge base d…-
<p class="wp-caption-text">Figure 3: Create knowledge base dialog with the Business Knowledge Base name and content options</p>-
</div> -
<h2 id="step-3-create-and-enrich-a-dataset">Step 3: Create and enrich a dataset</h2> -
<p>A <a href="https://docs.aws.amazon.com/quick/latest/userguide/working-with-datasets.html" target="_blank" rel="noopener">dataset</a> is your structured data file loaded into Amazon Quick. By enriching the dataset with semantic descriptions, custom instructions, and correct data …-
<h3 id="create-the-dataset">Create the dataset</h3> +
<p>Running <code>evaluate.py</code> without flags executes five steps:</p> <ol type="1"> -
<li>From the Amazon Quick home page, navigate to <strong>Data</strong> in the left navigation.</li> -
<li>Choose <strong>Create dataset</strong>, then select <strong>Upload file</strong>.</li> -
<li>Select the <strong>Product_Sales.csv</strong> file you downloaded earlier and choose <strong>Open</strong>.</li> -
<li>On the confirmation screen, review the column names and data types. Choose <strong>Next</strong> to create the dataset.</li> +
<li>Creates the Lambda execution role (<code>AgentCoreLambdaEvaluatorRole</code>).</li> +
<li>Packages and deploys two deterministic Lambda evaluators: <code>hr-response-length</code> (a trace-level evaluator that checks whether the agent’s response stays within length bounds) and <code>hr-fact-checker</code> (a session-level evaluator that regex-validates …+
<li>Registers both as AgentCore code-based evaluators through the control plane.</li> +
<li>Runs on-demand evaluation with <code>EvaluationClient</code> and <code>OnDemandEvaluationDatasetRunner</code> across five labeled scenarios, combining the code-based evaluators with <code>Builtin.Correctness</code>, <code>Builtin.Helpfulness</c…+
<li>Creates an online evaluation configuration that continuously scores live sessions at 100 percent sampling.</li> </ol> -
<p>Amazon Quick ingests the file into SPICE (the in-memory calculation engine) and the dataset appears in your dataset list.</p> -
<p>The dataset takes its name from the uploaded file, so it appears as Product_Sales.csv in your account.</p> -
<h3 id="enrich-the-dataset">Enrich the dataset</h3> -
<p>Enriching your dataset with semantic information helps the AI understand your data in business context. The agent uses this context to translate natural language questions into more accurate SQL queries. From the dataset page, choose <strong>Edit dataset</strong> to enter the da…-
<h3 id="transform-fix-data-types">Transform: Fix data types</h3> -
<p>In the <strong>Transform</strong> tab, update data types for columns that were auto-detected incorrectly. For this dataset:</p> -
<ul> -
<li>Change <strong>Order Date</strong> and <strong>Date Key</strong> from String to Date.</li> -
<li>Change <strong>Customer ID</strong> from String to Integer.</li> -
<li>Change <strong>Sales</strong>, <strong>Discount</strong>, and <strong>Profit</strong> from String to Decimal.</li> -
<li>Change <strong>Quantity</strong> and <strong>Unit Price</strong> from String to Integer.</li> -
</ul> +
<p>At this point you have a working evaluation pipeline with deterministic checks and LLM-as-a-judge scoring. Decision-model evaluators are opt-in. You can add them with a flag, and they run alongside the base evaluators with no changes to the existing flow. Using the flags <code>--with-…<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-4.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-4.png" alt="Transform tab showing d…-
<p class="wp-caption-text">Figure 4: Transform view showing data type changes applied to the Product_Sales.csv dataset</p>+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-2.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-2.png" alt="The evaluation pipeline…+
<p class="wp-caption-text">Figure 2: The evaluation pipeline deployed by evaluate.py, from deterministic checks through built-in and decision-model evaluators</p></div> -
<h3 id="output-add-semantic-context">Output: Add semantic context</h3> -
<p>Select the <strong>Output</strong> tab to access the dataset-level enrichment fields. Here you provide context that helps the AI understand what your data represents and how to use it.</p> -
<p><strong>Description:</strong> Add a dataset-level description:</p> -
<p>“Product sales transaction data for AnyCompany, covering orders from FY2024-FY2025 across North America, EMEA, Asia Pacific, and Latin America. Includes order details, customer information, product lines, and revenue metrics.”</p> -
<p><strong>Custom instructions:</strong> Add guidance for how the AI should query this data:</p> -
<p><strong>USAGE GUIDELINES:</strong></p> -
<ul> -
<li>Revenue values are in USD.</li> -
<li>When comparing performance, group by Region (North America, EMEA, Asia Pacific, Latin America).</li> -
<li>Order Date should be used for time-based trend analysis. The fiscal year matches the calendar year.</li> -
<li>Date Key is a numeric YYYYMMDD representation of Order Date.</li> -
<li>Sales = Quantity x Unit Price x (1 – Discount)</li> -
</ul> -
<p><strong>COMMON USE CASES:</strong></p> -
<ul> -
<li>Regional sales performance comparison.</li> -
<li>Product line revenue analysis.</li> -
<li>Quarterly trend identification.</li> -
<li>Customer segmentation by industry and segment.</li> -
</ul> -
<p><strong>TAGS:</strong> Sales, revenue, orders, regional performance, product analytics</p> -
<p><strong>Column descriptions:</strong> Select each column in the left panel and add a description. For example:</p> +
<p>Each decision-model backend contributes three evaluators:</p> +
<table border="1px" width="100%" cellpadding="10px"> +
<tbody> +
<tr> +
<td><strong>Evaluator</strong></td> +
<td><strong>Level</strong></td> +
<td><strong>Question type</strong></td> +
<td><strong>Label</strong></td> +
<td><strong>Value</strong></td> +
</tr> +
<tr> +
<td>JevGroundedness / DeciderGroundedness</td> +
<td>TRACE</td> +
<td><code>noul</code></td> +
<td>Grounded or Ungrounded</td> +
<td>P(grounded)</td> +
</tr> +
<tr> +
<td>JevHelpfulness / DeciderHelpfulness</td> +
<td>TRACE</td> +
<td><code>score</code> (4-level)</td> +
<td>Most probable level</td> +
<td>Expected level, scaled 0–1</td> +
</tr> +
<tr> +
<td>JevGoalCompletion / DeciderGoalCompletion</td> +
<td>SESSION</td> +
<td><code>choice</code> (3-way)</td> +
<td>Most probable option</td> +
<td>Probability-weighted value</td> +
</tr> +
</tbody> +
</table> +
<h2 id="build-the-evaluator-with-decision-models">Build the evaluator with decision models</h2> +
<p>Using the flags <code>--with-jev</code> and <code>--with-decider</code> creates a custom code-based evaluator, where you have complete control of the code you want to run as your evaluator. We create AWS Lambda functions where the decision model is invoked inside an …+
<p>The Lambda that is registered as the evaluator receives OpenTelemetry spans of the agent. The <code>spans.py</code> module normalizes the shapes OpenTelemetry content can arrive in and then formats it appropriately to make the call to the decision model. For trace-level evaluati…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-3.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-3.png" alt="The code-based evaluato…+
<p class="wp-caption-text">Figure 3: How the code-based evaluator Lambda adapts OpenTelemetry spans for the decision model and maps the result back to AgentCore</p>+
</div> +
<p>The adapter converts the model’s probability distribution into a result that AgentCore evaluation expects. For a <code>noul</code> question, the probability of <code>true</code> becomes the value and the most probable answer becomes the label. For a <code>score…+
<h3 id="validate-a-decision-model-evaluator-on-your-own-traffic">Validate a decision-model evaluator on your own traffic</h3> +
<p>When you rewrite a rubric as a decision model question, you’ve created a new measurement rather than a copy of the old one. A reframed rubric often lands stricter or looser than the original intended, and you won’t see that drift until you check it against real sessions. Before you gate a r…+
<p>The procedure has five steps.</p> +
<p><strong>Step 1.</strong> Build a labeled sample. Pull a set of stored sessions from your agent’s traffic, with enough coverage of the failures and edge cases you care about. A sample that covers only the happy path tells you nothing about where the evaluator breaks.</p> +
<p><strong>Step 2.</strong> Get human reference labels. Have three independent labelers score each turn for the property your question measures and take the majority vote as the reference label. Keep track of which turns were unanimous and which were split. Although unanimous turns…+
<p><strong>Step 3.</strong> Run both evaluators on the same inputs. Run the decision-model evaluator and the matching built-in evaluator over those same sessions in a single batch evaluation. Scoring identical inputs is what makes the comparison apples to apples.</p> +
<p><strong>Step 4.</strong> Read the results through three lenses. With the majority-vote labels and the easy and hard split in hand, you can now answer the three questions that tell you whether the evaluator is ready.</p> <ul> -
<li>Order ID: “Unique identifier for each sales transaction (format: <code>REGION-YYYY-NNNNNN</code>)”</li> -
<li>Region: “Geographic sales region: North America, EMEA, Asia Pacific, or Latin America”</li> -
<li>Product: “AnyCompany product name: Analytics Pro, Connect, Automate, AI Assistant, Data Catalog, or Integration Hub”</li> -
<li>Sales: “Order revenue in USD after discount”</li> -
<li>Segment: “Customer size segment: Enterprise, Mid-Market, or SMB”</li> +
<li>Agreement rate: does it match your humans? Measure how often each evaluator agrees with the majority-vote labels, and do it per question, since one question’s accuracy says nothing about another’s. Then break the numbers out by easy (unanimous) and hard (split-vote) cases. An evaluator th…+
<li>Calibration: do the confidence scores mean something? Group turns by the evaluator’s reported probability and compare each group’s agreement with the labels. Higher confidence should track higher accuracy. Then check whether easy cases consistently draw higher confidence than hard ones. I…+
<li>Disagreement analysis: where and why do they differ? Find the turns where the two evaluators disagree with each other, and check whether those are the same turns your humans found ambiguous. Read them. Most disagreements trace back to the wording of the question, and a revised question cl…</ul> +
<p><strong>Step 5</strong>. Re-run the check when anything changes. Treat this as a repeatable check rather than a one-time gate. Any time you reword a question or move to a new model version, run it again. Each result’s explanation records the model version that produced it, so yo…+
<h2 id="choosing-the-best-set-of-evaluators-for-your-agent">Choosing the best set of evaluators for your agent</h2> +
<p>The built-in evaluators remain the foundation of broad evaluation strategy in AgentCore Evaluations. They assess the response against a rubric and its supporting context, and explain the judgment so developers can investigate what went wrong. For the HR Assistant, a faithfulness evaluation …+
<p>Faithfulness, helpfulness, and correctness can require connecting evidence, identifying missing information, or working through a solution. The preprint <em>JEV-as-a-Judge: Accept When Confident, Escalate When Unsure</em> describes a related distinction as “reading versus derivi…<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-5.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-5.png" alt="Output tab showing the …-
<p class="wp-caption-text">Figure 5: Output view showing dataset details with Description and Custom instructions configured</p>-
</div> -
<p>Choose <strong>Save &amp; publish</strong> to save your changes. The dataset summary now shows the columns with their correct data types and the descriptions you added.</p> -
<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-6.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-6.png" alt="Enriched Product_Sales.…-
<p class="wp-caption-text">Figure 6: Enriched Product_Sales.csv dataset summary showing all 16 columns, data types, and descriptions</p>+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-4.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-4.png" alt="The cascade pattern, wh…+
<p class="wp-caption-text">Figure 4: The cascade pattern, where a calibrated decision model screens every turn and only low-confidence turns escalate to a built-in evaluator</p></div> -
<p>This enriched dataset provides the structured data layer for your solution. When the agent receives a natural language question about sales data, it uses these descriptions and instructions to generate more accurate SQL queries. Combined with the knowledge base from Step 2, you now have bot…-
<h2 id="step-4-create-a-space">Step 4: Create a Space</h2> -
<p><a href="https://docs.aws.amazon.com/quick/latest/userguide/working-with-spaces.html" target="_blank" rel="noopener">Spaces</a> in Amazon Quick act as unified data workspaces. A Space can hold file uploads, dashboards, knowledge bases, actions, and datasets, giving your chat age…-
<p>From the left navigation, select <strong>Spaces</strong> and create a new Space. Name it “Business Intelligence Hub”.</p> -
<p>Add your knowledge base and dataset to the Space:</p> +
<table border="1px" width="100%" cellpadding="10px"> +
<tbody> +
<tr> +
<td><strong>Criterion</strong></td> +
<td><strong>LLM as Judge</strong></td> +
<td><strong>Decision Models as Judge</strong></td> +
</tr> +
<tr> +
<td><strong>Output</strong></td> +
<td>Score + explanation</td> +
<td>Calibrated probability</td> +
</tr> +
<tr> +
<td><strong>Latency</strong></td> +
<td>2–10 s</td> +
<td>~hundreds of ms (network)</td> +
</tr> +
<tr> +
<td><strong>Customizable</strong></td> +
<td>Limited (fixed prompt templates)</td> +
<td>Question definitions</td> +
</tr> +
<tr> +
<td><strong>Best for</strong></td> +
<td>Qualitative nuance, explanation</td> +
<td>Calibrated judgment</td> +
</tr> +
</tbody> +
</table> +
<p>We recommend the following setup:</p> <ol type="1"> -
<li>Choose <strong>Add knowledge</strong> and select <strong>Knowledge bases</strong> from the list. Select “Business Knowledge Base” from the list.</li> -
<li>Choose <strong>Add knowledge</strong> again and select <strong>Datasets</strong> from the list. Select “Product_Sales.csv” from the list.</li> +
<li>Configure online evaluation with the built-in evaluators you rely on, as the standard measure of quality across your agents.</li> +
<li>Add decision-model evaluators for domain-specific yes/no and choice questions to the same configuration, along with deterministic code-based evaluators for anything you can compute exactly.</li> +
<li>Each week, or on whatever schedule suits your release cycle, query the online results for sessions where a decision-model check failed or scored near its threshold, and run batch evaluations on those sessions with the built-in evaluators to get explanations.</li> +
<li>Gate releases on the built-in evaluators and on the decision-model evaluators you have validated against labeled data, and track how often the two agree.</li> </ol> -
<p>The Space now shows both your knowledge base (unstructured documents) and your enriched dataset (structured sales data) linked as data sources. This is the central hub where both data types come together, giving your chat agent a single place to query across everything. Figure 7 shows the S…+
<h2 id="clean-up">Clean up</h2> +
<p>To avoid ongoing charges, run the <code>cleanup.py</code> that deletes the agent hosted on AgentCore runtime, the evaluators and evaluation config, the AWS Lambda functions that are registered as the evaluators, the EC2 instance that hosts the Strands Decider model, AWS Secrets …+
<h2 id="conclusion">Conclusion</h2> +
<p>In this post, you connected two decision models, Jev and Strands Decider, to AgentCore Evaluations using code-based evaluators, and the accompanying code sample gives you a framework for testing those models and seeing where they add value as evaluators. The built-in evaluators remain your …+
<p>To get started, review the <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html" target="_blank" rel="noopener">Amazon Bedrock AgentCore Evaluations</a> documentation, integrate custom evaluators using <a href="https://aws.amazon.com/blogs/m…+
<p>&nbsp;</p> +
<hr style="width: 100%"> +
<h2>About the authors</h2> +
<footer> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-5.jpg" alt="Sungyeon Kim" width="100" height="133">+
</div> +
<h3 class="lb-h4">Sungyeon Kim</h3> +
<p style="overflow: hidden">Sungyeon is an Applied Scientist at AWS, where he conducts research and develops machine learning solutions for agentic applications. His research interests include language model training and evaluation, generative AI, and agentic systems. He has worked across va…+
</div> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-6.jpg" alt="Bharathi Srinivasan" width="100" height="133">+
</div> +
<h3 class="lb-h4">Bharathi Srinivasan</h3> +
<p style="overflow: hidden">Bharathi is a Generative AI Data Scientist at AWS. She is passionate about Responsible AI to increase the reliability of AI agents in real world scenarios. Bharathi guides internal teams and AWS customers on their responsible AI journey. She has presented her work…+
</div> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-7.jpg" alt="Jonathan Buck" width="100" height="133">+
</div> +
<h3 class="lb-h4">Jonathan Buck</h3> +
<p style="overflow: hidden">Jonathan is a Principal Engineer at AWS. His work focuses on infrastructure and tooling to support AI agent environments and evaluation. Jonathan works with both research and engineering teams to shape the direction of next-generation AI products and services for …+
</div> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22136-8.jpg" alt="Haibo Ding" width="100" height="133">+
</div> +
<h3 class="lb-h4">Haibo Ding</h3> +
<p style="overflow: hidden">Haibo is a Principal Applied Scientist and Science Manager working on agentic AI at Amazon. He holds a Ph.D.&nbsp;from the University of Utah. His work focuses on large language models (LLMs) and AI agents, where he leads research in areas such as LLM post-tra…+
</div> +
</footer></content:encoded>+
+
+
+
</item>+
<item>+
<title>Smart Approvals in Amazon Quick: Human oversight where it matters</title>+
<link>https://aws.amazon.com/blogs/machine-learning/smart-approvals-in-amazon-quick-human-oversight-where-it-matters/</link>+
+
<dc:creator><![CDATA[Steve Vinsik]]></dc:creator>+
<pubDate>Wed, 04 Nov 2026 16:33:03 +0000</pubDate>+
<category><![CDATA[Amazon Quick Suite]]></category>+
<category><![CDATA[Intermediate (200)]]></category>+
<category><![CDATA[Thought Leadership]]></category>+
<guid isPermaLink="false">9c5c128f7a31e5e43f3eafba7ef1c3537f53e875</guid>+
+
<description>Smart Approvals in Amazon Quick reviews eligible agent actions within your existing controls, reducing unnecessary approval prompts while keeping consequential decisions with a person. See how policy sets the boundary, how a classifier reviews actions, and how AWS CloudTrail and Am…+
<content:encoded><p>How should security leaders preserve human oversight as AI agents take on more work? Permissions establish which actions agents can take, but a permitted action can still warrant a person’s judgment. Requiring approval for every step might seem like the safest app…+
<p>Smart Approvals in <a href="https://aws.amazon.com/quick/" target="_blank" rel="noopener">Amazon Quick</a> addresses this challenge by reviewing eligible actions within your existing controls. It evaluates each proposed action against the user’s request and either approves it au…<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-7.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-7.png" alt="Business Intelligence H…-
<p class="wp-caption-text">Figure 7: Business Intelligence Hub Space showing the Product_Sales.csv dataset and Business Knowledge Base</p>-
</div> -
<h2 id="step-5-create-a-custom-chat-agent">Step 5: Create a custom chat agent</h2> -
<p>Navigate to <strong>Chat agents</strong> in the left navigation and choose <strong>Create chat agent</strong>. Amazon Quick presents a prompt builder where you can describe your agent in natural language or choose from templates like Training Guide, Project Coordinat…-
<p>For this walkthrough, choose <strong>Skip</strong> to configure the agent manually.</p> -
<div style="width: 702px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-8.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-8.png" alt="Chat agent creation scr…-
<p class="wp-caption-text">Figure 8: Chat agent creation screen with the prompt builder and sample templates</p>-
</div> -
<h3 id="configure-the-agent-persona">Configure the agent persona</h3> -
<p>The configuration panel has several sections. Start with the Agent Persona:</p> +
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22232-1.png" target="_blank" rel="noopener"><img title="Smart Approvals within policy boundaries" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/1…+
<p class="wp-caption-text">Figure 1: Policy determines eligibility. Smart Approvals approves eligible actions or returns them to the user for review. Policy denials remain blocked</p>+
</div> +
<h2 id="policy-sets-the-boundary">Policy sets the boundary</h2> +
<p>Quick evaluates each proposed action against Custom Permissions, Quick Guardrails, connector action permissions, and the user’s settings before automated review. Denied actions remain denied, and mandatory human approvals still apply. With Smart Approvals enabled, an action is eligible only…+
<p>Administrators use Custom Permissions to determine who can use Smart Approvals. It’s on by default, but profiles with <a href="https://docs.aws.amazon.com/quick/latest/userguide/custom-permissions-governance.html" target="_blank" rel="noopener">deny-by-default</a> enabled must e…+
<h2 id="how-eligible-actions-are-reviewed">How eligible actions are reviewed</h2> +
<p>Smart Approvals uses a separate model, the classifier, to evaluate eligible actions against the user’s request and a bounded view of the conversation. It considers what the action does, where data goes, and who it affects. It can consult the user’s knowledge graph, which contains people and…+
<p>An action can match the request and still warrant human review. Examples include sending content outside the organization or to senior leaders in the user’s reporting chain, changing access, deleting records, or initiating payments. This list isn’t complete, and the classifier also weighs t…+
<p>The classifier provides judgment. The permission service enforces authority. Each automatic approval is single-use, applies only to the exact action and parameters reviewed, and grants no ongoing permission. Quick labels automatic approvals in the conversation. Actions the classifier doesn’…+
<h2 id="when-an-action-goes-beyond-the-request">When an action goes beyond the request</h2> +
<p>An account manager asks Quick to prepare a quarterly business review brief for AnyCompany using her team’s account notes and last quarter’s email. She then asks Quick to send it to two colleagues she names. Assuming the send is eligible for automated review, Smart Approvals evaluates the pr…+
<p>Now suppose an account note says the brief should also go to an all-sales distribution list of about 2,000 people. That would change the audience beyond the recipients she specified, so Smart Approvals is designed to return the proposed send to her for review, where she can decline it. An a…+
<h2 id="audit-evidence-and-investigation">Audit evidence and investigation</h2> +
<p>The Quick permission service records permission updates associated with Smart Approvals in <a href="https://aws.amazon.com/cloudtrail/" target="_blank" rel="noopener">AWS CloudTrail</a> in your account. Each event identifies the user, connector, action, and time, and marks the r…+
<p>Investigators can correlate these events by user and time with other Quick activity in CloudTrail and logs delivered through <a href="https://aws.amazon.com/cloudwatch/" target="_blank" rel="noopener">Amazon CloudWatch Logs</a>. Chat logs capture the user’s request, system respo…+
<h2 id="where-to-start">Where to start</h2> +
<p>Because Smart Approvals is on by default, begin by confirming that actions requiring human approval are configured through connector action permissions. Set up a CloudTrail trail and delivery of chat logs and agent metadata logs to CloudWatch Logs to support investigations. Chat logs captur…+
<p>Before the pilot, define what success looks like: fewer prompts for routine actions, with consequential actions still returning to a person for review. During the pilot, review a sample of automatic approvals and test high-impact cases, such as sending content outside the organization or ad…+
<p>The measure of effective oversight isn’t how often an agent asks for approval, but whether people retain control over consequential decisions. For security leaders, the case for expanding Smart Approvals should rest on evidence from the pilot. Fewer prompts are valuable when they preserve a…+
<p>For setup steps, see <a href="https://docs.aws.amazon.com/quick/latest/userguide/smart-approvals.html" target="_blank" rel="noopener">Smart Approvals</a> in the governance section of the Amazon Quick User Guide.</p> +
<p>&nbsp;</p> +
<hr style="width: 100%"> +
<h2>About the author</h2> +
<footer> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/11/04/ML-22232-2.jpg" alt="Steve Vinsik" width="100" height="133">+
</div> Diff display stops at 400 lines. The line counts above are from the whole diff. 73 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.