Change
f4934a9
f4934a9876ff4ba9a28a0763dbabfb1ca019dba4 · commit on GitHub
aws-blog-feed: changed (563905 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +3,953
- Lines removed
- -3,819
- Stored bytes at this commit
- 563,905
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-10-20T04:51:23.102Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Mon, 19 Oct 2026 23:32:25 GMT |
| date | Tue, 20 Oct 2026 04:51:23 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Fri, 16 Oct 2026 21:06:58 +0000</lastBuildDate>+
<lastBuildDate>Mon, 19 Oct 2026 23:32:19 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,422 +13,1162 @@ 1 </sy:updateFrequency> <item>-
<title>Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore</title>-
<link>https://aws.amazon.com/blogs/machine-learning/building-trust-in-ai-agents-a-production-ready-evaluation-framework-with-amazon-bedrock-agentcore/</link>+
<title>Structured outputs for Anthropic Claude models on Amazon Bedrock</title>+
<link>https://aws.amazon.com/blogs/machine-learning/structured-outputs-for-anthropic-claude-models-on-amazon-bedrock/</link> -
<dc:creator><![CDATA[Rui Li]]></dc:creator>-
<pubDate>Fri, 16 Oct 2026 20:48:16 +0000</pubDate>-
<category><![CDATA[Advanced (300)]]></category>-
<category><![CDATA[Amazon Bedrock AgentCore]]></category>-
<category><![CDATA[Best Practices]]></category>-
<guid isPermaLink="false">df0b1ae32a5ba63d5856db229a66706d4090e8ef</guid>+
<dc:creator><![CDATA[Anshika Saxena]]></dc:creator>+
<pubDate>Mon, 19 Oct 2026 23:32:19 +0000</pubDate>+
<category><![CDATA[Amazon Bedrock]]></category>+
<category><![CDATA[Announcements]]></category>+
<category><![CDATA[Intermediate (200)]]></category>+
<guid isPermaLink="false">a1099e0a38a56e4f447d2dfcf2f10f2a8cdb2b50</guid>-
<description>Autonomous AI agents fail softly: latency stays flat while answer quality quietly erodes. This post shares a nine-stage evaluation framework, built on Amazon Bedrock AgentCore, that catches silent quality regressions across the full agent lifecycle, from first commit to live produc…-
<content:encoded><h2 id="the-north-star">The North Star</h2>-
<p>Every organization racing to deploy AI agents faces the same reckoning: an agent that demos beautifully can quietly degrade in production, and by the time a human notices, the trust is already broken. Where traditional software fails loudly, agents fail softly. They return an answer that lo…-
<p>That’s the quality problem at its core. The challenge is therefore not building agents that work once, but proving they keep working, at scale, across every sub-agent, every workflow, and every release. The teams that win the agent era will be the ones who treat quality not as a final QA st…-
<h3 id="business-problem">Business problem</h3>-
<p>The solution uses Amazon Bedrock AgentCore and other AWS services for internal use, so sellers and sales leaders can ask business questions in plain English from Amazon Quick. Behind that single experience, several specialized sub-agents work together: the Performance Insights Agent for ter…-
<ul>-
<li>It reported $410.08 of a customer’s three-month revenue on a service as “$410.08 Million.”</li>-
<li>It reported Sales Revenue labeled as Sales Revenue excluding Adjustments.</li>-
<li>It offered an Annual Recurring Revenue estimate that appeared nowhere in the data.</li>-
<li>It presented its own calculations as sourced facts.</li>-
</ul>-
<p>Through the different runs, monitoring stayed green. It told us the system had responded, not whether the answer was right.</p>-
<p>Had these errors shipped uncaught, the field would have paid for them. A seller who repeats an inflated figure can lose credibility with the customer. This is exactly why we caught it before launch. The risk doesn’t end at launch: it grows with every new agent and every change to data, prom…-
<p>At AWS, the Sales, Marketing, and Global Services (SMGS) Operations team developed a production multi-agent system serving AWS Field Sales worldwide. It serves two distinct audiences. For field sales teams, it replaces hours of manual data analysis with on-demand, conversational intelligenc…-
<p>The agentic system orchestrates a growing portfolio of specialized sub-agents, each owning a distinct capability domain. Among them: the <a href="https://aws.amazon.com/blogs/machine-learning/how-aws-smgs-uses-an-ai-powered-conversational-assistant-to-transform-business-management-with-a…-
<p>Coordinating these agents reliably at scale made one thing clear: agentic quality wasn’t an operational concern to manage later. It was the prerequisite for operating. A single degraded sub-agent can silently corrupt every response downstream, and traditional monitoring won’t catch it. That…-
<p>In this post, we share how we tackled the hardest operational challenges in running AI agents at enterprise scale, and how Amazon Bedrock AgentCore Evaluations became the backbone of agent quality. Specifically, we cover:</p>-
<ul>-
<li><strong>Agents play by different rules </strong>–&nbsp;Why agents demand a different operational discipline, and why traditional monitoring (latency, error rates, uptime) leaves you blind to the failures that matter most.</li>-
<li><strong>A map for the whole journey </strong>–&nbsp;The nine-stage evaluation framework we built on AgentCore Evaluations, covering every phase of the agent development lifecycle from first commit to production.</li>-
<li><strong>From theory to practice </strong>–&nbsp;Concrete patterns (continuous integration and continuous delivery (CI/CD) regression gates, canary scoring, drift detection, and automated feedback loops) and exactly how AgentCore Evaluations powers each one.</li>-
<li><strong>Metrics that actually matter </strong>–&nbsp;The measurable impact on deployment confidence, regression prevention, and iteration speed across our agent portfolio.</li>-
</ul>-
<h2 id="why-agents-need-a-structured-evaluation-framework">Why agents need a structured evaluation framework</h2>-
<p>Traditional software is deterministic: the same input produces the same output. Machine learning (ML) models introduced probabilistic outputs, and MLOps emerged to manage their training, versioning, monitoring, and re-training. AI agents add a third layer of operational complexity: autonomo…-
<p>Agents inherit the hard problems of both predecessors, and introduce some new ones. They’re non-deterministic, following different reasoning paths for identical inputs. They make cascading decisions across routing, tool selection, and synthesis, where a single step can fail quietly and corr…-
<p>Agent evaluation is the practice of catching exactly that. It spans five dimensions: <strong>safety</strong> (is it behaving within defined boundaries?), <strong>quality</strong> (is the agent producing correct, coherent, grounded responses?), <strong>performan…-
<p>In this post, we share how we operationalize five evaluation dimensions across a production multi-agent system. Safety and security is our first priority. No agent ships, and no agent keeps running, if it can’t meet its safety bar. Quality gets the most space in this post because it’s the h…-
<p>The nine stages apply at three levels simultaneously, and this hierarchy is not specific to this product. It’s the right mental model when we build or integrate new production agents. Most teams start by evaluating at the agent level and stop there. But agent-level quality is only as good a…-
<h2 id="the-agent-evaluation-framework-nine-stages-one-closed-loop">The agent evaluation framework: nine stages, one closed loop</h2>-
<p>These nine stages trace the agent lifecycle from first commit to live production, and each one answers a question teams face as their agents mature.</p>-
<h3 id="the-agent-lifecycle-and-the-nine-common-questions">The agent lifecycle and the nine common questions</h3>-
<p>Every team building production agents encounters the following questions as their agent matures:</p>-
<p><strong>Stage 1: The Foundation:</strong> <em>“What does good look like?”</em></p>-
<p><strong>Stage 2: The Experiment:</strong> <em>“Which version is better?”</em></p>-
<p><strong>Stage 3: The Merge (CI):</strong> <em>“Did I break something?”</em></p>-
<p><strong>Stage 4: The Gate (CD):</strong> <em>“Is it safe for production?”</em></p>-
<p><strong>Stage 5: The Canary:</strong> <em>“Did something silently break?”</em></p>-
<p><strong>Stage 6: The Adversary:</strong> <em>“What if someone tries to break it?”</em></p>-
<p><strong>Stage 7: The Drift:</strong> <em>“Is quality drifting in live?”</em></p>-
<p><strong>Stage 8: The Pulse:</strong> <em>“How do we know it’s working right now?”</em></p>-
<p><strong>Stage 9: The Feedback Loop:</strong> <em>“How does it get better automatically?”</em></p>-
<p><strong>We build one framework for every stage of the agent lifecycle using AgentCore Evaluations.</strong> The nine stages are not separate tools stitched together after the fact. They’re nine points on a single, continuous evaluation surface, powered by AgentCore Evaluations f…-
<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/16/AgentCore-Evaluation-Journey.png" target="_blank" rel="noopener"><img title="Figure 1 Workflow diagram showing 9-stage AgentCore evaluation journey from Define to Improve" src="https://d2908q…-
<p class="wp-caption-text">Figure 1: The nine-stage AgentCore Evaluations journey, from Define through Improve</p>-
</div>-
<p>Here’s how we tackle each one: the pattern that works, the AgentCore services behind it, and what it looks like running in production, shown in system design in Figure 2.</p>-
<div id="attachment_141567" style="width: 1976px" class="wp-caption alignnone">-
<img aria-describedby="caption-attachment-141567" loading="lazy" class="size-full wp-image-141567" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/16/Screenshot-2026-10-16-at-3.46.59 PM.png" alt="" width="1966" height="1038">-
<p id="caption-attachment-141567" class="wp-caption-text">Figure 2: System design of the AgentCore evaluation framework</p>+
<description>Amazon Bedrock now supports structured outputs for all Anthropic Claude models on both the bedrock-runtime and bedrock-mantle endpoints. Receive schema-compliant JSON that conforms to a defined structure, without custom parsing, retry logic, or post-processing, using JSON Schema ou…+
<content:encoded><p>Amazon Bedrock now supports structured outputs for all Anthropic Claude models, on both the bedrock-runtime and bedrock-mantle endpoints, where the model is available.</p>+
<p>Whether you call Claude through Boto3 and the bedrock-runtime APIs or through the Messages API and the bedrock-mantle endpoint, you can now receive schema-compliant JSON responses that conform to a defined structure. You no longer need custom parsing, retry logic, or post-processing.</p&…+
<h2 id="practical-applications">Practical applications</h2>+
<p>With structured outputs for Claude on Amazon Bedrock, you can automate reliable, schema-compliant tasks across industries. Examples include extracting every required field from financial earnings reports and compliance documents, parsing clinical notes into Electronic Health Records (EHR), …+
<h2 id="what-changes-with-structured-outputs-on-amazon-bedrock">What changes with structured outputs on Amazon Bedrock</h2>+
<p>Structured outputs use constrained decoding to enforce your JSON schema during token generation, not after the fact. The model doesn’t produce tokens that would violate your schema. This changes output formatting from probabilistic to deterministic.</p>+
<p>Two complementary mechanisms are now available on both Amazon Bedrock endpoints:</p>+
<table border="1px" width="100%" cellpadding="10px">+
<tbody>+
<tr>+
<td><strong>Feature</strong></td>+
<td><strong>Purpose</strong></td>+
<td><strong>Use case</strong></td>+
</tr>+
<tr>+
<td>JSON Schema output format</td>+
<td>Control the model’s response structure</td>+
<td>Data extraction, report generation, API responses</td>+
</tr>+
<tr>+
<td>Strict tool use</td>+
<td>Validate tool parameters against schemas</td>+
<td>Agentic workflows, function calling, multi-step automation</td>+
</tr>+
</tbody>+
</table>+
<h2 id="what-this-means-for-you">What this means for you</h2>+
<p>Each response is a valid JSON, so you no longer encounter <code>JSON.parse()</code> errors or parsing exceptions. Responses are also type safe: field types are enforced, and required fields are always present. Because outputs always conform to your schema, you don’t need retries…+
<h2 id="supported-models-and-availability">Supported models and availability</h2>+
<p>Structured outputs works with <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html" target="_blank" rel="noopener">batch inference</a>, so you can process large volumes of requests asynchronously while still receiving schema-compliant outputs. On th…+
<h2 id="how-it-works">How it works</h2>+
<p>When you make a request with a JSON schema, here’s what happens behind the scenes:</p>+
<ol type="1">+
<li><strong>Schema validation –</strong>&nbsp;Amazon Bedrock validates your JSON schema against the supported JSON Schema Draft 2020-12 subset.</li>+
<li><strong>Grammar compilation –</strong>&nbsp;For new schemas, Amazon Bedrock compiles a grammar artifact (the first request might take slightly longer).</li>+
<li><strong>Caching –</strong>&nbsp;Compiled grammars are cached for 24 hours, making subsequent requests faster.</li>+
<li><strong>Constrained generation –</strong>&nbsp;The model generates tokens that produce valid JSON matching your schema.</li>+
</ol>+
<h2 id="performance-considerations">Performance considerations</h2>+
<p>For performance, the first request with a new schema might take longer because Amazon Bedrock compiles a grammar artifact before generation begins. After that initial compilation, subsequent requests using the same schema run with minimal overhead because the compiled grammar is cached. Thi…+
<h2 id="getting-started-structured-outputs-with-the-invokemodel-api-bedrock-runtime">Getting started: Structured outputs with the InvokeModel API (bedrock-runtime)</h2>+
<p>The InvokeModel API takes the Anthropic-native request body, so the schema goes in <code>output_config.format</code> as a JSON object rather than a JSON string. The following example shows how to extract the structured lead data from a customer email using boto3 with an Anthropi…+
<p>Python</p>+
<div class="hide-language">+
<pre><code class="language-python">import boto3+
import json+
+
REGION = "us-east-1" # Choose your preferred region+
bedrock_runtime = boto3.client(service_name="bedrock-runtime", region_name=REGION)+
+
# Define the JSON schema for structured output+
schema = {+
"type": "object",+
"properties": {+
"name": {"type": "string", "description": "Customer name"},+
"email": {"type": "string", "description": "Customer email address"},+
"plan_interest": {"type": "string", "description": "Product plan of interest"},+
"demo_requested": {"type": "boolean", "description": "Whether a demo was requested"}+
},+
"required": ["name", "email", "plan_interest", "demo_requested"],+
"additionalProperties": False+
}+
+
# Make a request with structured outputs via the InvokeModel API+
# (reusing the schema defined for the Converse example)+
response = bedrock_runtime.invoke_model(+
modelId="global.anthropic.claude-sonnet-5-5",+
body=json.dumps({+
"anthropic_version": "bedrock-2023-05-31",+
"max_tokens": 512,+
"messages": [+
{"role": "user", "content": (+
"Paulo Santos ([email redacted]) is interested in our Enterprise plan "+
"and wants to schedule a demo for next Tuesday at 2pm."+
)}+
],+
"output_config": {"format": {"type": "json_schema", "schema": schema}}+
})+
)+
+
body = json.loads(response["body"].read())+
+
# Check the stop reason before parsing+
if body["stop_reason"] == "end_turn":+
result_text = next(b["text"] for b in body["content"] if b["type"] == "text")+
print(json.dumps(json.loads(result_text), indent=2))+
elif body["stop_reason"] == "max_tokens":+
print("Response truncated: increase max_tokens and retry.")</code></pre></div>-
<h3 id="stage-1-the-foundation.-define-quality">Stage 1: The Foundation. Define quality</h3>-
<p>The foundation of evaluating an agent isn’t about choosing the right tools. It’s about asking the right question: what does success look like for this agent, and how do we know when it is failing? The answer must come from the customer, not from the model. This is an iterative process. Our …-
<p><strong>Stage 1.1: Define your business metrics first.</strong> Start by asking what the agent exists to deliver. The two end user personas define success differently:</p>-
<ul>-
<li>For sellers, success means walking into a customer conversation with accurate, current intelligence at the right detail level and at the right time.</li>-
<li>For leadership, success means reliable weekly and cross-business review inputs without lengthy manual preparation. These are business outcomes, not model properties.</li>-
</ul>-
<p><strong>Stage 1.2: Translate those business metrics into evaluation metrics.</strong> For each outcome, identify the signal that confirms the agent delivered it. Each signal becomes a tracked metric with an explicit threshold and an alarm condition. Accurate account intelligence…-
<p>For accuracy specifically, because the solution directly informs revenue decisions, renewal strategies, and executive reviews, the bar is high. A confidently wrong answer does not only waste a seller’s time. It leads to a wrong decision in a real business moment. We define six failure types…-
<ul>-
<li><strong>Metric swap:</strong> correct customer and period, wrong metric returned. Tracked as occurrences per 100 responses.</li>-
<li><strong>Wrong period:</strong> data from one time window is labeled as another, for example a year-over-year (YoY) figure presented to the seller as month-over-month (MoM), or the reverse. Tracked as % citing the correct window.</li>-
<li><strong>Wrong entity:</strong> conflated account names or subsidiaries. Tracked as entity resolution accuracy %.</li>-
<li><strong>Pure hallucination:</strong> ungrounded figures with no traceable source. Tracked as grounding rate %.</li>-
<li><strong>Wrong symbols:</strong> correct metric, wrong currency or unit symbol. Tracked as scope adherence rate %.</li>-
<li><strong>Unauthorized calculation:</strong> unsanctioned derived figures not present in the source data. Tracked as occurrences per 100 responses.</li>-
</ul>-
<p>The overall accuracy pass rate is the headline metric. Per-dimension tracking is what makes it actionable. Beyond accuracy, we also track <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/prompt-templates-builtin.html#response-relevance" target="_blank" rel="noopener…-
<p><strong>Stage 1.3: Build an evaluator stack that reflects those metrics.</strong> No single evaluator type covers the various failure modes possible. We combine four types:</p>-
<ul>-
<li><strong>AgentCore built-in evaluators</strong> for broad baseline coverage. Relevance, faithfulness, and helpfulness map directly to evaluation templates available natively in <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html" target="_…-
<li><strong>Customized evaluators</strong>, one per failure type in our taxonomy: a metric swap evaluator, a wrong period evaluator, a wrong customer evaluator, a pure hallucination evaluator, a wrong symbols evaluator, and an unauthorized calculation evaluator. None of the six ar…-
<li><strong>Third-party evaluators,</strong> we deployed the managed evaluators from third-party open source libraries DeepEval and AutoEval.</li>-
<li><strong>Deterministic evaluators</strong> for ground truth checks where the correct answer is known exactly, such as revenue, renewal date, and contract status.</li>-
</ul>-
<p>Where reasoning quality and helpfulness have no single correct answer, we use LLM-as-judge rubrics instead, with built-in rubrics for relevance, faithfulness, and helpfulness plus support for fully custom rubrics. AgentCore Evaluations provides both ground truth and LLM-as-judge evaluation …-
<p><strong>Stage 1.4: Build a golden set to exercise your evaluation metrics.</strong> At the <strong>tool level</strong>, each critical tool has its own deterministic ground truth pairs: customer identification must resolve to the exact canonical entity or fail. At the…-
<p>The golden set is a living document, built from three sources. Production-derived cases are sampled from real seller and leadership queries, including the low-scoring live sessions promoted in Stage 7. Failure-derived cases come from every regression caught in staging or production, includi…-
<h3 id="stage-2-the-experiment.-know-which-version-wins">Stage 2: The Experiment. Know which version wins</h3>-
<p>Every change (a prompt update, a new tool, a model upgrade, a dataset refresh) raises the same question: <em>is this version actually better?</em> In a system serving the AWS field at global scale, intuition isn’t an acceptable answer.</p>-
<p>Before a change merges, we run an <em>AgentCore On-Demand Evaluation</em>, scored against the same golden set with the same evaluator suite. The development team uses the quality, latency, cost metrics, and context token, to curate downstream comparisons which is evaluated acros…-
<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/15/ML-21894-3.png" target="_blank" rel="noopener"><img title="Figure 3. Side-by-side comparison of two candidate agent responses in an AgentCore On-Demand Evaluation, scored across quality, late…-
<p class="wp-caption-text">Figure 3: Side-by-side comparison of two candidate agent responses in an AgentCore On-Demand Evaluation across quality, latency, and cost. The numbers and values in the figure are illustrative.</p>+
<p>JSON</p>+
<div class="hide-language">+
<pre><code class="language-json">{+
"name": "Paulo Santos",+
"email": "[email redacted]",+
"plan_interest": "Enterprise",+
"demo_requested": true+
}</code></pre></div>-
<p><strong>We track quality as a comprehensive scorecard, not one single number.</strong> A pass rate alone doesn’t tell you enough. Each eval run produces a full breakdown across every failure category in our taxonomy: not just whether quality improved, but where and why, as captu…-
<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/15/ML-21894-4.png" target="_blank" rel="noopener"><img title="Figure 4. Evaluation scorecard showing per-category failure breakdown and pass rate trend. The numbers and values in the figure are …-
<p class="wp-caption-text">Figure 4: Evaluation scorecard with per-category failure breakdown and pass-rate trend. The numbers and values in the figure are illustrative.</p>+
<p>The response conforms to your schema. No additional validation is required.</p>+
<p>For additional API usage examples, including the Messages API, see the accompanying <a href="https://github.com/aws-samples/anthropic-on-aws/blob/main/notebooks/structured_outputs/structured-outputs-on-bedrock.ipynb" target="_blank" rel="noopener">sample notebook</a>.</p>+
<h2 id="strict-tool-use-for-agentic-workflows">Strict tool use for agentic workflows</h2>+
<p><a href="https://docs.aws.amazon.com/bedrock/latest/userguide/endpoints.html" target="_blank" rel="noopener">Amazon Bedrock Mantle</a> is designed for agentic applications with server-side tools, background inference, and stateful conversations. With strict tool use on the bedro…+
<div class="hide-language">+
<pre><code class="language-python">Import json+
from anthropic import AnthropicBedrockMantle+
+
# Reusing the client created in the Messages API example+
client = AnthropicBedrockMantle(aws_region="us-east-1")+
+
message = client.messages.create(+
model="anthropic.claude-sonnet-5",+
max_tokens=512,+
messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],+
tools=[+
{+
"name": "get_weather",+
"description": "Get the current weather for a specified location",+
"strict": True,+
"input_schema": {+
"type": "object",+
"properties": {+
"location": {+
"type": "string",+
"description": "The city and state, e.g., San Francisco, CA"+
},+
"unit": {+
"type": "string",+
"enum": ["celsius", "fahrenheit"],+
"description": "Temperature unit"+
}+
},+
"required": ["location", "unit"],+
"additionalProperties": False+
}+
}+
]+
)+
+
#Tool inputs conform to the schema+
for block in message.content:+
if block.type == "tool_use":+
print(f"Tool: {block.name}")+
print(f"Input: {json.dumps(block.input, indent=2)}")</code></pre></div>-
<h3 id="stage-3-the-merge-ci.-prevent-regression-at-merge-time">Stage 3: The Merge (CI). Prevent regression at merge time</h3>-
<p>Stage 3 is a pre-production gate. It runs in the staging environment, triggered by code and configuration changes, and its job is to block a bad change before it ever reaches production. If Stage 3 catches a regression, nothing ships.</p>-
<p>Engineers or scientists change the system in many ways: the prompt, the underlying data, the tools, or the model. The change looks clean locally, tests pass, but a prompt that improves responses for technology accounts could silently degrade responses for financial services accounts. Withou…-
<p>Our CI evaluation is triggered automatically on every staging deployment using AgentCore On-Demand Evaluation. Engineers do not need to remember to run an evaluation. No deployment can silently skip the gate. The run completes in minutes because it evaluates only the sentinel set, not the f…-
<p>The sentinel set is a curated selection of the highest-risk scenarios across five dimensions. Quality regressions beyond tolerance block the deployment. A safety boundary violation blocks just as hard. A change that doubles p90 latency does not proceed without a deliberate decision. Token c…-
<p><strong>One example of what this catches.</strong> After a data normalization update, a disambiguation tool started returning unresolved entities for a subset of account name formats. The agent, receiving an unresolved entity, responded with a best-guess match, returning real da…-
<h3 id="stage-4-the-gate-cd.-ship-with-quantitative-confidence">Stage 4: The Gate (CD). Ship with quantitative confidence</h3>-
<p>The sentinel covers 15 cases. Production covers tens of thousands of query patterns across hundreds of account types, geographies, and seller segments. Clearing the sentinel is necessary. It is not sufficient.</p>-
<p>Before a candidate ships, we run an AgentCore On-Demand Evaluation against the full golden set across the evaluator dimensions. We call this the <em>Quality Gate. </em>It replaces “it seems fine” with a quantitative shipping decision.</p>-
<p><strong>Sandboxed execution and tool mocking.</strong> Think of it as a replay, not a live call. Ahead of time, we freeze a known, versioned dataset of correct tool behavior. Every tool call the candidate makes during the gate is intercepted and answered from that frozen dataset…-
<p><strong>Aggregated end-to-end scoring.</strong> The gate also runs a holistic pass: the full candidate agent answers 200 held-out questions end-to-end, scored by an LLM-as-judge rubric across all evaluator dimensions. Any one of three conditions shuts the gate automatically: the…-
<p><strong>One example of what this catches.</strong> This is where mocking tests the agent’s own reasoning, not just the tools underneath it. Our golden set includes a hierarchy mismatch case: the Customer Insights Agent and Data Insights Agent return different, individually corre…-
<h3 id="stage-5-the-canary.-catch-silent-post-deploy-failures">Stage 5: The Canary. Catch silent post-deploy failures</h3>-
<p>Stage 5 is a post-production monitor. It runs continuously in the production environment against live traffic and synthetic evaluation cases, and its job is to catch issues that no code change triggered and that Stage 3 therefore didn’t see. Stage 3 guards against what engineers change. Sta…-
<p>The product operates against a data estate fed by thousands of upstream schemas and data pipelines. On a given day, dozens of those pipelines refresh without triggering a deployment or CI gate. The agent code is unchanged, the model is unchanged, but the data it reasons over has changed, an…-
<p>We run AgentCore Batch Evaluation on two complementary schedules, and the choice of cadence was deliberate.</p>-
<p>We settled on 30-minute cycles after learning from both directions: 15 minutes generated too many false alarms; 60 minutes let two hours pass before a real degradation registered. We call each 30-minute cycle an evaluation window. Thirty minutes catches a real trend within 90 minutes (three…-
<p>Thresholds are deliberately lower than CI/CD gates. A single window failure is an ops signal, not an incident. We only page oncall after three consecutive windows breach threshold. We learned this rule the hard way: the first month we used a single-window threshold, oncall stopped trusting …-
<p>Every run publishes to CloudWatch (accuracy, hallucination rates, grounding rate, latency, and cost), with alarm logic that distinguishes between the two cadences. The canary runs at three levels: tool-level scores surface a degraded retrieval tool before it corrupts agent-level responses.&…-
<p><strong>Another example of what this catches.</strong> After an upstream index refresh, a retrieval tool began returning loosely relevant chunks instead of the precise context needed to answer the query. Latency was nominal. Error rate was zero. The agent kept producing fluent, …-
<h3 id="stage-6-the-adversary.-protect-against-misuse">Stage 6: The Adversary. Protect against misuse</h3>-
<p>Not every bad output comes from a degraded model. Queries crafted to extract unauthorized account data, and prompt injections embedded in retrieved documents, are predictable patterns in a system serving thousands of users on sensitive business data.</p>-
<p>Safety isn’t a stage. It runs through all nine stages. We apply Amazon Bedrock Guardrails at two points in every interaction. At input, we validate query scope and screen for prompt injection. At output, we verify that no sensitive data is surfaced to unauthorized users and that the agent h…-
<p>We configure two guardrail layers, each covering a distinct threat class:</p>-
<p><strong>Content filters</strong> detect and block harmful or inappropriate content in both inputs and outputs. We enforce filters across categories including <code>VIOLENCE</code>, <code>HATE</code>, <code>SEXUAL</code>, <code>MISCONDUCT…-
<p><strong>Prompt security filters</strong> protect the agent from adversarial manipulation under a single <code>PROMPT_ATTACK</code> guardrail category, covering three sub-types: <code>JAILBREAK</code> (attempts to override the agent’s system instructions o…-
<p>Safety rules are defined at Stage 1, checked for regressions at Stages 3 and 4, and enforced continuously in production.</p>-
<h3 id="stage-7-the-drift.-sustain-quality-and-validate-with-live-traffic">Stage 7: The Drift. Sustain quality and validate with live traffic</h3>-
<p>Quality degrades from multiple directions: upstream data changes without triggering a deployment, models grow stale, and users push boundaries with queries no golden set fully anticipates. Stage 7 detects it before sellers do, and validates candidates against real user traffic before full r…-
<p><strong>Drift detection.</strong> Using AgentCore Online Evaluation, we sample 10 percent of live traffic continuously through the different evaluators defined at Stage 1, so the drift is immediately visible as a deviation from the baselines established offline. Scores feed into…-
<p>One production example. Our golden set for customer identification was built from synthetic edge cases: name variations, subsidiary lookups, abbreviations. It performed well offline. But in live traffic, sellers queried with stock ticker symbols (“what’s going on with AMZN?”), shorthand nam…-
<p>Live traffic feeds two continuous improvement loops: low-scoring sessions are promoted into the golden set as new edge cases, and clusters of low-scoring traces inform prompt revisions, tool updates, and routing adjustments.</p>-
<p><strong>A/B testing:</strong> After a candidate clears a quality gate, AgentCore Gateway, a capability of Amazon Bedrock AgentCore, routes a slice of real traffic between current and candidate agents. On-demand confirms no regression on known cases. Online confirms it holds up a…-
<h3 id="stage-8-the-pulse.-real-time-quality-visibility">Stage 8: The Pulse. Real-time quality visibility</h3>-
<p>A single call to the agent might touch three sub-agents, five tools, and two retrieval paths. When a quality alarm fires, “something in the agent pipeline” is not a runbook. Stage 8 is where signals from Stages 5, 6, and 7 converge into an observability layer with enough resolution to act.&…-
<p>AgentCore Observability, a capability of Amazon Bedrock AgentCore, is the foundation. For agents hosted on the AgentCore runtime, a capability of Amazon Bedrock AgentCore, every execution is instrumented automatically through OpenTelemetry, emitting structured telemetry for every reasoning …+
<p>Setting <code>strict: true</code> constrains the model’s tool input to match your schema exactly. The location field is always a string, and the unit field is always either celsius or fahrenheit. The model can’t add fields outside your schema, so the tool input contains only wha…+
<p>This is especially valuable for agentic use cases on Amazon Bedrock Mantle, where models orchestrate multi-step workflows with server-side tools like web search and code interpreter.</p>+
<h2 id="top-use-cases-for-claude-structured-outputs">Top use cases for Claude structured outputs</h2>+
<p>Structured outputs reduce error-handling complexity the most in three common patterns:</p><ul>-
<li>A conversational dashboard to query evaluation and observability results.</li>-
<li>An Amazon Quick dashboard for trend analysis, scorecard tracking, and cross-agent comparisons across the full history.</li>+
<li><strong>Data extraction:</strong> Extract fields from invoices, contracts, email, and transcripts into JSON that always matches your schema, ready to load into databases or customer relationship management (CRM) systems.</li>+
<li><strong>Reliable agent tool calls:</strong> Validate tool arguments against their schemas at generation time, so multi-step agent workflows run without failures from malformed input.</li>+
<li><strong>Classification at scale:</strong> Classify tickets, leads, or content by intent, priority, and sentiment for routing, moderation, and evaluation.</li></ul>-
<p>Role-appropriate views sit on top. Oncall engineers get a trace-first view: one click from a CloudWatch alarm to the AgentCore Observability trace that triggered it. The trace shows every tool called, in what order, with latency per step, token consumption, cost, and the full reasoning cont…+
<p>Claude Haiku 5.5 reliably handles quick and repetitive workloads such as summaries, compactions, database queries, and classification requests. The following example shows how to classify and route incoming data using the Invoke API and Haiku 5.5:</p>+
<div class="hide-language">+
<pre><code class="language-python">import boto3+
import json+
+
REGION = "us-east-1" # Choose your preferred region+
bedrock_runtime = boto3.client(service_name="bedrock-runtime", region_name=REGION)+
+
# Classification schema: enums restrict output to a fixed set of labels+
schema = {+
"type": "object",+
"properties": {+
"category": {+
"type": "string",+
"enum": ["billing", "technical", "account", "sales", "other"],+
"description": "Primary ticket category"+
},+
"priority": {+
"type": "string",+
"enum": ["low", "medium", "high", "urgent"],+
"description": "Urgency based on business impact"+
},+
"sentiment": {+
"type": "string",+
"enum": ["positive", "neutral", "negative"],+
"description": "Customer sentiment"+
},+
"escalate": {"type": "boolean", "description": "Whether a human should review immediately"},+
"reason": {"type": "string", "description": "One-sentence justification for the classification"}+
},+
"required": ["category", "priority", "sentiment", "escalate", "reason"],+
"additionalProperties": False+
}+
+
ticket = (+
"Our production API calls have been failing with 500 errors since this morning. "+
"We're losing orders every minute. This is the second outage this month!"+
)+
+
# Make a request with structured outputs via the InvokeModel API+
response = bedrock_runtime.invoke_model(+
modelId="global.anthropic.claude-haiku-5-5",+
body=json.dumps({+
"anthropic_version": "bedrock-2023-05-31",+
"max_tokens": 512,+
"system": "You are a support ticket classifier. Classify the ticket accurately.",+
"messages": [+
{"role": "user", "content": f"Classify this support ticket:\n\n{ticket}"}+
],+
"output_config": {"format": {"type": "json_schema", "schema": schema}}+
})+
)+
+
body = json.loads(response["body"].read())+
+
# Check the stop reason before parsing+
if body["stop_reason"] == "end_turn":+
result_text = next(b["text"] for b in body["content"] if b["type"] == "text")+
result = json.loads(result_text)+
print(json.dumps(result, indent=2))+
# Route the ticket based on the guaranteed-valid labels+
if result["escalate"]:+
print(f"Escalating to on-call: {result['category']} / {result['priority']}")+
elif body["stop_reason"] == "max_tokens":+
print("Response truncated: increase max_tokens and retry.")</code></pre>+
</div>+
<h2 id="requirements-and-best-practices">Requirements and best practices</h2>+
<p>To use structured outputs effectively on Amazon Bedrock, follow these guidelines:</p>+
<ol type="1">+
<li><strong>Set additionalProperties: false on all objects</strong>: This is required for structured outputs to work. Without it, your schema won’t be accepted.</li>+
<li><strong>Use descriptive field names and descriptions</strong>: Models use property names and descriptions to understand what data to extract. Clear names like <code>customer_email</code> outperform generic names like <code>field1</code>.</li>+
<li><strong>Use enum for constrained values</strong>: When a field has a limited set of valid values, use <code>enum</code> to constrain options.</li>+
<li><strong>Start basic, then add complexity</strong>: Begin with the minimum required fields and add complexity incrementally. Basic schemas compile faster, are easier to debug, and keep your request payload lean because the schema counts toward the Converse API payload size.<…+
<li><strong>Reuse schemas to benefit from caching</strong>: Structure your application to reuse schemas across requests. The 24-hour grammar cache improves performance for repeated queries.</li>+
<li><strong>Check the stop reason in every response</strong>: Two scenarios can produce non-conforming responses: refusals (when the model declines for safety reasons) and token limits (when <code>max_tokens</code> is reached before completing). Handle both cases in yo…+
<li><strong>Test with realistic data before deployment</strong>: Validate your schemas against production-representative inputs. Edge cases in real data often reveal schema design issues.</li>+
</ol>+
<p>For additional best practices and schema design recommendations, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/structured-output.html" target="_blank" rel="noopener">Structured outputs</a> in the Amazon Bedrock User Guide.</p>+
<h2 id="supported-json-schema-features">Supported JSON Schema features</h2>+
<ol type="1">+
<li>All basic types: <code>object</code>, <code>array</code>, <code>string</code>, <code>integer</code>, <code>number</code>, <code>boolean</code>, <code>null</code>.</li>+
<li><code>enum</code> (strings, numbers, booleans, or nulls only).</li>+
<li><code>const</code>, <code>anyOf</code>, <code>allOf</code> (with limitations).</li>+
<li><code>$ref</code>, <code>$def</code>, and <code>definitions</code> (internal references only).</li>+
<li>String formats: <code>date-time</code>, <code>time</code>, <code>date</code>, <code>duration</code>, <code>email</code>, <code>hostname</code>, <code>uri</code>, <code>ipv4</code>, <code>…+
<li>Array <code>minItems</code> (only values 0 and 1).</li>Diff display stops at 400 lines. The line counts above are from the whole diff. 63 lines shown here cut at 300 characters. 2 email addresss masked in this display. See the About page on personal data. The raw artifact at this commit is linked above.