Change
c667ad9
c667ad9ce659eb59dae63dc7340b994311f2a96d · commit on GitHub
aws-blog-feed: changed (571514 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +730
- Lines removed
- -612
- Stored bytes at this commit
- 571,514
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-10-17T04:26:12.779Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Fri, 16 Oct 2026 22:46:08 GMT |
| date | Sat, 17 Oct 2026 04:26:12 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Thu, 15 Oct 2026 17:43:35 +0000</lastBuildDate>+
<lastBuildDate>Fri, 16 Oct 2026 21:06:58 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,6 +13,735 @@ 1 </sy:updateFrequency> <item>+
<title>Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore</title>+
<link>https://aws.amazon.com/blogs/machine-learning/building-trust-in-ai-agents-a-production-ready-evaluation-framework-with-amazon-bedrock-agentcore/</link>+
+
<dc:creator><![CDATA[Rui Li]]></dc:creator>+
<pubDate>Fri, 16 Oct 2026 20:48:16 +0000</pubDate>+
<category><![CDATA[Advanced (300)]]></category>+
<category><![CDATA[Amazon Bedrock AgentCore]]></category>+
<category><![CDATA[Best Practices]]></category>+
<guid isPermaLink="false">df0b1ae32a5ba63d5856db229a66706d4090e8ef</guid>+
+
<description>Autonomous AI agents fail softly: latency stays flat while answer quality quietly erodes. This post shares a nine-stage evaluation framework, built on Amazon Bedrock AgentCore, that catches silent quality regressions across the full agent lifecycle, from first commit to live produc…+
<content:encoded><h2 id="the-north-star">The North Star</h2>+
<p>Every organization racing to deploy AI agents faces the same reckoning: an agent that demos beautifully can quietly degrade in production, and by the time a human notices, the trust is already broken. Where traditional software fails loudly, agents fail softly. They return an answer that lo…+
<p>That’s the quality problem at its core. The challenge is therefore not building agents that work once, but proving they keep working, at scale, across every sub-agent, every workflow, and every release. The teams that win the agent era will be the ones who treat quality not as a final QA st…+
<h3 id="business-problem">Business problem</h3>+
<p>The solution uses Amazon Bedrock AgentCore and other AWS services for internal use, so sellers and sales leaders can ask business questions in plain English from Amazon Quick. Behind that single experience, several specialized sub-agents work together: the Performance Insights Agent for ter…+
<ul>+
<li>It reported $410.08 of a customer’s three-month revenue on a service as “$410.08 Million.”</li>+
<li>It reported Sales Revenue labeled as Sales Revenue excluding Adjustments.</li>+
<li>It offered an Annual Recurring Revenue estimate that appeared nowhere in the data.</li>+
<li>It presented its own calculations as sourced facts.</li>+
</ul>+
<p>Through the different runs, monitoring stayed green. It told us the system had responded, not whether the answer was right.</p>+
<p>Had these errors shipped uncaught, the field would have paid for them. A seller who repeats an inflated figure can lose credibility with the customer. This is exactly why we caught it before launch. The risk doesn’t end at launch: it grows with every new agent and every change to data, prom…+
<p>At AWS, the Sales, Marketing, and Global Services (SMGS) Operations team developed a production multi-agent system serving AWS Field Sales worldwide. It serves two distinct audiences. For field sales teams, it replaces hours of manual data analysis with on-demand, conversational intelligenc…+
<p>The agentic system orchestrates a growing portfolio of specialized sub-agents, each owning a distinct capability domain. Among them: the <a href="https://aws.amazon.com/blogs/machine-learning/how-aws-smgs-uses-an-ai-powered-conversational-assistant-to-transform-business-management-with-a…+
<p>Coordinating these agents reliably at scale made one thing clear: agentic quality wasn’t an operational concern to manage later. It was the prerequisite for operating. A single degraded sub-agent can silently corrupt every response downstream, and traditional monitoring won’t catch it. That…+
<p>In this post, we share how we tackled the hardest operational challenges in running AI agents at enterprise scale, and how Amazon Bedrock AgentCore Evaluations became the backbone of agent quality. Specifically, we cover:</p>+
<ul>+
<li><strong>Agents play by different rules </strong>–&nbsp;Why agents demand a different operational discipline, and why traditional monitoring (latency, error rates, uptime) leaves you blind to the failures that matter most.</li>+
<li><strong>A map for the whole journey </strong>–&nbsp;The nine-stage evaluation framework we built on AgentCore Evaluations, covering every phase of the agent development lifecycle from first commit to production.</li>+
<li><strong>From theory to practice </strong>–&nbsp;Concrete patterns (continuous integration and continuous delivery (CI/CD) regression gates, canary scoring, drift detection, and automated feedback loops) and exactly how AgentCore Evaluations powers each one.</li>+
<li><strong>Metrics that actually matter </strong>–&nbsp;The measurable impact on deployment confidence, regression prevention, and iteration speed across our agent portfolio.</li>+
</ul>+
<h2 id="why-agents-need-a-structured-evaluation-framework">Why agents need a structured evaluation framework</h2>+
<p>Traditional software is deterministic: the same input produces the same output. Machine learning (ML) models introduced probabilistic outputs, and MLOps emerged to manage their training, versioning, monitoring, and re-training. AI agents add a third layer of operational complexity: autonomo…+
<p>Agents inherit the hard problems of both predecessors, and introduce some new ones. They’re non-deterministic, following different reasoning paths for identical inputs. They make cascading decisions across routing, tool selection, and synthesis, where a single step can fail quietly and corr…+
<p>Agent evaluation is the practice of catching exactly that. It spans five dimensions: <strong>safety</strong> (is it behaving within defined boundaries?), <strong>quality</strong> (is the agent producing correct, coherent, grounded responses?), <strong>performan…+
<p>In this post, we share how we operationalize five evaluation dimensions across a production multi-agent system. Safety and security is our first priority. No agent ships, and no agent keeps running, if it can’t meet its safety bar. Quality gets the most space in this post because it’s the h…+
<p>The nine stages apply at three levels simultaneously, and this hierarchy is not specific to this product. It’s the right mental model when we build or integrate new production agents. Most teams start by evaluating at the agent level and stop there. But agent-level quality is only as good a…+
<h2 id="the-agent-evaluation-framework-nine-stages-one-closed-loop">The agent evaluation framework: nine stages, one closed loop</h2>+
<p>These nine stages trace the agent lifecycle from first commit to live production, and each one answers a question teams face as their agents mature.</p>+
<h3 id="the-agent-lifecycle-and-the-nine-common-questions">The agent lifecycle and the nine common questions</h3>+
<p>Every team building production agents encounters the following questions as their agent matures:</p>+
<p><strong>Stage 1: The Foundation:</strong> <em>“What does good look like?”</em></p>+
<p><strong>Stage 2: The Experiment:</strong> <em>“Which version is better?”</em></p>+
<p><strong>Stage 3: The Merge (CI):</strong> <em>“Did I break something?”</em></p>+
<p><strong>Stage 4: The Gate (CD):</strong> <em>“Is it safe for production?”</em></p>+
<p><strong>Stage 5: The Canary:</strong> <em>“Did something silently break?”</em></p>+
<p><strong>Stage 6: The Adversary:</strong> <em>“What if someone tries to break it?”</em></p>+
<p><strong>Stage 7: The Drift:</strong> <em>“Is quality drifting in live?”</em></p>+
<p><strong>Stage 8: The Pulse:</strong> <em>“How do we know it’s working right now?”</em></p>+
<p><strong>Stage 9: The Feedback Loop:</strong> <em>“How does it get better automatically?”</em></p>+
<p><strong>We build one framework for every stage of the agent lifecycle using AgentCore Evaluations.</strong> The nine stages are not separate tools stitched together after the fact. They’re nine points on a single, continuous evaluation surface, powered by AgentCore Evaluations f…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/16/AgentCore-Evaluation-Journey.png" target="_blank" rel="noopener"><img title="Figure 1 Workflow diagram showing 9-stage AgentCore evaluation journey from Define to Improve" src="https://d2908q…+
<p class="wp-caption-text">Figure 1: The nine-stage AgentCore Evaluations journey, from Define through Improve</p>+
</div>+
<p>Here’s how we tackle each one: the pattern that works, the AgentCore services behind it, and what it looks like running in production, shown in system design in Figure 2.</p>+
<div id="attachment_141567" style="width: 1976px" class="wp-caption alignnone">+
<img aria-describedby="caption-attachment-141567" loading="lazy" class="size-full wp-image-141567" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/16/Screenshot-2026-10-16-at-3.46.59 PM.png" alt="" width="1966" height="1038">+
<p id="caption-attachment-141567" class="wp-caption-text">Figure 2: System design of the AgentCore evaluation framework</p>+
</div>+
<h3 id="stage-1-the-foundation.-define-quality">Stage 1: The Foundation. Define quality</h3>+
<p>The foundation of evaluating an agent isn’t about choosing the right tools. It’s about asking the right question: what does success look like for this agent, and how do we know when it is failing? The answer must come from the customer, not from the model. This is an iterative process. Our …+
<p><strong>Stage 1.1: Define your business metrics first.</strong> Start by asking what the agent exists to deliver. The two end user personas define success differently:</p>+
<ul>+
<li>For sellers, success means walking into a customer conversation with accurate, current intelligence at the right detail level and at the right time.</li>+
<li>For leadership, success means reliable weekly and cross-business review inputs without lengthy manual preparation. These are business outcomes, not model properties.</li>+
</ul>+
<p><strong>Stage 1.2: Translate those business metrics into evaluation metrics.</strong> For each outcome, identify the signal that confirms the agent delivered it. Each signal becomes a tracked metric with an explicit threshold and an alarm condition. Accurate account intelligence…+
<p>For accuracy specifically, because the solution directly informs revenue decisions, renewal strategies, and executive reviews, the bar is high. A confidently wrong answer does not only waste a seller’s time. It leads to a wrong decision in a real business moment. We define six failure types…+
<ul>+
<li><strong>Metric swap:</strong> correct customer and period, wrong metric returned. Tracked as occurrences per 100 responses.</li>+
<li><strong>Wrong period:</strong> data from one time window is labeled as another, for example a year-over-year (YoY) figure presented to the seller as month-over-month (MoM), or the reverse. Tracked as % citing the correct window.</li>+
<li><strong>Wrong entity:</strong> conflated account names or subsidiaries. Tracked as entity resolution accuracy %.</li>+
<li><strong>Pure hallucination:</strong> ungrounded figures with no traceable source. Tracked as grounding rate %.</li>+
<li><strong>Wrong symbols:</strong> correct metric, wrong currency or unit symbol. Tracked as scope adherence rate %.</li>+
<li><strong>Unauthorized calculation:</strong> unsanctioned derived figures not present in the source data. Tracked as occurrences per 100 responses.</li>+
</ul>+
<p>The overall accuracy pass rate is the headline metric. Per-dimension tracking is what makes it actionable. Beyond accuracy, we also track <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/prompt-templates-builtin.html#response-relevance" target="_blank" rel="noopener…+
<p><strong>Stage 1.3: Build an evaluator stack that reflects those metrics.</strong> No single evaluator type covers the various failure modes possible. We combine four types:</p>+
<ul>+
<li><strong>AgentCore built-in evaluators</strong> for broad baseline coverage. Relevance, faithfulness, and helpfulness map directly to evaluation templates available natively in <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html" target="_…+
<li><strong>Customized evaluators</strong>, one per failure type in our taxonomy: a metric swap evaluator, a wrong period evaluator, a wrong customer evaluator, a pure hallucination evaluator, a wrong symbols evaluator, and an unauthorized calculation evaluator. None of the six ar…+
<li><strong>Third-party evaluators,</strong> we deployed the managed evaluators from third-party open source libraries DeepEval and AutoEval.</li>+
<li><strong>Deterministic evaluators</strong> for ground truth checks where the correct answer is known exactly, such as revenue, renewal date, and contract status.</li>+
</ul>+
<p>Where reasoning quality and helpfulness have no single correct answer, we use LLM-as-judge rubrics instead, with built-in rubrics for relevance, faithfulness, and helpfulness plus support for fully custom rubrics. AgentCore Evaluations provides both ground truth and LLM-as-judge evaluation …+
<p><strong>Stage 1.4: Build a golden set to exercise your evaluation metrics.</strong> At the <strong>tool level</strong>, each critical tool has its own deterministic ground truth pairs: customer identification must resolve to the exact canonical entity or fail. At the…+
<p>The golden set is a living document, built from three sources. Production-derived cases are sampled from real seller and leadership queries, including the low-scoring live sessions promoted in Stage 7. Failure-derived cases come from every regression caught in staging or production, includi…+
<h3 id="stage-2-the-experiment.-know-which-version-wins">Stage 2: The Experiment. Know which version wins</h3>+
<p>Every change (a prompt update, a new tool, a model upgrade, a dataset refresh) raises the same question: <em>is this version actually better?</em> In a system serving the AWS field at global scale, intuition isn’t an acceptable answer.</p>+
<p>Before a change merges, we run an <em>AgentCore On-Demand Evaluation</em>, scored against the same golden set with the same evaluator suite. The development team uses the quality, latency, cost metrics, and context token, to curate downstream comparisons which is evaluated acros…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/15/ML-21894-3.png" target="_blank" rel="noopener"><img title="Figure 3. Side-by-side comparison of two candidate agent responses in an AgentCore On-Demand Evaluation, scored across quality, late…+
<p class="wp-caption-text">Figure 3: Side-by-side comparison of two candidate agent responses in an AgentCore On-Demand Evaluation across quality, latency, and cost. The numbers and values in the figure are illustrative.</p>+
</div>+
<p><strong>We track quality as a comprehensive scorecard, not one single number.</strong> A pass rate alone doesn’t tell you enough. Each eval run produces a full breakdown across every failure category in our taxonomy: not just whether quality improved, but where and why, as captu…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/15/ML-21894-4.png" target="_blank" rel="noopener"><img title="Figure 4. Evaluation scorecard showing per-category failure breakdown and pass rate trend. The numbers and values in the figure are …+
<p class="wp-caption-text">Figure 4: Evaluation scorecard with per-category failure breakdown and pass-rate trend. The numbers and values in the figure are illustrative.</p>+
</div>+
<h3 id="stage-3-the-merge-ci.-prevent-regression-at-merge-time">Stage 3: The Merge (CI). Prevent regression at merge time</h3>+
<p>Stage 3 is a pre-production gate. It runs in the staging environment, triggered by code and configuration changes, and its job is to block a bad change before it ever reaches production. If Stage 3 catches a regression, nothing ships.</p>+
<p>Engineers or scientists change the system in many ways: the prompt, the underlying data, the tools, or the model. The change looks clean locally, tests pass, but a prompt that improves responses for technology accounts could silently degrade responses for financial services accounts. Withou…+
<p>Our CI evaluation is triggered automatically on every staging deployment using AgentCore On-Demand Evaluation. Engineers do not need to remember to run an evaluation. No deployment can silently skip the gate. The run completes in minutes because it evaluates only the sentinel set, not the f…+
<p>The sentinel set is a curated selection of the highest-risk scenarios across five dimensions. Quality regressions beyond tolerance block the deployment. A safety boundary violation blocks just as hard. A change that doubles p90 latency does not proceed without a deliberate decision. Token c…+
<p><strong>One example of what this catches.</strong> After a data normalization update, a disambiguation tool started returning unresolved entities for a subset of account name formats. The agent, receiving an unresolved entity, responded with a best-guess match, returning real da…+
<h3 id="stage-4-the-gate-cd.-ship-with-quantitative-confidence">Stage 4: The Gate (CD). Ship with quantitative confidence</h3>+
<p>The sentinel covers 15 cases. Production covers tens of thousands of query patterns across hundreds of account types, geographies, and seller segments. Clearing the sentinel is necessary. It is not sufficient.</p>+
<p>Before a candidate ships, we run an AgentCore On-Demand Evaluation against the full golden set across the evaluator dimensions. We call this the <em>Quality Gate. </em>It replaces “it seems fine” with a quantitative shipping decision.</p>+
<p><strong>Sandboxed execution and tool mocking.</strong> Think of it as a replay, not a live call. Ahead of time, we freeze a known, versioned dataset of correct tool behavior. Every tool call the candidate makes during the gate is intercepted and answered from that frozen dataset…+
<p><strong>Aggregated end-to-end scoring.</strong> The gate also runs a holistic pass: the full candidate agent answers 200 held-out questions end-to-end, scored by an LLM-as-judge rubric across all evaluator dimensions. Any one of three conditions shuts the gate automatically: the…+
<p><strong>One example of what this catches.</strong> This is where mocking tests the agent’s own reasoning, not just the tools underneath it. Our golden set includes a hierarchy mismatch case: the Customer Insights Agent and Data Insights Agent return different, individually corre…+
<h3 id="stage-5-the-canary.-catch-silent-post-deploy-failures">Stage 5: The Canary. Catch silent post-deploy failures</h3>+
<p>Stage 5 is a post-production monitor. It runs continuously in the production environment against live traffic and synthetic evaluation cases, and its job is to catch issues that no code change triggered and that Stage 3 therefore didn’t see. Stage 3 guards against what engineers change. Sta…+
<p>The product operates against a data estate fed by thousands of upstream schemas and data pipelines. On a given day, dozens of those pipelines refresh without triggering a deployment or CI gate. The agent code is unchanged, the model is unchanged, but the data it reasons over has changed, an…+
<p>We run AgentCore Batch Evaluation on two complementary schedules, and the choice of cadence was deliberate.</p>+
<p>We settled on 30-minute cycles after learning from both directions: 15 minutes generated too many false alarms; 60 minutes let two hours pass before a real degradation registered. We call each 30-minute cycle an evaluation window. Thirty minutes catches a real trend within 90 minutes (three…+
<p>Thresholds are deliberately lower than CI/CD gates. A single window failure is an ops signal, not an incident. We only page oncall after three consecutive windows breach threshold. We learned this rule the hard way: the first month we used a single-window threshold, oncall stopped trusting …+
<p>Every run publishes to CloudWatch (accuracy, hallucination rates, grounding rate, latency, and cost), with alarm logic that distinguishes between the two cadences. The canary runs at three levels: tool-level scores surface a degraded retrieval tool before it corrupts agent-level responses.&…+
<p><strong>Another example of what this catches.</strong> After an upstream index refresh, a retrieval tool began returning loosely relevant chunks instead of the precise context needed to answer the query. Latency was nominal. Error rate was zero. The agent kept producing fluent, …+
<h3 id="stage-6-the-adversary.-protect-against-misuse">Stage 6: The Adversary. Protect against misuse</h3>+
<p>Not every bad output comes from a degraded model. Queries crafted to extract unauthorized account data, and prompt injections embedded in retrieved documents, are predictable patterns in a system serving thousands of users on sensitive business data.</p>+
<p>Safety isn’t a stage. It runs through all nine stages. We apply Amazon Bedrock Guardrails at two points in every interaction. At input, we validate query scope and screen for prompt injection. At output, we verify that no sensitive data is surfaced to unauthorized users and that the agent h…+
<p>We configure two guardrail layers, each covering a distinct threat class:</p>+
<p><strong>Content filters</strong> detect and block harmful or inappropriate content in both inputs and outputs. We enforce filters across categories including <code>VIOLENCE</code>, <code>HATE</code>, <code>SEXUAL</code>, <code>MISCONDUCT…+
<p><strong>Prompt security filters</strong> protect the agent from adversarial manipulation under a single <code>PROMPT_ATTACK</code> guardrail category, covering three sub-types: <code>JAILBREAK</code> (attempts to override the agent’s system instructions o…+
<p>Safety rules are defined at Stage 1, checked for regressions at Stages 3 and 4, and enforced continuously in production.</p>+
<h3 id="stage-7-the-drift.-sustain-quality-and-validate-with-live-traffic">Stage 7: The Drift. Sustain quality and validate with live traffic</h3>+
<p>Quality degrades from multiple directions: upstream data changes without triggering a deployment, models grow stale, and users push boundaries with queries no golden set fully anticipates. Stage 7 detects it before sellers do, and validates candidates against real user traffic before full r…+
<p><strong>Drift detection.</strong> Using AgentCore Online Evaluation, we sample 10 percent of live traffic continuously through the different evaluators defined at Stage 1, so the drift is immediately visible as a deviation from the baselines established offline. Scores feed into…+
<p>One production example. Our golden set for customer identification was built from synthetic edge cases: name variations, subsidiary lookups, abbreviations. It performed well offline. But in live traffic, sellers queried with stock ticker symbols (“what’s going on with AMZN?”), shorthand nam…+
<p>Live traffic feeds two continuous improvement loops: low-scoring sessions are promoted into the golden set as new edge cases, and clusters of low-scoring traces inform prompt revisions, tool updates, and routing adjustments.</p>+
<p><strong>A/B testing:</strong> After a candidate clears a quality gate, AgentCore Gateway, a capability of Amazon Bedrock AgentCore, routes a slice of real traffic between current and candidate agents. On-demand confirms no regression on known cases. Online confirms it holds up a…+
<h3 id="stage-8-the-pulse.-real-time-quality-visibility">Stage 8: The Pulse. Real-time quality visibility</h3>+
<p>A single call to the agent might touch three sub-agents, five tools, and two retrieval paths. When a quality alarm fires, “something in the agent pipeline” is not a runbook. Stage 8 is where signals from Stages 5, 6, and 7 converge into an observability layer with enough resolution to act.&…+
<p>AgentCore Observability, a capability of Amazon Bedrock AgentCore, is the foundation. For agents hosted on the AgentCore runtime, a capability of Amazon Bedrock AgentCore, every execution is instrumented automatically through OpenTelemetry, emitting structured telemetry for every reasoning …+
<ul>+
<li>A conversational dashboard to query evaluation and observability results.</li>+
<li>An Amazon Quick dashboard for trend analysis, scorecard tracking, and cross-agent comparisons across the full history.</li>+
</ul>+
<p>Role-appropriate views sit on top. Oncall engineers get a trace-first view: one click from a CloudWatch alarm to the AgentCore Observability trace that triggered it. The trace shows every tool called, in what order, with latency per step, token consumption, cost, and the full reasoning cont…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/15/ML-21894-5.png" target="_blank" rel="noopener"><img title="Figure 5. AgentCore execution trace for the Customer Insights Agent. The left panel shows the full span tree across tool calls (illu…+
<p class="wp-caption-text">Figure 5: AgentCore execution trace for the Customer Insights Agent, showing the span tree, trajectory, and span-level metrics. The numbers and values in the figure are illustrative.</p>+
</div>+
<p>Product managers get Amazon Quick trend dashboards: quality scores by sub-agent and feature, release-over-release. Leadership gets a single-page health summary aligned to weekly and cross-business reviews across all five dimensions, a one-minute spot check that answers “how are our agents p…+
<p>Figure 6 shows the top-line view of our weekly evaluation dashboard: one combined pass rate drawn from both custom evaluators and AgentCore built-in evaluators, alongside latency, for the week’s run. The numbers and values in the figure are illustrative.</p>+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/15/ML-21894-6.png" target="_blank" rel="noopener"><img title="Dashboard screenshot showing agent evaluation metrics with 97.3% pass rate across 183 questions. Displays KPIs for latency, groundin…+
<p class="wp-caption-text">Figure 6: Top-line weekly evaluation dashboard with combined pass rate and latency. The numbers and values in the figure are illustrative.</p>+
</div>+
<p>Figure 7 drills into that same run at the custom-evaluator, failure-category level, breaking down exactly which error types (metric swaps, hallucinations, customer ID misses, and others) are driving the gap and how many questions each one affects. The numbers and values in the figure are il…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/15/ML-21894-7.png" target="_blank" rel="noopener"><img title="Error reporting dashboard showing 183 evaluated questions with 97.3% pass rate. Bar chart displays error categories: Metric Swap 8 i…+
<p class="wp-caption-text">Figure 7: Weekly error-reporting dashboard breaking down failures by category. The numbers and values in the figure are illustrative.</p>+
</div>+
<p>Before Stage 8, mean time to detect (MTTD) was days to weeks. After Stage 8, it is under four hours, and the first signal is a CloudWatch alarm with an AgentCore trace attached.</p>+
<h3 id="stage-9-the-feedback-loop.-continuous-improvement">Stage 9: The Feedback Loop. Continuous improvement</h3>+
<p>This stage turns observations into actions. The system compounds quality over time. We use AgentCore optimization, a capability of Amazon Bedrock AgentCore, to close the loop.</p>+
<ul>+
<li><strong>System prompt recommendation:</strong> Analyzes agent traces and generates a data driven optimized system prompt that improves performance on the target evaluator. The solution uses failure patterns and adds specific behavioral instructions.</li>+
<li><strong>Tool description recommendation:</strong> Analyzes agent traces and generates sharpened tool descriptions that reduce tool selection confusion. Particularly when an agent selects the wrong tool for ambiguous requests.</li>+
</ul>+
<p>Let’s look at an example:</p>+
<blockquote>+
<p><strong>AgentCore optimization in action</strong></p>+
<p><strong>What to improve on:</strong>+
<br>+
<strong>Target evaluator:</strong> <strong>hallucination</strong>: model reproducing hardcoded values from few-shot examples verbatim in unrelated contexts.</p>+
<p><strong>Why this happened:</strong> Few-shot examples contain real dollar amounts and customer names (for example, “$4.2M”, “Acme Corp”) that the model memorizes and leaks into unrelated queries.</p>+
<p><strong>How to fix:</strong> Replace hardcoded values in few-shot examples with generic placeholders (for example, <code>$[AMOUNT]</code>, <code>[CUSTOMER_NAME]</code>) so the model learns response structure without memorizing specific data points.</p…+
<p><strong>Impact and result:</strong> Fix applied. Pure hallucination occurrences dropped from 138 to under 52 per 10,000 responses, and overall grounding rate improved from 93 percent to 96 percent.</p>+
</blockquote>+
<p>AgentCore optimization uses recommendations to produce data driven system prompt and tool descriptions changes based on real agent traces and a target evaluator. AgentCore optimization builds on <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html" targ…+
<h2 id="agent-evaluation-key-metrics-at-a-glance">Agent evaluation key metrics at a glance</h2>+
<p>The evaluation metrics are defined at Stage 1, enforced across the nine stages, and monitored continuously in production. This set is not static. As we learn from the agent in production, new failure patterns surface, new dimensions become measurable, and the metrics evolve accordingly. Wha…+
<table border="1px" width="100%" cellpadding="10px">+
<tbody>+
<tr>+
<td><strong>Level</strong></td>+
<td><strong>Metric</strong></td>+
<td><strong>What it measures</strong></td>+
<td><strong>Tracked as</strong></td>+
</tr>+
<tr>+
<td>Quality: Agent</td>+
<td>Overall accuracy pass rate</td>+
<td>% of responses with zero taxonomy failures</td>+
<td>Headline scorecard metric</td>+
</tr>+
<tr>+
<td>Quality: Agent</td>+
<td>Metric swap rate</td>+
<td>Correct customer and period, wrong metric returned</td>+
<td>Occurrences per 100 responses</td>+
</tr>+
<tr>+
<td>Quality: Agent</td>+
<td>Wrong period rate</td>+
<td>Data from the wrong time window</td>+
<td>% citing the correct time window</td>+
</tr>+
<tr>+
<td>Quality: Agent</td>+
<td>Wrong customer rate</td>+
<td>Conflated account names or subsidiaries</td>+
<td>Entity resolution accuracy %</td>+
</tr>+
<tr>+
<td>Quality: Agent</td>+
<td>Grounding rate</td>+
<td>Figures traceable to a retrieved source</td>+
<td>% of reported figures grounded</td>+
</tr>+
<tr>+
<td>Quality: Agent</td>+
<td>Scope adherence rate</td>+
<td>Responses that address the question asked</td>+
<td>% on-topic responses</td>+
</tr>+
<tr>+
<td>Quality: Agent</td>+
<td>Unauthorized calculation rate</td>+
<td>Unsanctioned derived figures</td>+
<td>Occurrences per 100 responses</td>+
</tr>+
<tr>+
<td>Quality: Agent</td>+
<td>Relevance / Faithfulness / Helpfulness</td>+
<td>Response quality across AgentCore built-in dimensions</td>+
<td>AgentCore built-in evaluator score (0–1)</td>+
</tr>+
<tr>+
<td>Quality: Tool</td>+
<td>Customer identification accuracy</td>+
<td>Tool resolves to the exact canonical entity</td>+
<td>% correct resolutions (deterministic)</td>+
</tr>+
<tr>+
<td>Quality: Tool</td>+
<td>Tool selection accuracy</td>+
<td>Agent selects the correct tool for the query</td>+
<td>AgentCore built-in evaluator score (0–1)</td>+
</tr>+
<tr>+
<td>Quality: Tool</td>+
<td>Retrieval precision</td>+
<td>Retrieved context contains the information needed</td>+
<td>% retrievals with relevant context</td>+
</tr>+
<tr>+
<td>Quality: Tool</td>+
<td>SQL generation accuracy</td>+
<td>Data Insights Agent generates correct SQL</td>+
<td>% correct result sets (deterministic)</td>+
</tr>+
<tr>+
<td>Quality: Tool</td>+
<td>Tool pass rate</td>+
<td>% of tool calls returning a valid response</td>+
<td>% success per tool</td>+
</tr>+
<tr>+
<td>Quality: Multi-agent</td>+
<td>End-to-end accuracy pass rate</td>+
<td>Full orchestration path produces a correct response</td>+
<td>% of scenarios passing all evaluators</td>+
</tr>+
<tr>+
<td>Quality: Multi-agent</td>+
<td>Routing accuracy</td>+
<td>Orchestrator routes to the correct sub-agent(s)</td>+
<td>% correct routing (deterministic)</td>+
</tr>+
<tr>+
<td>Quality: Multi-agent</td>+
<td>Synthesis faithfulness</td>+
<td>Sub-agent outputs combined without distortion</td>+
<td>LLM-as-judge score across agent boundaries</td>+
</tr>+
<tr>+
<td>Quality: Multi-agent</td>+
<td>Context propagation accuracy</td>+
<td>Context passed between agents is correct and complete</td>+
<td>% traces with no context loss</td>+
</tr>+
<tr>+
<td>Safety</td>+
<td>Guardrail block rate</td>+
<td>% of requests blocked at input or output</td>+
<td>Per 1,000 requests</td>+
</tr>+
<tr>+
<td>Safety</td>+
<td>Input / Output violation rate</td>+
<td>% of queries or responses flagged as policy-violating</td>+
<td>% of total requests / responses</td>+
</tr>+
<tr>+
<td>Safety</td>+
<td>Guardrail latency</td>+
<td>Latency overhead added by guardrail checks</td>+
<td>p99 (target: under 50ms)</td>+
</tr>+
<tr>+
<td>Performance</td>+
<td>End-to-end latency</td>+
<td>Query received to response delivered</td>+
<td>p50, p90, p99</td>+
</tr>+
<tr>+
<td>Performance</td>+
<td>Per sub-agent / tool latency</td>+
<td>Per sub-agent and per tool execution</td>+
<td>p50, p90, p99 independently</td>+
</tr>+
<tr>+
<td>Performance</td>+
<td>Latency regression rate</td>+
<td>CI runs that breach the Stage 1 latency threshold</td>+
<td>% of CI runs</td>+
</tr>+
<tr>+
<td>Cost</td>+
<td>Cost per query</td>+
<td>Total token cost per end-to-end user request</td>+
<td>Input + output tokens × price</td>+
</tr>+
<tr>+
<td>Cost</td>+
<td>Cost per successful response</td>+
<td>Cost per query divided by pass rate</td>+
<td>Penalizes costly failures</td>+
</tr>+
<tr>+
<td>Cost</td>+
<td>Avg tool calls per query</td>+
<td>Reasoning efficiency. Rising calls signal degradation</td>+
<td>Average per query</td>+
</tr>+
<tr>+
<td>Ops control</td>+
<td>MTTD / MTTR</td>+
<td>Time to detect and remediate, per failure class</td>+
<td>Hours (MTTD target: under 4h)</td>+
</tr>+
<tr>+
<td>Ops control</td>+
<td>Automation detection rate</td>+
<td>Incidents detected by evaluation compared to user complaints</td>+
<td>% (target: above 95%)</td>+
</tr>+
<tr>+
<td>Ops control</td>+
<td>Alarm false positive rate</td>+
<td>Alarms not corresponding to real degradation</td>+
<td>% of total alarms</td>+
</tr>+
</tbody>+
</table>+
<h2 id="conclusion">Conclusion</h2>+
<p>The impact of the nine-stage agent evaluation framework was concrete and measurable. Quality regressions that previously took days or weeks to surface, and usually arrived through a seller complaint, are now caught in under four hours by automated evaluation. Deployment decisions that once …+
<p>The broader lesson is one the industry has learned before. DevOps taught us that shipping code without automated testing is reckless. MLOps taught us that deploying models without monitoring is dangerous. AgentOps teaches us the same thing about AI agents: operating them without continuous,…+
<p>The nine-stage framework described here is not specific to sales intelligence agents. Team deploying multi-agent systems that reason, call sub-agents, tools, and synthesize responses will face these same questions across the same lifecycle. Your failure taxonomy will look different. Your go…+
<p>If you are building production agents and want to bring this kind of evaluation discipline to your system, Amazon Bedrock AgentCore Evaluations gives you the platform to do it: on-demand evaluation for experiments, batch evaluation for continuous monitoring, online evaluation for live traff…+
<p><strong>Ready to start?</strong> See <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html" target="_blank" rel="noopener">Evaluate agent performance</a> in the Amazon Bedrock AgentCore documentation. And if you’re building productio…+
<p>To learn more about Amazon Bedrock AgentCore Evaluations, see <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html" target="_blank" rel="noopener">Evaluate agent performance</a> in the AWS documentation. For a technical introduction, read <a…+
<h3 id="acknowledgement">Acknowledgement</h3>+
<p>We extend our sincere gratitude to our executive sponsors and mentors whose vision and guidance made this initiative possible: <a href="https://www.linkedin.com/in/dforthomme/" target="_blank" rel="noopener">Damien Forthomme</a>, Manager III, Applied Science; <a href="https:/…+
<hr style="width: 100%">+
<h2>About the authors</h2>+
<footer>+
<div class="blog-author-box" style="padding-top: 2.0em">+
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/15/ML-21894-8.jpg" alt="Rui Li" width="100" height="133">+
</div>+
<h3 class="lb-h4">Rui Li</h3>+
<p style="overflow: hidden"><a href="https://www.linkedin.com/in/ruili2011/">Rui</a> is an Applied Scientist in AWS Sales, Marketing, and Global Services (SMGS). She leads the science behind making AI agents trustworthy, so AWS sellers, leaders and customers can rely on them with…+
</div>+
<div class="blog-author-box" style="padding-top: 2.0em">+
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/15/ML-21894-9.jpg" alt="Chandana Srinivas" width="100" height="133">+
</div>+
<h3 class="lb-h4">Chandana Srinivas</h3>Diff display stops at 400 lines. The line counts above are from the whole diff. 67 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.