Change
d274fb8
d274fb873d98e4e7407823dafcadfa7654e83486 · commit on GitHub
aws-blog-feed: changed (559361 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +681
- Lines removed
- -1,099
- Stored bytes at this commit
- 559,361
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-09-12T04:35:01.553Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Fri, 11 Sep 2026 22:28:36 GMT |
| date | Sat, 12 Sep 2026 04:35:01 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Thu, 10 Sep 2026 21:58:09 +0000</lastBuildDate>+
<lastBuildDate>Fri, 11 Sep 2026 22:28:31 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,6 +13,684 @@ 1 </sy:updateFrequency> <item>+
<title>Monitoring production agent lifecycle with AWS DevOps Agent and AgentCore Evaluations</title>+
<link>https://aws.amazon.com/blogs/machine-learning/monitoring-production-agent-lifecycle-with-aws-devops-agent-and-agentcore-evaluations/</link>+
+
<dc:creator><![CDATA[Meghana Ashok]]></dc:creator>+
<pubDate>Fri, 11 Sep 2026 18:26:38 +0000</pubDate>+
<category><![CDATA[Advanced (300)]]></category>+
<category><![CDATA[Amazon Bedrock AgentCore]]></category>+
<category><![CDATA[Best Practices]]></category>+
<category><![CDATA[DevOps]]></category>+
<guid isPermaLink="false">9da22a53a45cf214e154c67bafd0a4bd7eefa26b</guid>+
+
<description>Multi-agent systems fail in ways traditional monitoring misses. This post presents a dual-layer approach to monitoring production agents: Amazon Bedrock AgentCore Evaluations for continuous quality scoring and AWS DevOps Agent for autonomous infrastructure investigation, shown on a…+
<content:encoded><p>Multi-agent systems in production experience issues in ways that traditional monitoring misses. For example, the agent can’t invoke its foundation model (FM) and returns an empty response. This could be because of a missing AWS Identity and Access Management (IAM)…+
<p>Infrastructure monitoring and agent effectiveness monitoring require different approaches. <a href="https://aws.amazon.com/cloudwatch/" target="_blank" rel="noopener">Amazon CloudWatch</a> metrics show whether systems executed correctly, but not whether agents helped users accom…+
<p>These problems compound in multi-agent systems where a single user request triggers a supervisor agent that routes work to multiple specialists, each with their own tools and model invocations. There’s typically no fixed execution graph to instrument, failures can occur at multiple handoff …+
<p>We built a production airline reservation system with four specialized agents that combine <a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html" target="_blank" rel="noopener">Amazon Bedrock AgentCore Evaluations</a> for continuous agent qualit…+
<h2 id="key-technologies">Key technologies</h2> +
<p>The system uses several AWS services:</p> +
<ul> +
<li><a href="https://docs.aws.amazon.com/bedrock/" target="_blank" rel="noopener">Amazon Bedrock</a> provides API access to foundation models from leading AI companies including Anthropic, Meta, Mistral, and Amazon. In our airline reservation system built on AgentCore runtime, Ama…+
<li><a href="https://aws.amazon.com/bedrock/agentcore/" target="_blank" rel="noopener">AgentCore runtime</a> handles agent orchestration and manages interaction lifecycles, with built-in observability through OpenTelemetry instrumentation.</li> +
<li>With the <a href="https://github.com/awslabs/fullstack-solution-template-for-agentcore" target="_blank" rel="noopener">Fullstack AgentCore Solution Template (FAST)</a>, teams can quickly deploy a secured React frontend connected to an AgentCore backend.</li> +
<li><a href="https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html" target="_blank" rel="noopener">AgentCore Evaluations</a> is a quality assessment framework integrated into AgentCore runtime. It continuously scores agent interactions using <a href="ht…+
<li><a href="https://aws.amazon.com/devops-guru/" target="_blank" rel="noopener">AWS DevOps Agent</a> is an autonomous investigation tool that acts as an on-call engineer for your infrastructure. When incidents occur, it automatically analyzes Amazon CloudWatch logs, traces failur…+
<li><a href="https://strandsagents.com/" target="_blank" rel="noopener">Strands Agents</a> is an open source SDK for building AI agents with a model-driven approach, supporting multi-agent collaboration patterns including Swarm, Graph, and Agents-as-Tools.</li> +
<li><a href="https://opentelemetry.io/" target="_blank" rel="noopener">OpenTelemetry</a> is an open source observability framework that provides standardized instrumentation for collecting traces, metrics, and logs. AgentCore runtime uses OpenTelemetry to emit monitoring data to A…+
<li><a href="https://strandsagents.com/docs/user-guide/concepts/multi-agent/swarm/" target="_blank" rel="noopener">The Swarm Pattern</a> is a multi-agent orchestration approach where a supervisor agent dynamically routes work to specialized agents based on the task at hand. Unlike…+
</ul> +
<h2 id="dual-layer-monitoring-quality-and-infrastructure">Dual-layer monitoring: quality and infrastructure</h2> +
<p>The monitoring architecture answers two questions: Is the agent working well? Is the infrastructure healthy?</p> +
<p><strong>Amazon Bedrock AgentCore Evaluations: Continuous quality monitoring</strong></p> +
<p>AgentCore Evaluations scores live agent interactions on helpfulness, correctness, and goal completion. The system samples a configurable percentage of production requests and evaluates them in the background. Every score comes with reasoning that explains why that score was assigned based o…+
<p>When quality metrics drop, it runs pattern analysis across recent low-scoring sessions to identify common failure modes. If the agent consistently chooses the wrong tool for a specific request type, or provides correct information in an unhelpful format, the pattern analysis surfaces that. …+
<p><strong>AWS DevOps Agent: Autonomous infrastructure investigation</strong></p> +
<p>AWS DevOps Agent monitors system health across metrics, logs, and error patterns. When something breaks, the agent investigates on its own. It pulls relevant Amazon CloudWatch logs, builds a topology graph of affected resources, correlates errors across services (IAM, Amazon Bedrock, your a…+
<p><strong>How the layers work together</strong></p> +
<p>AgentCore Evaluations replaces intuition with quantitative quality metrics, so teams can measure the actual impact of changes. AWS DevOps Agent automates much of the investigation that previously required manual war rooms by autonomously investigating infrastructure incidents the moment the…+
<h2 id="the-airline-reservation-system-why-its-hard-to-monitor">The airline reservation system: why it’s hard to monitor</h2> +
<p>To demonstrate this architecture end to end, we built an airline reservation system that handles complex, dynamic queries: multi-city bookings, loyalty program benefit application, and company travel policy compliance, all within a single conversational turn. This use case demands exactly t…+
<p>Consider a request like this one: <em>“Book me from Seattle to Boston on March 15th, then Boston to Miami on March 18th. Use my companion certificate for the second leg and make sure both flights comply with my company’s travel policy. I’m Gold status so apply any eligible upgrades.”&…+
<p>Handling this in a single conversational turn requires searching two separate routes and retrieving loyalty status and certificates from a different data source while the flight search runs. The bookings must then be sequenced in the right order, because the companion certificate can’t be a…+
<p>We built this with four specialized agents using the <strong>Swarm pattern</strong> (Figure 1). The Supervisor Agent receives requests and acts as an entry point, uses a think tool to plan subtasks, routes work to other agents which can hand off tasks to each other. The Flight A…+
<p>In a Swarm, agents share working memory and hand off work to one another dynamically. Each specialist decides who should act next based on what it finds, rather than following a predetermined execution plan. The supervisor is only the entry point. After a request is in flight, control passe…+
<p>Failures can occur at any handoff point, and because execution paths change based on runtime decisions, the failure path changes each time too. A quality failure and an infrastructure failure can look nearly identical from the outside, but they require completely different responses. From t…+
<div style="width: 516px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-1-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-1-1.png" alt="Diagram of the Swar…+
<p class="wp-caption-text">Figure 1: Swarm multi-agent pattern</p>+
</div> +
<h2 id="dual-monitoring-architecture">Dual monitoring architecture</h2> +
<p>The following diagram shows how these components connect.</p> +
<div style="width: 696px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-2-1.jpg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-2-1.jpg" alt="Full system archite…+
<p class="wp-caption-text">Figure 2: Full system architecture showing the React frontend, AgentCore runtime, Amazon CloudWatch, and AWS DevOps Agent</p>+
</div> +
<p>We built a React frontend hosted on <a href="https://aws.amazon.com/amplify/" target="_blank" rel="noopener">AWS Amplify</a> that connects through Amazon Bedrock AgentCore Identity, a capability of Amazon Bedrock AgentCore, to Amazon Bedrock AgentCore runtime, where the four-age…+
<p>Monitoring data originates from a single source: the Amazon Bedrock AgentCore runtime hosting the four-agent swarm. Amazon Bedrock AgentCore Observability, a capability of Amazon Bedrock AgentCore, instruments the runtime directly, capturing traces and metrics in OpenTelemetry format and fo…+
<p>The second monitoring layer connects to this same backend. When an incident occurs, a team member submits it to AWS DevOps Agent through a signed webhook. The agent pulls Amazon CloudWatch logs and metrics, investigates autonomously, and returns findings and remediation steps without requir…+
<h3 id="getting-started">Getting started</h3> +
<p><strong>Open source</strong>: We built this system on FAST and the complete source code for this system including CDK infrastructure, evaluation dashboard, and AWS DevOps Agent integration is available in the <a href="https://github.com/aws-samples/sample-FAST-applications/tr…+
<p>To use Amazon Bedrock AgentCore Evaluations, you need:</p> +
<ul> +
<li>AgentCore CLI (<a class="uri" href="https://github.com/aws/agentcore-cli" target="_blank" rel="noopener">https://github.com/aws/agentcore-cli</a>)</li> +
<li>AWS credentials with bedrock-agentcore and Amazon CloudWatch permissions.</li> +
<li>The bedrock-agentcore Python SDK (Boto3 client)</li> +
</ul> +
<h3 id="layer-1-amazon-bedrock-agentcore-evaluations-is-the-agent-working-well">Layer 1: Amazon Bedrock AgentCore Evaluations, is the agent working well?</h3> +
<p>Silent quality issues in production agents impact team efficiency, user trust, and business outcomes. AgentCore Evaluations catches these failures the moment they begin by continuously scoring live interactions against structured quality dimensions.</p> +
<div style="width: 640px;" class="wp-video">+
<video class="wp-video-shortcode" id="video-138259-1" width="640" height="360" preload="metadata" controls="controls">+
<source type="video/mp4" src="https://d2908q01vomqb2.cloudfront.net/artifacts/DBSBlogs/ML-20434/blogpost_eval_video.mp4?_=1">+
</video>+
</div> +
<p><em>Video 1: Extracting evaluation metrics for sessions, traces, and spans; viewing metrics on the dashboard to understand agent performance; performing on-demand evaluation by exploring session traces and spans; running the AI engine to identify patterns across low-scoring sessions; …+
<p>Amazon Bedrock AgentCore provides 16 built-in evaluators organized by what they measure and when they run. There are 13 LLM-as-a-Judge evaluators to score interactions with detailed explanations, so teams can verify judgments and understand exactly why an interaction received a particular r…+
<table border="1px" width="100%" cellpadding="10px"> +
<tbody> +
<tr> +
<td><strong>Evaluator</strong></td> +
<td><strong>Definition</strong></td> +
<td><strong>Evaluation Level</strong></td> +
</tr> +
<tr> +
<td>Goal Success Rate</td> +
<td>Assesses whether the AI assistant successfully completed the user goals within a conversation session by analyzing the entire conversation end-to-end.</td> +
<td>Session</td> +
</tr> +
<tr> +
<td>Coherence</td> +
<td>Assesses the logical consistency and cohesion of a response, checking for self-contradictions, logic gaps, and soundness of reasoning without evaluating factual accuracy.</td> +
<td>Trace</td> +
</tr> +
<tr> +
<td>Conciseness</td> +
<td>Measures how efficiently the assistant communicates, assessing whether responses provide necessary information using minimal words without unnecessary elaboration.</td> +
<td>Trace</td> +
</tr> +
<tr> +
<td>Correctness</td> +
<td>Assesses the factual accuracy of a response to a given task, focusing on whether the content and solution are accurate regardless of style or presentation.</td> +
<td>Trace</td> +
</tr> +
<tr> +
<td>Faithfulness</td> +
<td>Assesses whether a response remains consistent with the conversation history, identifying conflicts between the current response and previous interactions within the same conversation.</td> +
<td>Trace</td> +
</tr> +
<tr> +
<td>Harmfulness</td> +
<td>Detects potentially harmful content in a response, including insults, hate speech, violence, inappropriate sexual content, and stereotyping.</td> +
<td>Trace</td> +
</tr> +
<tr> +
<td>Helpfulness</td> +
<td>Assesses how effectively a response helps users progress toward their goals, evaluated purely from the user’s perspective on whether the response moves them closer to their objectives.</td> +
<td>Trace</td> +
</tr> +
<tr> +
<td>Instruction Following</td> +
<td>Assesses whether a response adheres to the explicit instructions provided in the user’s input, focusing on compliance with specific directives regardless of overall response quality.</td> +
<td>Trace</td> +
</tr> +
<tr> +
<td>Refusal</td> +
<td>Detects when the assistant declines to address or fulfill a user’s request, identifying both direct declines and indirect avoidance of the requested task.</td> +
<td>Trace</td> +
</tr> +
<tr> +
<td>Response Relevance</td> +
<td>Assesses how well a response addresses the specific question or request, measuring the focus and relevance of the response to the given input.</td> +
<td>Trace</td> +
</tr> +
<tr> +
<td>Stereotyping</td> +
<td>Detects bias and stereotypical content in a response, identifying prejudicial assumptions or generalizations about specific groups of people.</td> +
<td>Trace</td> +
</tr> +
<tr> +
<td>Tool Parameter Accuracy</td> +
<td>Assesses whether the assistant correctly uses contextual information when making tool calls, verifying that tool parameters are accurately derived from the conversation context.</td> +
<td>Tool</td> +
</tr> +
<tr> +
<td>Tool Selection Accuracy</td> +
<td>Assesses whether the assistant chooses the appropriate tool for a given situation, determining if the selected action is justified and optimal at a specific point in the conversation.</td> +
<td>Tool</td> +
</tr> +
<tr> +
<td>Trajectory Any Order Match</td> +
<td>Validates that expected tools are present regardless of order.</td> +
<td>Session</td> +
</tr> +
<tr> +
<td>Trajectory Exact Order Match</td> +
<td>Validates that actual tools match expected tools in exact order with no extras.</td> +
<td>Session</td> +
</tr> +
<tr> +
<td>Trajectory In Order Match</td> +
<td>Validates that expected tools appear in order within actual trajectory, extras allowed between.</td> +
<td>Session</td> +
</tr> +
</tbody> +
</table> +
<p><em>Table 1: Built-in evaluator metrics supported by AgentCore Evaluations</em></p> +
<h4 id="online-evaluation-continuous-production-monitoring">Online evaluation: continuous production monitoring</h4> +
<p>Online evaluation monitors live agent interactions by continuously sampling a configurable percentage of traces (from 0.01–100 percent) and scoring them asynchronously against your chosen evaluators. This asynchronous, event-driven design means evaluation runs alongside production traffic w…+
<p>Amazon Bedrock AgentCore emits evaluation metrics in real time to Amazon CloudWatch through OpenTelemetry. If you’re already collecting traces for observability, online evaluation adds quality scores alongside your existing operational metrics without requiring code changes or redeployments…+
<p>For our airline reservation system online evaluation, we selected three built-in Amazon Bedrock AgentCore evaluators that provide comprehensive coverage of agent quality: <strong>Helpfulness, Correctness, and Goal Success Rate</strong>. We chose these metrics because they repres…+
<table border="1px" width="100%" cellpadding="10px"> +
<tbody> +
<tr> +
<td><strong>Metric</strong></td> +
<td><strong>Why it matters</strong></td> +
<td><strong>Example scenarios</strong></td> +
</tr> +
<tr> +
<td><strong>Helpfulness</strong></td> +
<td>Captures user satisfaction beyond correctness. Identifies responses that are technically accurate but not useful. Helps optimize for user experience, not only accuracy. Detects when agents provide too much or too little information.</td> +
<td><img src="https://s.w.org/images/core/emoji/14.0.0/72x72/2705.png" alt="✅" class="wp-smiley" style="height: 1em; max-height: 1em;"> High helpfulness: Agent provides a clear, actionable answer with context. <img src="https://s.w.org/images/core/emoji/14.0.0/72x72/274c.png" alt="…+
</tr> +
<tr> +
<td><strong>Correctness</strong></td> +
<td>Facilitates reliability and trustworthiness. Catches hallucinations and factual errors. Critical for domains requiring accuracy (finance, healthcare, legal). Builds user confidence in the agent.</td> +
<td><img src="https://s.w.org/images/core/emoji/14.0.0/72x72/2705.png" alt="✅" class="wp-smiley" style="height: 1em; max-height: 1em;"> High correctness: Agent provides accurate data and valid reasoning. <img src="https://s.w.org/images/core/emoji/14.0.0/72x72/274c.png" alt="❌" cla…+
</tr> +
<tr> +
<td><strong>Goal Success Rate</strong></td> +
<td>Measures actual business value delivered. Captures multi-turn conversation effectiveness. Identifies when agents get stuck or fail to complete tasks. Aligns with user intent and business objectives.</td> +
<td><img src="https://s.w.org/images/core/emoji/14.0.0/72x72/2705.png" alt="✅" class="wp-smiley" style="height: 1em; max-height: 1em;"> High goal success: User wanted to book a flight, and the agent completed the booking. <img src="https://s.w.org/images/core/emoji/14.0.0/72x72/274…+
</tr> +
</tbody> +
</table> +
<p><strong>Workflow to set up the online evaluation configuration:</strong></p> +
<p>Online evaluation runs in production without interruption in the background, automatically sampling sessions at your configured rate and writing results to Amazon CloudWatch Logs without impacting production latency.</p> +
<figure>+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-4.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-4.png" alt="Workflow diagram for se…+
<figcaption aria-hidden="true">+
Workflow diagram for setting up the online evaluation configuration+
</figcaption>+
</figure> +
<p><strong>Why not more metrics?</strong></p> +
<ul> +
<li><strong>Avoid metric overload</strong>: Too many metrics make it hard to identify what matters.</li> +
<li><strong>Reduce evaluation costs</strong>: Each evaluator adds latency and cost per invocation.</li> +
<li><strong>Focus on actionable insights</strong>: These three cover the dimensions that matter most to users.</li> +
<li><strong>Enable quick iteration</strong>: Teams can quickly understand and act on these metrics.</li> +
<li><strong>Additional evaluators</strong> (Faithfulness, Instruction Following, Tool Use Quality) are available but not enabled by default. Teams can add them based on their specific needs.</li> +
</ul> +
<figure>+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-5.jpg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-5.jpg" alt="Evaluation dashboard sh…+
<figcaption aria-hidden="true">+
Evaluation dashboard showing agent quality score summaries+
</figcaption>+
</figure> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-6-1.jpg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-6-1.jpg" alt="Evaluation dashboar…+
<p class="wp-caption-text">Figure 3: Evaluation dashboard displaying average scores across sessions with distribution breakdown by score range (0.0–1.0)</p>+
</div> +
<p>The evaluation dashboard transforms raw Amazon CloudWatch logs and OpenTelemetry traces into an actionable view of how your agent is actually performing (Figure 3). Instead of sifting through thousands of JSON log entries across multiple log groups to piece together what happened in a singl…+
<h4 id="responsible-ai-safeguards">Responsible AI safeguards</h4> +
<p>Evaluators like Correctness and Faithfulness catch hallucinated or inaccurate outputs after the fact, but because online evaluation runs asynchronously on a sample of sessions, a problematic response can still reach the user before it’s scored. For production agent systems, <a href="http…+
<p>For a system like the airline reservation agent, these controls address real-time risks that asynchronous evaluation cannot: a model fabricating flight pricing that sounds plausible but wasn’t returned by any tool, an agent offering legal commitments about refund policies it has no authorit…+
<h4 id="on-demand-evaluation-development-and-cicd-integration">On-demand evaluation: development and CI/CD integration</h4> +
<p>While online evaluation provides continuous monitoring, on-demand evaluation helps you investigate specific sessions: a user complaint, an edge case, or a session flagged by your monitoring. Production metrics operate on a sampling rate (typically 10 percent), so not every session gets scor…+
<p>You can also create custom evaluators with your own scoring rubrics and instructions tailored to your domain. The dashboard surfaces these through the evaluators API, so you can browse what’s available and run combinations against individual sessions or in batch across multiple sessions (Fi…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-7.jpg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-7.jpg" alt="Trace and span view for…+
<p class="wp-caption-text">Figure 4: Viewing trace and span data and performing on-demand evaluation against a session’s trace, spans, and tool calls</p>+
</div> +
<p><strong>Workflow for on-demand evaluation:</strong></p> +
<p>On-demand evaluation follows a synchronous workflow where you request evaluation of a specific session and receive immediate results with scores and explanations.</p> +
<figure>+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-8.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-8.png" alt="Workflow diagram for on…+
<figcaption aria-hidden="true">+
Workflow diagram for on-demand evaluation+
</figcaption>+
</figure> +
<h4 id="the-ai-analysis-engine-from-scores-to-improvements">The AI analysis engine: from scores to improvements</h4> +
<p>After retrieving evaluation metrics, build an analysis layer that detects patterns in low-performing sessions, runs statistical analysis to separate systemic issues from isolated incidents, and generates concrete prompt improvements. This layer should apply:</p> +
<ul> +
<li><strong>Unsupervised pattern detection</strong> to surface recurring failure modes across evaluation dimensions.</li> +
<li><strong>Statistical analysis</strong> (frequency, correlation) to identify which failure patterns are systemic versus isolated.</li> +
<li><strong>LLM-based reasoning</strong> to generate concrete prompt optimization recommendations grounded in production evidence.</li> +
</ul> +
<p>The AI Engine identifies common failure patterns: poor tool selection, missing context, or specific criteria that score low. Configure it to return structured findings with frequency counts, affected session IDs, and concrete evidence from the traces. For example, your engine might identify…+
<p><strong>Prompt improvement layer</strong>: Build on the pattern analysis by implementing a prompt improvement feature that generates revised versions of your original prompt. This feature should directly address the identified patterns with clear explanations of what changed, wh…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-9-1.jpg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-9-1.jpg" alt="AI pattern analysis…+
<p class="wp-caption-text">Figure 5: AI pattern analysis based on the low-scoring sessions</p>+
</div> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-10.jpg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-10.jpg" alt="Prompt improvement re…+
<p class="wp-caption-text">Figure 6: Prompt improvement recommendations and the improved prompt</p>+
</div> +
<h3 id="layer-2-aws-devops-agent-is-the-system-healthy">Layer 2: AWS DevOps Agent, is the system healthy?</h3> +
<p>AgentCore Evaluations monitors agent quality, but infrastructure issues like permissions and tool errors need a different approach. The AWS DevOps Agent acts as an autonomous on-call engineer, investigating infrastructure issues automatically. When anomalies occur, it analyzes system logs, …+
<p>The following demo video (Video 2) showcases how the AWS DevOps Agent can be triggered through a signed webhook for the Travel Agent:</p> +
<div style="width: 640px;" class="wp-video">+
<video class="wp-video-shortcode" id="video-138259-2" width="640" height="360" preload="metadata" controls="controls">+
<source type="video/mp4" src="https://d2908q01vomqb2.cloudfront.net/artifacts/DBSBlogs/ML-20434/devops_blogpost_final.mp4?_=2">+
</video>+
</div> +
<p><em>Video 2: How the AWS DevOps Agent performs an investigation, looking into relevant Amazon CloudWatch logs and AWS service gaps to identify the root cause and provide remediation steps</em></p> +
<p>As shown in the video, after an incident is submitted through a signed webhook, the AWS DevOps Agent first identifies relevant logs from Amazon CloudWatch, then analyzes it to check for common errors such as IAM permission issues, tool failures or other hidden errors, and finally applies la…+
<p>Without the AWS DevOps Agent, you would see that the airline swarm agent suddenly stopped responding to flight booking requests, returning either a generic error message or a completely blank output instead of helping the user.</p> +
<p>At this point, you would:</p> +
<ol type="1"> +
<li>Check application logs for error patterns.</li> +
<li>Review recent deployments for potential causes.</li> +
<li>Examine IAM policies and permissions manually.</li> +
<li>Correlate Amazon CloudWatch metrics across multiple services.</li> +
<li>Trace the execution flow through multiple agent interactions.</li> +
</ol> +
<p>This process could take 30–60 minutes, assuming there is a deep knowledge of the system architecture.</p> +
<p>After we submitted the incident directly to our AWS DevOps Agent Space through a signed webhook connected to the AWS DevOps Agent Space, an investigation was kicked off, resulting in a topology graph of the affected AWS resources as well as a comprehensive analysis of the Amazon CloudWatch …+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-12.jpg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-12.jpg" alt="AWS DevOps Agent UI s…+
<p class="wp-caption-text">Figure 7: The AWS DevOps Agent UI showing the topology graph construction and Amazon CloudWatch log analysis beginning simultaneously</p>+
</div> +
<p>Then, by examining the errors that occurred across the invocation chain, the AWS DevOps Agent identified the root cause.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-13.jpg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-13.jpg" alt="AWS DevOps Agent UI s…+
<p class="wp-caption-text">Figure 8: The AWS DevOps Agent UI showing the identified root cause, a missing bedrock:InvokeModel permission, with the complete failure path traced from the user request through AgentCore runtime to the Amazon Bedrock API denial</p>+
</div> +
<p>It identified a missing <code>bedrock:InvokeModel</code> permission on the execution role. Every agent invocation was calling Amazon Bedrock to run its language model, and every call was being denied at the IAM layer. The Supervisor Agent couldn’t invoke its model to process the…+
<p>Critically, the AWS DevOps Agent did not only identify an IAM error. It traced the complete failure path starting from the user request:</p> +
<p><strong>User request → AgentCore runtime → Amazon Bedrock API call → Access denied → Agent failure</strong></p> +
<p>Lastly, the investigation provided specific remediation steps in the prevention tab.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-14-1.jpg" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-14-1.jpg" alt="AWS DevOps Agent …+
<p class="wp-caption-text">Figure 9: The AWS DevOps Agent UI showing the prevention tab with specific, actionable remediation steps and a high confidence rating</p>+
</div> +
<p>At a high level, recommended actions were to add the required Amazon Bedrock permissions to the particular execution role, and scope permissions to the specific foundation model resource in use. These provide the user with things to think about for future development to help prevent failure…+
<p>The AWS DevOps Agent improves developer efficiency by automating failure diagnosis across the multi-agent system. For example, it maps blank agent outputs to a missing Amazon Bedrock permission at the IAM layer. The AWS DevOps Agent coordinates between Amazon Bedrock AgentCore runtime, Amaz…+
<p>Beyond IAM permission issues, the AWS DevOps Agent is designed to handle the full range of complex failure modes common in multi-agent systems. Such examples include:</p> +
<ul> +
<li><strong>Model throttling under load</strong>: When subagents scale up and hit Amazon Bedrock token-per-minute limits, individual subagents begin failing intermittently. The AWS DevOps Agent correlates throttling metrics with agent invocation timelines to identify which model, …+
<li><strong>Tool call failures masked by retry logic</strong>: If a subagent’s tool integration begins returning errors and the Supervisor Agent silently reroutes, the AWS DevOps Agent surfaces downstream tool errors from logs and connects them to degraded user experience.</li&…+
<li><strong>Memory and context issues</strong>: If an agent loses access to its memory resource mid-session, responses become incoherent rather than failing outright. The AWS DevOps Agent detects anomalies in response quality patterns and correlates them with configuration changes…+
<li><strong>Cross-agent communication breakdowns</strong>: When handoffs between agents fail because of network issues or authentication problems, the AWS DevOps Agent tracks the complete handoff chain and identifies exactly where communication broke down.</li> +
</ul> +
<p><strong>Note:</strong> Before adopting this architecture, keep the following in mind:</p> +
<ul> +
<li><strong>LLM-as-judge reliability</strong>: AgentCore Evaluations uses LLM-based scoring, which lacks ground truth. Treat scores as signals, not absolute measures. Calibrate evaluators with subject matter experts to align automated judgments with human expectations in your doma…+
<li><strong>Service maturity</strong>: AWS DevOps Agent is actively evolving. Currently, webhook credential generation is done through the console, but Agent Space creation and management can be automated with the <a href="https://docs.aws.amazon.com/devopsagent/latest/userguid…+
<li><strong>Latency trade-offs</strong>: Online evaluation adds processing overhead. Lower sampling rates reduce this overhead but may miss edge cases. Tune your sampling rate based on your traffic volume and how comprehensively you need to cover the interaction space.</li> +
<li><strong>Security</strong>: Careful IAM policy configuration is required. The AWS DevOps Agent needs broad read access to logs and metrics, scope permissions appropriately while making sure it has access to everything needed to trace cross-service failures.</li> +
<li><strong>Responsible AI controls</strong>: For production deployments, complement evaluation with <a href="https://aws.amazon.com/bedrock/guardrails/" target="_blank" rel="noopener">Amazon Bedrock Guardrails</a> to enforce content filtering, denied topics, grounding…+
</ul> +
<h2 id="conclusion">Conclusion</h2> +
<p>Deploying agents to production is only the start. This dual-monitoring approach, quality and infrastructure, gives teams the feedback loop they need to improve continuously. AgentCore Evaluations monitors quality. AWS DevOps Agent investigates infrastructure. Together, they turn production …+
<p>Our airline reservation system, built on the Swarm pattern with a Strands agent and deployed to AgentCore, demonstrates that this dual-monitoring approach is practical, scalable, and effective. The result is faster incident resolution, higher policy compliance, and an iterative improvement …+
<p>Whether you’re building your first multi-agent system or scaling an existing deployment, the same principles apply: measure quality continuously, investigate failures autonomously, and let production data drive your next iteration.</p> +
<p><strong>Ready to get started?</strong></p> +
<ul> +
<li>Check out our <a href="https://github.com/aws-samples/sample-FAST-applications/tree/main/samples/dual-monitoring-system" target="_blank" rel="noopener">Open Sourced Code</a> to demo our dual-layer monitoring architecture for multi-agent systems.</li> +
<li>Explore <a href="https://aws.amazon.com/bedrock/agentcore/" target="_blank" rel="noopener">AgentCore</a> for agent deployment and evaluation.</li> +
<li>Learn more about <a href="https://strandsagents.com" target="_blank" rel="noopener">Strands Agents</a> for building multi-agent systems.</li> +
<li>Review the <a href="https://github.com/sierra-research/tau-bench" target="_blank" rel="noopener">τ-Bench benchmark</a> for evaluating agent architectures. tau-Bench includes a tau-airline domain that benchmarks exactly the kind of tool-calling and policy-following behaviors ou…+
</ul> +
<hr style="width: 100%"> +
<h2>About the authors</h2> +
<footer> +
<div class="blog-author-box"> +
<div class="blog-author-image"> +
<p><img loading="lazy" class="alignleft size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-15-1.png" alt="Meghana Ashok" width="100" height="100"></p> +
</div> +
<h3 class="lb-h4">Meghana Ashok</h3> +
<p>Meghana is a Senior Machine Learning Engineer at the AWS Generative AI Innovation Center (GenAIIC), where she operates at the intersection of engineering, research, and GenAI solutions. She specializes in designing and building scalable AI architectures that bridge the gap between cutting…+
</div> +
<div class="blog-author-box"> +
<div class="blog-author-image"> +
<p><img loading="lazy" class="alignleft size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-16-1.png" alt="Suren Gunturu" width="100" height="100"></p> +
</div> +
<h3 class="lb-h4">Suren Gunturu</h3> +
<p>Suren is a Data Scientist II working in the AWS Generative AI Innovation Center, where he works with various AWS customers to solve high-value business problems. He specializes in building AI/ML pipelines using agentic frameworks, primarily through Amazon Bedrock AgentCore and other AI/ML…+
</div> +
<div class="blog-author-box"> +
<div class="blog-author-image"> +
<p><img loading="lazy" class="alignleft size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-17-1.png" alt="Samaneh Aminikhanghahi" width="100" height="100"></p> +
</div> +
<h3 class="lb-h4">Samaneh Aminikhanghahi</h3> +
<p>Samaneh is an Applied Scientist at the AWS Generative AI Innovation Center, where she works with customers across different verticals to accelerate their adoption of generative AI. She specializes in agentic AI frameworks, building robust evaluation systems, and implementing responsible A…+
</div> +
<div class="blog-author-box"> +
<div class="blog-author-image"> +
<p><img loading="lazy" class="alignleft size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/08/31/ML-20434-18-1.png" alt="Vidya Sagar Ravipati" width="100" height="100"></p> +
</div> +
<h3 class="lb-h4">Vidya Sagar Ravipati</h3> +
<p>Vidya Sagar is a Science Manager at the AWS Generative AI Innovation Center, where he uses his vast experience in large-scale distributed systems and his passion for machine learning to help AWS customers across different industry verticals accelerate their AI and cloud adoption.</p>…+
</div> +
</footer></content:encoded>+
+
+
<enclosure length="94894657" type="video/mp4" url="https://d2908q01vomqb2.cloudfront.net/artifacts/DBSBlogs/ML-20434/blogpost_eval_video.mp4"/>+
<enclosure length="53965455" type="video/mp4" url="https://d2908q01vomqb2.cloudfront.net/artifacts/DBSBlogs/ML-20434/devops_blogpost_final.mp4"/>+
+
</item>+
<item>+
<title>Beyond the price per token: Choosing the right OpenAI model on Amazon Bedrock for your workload</title>+
<link>https://aws.amazon.com/blogs/machine-learning/beyond-the-price-per-token-choosing-the-right-openai-model-on-amazon-bedrock-for-your-workload/</link>+
+
<dc:creator><![CDATA[Nick McCarthy]]></dc:creator>+
<pubDate>Fri, 11 Sep 2026 18:24:38 +0000</pubDate>+
<category><![CDATA[Advanced (300)]]></category>+
<category><![CDATA[Amazon Bedrock]]></category>+
<category><![CDATA[Technical How-to]]></category>+
<guid isPermaLink="false">4fe7cafaba3f92f547e3e291a89c2e4e2100f953</guid>+
+
<description>Comparing models on dollars per million tokens misses what production workloads actually pay for: outcomes. This post shares an open-source benchmarking harness that measures cost per correct answer, agent trajectory cost, and rubric-graded deliverable quality across OpenAI models …+
<content:encoded><p>Organizations building generative AI applications usually compare models the same way: dollars per million tokens. It’s the number on every pricing page, so it becomes the number in every spreadsheet. But production workloads don’t buy tokens. They buy outcomes: a…+
<p>In this post, we share results from an open-source benchmarking harness that measures those multipliers across OpenAI models on Amazon Bedrock (gpt-5.6-luna, gpt-5.6-terra, and gpt-5.6-sol) and two widely used cost-efficient models on the OpenAI API (gpt-5.4-mini and gpt-5.4-nano). We chose…+
<ol type="1"> +
<li><strong>What does a correct answer cost?</strong> Not what a token costs.</li> +
<li><strong>What do agent trajectories cost?</strong> Turn count can dominate the bill.</li> +
<li><strong>Can it produce work a professional would accept?</strong> This is measured on real occupational deliverables, not quiz questions.</li> +
</ol> +
<p>The core benchmarks here are reproducible: the harness, <a href="https://github.com/openai-on-aws/benchmarks-openai" target="_blank" rel="noopener">openai-on-aws/benchmarks-openai</a>, runs one identical code path (the OpenAI Responses API) against both applications, and we enco…+
<h2 id="solution-overview">Solution overview</h2> +
<p>The harness evaluates all five models through the same Responses API client, switching the backend and model ID while holding the evaluation logic constant. The results still reflect differences in the models, provider infrastructure, and model-specific configuration. In particular, the mod…+
<p>We measure three things: single-call accuracy and cost on benchmarks that still separate frontier models (AIME competition mathematics, GPQA Diamond graduate-level science, and MMLU-Pro), multi-turn agent trajectories on live web-research tasks, and rubric-graded professional deliverables. …+
<p>How to read the results: sample sizes range from 48–198 items. Treat small gaps as directional unless accompanied by uncertainty estimates, and reproduce the evaluation on your own workload before selecting a model.</p> +
<h2 id="measuring-the-cost-of-a-correct-answer">Measuring the cost of a correct answer</h2> +
<p>For each benchmark we divide a model’s total spend, across right and wrong attempts alike, by its number of correct answers. This estimates the observed cost of a correct answer in our sample. The following graph shows accuracy and cost per correct answer on AIME for all five models. The pa…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/09/ML-21634-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/09/ML-21634-1.png" alt="Chart of AIME accuracy …+
<p class="wp-caption-text">Figure 1: AIME accuracy and cost per correct answer across the five models</p>Diff display stops at 400 lines. The line counts above are from the whole diff. 78 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.