Change
227f63b
227f63b06fc31ec5e830330ef71c949ac682fa63 · commit on GitHub
aws-blog-feed: changed (627401 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +649
- Lines removed
- -1,149
- Stored bytes at this commit
- 627,401
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-10-29T05:48:14.193Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Wed, 28 Oct 2026 23:29:23 GMT |
| date | Thu, 29 Oct 2026 05:48:14 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Tue, 27 Oct 2026 18:03:12 +0000</lastBuildDate>+
<lastBuildDate>Wed, 28 Oct 2026 18:28:26 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,6 +13,653 @@ 1 </sy:updateFrequency> <item>+
<title>Building an Agentic AI Platform on Amazon Bedrock AgentCore</title>+
<link>https://aws.amazon.com/blogs/machine-learning/building-an-agentic-ai-platform-on-amazon-bedrock-agentcore/</link>+
+
<dc:creator><![CDATA[Omar Elkharbotly]]></dc:creator>+
<pubDate>Wed, 28 Oct 2026 18:28:26 +0000</pubDate>+
<category><![CDATA[Amazon Bedrock AgentCore]]></category>+
<category><![CDATA[Intermediate (200)]]></category>+
<category><![CDATA[Technical How-to]]></category>+
<guid isPermaLink="false">8bd827344f7a730b420aced0542c5d0dd666efbf</guid>+
+
<description>Stand up a governed, multi-account Agentic AI Platform on Amazon Bedrock AgentCore. This post walks through the architecture and an open-source reference blueprint: the operating model, control plane, per-Workstream cells, separate inference and tool gateways, and a CI/CD evaluatio…+
<content:encoded><p>If you’ve tried to get an agent into production, the model and tool use are not the blocker. The main bottleneck is everything around them: how many teams can build within a secure environment, which team owns each model and tool, how AgentOps and error handling s…+
<p>Each Workstream team needs a paved road with approved models and tools, short-lived workload identity, isolated runtime and memory, automated evaluation, rollback, telemetry, and cost attribution. These controls need to remain consistent as Workstream teams customize prompts, tools, data ac…+
<p>In this post, we walk through the architecture and reference implementation of an enterprise Agentic AI Platform on <a href="https://aws.amazon.com/bedrock/" target="_blank" rel="noopener">Amazon Bedrock</a> and <a href="https://aws.amazon.com/bedrock/agentcore/" target="_bla…+
<p><a href="https://aws.amazon.com/bedrock/agentcore/" target="_blank" rel="noopener">Amazon Bedrock AgentCore</a> provides modular services for building, deploying, and operating agents at scale, including runtime, gateway, identity, memory, policy, registry, and evaluations. The …+
<h2 id="architectural-principles">Architectural principles</h2> +
<p>Before diving into the solution details, it helps to understand the organizational needs that drive the platform architecture. The common challenges we have heard from customers who establish an agent platform fall into these five categories:</p> +
<ol type="1"> +
<li><strong>Governed model access:</strong> Every untrusted inference request must pass through an approved inference gateway. The gateway applies the organization’s guardrails (for example, <a href="https://aws.amazon.com/bedrock/guardrails/" target="_blank" rel="noopener">…+
<li><strong>Governed tool access:</strong> Agents must discover and invoke only approved tools through an authenticated Model Context Protocol (MCP) gateway, with exact targets, least-privilege execution roles, policy decisions, and an auditable owner.</li> +
<li><strong>Delegated delivery:</strong> Workstream teams (for example, product teams) need to ship and roll back independently inside pre-approved boundaries, without waiting for the platform team to deploy each application or granting developers a direct infrastructure write pat…+
<li><strong>Fleet operations and cost:</strong> Several personas need visibility across accounts without gaining write access to team workloads. Agent operations engineers need logs, metrics, traces, and quotas. Workstream owners need cost per agent against their budgets. FinOps a…+
<li><strong>Bounded evidence and lifecycle:</strong> Every agent release needs proof at each boundary it touches: +
<ol type="a"> +
<li><strong>Model access:</strong> An approved model call succeeds, and a call to an unapproved model or Region is denied.</li> +
<li><strong>Tool access:</strong> A subscribed tool call succeeds, and a call to a tool outside the agent’s subscription is denied with the expected error.</li> +
<li><strong>Identity:</strong> A request from another workstream’s account or identity is rejected.</li> +
<li><strong>Release:</strong> The previous agent runtime version can be restored.</li> +
<li><strong>Retirement:</strong> Teardown removes the agent’s runtime, memory, tool gateway, and tool grants in dependency order, and confirms nothing is left behind.</li> +
</ol> </li> +
</ol> +
<p>The blueprint addresses these needs through four connected planes: developer experience, platform control, Workstream execution, and assurance and operations. The unit of scale is the Workstream cell, not a manually configured agent or one shared workload account. The blueprint <a href="…+
<h2 id="operating-models-and-platform-components">Operating models and platform components</h2> +
<p>Enterprises usually adopt one of three operating models for agentic AI:</p> +
<ul> +
<li><strong>Centralized:</strong> A single artificial intelligence and machine learning (AI/ML) team builds and runs every agent. Governance is consistent, but that team quickly becomes a bottleneck as the number of use cases grows.</li> +
<li><strong>Decentralized:</strong> Each business unit builds and operates its own agents. Teams can move quickly, but without shared standards, controls tend to drift, effort gets duplicated, and cost visibility suffers.</li> +
<li><strong>Federated:</strong> A central Platform team provides shared, governed services while domain teams retain ownership of their agents, tools, and data. This model balances team autonomy with consistent guardrails and is where most enterprises land.</li> +
<li>For assistance choosing, see the <a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/govern-architect-agentic-ai/governance-models.html" target="_blank" rel="noopener">AWS Prescriptive Guidance page on Governance Models</a>. </li> +
</ul> +
<p>Following patterns we have observed in customer engagements, the blueprint implements the federated model. The Platform team is the hub, typically a Platform product manager, architect, platform engineers, agent operations engineers, and a security and governance lead. Workstream teams are …+
<p>In this blueprint, the Platform team owns the definition of each component: templates, versions, policies, and admission. Each Workstream account runs its own instance of the per-team components (Runtime, Memory, Tool Gateway), so a failure or rollback stays inside one team’s boundary.</…+
<h2 id="solution-overview">Solution overview</h2> +
<p>This solution delivers a governed, multi-account Agentic AI Platform that platform teams operate, and workstream teams such as internal product and engineering teams consume through versioned, self-service interfaces.</p> +
<p>At enterprise scale, the developer experience plane is a catalog of organization-approved, versioned agent templates. The blueprint ships reference templates for <a href="https://github.com/strands-agents" target="_blank" rel="noopener">Strands</a>, <a href="https://www.langc…+
<p>Out of the four planes, two carry most of the architecture:</p> +
<ol type="1"> +
<li><strong>Agentic AI Platform control plane:</strong> The Platform account publishes versioned templates and interfaces, operates AWS Agent Registry governance records and shared Inference Gateways, admits models and Guardrails, orchestrates reviewed releases, and aggregates fle…+
<li><strong>Repeatable Workstream cell:</strong> Each product or domain receives an isolated boundary with a team-owned source repository, a Workload pipeline, and three Amazon Bedrock AgentCore capabilities: AgentCore Runtime, actor-scoped AgentCore Memory, and an AgentCore Gatew…+
</ol> +
<h3 id="key-components">Key components</h3> +
<p>The following diagram illustrates the operating model and governed flow of the solution.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/22/ML-21798-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/22/ML-21798-1.png" alt="Operating model linking…+
<p class="wp-caption-text">Figure 1: Enterprise Agentic AI Platform operating model, showing the governance foundation, the Platform control plane, repeatable team-owned Workstream cells, and the software delivery, inference, tool, telemetry, and ownership flows between them.</p>+
</div> +
<p>The architecture consists of the following account layers:</p> +
<ol type="1"> +
<li><strong>Management and governance foundation.</strong> AWS Organizations, AWS IAM Identity Center, preventive policy, audit, retention, and cost policy establish the enterprise boundary. These are logical responsibilities that can map to an existing AWS Control Tower landing z…+
<li><strong>Security, audit, and observability layer.</strong> AWS CloudTrail, AWS Config, AWS Security Hub, Amazon GuardDuty, log archive, and an Amazon CloudWatch OAM sink provide independent control-plane evidence and fleet visibility. Central operators can query shared signals…+
<li><strong>Platform layer of Agentic AI Platform</strong>. Separate Platform environments own golden paths, the governance catalog, Guardrails, shared Inference Gateways, exact tool aliases, pipelines, compatibility, quotas, and common operational interfaces. Platform changes are…+
<li><strong>Workstream fleet layer.</strong> Each product, domain, or regulated boundary receives one or more nonproduction and production cells. Teams own Runtime, Memory, Tool Gateway, tools, data, prompts, evaluation cases, application incidents, and service-level objectives (S…+
</ol> +
<h2 id="solution-walkthrough">Solution walkthrough</h2> +
<p>The following sections trace the key design decisions: account topology, preventive and detective controls, the AgentCore execution path, and the delivery pipeline.</p> +
<h3 id="account-topology-and-organizational-units">Account topology and organizational units</h3> +
<p>The blueprint defines logical account roles that separate enterprise governance, shared Platform services, and team-owned execution:</p> +
<table border="1px" width="100%" cellpadding="10px"> +
<tbody> +
<tr> +
<td><strong>Role</strong></td> +
<td><strong>Landing-zone placement</strong></td> +
<td><strong>Purpose</strong></td> +
</tr> +
<tr> +
<td>Management foundation</td> +
<td>Root / Security and Management</td> +
<td>Organizations, Identity Center, preventive policy, audit, retention, and cost policy</td> +
</tr> +
<tr> +
<td>Security and log archive</td> +
<td>Security</td> +
<td>CloudTrail, AWS Config, Security Hub, GuardDuty, and retained audit data</td> +
</tr> +
<tr> +
<td>Observability</td> +
<td>Security or Operations</td> +
<td>CloudWatch OAM sink and fleet logs, metrics, traces, and cost views</td> +
</tr> +
<tr> +
<td>Agentic AI Platform</td> +
<td>Agent Platform</td> +
<td>Golden paths, AWS Agent Registry, Guardrails, Inference Gateways, pipelines, and fleet controls</td> +
</tr> +
<tr> +
<td>Workstream cell</td> +
<td>Agent Workstreams</td> +
<td>Team-owned Runtime, Memory, Tool Gateway, tools, data, evaluation, and application SLOs</td> +
</tr> +
<tr> +
<td>Additional Workstream cells</td> +
<td>Agent Workstreams</td> +
<td>Repeat the same baseline per product, domain, environment, or regulated boundary</td> +
</tr> +
</tbody> +
</table> +
<p>This topology integrates into an existing landing zone, a pre-configured, secure, and scalable multi-account AWS environment, where the Management, Platform, and Workstream labels describe responsibilities rather than a fixed three-account estate. An organization can operate multiple valida…+
<h3 id="service-control-policies">Service control policies</h3> +
<p>Service control policies constrain the maximum permissions available to Platform and Workstream accounts. They complement, rather than replace, IAM, gateway authorization, Guardrails, resource policies, network controls, and pipeline evidence. The blueprint renders and reviews the effective…+
<ol type="1"> +
<li><strong>Model and Region admission:</strong> Organization policy and Gateway execution roles restrict inference to approved models and the supported Region. Adding a model or Region requires service availability, residency, IAM, quality, cost, rollback, observability, and deni…+
<li><strong>Guardrail and inference enforcement:</strong> Agents do not call model providers directly. Inference crosses the governed gateway, where the stage Guardrail and model allow-list are applied.</li> +
<li><strong>Pipeline-only Workstream mutation:</strong> Workstream teams open pull requests and use the Workload pipeline. Direct developer deployment into Workstream accounts is outside the supported operating model, which keeps the reviewed release and evidence path non-bypassab…+
<li><strong>Registry and Gateway governance:</strong> Workstream Gateway synthesis fails when a subscribed Registry record is missing, changed, or not approved.</li> +
<li><strong>Scoped deployment roles:</strong> Bootstrap policies are generated per account and Region, validated with AWS Identity and Access Management Access (IAM) Analyzer, and checked through effective-policy simulation. Administrator access is reserved for initial setup rathe…+
<li><strong>Separation of duties and adversarial proof:</strong> Platform controls model, Guardrail, Registry, and shared-service admission. Workstream teams control application behavior. Every denied path is paired with an authorized positive call and a mutation that proves the i…+
</ol> +
<p>An administrator inside a Workstream account can’t create a supported fast path around these controls. New or changed organization policies need a sandbox or representative-cell soak, exact allow and deny tests, and a rollback plan before broad attachment.</p> +
<h3 id="cicd-pipeline-with-evaluation-gates">CI/CD pipeline with evaluation gates</h3> +
<p>The blueprint uses reviewed Platform and Workload pipelines to enforce the following promotion sequence:</p> +
<ol type="1"> +
<li><strong>Source and review:</strong> A pull request establishes the immutable source revision. The pipeline pins the intended branch so an unmerged change can’t silently build the repository default branch.</li> +
<li><strong>Build and synth:</strong> The pipeline compiles code, runs tests and linting, creates the agent manifest, scans the digest-bound image, validates policies, resolves approved Registry descriptors, and inspects synthesized AWS CloudFormation and cdk-nag output.</li>…+
<li><strong>Stable roles and permission handoff:</strong> The Workload pipeline creates stable Workstream roles and pauses. The Platform pipeline grants exact tool aliases to the exact Gateway role, live policies are verified, and only then does the Workload pipeline continue.<…+
<li><strong>Deploy to nonproduction:</strong> The pipeline deploys Runtime, Memory, Tool Gateway, observability, alarms, budgets, tags, and application resources into the cell’s nonproduction boundary.</li> +
<li><strong>Evaluate the deployed agent:</strong> The gate invokes the real Runtime and measures regression, quality, tool success, refusal behavior, first-token latency, and cost. Every authorized call is paired with an adversarial twin: the gate calls an approved model and confi…+
<li><strong>Human approval:</strong> A reviewer examines the deployed evidence, known limitations, and residual risk. A generic green build is not enough when a required probe, resource, or expected denial is missing.</li> +
<li><strong>Promote, observe, and roll back:</strong> Production promotion uses the reviewed revision and continuity checks. Runtime sessions are sampled through update, cancellation, rollback, restore, and re-run so the pipeline proves both the forward path and recovery path.<…+
</ol> +
<p>No Workstream deployment step has a manual fast track. The Platform team owns the release contract, but Workstream teams operate their own pipeline instances and deployment cadence. Retirement follows a strict dependency order: Platform alias grants are revoked first, then Workstream Gatewa…+
<h2 id="security-and-governance">Security and governance</h2> +
<p>The blueprint implements defense in depth through reinforcing identity, policy, delivery, data, and evidence controls:</p> +
<ol type="1"> +
<li><strong>Identity and trust boundaries:</strong> AgentCore Identity, a capability of Amazon Bedrock AgentCore, and Amazon Cognito provide short-lived inference credentials. AWS_IAM and SigV4 authenticate tool traffic. Exact Runtime and Gateway roles, issuer and audience checks,…+
<li><strong>Model safety and admission:</strong> Inference flows through the governed gateway described in the Solution overview, applying Guardrails and model allow-lists before every call.</li> +
<li><strong>Tool authorization:</strong> The Tool Gateway invokes only Registry-approved aliases, as detailed in the Solution overview. Cedar policies provide authorization, with a retained AWS Lambda wrapper for defense in depth.</li> +
<li><strong>Tenant data protection:</strong> Workstream accounts isolate application state. AgentCore memory uses actor-scoped namespaces and customer-managed KMS keys. Tools and application data remain team-owned, and retention, deletion, and access policies are explicit rather t…+
<li><strong>Software supply chain and release:</strong> Reviewed source, manifest hashing, reproducible builds, digest-bound image scanning, policy validation, cdk-nag checks, deployed-runtime evaluation, human approval, rollback, and dependency-ordered teardown form one release c…+
<li><strong>Audit, observability, and cost governance:</strong> CloudTrail and retained control-plane data corroborate pipeline evidence. OAM shares logs, metrics, and traces with central operators. Standard allocation tags, budgets, quotas, and AWS Cost and Usage Report reconcili…+
</ol> +
<p>The repository is a reference implementation. Developers can adjust it to address their needs. The deployment and available features are based on the eu-west-1 (Ireland) Region. A substituted component, another Region, or a wider organizational rollout requires its own availability, securit…+
<h2 id="conclusion">Conclusion</h2> +
<p>In this post, we walked through an enterprise Agentic AI Platform built on Amazon Bedrock AgentCore. The architecture separates a centrally governed Platform control plane from repeatable, team-owned Workstream cells, with distinct inference and tool gateways, and production promotion gated…+
<p>The companion repository includes a blueprint that validates these controls end to end, from pipeline gates and adversarial boundary tests through zero-residual teardown. The support claim stops at that tested boundary.</p> +
<p>To get started, deploy one representative Workstream cell from the companion repository, prove its full lifecycle, and then automate additional cells from the same versioned baseline.</p> +
<p>Related resources:</p> +
<ol type="1"> +
<li><a href="https://github.com/aws-samples/sample-ai-agent-factory/tree/main/enterprise-agentic-ai-platform-blueprint" target="_blank" rel="noopener">Enterprise Agentic AI Platform Blueprint on GitHub</a></li> +
<li><a href="https://docs.aws.amazon.com/prescriptive-guidance/latest/agent-ops-maturity-roadmap-for-enterprise-ai/introduction.html" target="_blank" rel="noopener">Enterprise multi-account architecture for running agentic AI in production on AWS (Prescriptive Guidance)</a></li…+
<li><a href="https://catalog.us-east-1.prod.workshops.aws/workshops/3f49be39-c62b-40a2-975b-be9bf626526a/en-US" target="_blank" rel="noopener">Building an Enterprise Agentic AI Platform on Amazon Bedrock AgentCore Workshop</a></li> +
<li><a href="https://aws.amazon.com/bedrock/agentcore/" target="_blank" rel="noopener">Amazon Bedrock AgentCore documentation</a></li> +
</ol> +
<p>&nbsp;</p> +
<p style="clear: both"></p> +
<hr style="width: 100%"> +
<h2>About the authors</h2> +
<footer> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/07/10/ML-21281-3-1.jpg" alt="Omar Elkharbotly" width="100" height="133">+
</div> +
<h3 class="lb-h4">Omar Elkharbotly</h3> +
<p style="overflow: hidden">Omar is a Senior GenAI/ML Specialist Solutions Architect at AWS. He works with enterprise customers across the UK and Ireland to design and deploy generative AI and machine learning solutions using Amazon Bedrock and Amazon Bedrock AgentCore to build scalable, pro…+
</div> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/22/ML-21798-2.jpg" alt="Mia Chang" width="100" height="133">+
</div> +
<h3 class="lb-h4">Mia Chang</h3> +
<p style="overflow: hidden"><a href="https://de.linkedin.com/in/mia-chang" target="_blank" rel="noopener">Mia</a> is a Specialist Solutions Architect for Agentic AI, based in Berlin, Germany. She helps enterprise customers across EMEA design, operationalize, and scale agentic AI …+
</div> +
<div class="blog-author-box" style="padding-top: 2.0em"> +
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/22/ML-21798-3.jpg" alt="Siva Shanmugam" width="100" height="133">+
</div> +
<h3 class="lb-h4">Siva Shanmugam</h3> +
<p style="overflow: hidden">Siva is a Senior Solutions Architect at Amazon Web Services, specializing in AI/ML, Data, and cloud-native architecture. His expertise spans Generative and Agentic AI applications, Developer Experience Transformation — including Agentic Coding and AI-powered engin…+
</div> +
</footer></content:encoded>+
+
+
+
</item>+
<item>+
<title>Distributed training with Ray Train DLCs and EFA on EKS</title>+
<link>https://aws.amazon.com/blogs/machine-learning/distributed-training-with-ray-train-dlcs-and-efa-on-eks/</link>+
+
<dc:creator><![CDATA[Sunita Koppar]]></dc:creator>+
<pubDate>Wed, 28 Oct 2026 18:19:54 +0000</pubDate>+
<category><![CDATA[Amazon Elastic Kubernetes Service]]></category>+
<category><![CDATA[Expert (400)]]></category>+
<category><![CDATA[Technical How-to]]></category>+
<guid isPermaLink="false">ad7c54a5a92c3c3ab550b696c83e5204108e2c13</guid>+
+
<description>The Ray Train Deep Learning Container ships the full CUDA-through-EFA stack, so you build no image. This post walks through a reproducible sample that fine-tunes Qwen2.5-1.5B with PyTorch FSDP across eight GPUs on two EFA-enabled g6.12xlarge instances on Amazon EKS, using the conta…+
<content:encoded><p>What usually stops <a href="https://pytorch.org/" target="_blank" rel="noopener">PyTorch</a> training from scaling out to a cluster is the container, not the model code. Every container needs the same <a href="https://docs.nvidia.com/cuda/cuda-progr…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/19/ML-22002-1-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/19/ML-22002-1-1.png" alt="EKS cluster with a …+
<p class="wp-caption-text">Figure 1: One EKS cluster with a control plane managed by AWS, a CPU system node (m7i.xlarge) running the KubeRay operator and Ray head, and two EFA-enabled g6.12xlarge nodes in one Availability Zone. KubeRay builds it from one RayCluster manifest, Ray Train places …+
</div> +
<p>Ray runs a job across machines as a single cluster with two roles. The Ray head coordinates and runs the global control store that tracks cluster state, the dashboard and job server you submit jobs to. Ray worker nodes supply the GPUs and run the training processes. Each worker joins the cl…+
<p>A single tagged image runs as both the Ray head and the Ray workers, whether KubeRay starts it on <a href="https://aws.amazon.com/eks/" target="_blank" rel="noopener">Amazon Elastic Kubernetes Service (Amazon EKS)</a> or you start it yourself on Amazon Elastic Compute Cloud (Ama…+
<p>This post walks through a sample that uses the DLC exactly as published to fine-tune Qwen2.5-1.5B with PyTorch Fully Sharded Data Parallel (FSDP) across eight NVIDIA L4 GPUs on two g6.12xlarge instances. The training code is one function, the cluster is one RayCluster manifest plus a file o…+
<h2 id="what-you-get-without-building-anything">What you get without building anything</h2> +
<p>The Ray Train DLC is the PyTorch training image with Ray’s training libraries on top. The tag this sample pulls, <code>public.ecr.aws/deep-learning-containers/ray:train-ml-cuda-v1.1</code>, replaces a build that would otherwise look like this:</p> +
<table border="1px" width="100%" cellpadding="10px"> +
<tbody> +
<tr> +
<td><strong>Layer</strong></td> +
<td><strong>What you would otherwise build and keep aligned</strong></td> +
<td><strong>In the Ray Train DLC</strong></td> +
</tr> +
<tr> +
<td>Distributed runtime</td> +
<td>Ray with the dashboard and job server, at a version KubeRay supports</td> +
<td>Ray 2.58.0 with Train, Tune and Data</td> +
</tr> +
<tr> +
<td>Framework</td> +
<td>PyTorch compiled for your CUDA version</td> +
<td>PyTorch 2.13.0, torchvision 0.28.0</td> +
</tr> +
<tr> +
<td>GPU runtime</td> +
<td>CUDA, cuDNN and NCCL releases that agree with each other</td> +
<td>CUDA 13.0.2, cuDNN, NCCL 2.29.7</td> +
</tr> +
<tr> +
<td>Networking</td> +
<td>EFA installer, libfabric and <code>aws-ofi-nccl</code>, built against that NCCL</td> +
<td>EFA latest, AWS OFI NCCL plugin, GDRCopy</td> +
</tr> +
<tr> +
<td>Training libraries</td> +
<td>DeepSpeed, flash-attn and Transformer Engine compiled for your CUDA</td> +
<td>DeepSpeed, flash-attn, Transformer Engine, PyTorch Lightning, Hugging Face libraries</td> +
</tr> +
<tr> +
<td>Diagnostics</td> +
<td>NCCL tests built from source</td> +
<td><code>all_reduce_perf</code> already on the path</td> +
</tr> +
</tbody> +
</table> +
<p>The networking row is the one most teams would least like to own. A mismatch there doesn’t produce an error. NCCL drops to TCP sockets, and the job runs to completion at a fraction of the speed. In the DLC, those libraries are built and tested together, and AWS gates each release on multi-n…+
<p>A few more properties remove work that you would otherwise do per cluster:</p> +
<ul> +
<li><strong>No registry setup –</strong>&nbsp;The image is public, so nodes pull it without credentials and nothing is built or pushed on the way.</li> +
<li><strong>No driver matching –</strong>&nbsp;On startup the entrypoint checks the host’s NVIDIA driver and, if it is older than the image’s CUDA, turns on the bundled CUDA forward-compatibility libraries automatically.</li> +
<li><strong>No head-versus-worker images –</strong>&nbsp;The image has no fixed role. Whatever starts the container supplies the <code>ray start</code> command, so one tag serves every pod.</li> +
<li><strong>No separate inference image family –</strong>&nbsp;Ray Train shares the <code>ray</code> ECR repository with the <a href="https://aws.github.io/deep-learning-containers/ray/" target="_blank" rel="noopener">Ray Serve DLC</a> and <a href="h…+
</ul> +
<h2 id="configure-it-dont-rebuild-it">Configure it, don’t rebuild it</h2> +
<p>Everything that differs between one training job and the next is set around the image. The sample uses each of these mechanisms:</p> +
<table border="1px" width="100%" cellpadding="10px"> +
<tbody> +
<tr> +
<td><strong>What you control</strong></td> +
<td><strong>How you set it</strong></td> +
<td><strong>In this sample</strong></td> +
</tr> +
<tr> +
<td>Head or worker role</td> +
<td>KubeRay <code>rayStartParams</code> per pod group</td> +
<td>Same <code>DLC_IMAGE</code> for both. Head runs with <code>num-gpus: "0"</code></td> +
</tr> +
<tr> +
<td>GPUs and EFA per pod</td> +
<td>Kubernetes resource requests</td> +
<td><code>nvidia.com/gpu: 4</code>, <code>vpc.amazonaws.com/efa: 1</code></td> +
</tr> +
<tr> +
<td>Network transport</td> +
<td>Environment variables on the pod or the job</td> +
<td><code>FI_PROVIDER</code>, <code>NCCL_NET_PLUGIN</code>, <code>NCCL_SOCKET_IFNAME</code>, <code>NCCL_DEBUG</code></td> +
</tr> +
<tr> +
<td>Extra Python packages</td> +
<td>Ray <code>runtime_env</code> at submit time</td> +
<td>Any package a job needs, installed at submit time</td> +
</tr> +
<tr> +
<td>Image version</td> +
<td>Versioned or digest-pinned tag in <code>DLC_IMAGE</code></td> +
<td><code>train-ml-cuda-v1.1</code></td> +
</tr> +
<tr> +
<td>Cluster shape</td> +
<td>Variables in scripts/<code>env.sh</code></td> +
<td>Instance type, node count, GPUs per node, Region, AZ</td> +
</tr> +
<tr> +
<td>Training job</td> +
<td>Arguments to <code>train.py</code></td> +
<td>Model, steps, sequence length, batch size</td> +
</tr> +
</tbody> +
</table> +
<p><strong>Packages per job.</strong> The image ships the HuggingFace training stack, so this sample’s job needs nothing added. When an experiment does need a library the image does not carry, Ray installs it for that job at submit time through <code>runtime_env</code> …+
<p><strong>Transport per job.</strong> The image defaults to <code>FI_PROVIDER=efa</code>, and its NCCL configuration excludes the loopback and Docker bridge interfaces. On EKS you point <code>NCCL_SOCKET_IFNAME</code> at the pod’s <code>eth0</code&…+
<p><strong>Versions you choose.</strong> The sample pins <code>train-ml-cuda-v1.1</code>, the current minor release, so a cluster is reproducible from one run to the next. The <code>vX.Y</code> tags move with security rebuilds. The fully qualified <code&g…+
<h2 id="how-the-cluster-is-laid-out">How the cluster is laid out</h2> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/19/ML-22002-2-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/19/ML-22002-2-1.png" alt="Sample cluster: a C…+
<p class="wp-caption-text">Figure 2: The sample’s cluster. Every pod runs the same Ray Train DLC from Amazon ECR Public. The CPU system node runs the KubeRay operator and a Ray head with no GPUs. Each EFA-enabled g6.12xlarge, both in one Availability Zone, runs a single four-GPU Ray worker po…+
</div> +
<p>All nodes sit in private subnets of one virtual private cloud (VPC) and reach the internet only through a NAT Gateway, which is how every node pulls the DLC from Amazon ECR Public and how workers download model weights from Hugging Face. One m7i.xlarge forms the system node group, hosting k…+
<p>Training runs on two <a href="https://aws.amazon.com/ec2/instance-types/g6/" target="_blank" rel="noopener">g6.12xlarge</a> instances, each with four NVIDIA L4 GPUs, 48 vCPUs, 192 GiB of memory, 40 Gbps of networking and one EFA interface. Each runs one Ray worker pod that claim…+
<h2 id="what-ray-train-does-inside-the-container">What Ray Train does inside the container</h2> +
<p>The DLC provides the stack. <a href="https://docs.ray.io/en/latest/train/train.html" target="_blank" rel="noopener">Ray Train</a> provides the part that would otherwise be launcher code. Multi-node PyTorch normally means starting a process per GPU on every host and giving each o…+
<p>This sample asks Ray Train for <a href="https://docs.pytorch.org/docs/stable/fsdp.html" target="_blank" rel="noopener">FSDP</a>, which splits the model’s weights, gradients and optimizer state across all eight GPUs rather than copying them to each one. Qwen2.5-1.5B in bfloat16 n…+
<h3 id="before-you-start">Prerequisites</h3> +
<p>You must have the following in place before you start:</p> +
<ul> +
<li><strong>Tools:</strong> AWS Command Line Interface (AWS CLI) v2 signed in to the target account, eksctl, kubectl, Helm, and <code>envsubst</code> from GNU gettext.</li> +
<li><strong>Quota:</strong> 96 vCPUs of G-family On-Demand capacity for the two g6.12xlarge instances, in an AWS Region where at least one zone offers them. If you already hold an On-Demand Capacity Reservation for the instances, set <code>CAPACITY_RESERVATION_ID</code>…+
<li><strong>Budget:</strong> about $9.50 an hour while the GPU instances are up, covering both instances, the EKS control plane and the NAT Gateway.</li> +
</ul> +
<p>A quick way to confirm that the CLI is pointed at the account that you expect:</p> +
<div class="hide-language"> +
<pre><code class="language-bash">aws sts get-caller-identity</code></pre> +
</div> +
<h2 id="the-training-code-one-function-and-a-trainer">The training code: One function and a trainer</h2> +
<p><code>code/train.py</code> is the whole job, and it contains nothing specific to the container, the cluster or the network. Ray runs <code>train_func</code> once per GPU. It asks Ray for its rank, loads the model, hands it to <code>prepare_model</code> fo…+
<p><strong><code>code/train.py</code> — the per-GPU function (abridged)</strong></p> +
<div class="hide-language"> +
<pre><code class="language-python">def train_func(config):+
ctx = ray.train.get_context()+
world_size, rank = ctx.get_world_size(), ctx.get_world_rank()+
device = ray.train.torch.get_device()+
+
model = AutoModelForCausalLM.from_pretrained(config["model_id"], torch_dtype=torch.bfloat16)+
+
# Any submodule above 10M parameters becomes its own FSDP unit. Wrapping by+
# size, not by a model-specific layer class, keeps the script model-agnostic.+
auto_wrap_policy = partial(size_based_auto_wrap_policy, min_num_params=int(1e7))+
model = ray.train.torch.prepare_model(+
model,+
parallel_strategy="fsdp",+
parallel_strategy_kwargs={"auto_wrap_policy": auto_wrap_policy},+
)+
optimizer = torch.optim.AdamW(model.parameters(), lr=config["learning_rate"])+
+
for step in range(config["steps"]):+
input_ids = torch.randint(0, model.config.vocab_size,+
(config["batch_size"], config["seq_len"]), device=device)+
optimizer.zero_grad()+
loss = model(input_ids=input_ids, labels=input_ids).loss+
loss.backward() # parameters gathered again, gradients reduce-scattered+
optimizer.step() # local: each rank updates only its own slice+
ray.train.report({"step": step, "loss": loss.item()})</code></pre> +
</div> +
<p>The driver joins the running cluster, counts its GPUs and starts the trainer. Because the worker count comes from the cluster, the same script runs unchanged on one node or many:</p> +
<p><strong><code>code/train.py</code> — the driver (abridged)</strong></p> +
<div class="hide-language"> +
<pre><code class="language-python">ray.init(address="auto") # join the RayCluster+
num_workers = int(ray.cluster_resources().get("GPU", 0)) # 8 here+
+
trainer = TorchTrainer(+
train_func,+
train_loop_config={...},+
scaling_config=ScalingConfig(num_workers=num_workers, use_gpu=True),+
run_config=RunConfig(name="ray-train-fsdp-llm"),+
)+
result = trainer.fit()</code></pre> +
</div> +
<p>The batches are random token IDs so that the run depends on nothing but the cluster. The sample is judged on whether real FSDP collectives crossed both nodes and the job completed. For a real fine-tune, replace the random tensor with a Ray Data or DataLoader pipeline. Nothing else in the fu…+
<h2 id="the-manifest-the-only-place-the-dlc-is-configured">The manifest: The only place the DLC is configured</h2> +
<p><code>manifest/raycluster.yaml</code> is where the image meets the cluster, and it is short because the image needs so little. The head’s only special setting is <code>NVIDIA_VISIBLE_DEVICES=void</code>, which stops the container runtime’s NVIDIA hook from failing on…+
<p><strong><code>manifest/raycluster.yaml</code> — worker group (abridged)</strong></p> +
<div class="hide-language"> +
<pre><code class="language-yaml">workerGroupSpecs:+
- groupName: gpu-workers+
replicas: 2+
rayStartParams:+
num-gpus: "${GPUS_PER_NODE}" # keeps Ray's GPU count equal to the pod's+
template:+
spec:+
nodeSelector: { role: gpu-worker }+
containers:+
- name: ray-worker+
image: "${DLC_IMAGE}" # the DLC as published; same tag as the head+
env:+
- { name: NCCL_DEBUG, value: "INFO" } # print the chosen transport+
- { name: NCCL_SOCKET_IFNAME, value: "eth0" } # pod network on EKS+
- { name: FI_PROVIDER, value: "efa" } # libfabric: EFA, nothing else+
- { name: NCCL_NET_PLUGIN, value: "ofi" } # load the AWS OFI pluginDiff display stops at 400 lines. The line counts above are from the whole diff. 62 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.