Change
297e09c
297e09cba67b61466c7a01eadfeb499c612b9288 · commit on GitHub
aws-blog-feed: changed (538956 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +845
- Lines removed
- -558
- Stored bytes at this commit
- 538,956
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-10-13T06:47:22.899Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Mon, 12 Oct 2026 19:06:06 GMT |
| date | Tue, 13 Oct 2026 06:47:22 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Fri, 09 Oct 2026 15:38:39 +0000</lastBuildDate>+
<lastBuildDate>Mon, 12 Oct 2026 17:52:22 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,6 +13,849 @@ 1 </sy:updateFrequency> <item>+
<title>Accessing OpenAI GPT models on Amazon Bedrock with global cross-Region inference in Thailand, Malaysia, Singapore, Indonesia, and Taiwan</title>+
<link>https://aws.amazon.com/blogs/machine-learning/accessing-openai-gpt-models-on-amazon-bedrock-with-global-cross-region-inference-in-thailand-malaysia-singapore-indonesia-and-taiwan/</link>+
+
<dc:creator><![CDATA[Traci Lim]]></dc:creator>+
<pubDate>Mon, 12 Oct 2026 17:52:22 +0000</pubDate>+
<category><![CDATA[Advanced (300)]]></category>+
<category><![CDATA[Amazon Bedrock]]></category>+
<category><![CDATA[Technical How-to]]></category>+
<guid isPermaLink="false">e20b8171cff84e878ab96622856dfe3f22a41ff0</guid>+
+
<description>Learn how to run OpenAI GPT-6 Sol, Luna, and Astra on Amazon Bedrock through global cross-Region inference from Singapore, Jakarta, Thailand, Malaysia, and Taipei. This post covers the Converse and OpenAI-compatible APIs, IAM and SCP permissions, quota sizing, and monitoring infere…+
<content:encoded><p>Single-Region inference can throttle when demand exceeds local capacity. Teams in Thailand, Malaysia, Singapore, Indonesia, and Taiwan need access to the newest OpenAI models beyond that capacity. <a href="https://aws.amazon.com/bedrock/" target="_blank" rel="n…+
<p>GPT-6 Sol, GPT-6 Luna, and GPT-6 Astra are all available through global cross-Region inference from the preceding Regions, on the <code>bedrock-runtime</code> endpoint. See <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html" target="_bla…+
<p>In this post, we introduce the global cross-Region inference profiles and invoke them from Asia Pacific (Singapore) using the Converse API, the OpenAI-compatible APIs, and coding clients. We then cover the <a href="https://aws.amazon.com/iam/" target="_blank" rel="noopener">AWS Identi…+
<h2 id="global-cross-region-inference-profiles-for-openai-gpt-6">Global cross-Region inference profiles for OpenAI GPT-6</h2>+
<p>The GPT-6 family offers three models, each available through global cross-Region inference.</p>+
<ul>+
<li>GPT-6 Sol brings strong reasoning and coding capabilities to complex tasks performed throughout the week.</li>+
<li>GPT-6 Luna makes focused, repeatable tasks practical at high volume.</li>+
<li>GPT-6 Astra handles complex reasoning, coding, and long-document work.</li>+
</ul>+
<p>All three accept text and image input, client-side tool use, and prompt caching. They differ on reasoning effort. GPT-6 Sol and GPT-6 Luna accept none, low, medium, high, xhigh, and max. GPT-6 Astra accepts the same levels except none. For GPT-6 Sol and GPT-6 Luna, see <a href="https://a…+
<p>Global cross-Region inference routes requests across supported commercial AWS Regions where the model is deployed, giving you broad access to Amazon Bedrock capacity. You invoke it by passing a global cross-Region inference profile ID in place of a model ID. Your source Region is where you …+
<p>The following table lists the global cross-Region inference profile IDs and the source Regions covered in this post.</p>+
<table border="1px" width="100%" cellpadding="10px">+
<tbody>+
<tr>+
<td><strong>Global cross-Region inference profiles</strong></td>+
<td><strong>Source Regions</strong></td>+
<td><strong>Destination Regions</strong></td>+
</tr>+
<tr>+
<td>+
<p>global.openai.gpt-6-sol</p>+
<p>global.openai.gpt-6-luna</p>+
</td>+
<td>+
<p>Asia Pacific (Singapore) ap-southeast-1</p>+
<p>Asia Pacific (Jakarta) ap-southeast-3</p>+
<p>Asia Pacific (Malaysia) ap-southeast-5</p>+
<p>Asia Pacific (Thailand) ap-southeast-7</p>+
<p>Asia Pacific (Taipei) ap-east-2</p>+
</td>+
<td>Supported commercial AWS Regions where the model is deployed</td>+
</tr>+
<tr>+
<td>global.openai.gpt-6-astra</td>+
<td>Asia Pacific (Singapore) ap-southeast-1</td>+
<td>Supported commercial AWS Regions where the model is deployed</td>+
</tr>+
</tbody>+
</table>+
<p>Inference profile availability and Region support change as AWS adds Regions and profiles, so confirm both on the model card before you build: <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-sol.html" target="_blank" rel="noopener">GPT-6 Sol</a…+
<h2 id="getting-started-with-openai-models-on-amazon-bedrock">Getting started with OpenAI models on Amazon Bedrock</h2>+
<p>Start with the prerequisites, then invoke any of the three profiles through the Converse API, the OpenAI-compatible APIs, or your existing coding client.</p>+
<h3 id="prerequisites">Prerequisites</h3>+
<ol type="1">+
<li>An IAM identity with permission to invoke the profile. The IAM policy in the Security and permissions section grants only what these examples need.</li>+
<li>Your source Region activated. Only the Region you call from needs activating. You do not activate destination Regions, which run on Amazon Bedrock managed capacity. Singapore (<code>ap-southeast-1</code>) is a Region that is enabled by default. Jakarta (<code>ap-southeas…+
<li>A current AWS SDK or CLI. The OpenAI-compatible examples also require the OpenAI SDK and the Amazon Bedrock token generator. Install them with <code>pip install boto3 openai aws-bedrock-token-generator</code>.</li>+
</ol>+
<p>The Amazon Bedrock console lists all three global cross-Region inference profiles under system-defined inference profiles. See Figure 1.</p>+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-21759-1-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-21759-1-1.png" alt="The three global Op…+
<p class="wp-caption-text">Figure 1: The three global OpenAI GPT-6 inference profiles among the system-defined profiles in the Amazon Bedrock console</p>+
</div>+
<h3 id="invoke-gpt-6-models-through-amazon-bedrock-runtime">Invoke GPT-6 models through Amazon Bedrock Runtime</h3>+
<p>Each model supports three access paths on this endpoint: the Amazon Bedrock Converse API, the OpenAI Responses API, and the OpenAI Chat Completions API. The OpenAI-compatible APIs are called on the <code>/openai/v1</code> paths rather than through the AWS SDKs, and the endpoint …+
<p><strong>Converse API.</strong> This example calls GPT-6 Sol from Asia Pacific (Singapore) and logs the response. Set <code>BEDROCK_PROFILE_ID</code> to <code>global.openai.gpt-6-luna</code> or <code>global.openai.gpt-6-astra</code> to switch m…+
<div class="hide-language">+
<pre><code class="language-python">import logging+
import os+
import boto3+
+
# AWS_REGION overrides this default, so set it to your source Region or unset it.+
REGION = os.environ.get("AWS_REGION", "ap-southeast-1")+
PROFILE_ID = os.environ.get("BEDROCK_PROFILE_ID", "global.openai.gpt-6-sol")+
PROMPT = (+
"In three bullet points, explain when a workload should use "+
"cross-Region inference instead of calling a model in a single Region."+
)+
+
logging.basicConfig(format="%(levelname)s %(message)s")+
log = logging.getLogger("bedrock-cris")+
log.setLevel(logging.INFO)+
+
client = boto3.client("bedrock-runtime", region_name=REGION)+
+
# AccessDeniedException points at the IAM policy. ValidationException usually+
# means the model ID is not a global cross-Region inference profile.+
response = client.converse(+
modelId=PROFILE_ID,+
messages=[{"role": "user", "content": [{"text": PROMPT}]}],+
inferenceConfig={"maxTokens": 1024},+
# GPT-6 Sol and Luna accept none through max. GPT-6 Astra rejects none.+
additionalModelRequestFields={"reasoning": {"effort": "low"}},+
)+
+
# Any effort above none can prepend a reasoningContent block, so pick the text+
# block by key rather than by position.+
text = "".join(b["text"] for b in response["output"]["message"]["content"] if "text" in b)+
+
log.info("Called %s from %s", PROFILE_ID, REGION)+
log.info("Stop reason: %s", response["stopReason"])+
log.info("Response:\n%s", text)</code></pre>+
</div>+
<p><strong>OpenAI Chat Completions API.</strong> Point the OpenAI SDK at the Regional endpoint and pass the same profile ID.</p>+
<div class="hide-language">+
<pre><code class="language-python">import logging+
import os+
from aws_bedrock_token_generator import provide_token+
from openai import OpenAI+
+
REGION = os.environ.get("AWS_REGION", "ap-southeast-1")+
PROFILE_ID = os.environ.get("BEDROCK_PROFILE_ID", "global.openai.gpt-6-sol")+
PROMPT = "List three uses for a message queue."+
+
logging.basicConfig(format="%(levelname)s %(message)s")+
log = logging.getLogger("bedrock-cris")+
log.setLevel(logging.INFO)+
+
# provide_token mints a short-term key from your AWS credentials, valid up to 12 hours.+
client = OpenAI(+
base_url=f"https://bedrock-runtime.{REGION}.amazonaws.com/openai/v1",+
api_key=provide_token(region=REGION),+
)+
+
response = client.chat.completions.create(+
model=PROFILE_ID,+
messages=[{"role": "user", "content": PROMPT}],+
max_completion_tokens=256,+
)+
+
log.info("Called %s from %s", PROFILE_ID, REGION)+
log.info("Response:\n%s", response.choices[0].message.content)</code></pre>+
</div>+
<p>When migrating an existing OpenAI client:</p>+
<ul>+
<li>Set the model parameter on every request, including those referencing a previous response.</li>+
<li>Use supported tool types: <code>function</code>, <code>mcp</code>, <code>custom</code>, <code>namespace</code>, and <code>tool_search</code>.</li>+
<li>Configure the inference profile ID directly in your client rather than querying the models endpoint.</li>+
</ul>+
<h3 id="set-up-the-chatgpt-desktop-app-and-codex-cli">Set up the ChatGPT desktop app and Codex CLI</h3>+
<p>The ChatGPT desktop app and the Codex CLI share <code>~/.codex/config.toml</code>, so one change covers both. Of the two built-in Amazon Bedrock providers, only <code>amazon-bedrock-runtime</code> accepts an inference profile.</p>+
<div class="hide-language">+
<pre><code class="language-toml">model = "global.openai.gpt-6-astra"+
model_provider = "amazon-bedrock-runtime"+
[model_providers.amazon-bedrock-runtime.aws]+
region = "ap-southeast-1"</code></pre>+
</div>+
<p>SigV4 signing means there is no key to rotate. Add <code>profile = "&lt;your-profile&gt;"</code> for a non-default profile, and set <code>region</code> explicitly, because the client otherwise falls back to <code>AWS_REGION</code>. Substitute anot…+
<p>Codex reports a 272,000-token window for these profiles, which is its own bundled figure rather than a service limit. Raise it with <code>model_context_window</code>. This configuration needs Codex CLI 0.148 or later.</p>+
<p>Figure 2 shows the result in the app.</p>+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-21759-2-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-21759-2-1.png" alt="The ChatGPT desktop…+
<p class="wp-caption-text">Figure 2: The ChatGPT desktop app running GPT-6 Astra on Amazon Bedrock</p>+
</div>+
<h2 id="security-and-permissions">Security and permissions</h2>+
<p>Amazon Bedrock offers a range of controls over inference. Two of them matter here. AWS Identity and Access Management (IAM) policies control which roles can invoke which profiles, and service control policies (SCPs) apply organization-wide controls, including Region scope. This section cove…+
<h3 id="iam-permissions-for-global-cross-region-inference">IAM permissions for global cross-Region inference</h3>+
<p>To call any of these models through a global cross-Region inference profile, your IAM policy needs four statements:</p>+
<ol type="1">+
<li>Access to the inference profile and the project/default resource in your source Region.</li>+
<li>Access to the foundation model in your source Region, scoped to the inference profile.</li>+
<li>The Region-agnostic Amazon Resource Name (ARN) for the foundation model, <code>arn:aws:bedrock:::foundation-model/openai.gpt-6-sol</code>, with <code>aws:RequestedRegion</code> set to <code>unspecified</code> to permit cross-Region routing.</li>+
<li><code>bedrock:CallWithBearerToken</code> for API-key authentication.</li>+
</ol>+
<p>This IAM policy grants permission to invoke GPT-6 Sol through global cross-Region inference from <code>ap-southeast-1</code>. Replace with your AWS account ID and <code>ap-southeast-1</code> with your source Region.</p>+
<div class="hide-language">+
<pre><code class="language-json">{+
"Version": "2012-10-17",+
"Statement": [+
{+
"Sid": "GrantGlobalCrisProfileAndProjectAccess",+
"Effect": "Allow",+
"Action": ["bedrock:InvokeModel"],+
"Resource": [+
"arn:aws:bedrock:ap-southeast-1:&lt;ACCOUNT&gt;:inference-profile/global.openai.gpt-6-sol",+
"arn:aws:bedrock:ap-southeast-1:&lt;ACCOUNT&gt;:project/default"+
],+
"Condition": {+
"StringEquals": { "aws:RequestedRegion": "ap-southeast-1" }+
}+
},+
{+
"Sid": "GrantGlobalCrisInRegionModelAccess",+
"Effect": "Allow",+
"Action": ["bedrock:InvokeModel"],+
"Resource": ["arn:aws:bedrock:ap-southeast-1::foundation-model/openai.gpt-6-sol"],+
"Condition": {+
"StringEquals": {+
"aws:RequestedRegion": "ap-southeast-1",+
"bedrock:InferenceProfileArn": "arn:aws:bedrock:ap-southeast-1:&lt;ACCOUNT&gt;:inference-profile/global.openai.gpt-6-sol"+
}+
}+
},+
{+
"Sid": "GrantGlobalCrisGlobalModelAccess",+
"Effect": "Allow",+
"Action": ["bedrock:InvokeModel"],+
"Resource": ["arn:aws:bedrock:::foundation-model/openai.gpt-6-sol"],+
"Condition": {+
"StringEquals": {+
"aws:RequestedRegion": "unspecified",+
"bedrock:InferenceProfileArn": "arn:aws:bedrock:ap-southeast-1:&lt;ACCOUNT&gt;:inference-profile/global.openai.gpt-6-sol"+
}+
}+
},+
{+
"Sid": "AllowBearerTokenAuth",+
"Effect": "Allow",+
"Action": ["bedrock:CallWithBearerToken"],+
"Resource": "*"+
}+
]+
}</code></pre>+
</div>+
<p>If you use streaming (<code>converse_stream</code> or <code>stream=True</code>), add <code>bedrock:InvokeModelWithResponseStream</code> to the Action in the first three statements. Converse authorizes as <code>bedrock:InvokeModel</code>. If yo…+
<p>To switch models, replace <code>openai.gpt-6-sol</code> with <code>openai.gpt-6-luna</code> or <code>openai.gpt-6-astra</code> throughout. For the complete reference, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/global-cross-region…+
<h3 id="service-control-policy-considerations-for-global-cross-region-inference">Service control policy considerations for global cross-Region inference</h3>+
<p>A global cross-Region inference request carries no Region name. Amazon Bedrock sets <code>aws:RequestedRegion</code> to <code>unspecified</code>, so a Region allowlist in an SCP denies the call, and the error names only the policy type, not which policy. That is a co…+
<p>Rather than widening the allowlist, scope the exception. Amazon Bedrock populates the <code>bedrock:InferenceProfileArn</code> condition key during authorization, so an SCP can allow these profiles while keeping Region controls for everything else. The following SCP does that.&l…+
<div class="hide-language">+
<pre><code class="language-json">{+
"Version": "2012-10-17",+
"Statement": [+
{+
"Sid": "DenyOutsideApprovedRegionsExceptBedrockInvoke",+
"Effect": "Deny",+
"NotAction": [+
"iam:*", "sts:*", "organizations:*", "account:*",+
"route53:*", "cloudfront:*", "waf:*", "wafv2:*", "shield:*",+
"support:*", "trustedadvisor:*", "health:*",+
"budgets:*", "ce:*", "cur:*", "tax:*", "billing:*",+
"bedrock:Invoke*"+
],+
"Resource": "*",+
"Condition": {+
"StringNotEquals": { "aws:RequestedRegion": ["ap-southeast-1"] }+
}+
},+
{+
"Sid": "DenyBedrockInferenceOutsideApprovedRegionsUnlessCris",+
"Effect": "Deny",+
"Action": ["bedrock:Invoke*"],+
"Resource": "*",+
"Condition": {+
"StringNotEquals": { "aws:RequestedRegion": ["ap-southeast-1"] },+
"ArnNotLike": {+
"bedrock:InferenceProfileArn": [+
"arn:aws:bedrock:*:*:inference-profile/global.*"+
]+
}+
}+
}+
]+
}</code></pre>+
</div>+
<p>The first statement enforces your Region allowlist for every service except Amazon Bedrock inference. The second denies that inference outside those Regions unless it routes through a global cross-Region inference profile. For geographic profiles, add their prefix (for example <code>i…+
<p>A blunter alternative is to add <code>unspecified</code> to your existing allowlist, which works but opens the same exception to every other service. For a ready-to-use policy file, see the <a href="https://github.com/aws-samples/service-control-policy-examples/blob/main/Regi…+
<h3 id="data-retention-and-abuse-detection">Data retention and abuse detection</h3>+
<p>For automated offline abuse detection, classifier-flagged traffic is retained by AWS for up to 30 days and processed programmatically. Under cross-Region inference that retention happens in the destination Region, which for a global profile is any supported commercial AWS Region where the m…+
<p>If your organization has data residency requirements, take this to your risk and compliance team early. See the <a href="https://aws.amazon.com/financial-services/security-compliance/compliance-center/" target="_blank" rel="noopener">AWS Financial Services Compliance Center</a> …+
<h2 id="quota-management">Quota management</h2>+
<p>GPT-6 quotas on <code>bedrock-runtime</code> are measured in tokens per minute (TPM), tracked against the source Region you call from. Input and cache-write tokens count 1:1, and each output token consumes 10 tokens of quota. For example, a request with 2,000 input tokens and 1,…+
<p>Amazon Bedrock reserves the full <code>max_tokens</code> up front, so set it close to the completions you expect. One quota covers <code>InvokeModel</code>, Converse, Responses, and Chat Completions. Review and raise it on the <a href="https://console.aws.amazon.c…+
<h2 id="monitoring-and-logging">Monitoring and logging</h2>+
<p><strong>Your operational surface stays in your source Region.</strong> Amazon CloudWatch metrics, AWS CloudTrail events, model invocation logs, quota accounting, and billing all remain in the Region you called from. You pay that Region’s rates, not the processing Region’s.</p…+
<p><strong>CloudTrail records where each request was processed.</strong> AWS documents a global cross-Region inference profile’s destinations as its commercial Regions rather than enumerating them. <code>GetInferenceProfile</code> returns the Region-agnostic and source-…+
<div class="hide-language">+
<pre><code class="language-bash">aws cloudtrail lookup-events \+
--region ap-southeast-1 \+
--lookup-attributes AttributeKey=EventSource,AttributeValue=bedrock.amazonaws.com \+
--start-time &lt;START&gt; \+
--end-time &lt;END&gt; \+
--max-results 1 \+
--query 'Events[0].CloudTrailEvent' --output text | python3 -m json.tool</code></pre>+
</div>+
<p>Set the window around a call you just made, because CloudTrail delivery lags by a few minutes. If the most recent event was a failed call, it carries an <code>errorCode</code> and no <code>inferenceRegion</code>. Raise <code>--max-results</code> and pick …+
<div class="hide-language">+
<pre><code class="language-json">{+
"eventName": "Converse",+
"eventSource": "bedrock.amazonaws.com",+
"awsRegion": "ap-southeast-1",+
"managementEvent": true,+
"requestParameters": {+
"modelId": "global.openai.gpt-6-sol",+
"inferenceConfig": { "maxTokens": 1024 }+
},+
"additionalEventData": {+
"inferenceRegion": "us-east-1"+
}+
}</code></pre>+
</div>+
<p><strong>Metrics and coding-agent usage stay in CloudWatch.</strong> Where developers reach GPT-6 through a coding agent, CloudWatch <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/coding-agents-insights.html" target="_blank" rel="noopener">Coding Ag…+
<h2 id="clean-up">Clean up</h2>+
<p>This post adds an IAM policy and optionally an SCP statement and a logging destination. Detach or delete the IAM policy you attached, and revert the SCP entry if you added one. Delete the model invocation logging destination in <a href="https://aws.amazon.com/s3/" target="_blank" rel="no…+
<h2 id="conclusion">Conclusion</h2>+
<p>This post showed how to invoke GPT-6 Sol and GPT-6 Luna from five Asia Pacific source Regions, and GPT-6 Astra from Asia Pacific (Singapore), through global cross-Region inference on <code>bedrock-runtime</code>. Pass a global cross-Region inference profile ID as the <code>…+
<p>To get started, attach the IAM policy from this post, and send a first request through <code>global.openai.gpt-6-sol</code>. Size your tokens-per-minute quota at the 10x burndown rate, and confirm in CloudTrail that <code>additionalEventData.inferenceRegion</code> ap…+
<p>For more information, see the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html" target="_blank" rel="noopener">Amazon Bedrock User Guide</a>, <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/cross-region-inference.html" target="_…+
<p>&nbsp;</p>+
<p style="clear: both"></p>+
<hr style="width: 100%">+
<h2>About the authors</h2>+
<footer>+
<div class="blog-author-box" style="padding-top: 2.0em">+
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-21759-3-1.jpg" alt="Traci Lim" width="100" height="133">+
</div>+
<h3 class="lb-h4">Traci Lim</h3>+
<p style="overflow: hidden">Traci is a Senior AI/ML Specialist Technical Account Manager at AWS based in Singapore. A machine learning engineer by trade, he works with startups and enterprises to operationalize and scale AI/ML applications in production, with a focus on GenAIOps, Agentic Ops…+
</div>+
<div class="blog-author-box" style="padding-top: 2.0em">+
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-21759-4-1.jpg" alt="Karan Jhaveri" width="100" height="133">+
</div>+
<h3 class="lb-h4">Karan Jhaveri</h3>+
<p style="overflow: hidden">Karan is a Senior Technical Account Manager at AWS based in Singapore. He works with public sector and enterprise customers to take generative AI from experimentation to production at scale. He likes to stay hands-on, getting into the details and building out solu…+
</div>+
<div class="blog-author-box" style="padding-top: 2.0em">+
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-21759-5-1.jpg" alt="Melanie Li" width="100" height="133">+
</div>+
<h3 class="lb-h4">Melanie Li</h3>+
<p style="overflow: hidden">Melanie, PhD, is a Senior Generative AI Specialist Solutions Architect at AWS based in Sydney, Australia, where her focus is on working with customers to build solutions using state-of-the-art AI/ML tools. She has been actively involved in multiple generative AI i…+
</div>+
<div class="blog-author-box" style="padding-top: 2.0em">+
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-21759-6-1.jpg" alt="Zohreh Norouzi" width="100" height="133">+
</div>+
<h3 class="lb-h4">Zohreh Norouzi</h3>+
<p style="overflow: hidden">Zohreh is a Senior Security Solutions Architect at Amazon Web Services (AWS). She helps customers make good security choices and accelerate their journey to the AWS Cloud. She has been actively involved in AI security initiatives, using her expertise to help custo…+
</div>+
<div class="blog-author-box" style="padding-top: 2.0em">+
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-21759-7-1.jpg" alt="Saurabh Trikande" width="100" height="133">+
</div>+
<h3 class="lb-h4">Saurabh Trikande</h3>+
<p style="overflow: hidden">Saurabh is a Senior Product Manager for Amazon Bedrock and Amazon SageMaker Inference. He is passionate about working with customers and partners, motivated by the goal of democratizing AI. He focuses on core challenges related to deploying complex AI applications…+
</div>+
<div class="blog-author-box" style="padding-top: 2.0em">+
<div class="blog-author-image" style="margin-right: 1.0em">+
<img loading="lazy" class="alignnone size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-21759-8-1.jpg" alt="Chris Dickens" width="100" height="133">+
</div>+
<h3 class="lb-h4">Chris Dickens</h3>+
<p style="overflow: hidden">Chris is a Member of Product Staff at OpenAI focused on the OpenAI APIs. His work includes collaboration with AWS on Amazon Bedrock to make OpenAI’s frontier models widely accessible to developers.</p>+
</div>+
</footer></content:encoded>+
+
+
+
</item>+
<item>+
<title>Deploy decision models with llama.cpp on Amazon SageMaker AI</title>+
<link>https://aws.amazon.com/blogs/machine-learning/deploy-decision-models-with-llama-cpp-on-amazon-sagemaker-ai/</link>+
+
<dc:creator><![CDATA[Jyothirmai Kottu]]></dc:creator>+
<pubDate>Mon, 12 Oct 2026 15:51:46 +0000</pubDate>+
<category><![CDATA[Advanced (300)]]></category>+
<category><![CDATA[Amazon SageMaker AI]]></category>+
<category><![CDATA[Technical How-to]]></category>+
<guid isPermaLink="false">f8a20bd6f8bcd8037ef8136ecd594ea98c83c9d3</guid>+
+
<description>Learn how to deploy decision models with llama.cpp on Amazon SageMaker AI. This technical how-to serves OpenAI-compatible chat completions with the AWS Deep Learning Container for llama.cpp, then extends the runtime with SemIf to return probabilities over declared options on CPU or…+
<content:encoded><p>Applications that route requests, select agent actions, or assign policy outcomes often need scores over a fixed set of choices instead of free-form text. Deploying these decision models with <a href="https://github.com/ggml-org/llama.cpp" target="_blank" rel="…+
<p>This post continues a series on specialized AWS Deep Learning Containers (DLCs) for generative AI inference. Previous posts cover <a href="https://aws.amazon.com/blogs/machine-learning/speaker-labeled-transcription-with-whisperx-on-sagemaker-ai/" target="_blank" rel="noopener">Whisper…+
<p>In this post, you first use the <a href="https://aws.github.io/deep-learning-containers/llama-cpp/" target="_blank" rel="noopener">AWS DLC for llama.cpp</a> to serve OpenAI-compatible chat completions through SageMaker AI <code>InvokeEndpoint</code>. You then extend …+
<h2 id="solution-overview">Solution overview</h2>+
<p>Figure 1 shows two deployment paths from the same packaged model. The text-generation path uses the AWS DLC for llama.cpp and returns generated responses through the OpenAI-compatible API. The structured-decision path builds a child image that replaces the default server with a SemIf adapte…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-22079-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/09/ML-22079-1.png" alt="An application invokes …+
<p class="wp-caption-text">Figure 1: Run the AWS DLC for llama.cpp or the SemIf child image on CPU or GPU, depending on the API and latency requirements of your workload</p>+
</div>+
<p>For the standard path, package the model as <code>model.tar.gz</code> in <a href="https://aws.amazon.com/s3/" target="_blank" rel="noopener">Amazon Simple Storage Service (Amazon S3)</a> and reference the published DLC image directly. The container finds the first &l…+
<p>The <code>llama-server</code> HTTP API doesn’t expose the final-position logits and state operations that SemIf uses. The structured-decision path therefore uses <a href="https://aws.amazon.com/codebuild/" target="_blank" rel="noopener">AWS CodeBuild</a> to build a c…+
<h2 id="understand-the-input-and-output-schemas">Understand the input and output schemas</h2>+
<p>The AWS DLC for llama.cpp accepts the OpenAI Chat Completions schema. Send a <code>messages</code> array with <code>role</code> and <code>content</code>, plus generation controls such as <code>max_tokens</code> and <code>stream</code&…+
<div class="hide-language">+
<pre><code class="language-json">{+
"messages": [+
{+
"role": "user",+
"content": "A customer cannot receive a password reset email. Reply with the best support queue."+
}+
],+
"max_tokens": 64,Diff display stops at 400 lines. The line counts above are from the whole diff. 46 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.