Change
14d198e
14d198e647c53364e04e253601fe3bf9abf1fed6 · commit on GitHub
aws-blog-feed: changed (578645 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +3,504
- Lines removed
- -3,631
- Stored bytes at this commit
- 578,645
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-11-03T06:47:08.231Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Tue, 03 Nov 2026 00:36:49 GMT |
| date | Tue, 03 Nov 2026 06:47:08 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Fri, 30 Oct 2026 22:42:24 +0000</lastBuildDate>+
<lastBuildDate>Mon, 02 Nov 2026 20:28:52 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,223 +13,313 @@ 1 </sy:updateFrequency> <item>-
<title>Managing Amazon Bedrock costs end to end: A playbook for FinOps and platform teams</title>-
<link>https://aws.amazon.com/blogs/machine-learning/managing-amazon-bedrock-costs-end-to-end-a-playbook-for-finops-and-platform-teams/</link>+
<title>Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI</title>+
<link>https://aws.amazon.com/blogs/machine-learning/prepare-web-content-for-amazon-bedrock-knowledge-bases-with-crawl4ai/</link> -
<dc:creator><![CDATA[Aswath Ram A Srinivasan]]></dc:creator>-
<pubDate>Fri, 30 Oct 2026 22:42:24 +0000</pubDate>+
<dc:creator><![CDATA[Daxeshkumar Patel]]></dc:creator>+
<pubDate>Mon, 02 Nov 2026 17:11:21 +0000</pubDate> <category><![CDATA[Advanced (300)]]></category>-
<category><![CDATA[Amazon Bedrock]]></category>+
<category><![CDATA[Amazon Bedrock Knowledge Bases]]></category> <category><![CDATA[Technical How-to]]></category>-
<guid isPermaLink="false">5542e879e541b56f1c06a9f2c71c7c3f37f6d5e3</guid>+
<guid isPermaLink="false">43ca16f9eb64c74c6edd06ac90851b607a986aa7</guid>-
<description>This post presents a five-stage operating model for managing Amazon Bedrock cost: attribute every dollar to an owner, make every request observable, enforce budgets, optimize platform defaults, and operate a review cadence. Written for FinOps and platform engineering teams.</descri…-
<content:encoded><p>Cost management for generative AI is a shared responsibility between the teams that build applications and the teams accountable for cloud spend. Many teams start by comparing per-model token rates and asking application teams to shorten prompts. Those steps help,…-
<p>Agentic workloads make those questions more pressing. A conversational assistant typically issues one model invocation per user turn. An agentic workflow, such as an AI coding assistant that plans a change, reads files, calls tools, and iterates, issues many invocations per task and carries…-
<p>The fix isn’t a cheaper model or a smaller prompt. It’s an operating model: a repeatable loop that attributes every dollar to an owner, makes every request observable, alerts on and enforces budgets before overspend compounds, optimizes the platform defaults that drive most waste, and revie…-
<p>In this post, you learn how to manage <a href="https://aws.amazon.com/bedrock/" target="_blank" rel="noopener">Amazon Bedrock</a> cost in five steps. You attribute every dollar to an owner, make every request observable, control spend with budgets and enforcement, optimize the p…-
<h2 id="the-operating-model-in-one-view">The operating model in one view</h2> -
<p>Five stages, run as a loop. Attribute: every Amazon Bedrock dollar lands on an identity, team, and application in the billing pipeline. Observe: every request is logged with enough metadata to explain what generated the spend. Enforce: budgets have teeth, with alerts at minimum and hard blo…+
<description>This post shows how to use Crawl4AI, an open source web crawler, as a content preparation layer for Amazon Bedrock Knowledge Bases, handling JavaScript rendering, selective extraction, and custom metadata enrichment before web content reaches an Amazon S3 data source.</description>+
<content:encoded><p>When you build Retrieval Augmented Generation (RAG) applications, answer quality depends on the data that you ingest. <a href="https://aws.amazon.com/bedrock/knowledge-bases/" target="_blank" rel="noopener">Amazon Bedrock Knowledge Bases</a> is the ful…+
<p>For web content, start with the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-ds-webcrawler.html" target="_blank" rel="noopener">Amazon Bedrock Web Crawler</a> data source connector for managed knowledge bases. You give it the seed URLs or sitemaps tha…+
<p>Some sources need more control over what reaches the knowledge base. You might want to index only the main content region of each page, remove navigation and other boilerplate, or attach your own metadata attributes to every document. For that class of source material, you can extend the re…+
<p>In this post, we show you how to choose the appropriate web ingestion path and implement custom preparation for content that needs additional processing. You then start ingestion into a managed knowledge base and validate retrieval quality. The companion <a href="https://github.com/aws-s…+
<p>This is batch ingestion of known web properties, not real-time web access at inference time such as <a href="https://docs.aws.amazon.com/nova/latest/nova2-userguide/web-grounding.html" target="_blank" rel="noopener">Web Grounding</a> with Amazon Nova.</p> +
<h2 id="solution-overview">Solution overview</h2> +
<p>Start with the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-ds-webcrawler.html" target="_blank" rel="noopener">Amazon Bedrock Web Crawler</a> when it can reach and index the pages you need. It respects <code>robots.txt</code>, and after th…+
<p>Use the Crawl4AI preparation path when public source material needs one or more of the following before ingestion:</p> +
<ul> +
<li><strong>Structured Markdown preparation</strong>: Use <code>DefaultMarkdownGenerator</code> with a pruning content filter to reduce navigation and other boilerplate while preserving headings, lists, and tables. Because pruning is heuristic, review representative ou…+
<li><strong>Bounded, targeted extraction</strong>: Extract only the page region that an optional CSS selector matches. Constrain breadth-first traversal by seed origin, page/path/host scope, depth and page caps, and URL patterns.</li> +
<li><strong>Custom metadata</strong>: Attach source URLs, crawl timestamps, content type, and your own attributes to each document, so that you can filter retrieval results on them.</li> +
</ul> +
<p>Because managed and customer-managed knowledge bases both support Amazon S3 data sources, the Crawl4AI path works with either type. This post uses a managed knowledge base and notes where a customer-managed knowledge base differs.</p> +
<p>The rest of this post follows the companion sample. Treat its architecture and defaults as a starting point, not as requirements of Amazon Bedrock Knowledge Bases.</p> +
<p>By default, the sample runs the AWS Fargate task in a public subnet, where it reaches your configured websites through an internet gateway. Crawl4AI’s role ends once it has prepared the source documents and metadata sidecars. From ingestion onward, Amazon Bedrock Knowledge Bases handles par…<div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/29/ML-21654-1-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/29/ML-21654-1-1.png" alt="Five-stage Amazon B…-
<p class="wp-caption-text">Figure 1: The five-stage operating model, run as a continuous loop</p>+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-21438-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-21438-1.png" alt="Architecture diagram sh…+
<p class="wp-caption-text">Figure 1: The Crawl4AI-to-Amazon S3 ingestion path for Amazon Bedrock Knowledge Bases</p></div> -
<h2 id="stage-1-attribute-every-dollar">Stage 1: Attribute every dollar</h2> -
<p>You can’t manage what you can’t attribute, and everything downstream (chargeback, budgets, enforcement, anomaly detection) depends on this stage. The foundation is the granular cost attribution of Amazon Bedrock (see <a href="https://aws.amazon.com/blogs/machine-learning/introducing-gran…-
<p>Setup is four steps:</p> +
<p>The numbered steps in the diagram map to the following workflow:</p> <ol type="1"> -
<li>Turn on IAM principal data in your CUR 2.0 data export.</li> -
<li>Tag your IAM principals with business dimensions such as team, application, and cost center. For federated access, configure your identity provider to pass session names and tags through OIDC or SAML claims.</li> -
<li>Activate those tags as cost allocation tags in AWS Billing (console or the <code>UpdateCostAllocationTagsStatus</code> API).</li> -
<li>Analyze the tags: they surface in CUR 2.0 with an <code>iamPrincipal/</code> prefix (for example, <code>iamPrincipal/team</code>) and appear in <a href="https://aws.amazon.com/aws-cost-management/aws-cost-explorer/" target="_blank" rel="noopener">AWS Cost Exp…+
<li>Amazon EventBridge starts the Amazon Elastic Container Service (Amazon ECS) task on AWS Fargate on the configured schedule.</li> +
<li>The task crawls the public websites you configured and prepares Markdown with a metadata sidecar for each page.</li> +
<li>The task uploads the prepared files to the Amazon S3 bucket that backs the knowledge base data source.</li> +
<li>The task starts an Amazon Bedrock Knowledge Bases ingestion job.</li> +
<li>Amazon Bedrock Knowledge Bases reads the new and changed objects, parses them, and applies the chunking strategy you selected.</li> +
<li>It creates embeddings and indexes them in the knowledge base’s vector store, which Amazon Bedrock manages for a managed knowledge base.</li> +
<li>Your generative AI application queries the knowledge base for relevant chunks or for a generated response with citations.</li> </ol> -
<p>This is invoice-grade, per-identity attribution with no custom infrastructure.</p> -
<h3 id="the-shared-role-problem">The shared-role problem</h3> -
<p>The pattern that deserves the most attention is the shared-role problem, because it is where most platforms silently lose attribution. A large language model (LLM) gateway that calls Amazon Bedrock through a single IAM role attributes every dollar to that one role, eliminating per-user visi…-
<p>This needs two roles: a gateway execution role holding <code>sts:AssumeRole</code> and <code>sts:TagSession</code> permissions, and the Amazon Bedrock invocation role it assumes. The result is <code>line_item_iam_principal</code> entries like <code>…+
<p>The task reads its non-secret pipeline configuration from AWS Systems Manager Parameter Store at startup.</p> +
<p>The workflow processes public pages only and doesn’t implement authenticated sessions. For sites that require sign-in, evaluate the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-ds-webcrawler.html" target="_blank" rel="noopener">Amazon Bedrock Web Crawler au…+
<h2 id="how-the-workflow-works">How the workflow works</h2> +
<p>The AWS Cloud Development Kit (AWS CDK) app packages the implementation in a container image during deployment. When the schedule fires, the container runs <code>scripts/container_entrypoint.py</code> and executes crawl, upload, and ingestion in sequence. The following three ste…+
<h3 id="step-1-crawl-and-extract-content">Step 1: Crawl and extract content</h3> +
<p>The crawl step performs bounded crawling and writes a Markdown file with a metadata sidecar for each successfully prepared page. The following excerpt is simplified. The module <code>src/crawl4ai_bedrock_kb/crawl.py</code> contains the complete seed pacing, URL filtering and sco…<div class="hide-language"> -
<pre><code class="language-python">import boto3+
<pre><code class="language-python">async def crawl_and_prepare(config: PipelineConfig, output_dir: str) -&gt; list[str]:+
written, seen = [], set()+
pages_crawled = 0+
browser_config = BrowserConfig(headless=True, java_script_enabled=True)-
sts = boto3.client("sts")+
async with AsyncWebCrawler(config=browser_config) as crawler:+
for seed in config.seed_urls:+
remaining = config.crawl.max_pages - pages_crawled+
if remaining &lt;= 0:+
break-
# One call per user; cache the returned credentials up to 1 hour-
response = sts.assume_role(-
RoleArn="arn:aws:iam::123456789012:role/BedrockRuntimeRole",-
RoleSessionName="gw-alex", # the user, not the gateway-
Tags=[-
{"Key": "team", "Value": "sales-ai"},-
{"Key": "application", "Value": "renewal-agent"},-
{"Key": "costCenter", "Value": "CC-2041"}-
]-
)</code></pre> -
</div> -
<h3 id="a-note-on-application-inference-profiles-and-projects">A note on application inference profiles and Projects</h3> -
<p>Amazon Bedrock offers two more attribution mechanisms, each suited to a different pattern. <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-create.html" target="_blank" rel="noopener">Application inference profiles</a> are pre-created, per-applica…-
<p>For <code>bedrock-runtime</code> workloads, we recommend IAM principal attribution through session tags as the primary mechanism. It scales with your identity system, requires no pre-created resources, and covers every model and API operation. Application inference profiles work…-
<h2 id="stage-2-make-every-request-observable">Stage 2: Make every request observable</h2> -
<p>CUR tells you who spent. Invocation logs tell you what happened. Amazon Bedrock supports <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html" target="_blank" rel="noopener">model invocation logging</a> of request data, response data, and m…-
<p>Logging also closes the latency gap: CUR attribution lags roughly 24 hours, but CloudWatch plus OpenTelemetry can power near-real-time dashboards showing per-developer spend within minutes (<a href="https://aws.amazon.com/blogs/mt/analyzing-claude-code-usage-with-cloudwatch-and-opentelem…-
<h3 id="request-metadata-for-workload-level-attribution">Request metadata for workload-level attribution</h3> -
<p>Raw logs answer identity-level questions. To answer workload-level questions (which feature, workflow, or tenant generated the tokens), stamp <a href="https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html#bedrock-runtime_Converse-request-requestMetadata" targe…-
<p>The following example uses Anthropic Claude Sonnet 5.5 on Amazon Bedrock through the US cross-Region inference profile. For the full list of models, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html" target="_blank" rel="noopener">Supported founda…+
strategy = BFSDeepCrawlStrategy(+
max_depth=config.crawl.max_depth,+
max_pages=remaining,+
filter_chain=FilterChain([+
SeedScopeFilter(seed, config.crawl.scope),+
# Optional include/exclude URLPatternFilter objects are added here.+
]),+
)+
run_config = CrawlerRunConfig(+
deep_crawl_strategy=strategy,+
markdown_generator=DefaultMarkdownGenerator(+
content_filter=PruningContentFilter()+
),+
css_selector=config.crawl.css_selector, # Optional page region.+
check_robots_txt=config.crawl.check_robots_txt,+
mean_delay=config.crawl.request_delay_seconds,+
max_range=0,+
semaphore_count=1,+
)+
+
results = await crawler.arun(url=seed, config=run_config)+
for result in results:+
if pages_crawled &gt;= config.crawl.max_pages:+
break+
pages_crawled += 1+
requested_url = result.url+
source_url = getattr(result, "redirected_url", None) or requested_url+
if not SeedScopeFilter(seed, config.crawl.scope).apply(source_url):+
continue # Recheck the effective destination after redirects.+
canonical = _canonical_url(source_url)+
if canonical in seen:+
continue+
seen.add(canonical)+
if not result.success or not result.markdown:+
continue+
+
md_result = result.markdown+
markdown = (+
getattr(md_result, "fit_markdown", None)+
or getattr(md_result, "raw_markdown", None)+
or (md_result if isinstance(md_result, str) else "")+
)+
if not markdown:+
continue+
+
# The complete implementation adds title, crawl timestamps, content+
# type, and configured custom attributes to this metadata.+
written.extend(+
_write_document(+
output_dir,+
_base_name(source_url),+
markdown,+
{"source_url": source_url},+
)+
)+
return written</code></pre> +
</div> +
<p>Amazon Bedrock Knowledge Bases supports <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-ds.html" target="_blank" rel="noopener">several document formats</a> for Amazon S3 data sources, including plain text, HTML, Markdown, PDF, and DOCX. The workflow…+
<h3 id="step-2-upload-to-amazon-s3">Step 2: Upload to Amazon S3</h3> +
<p>The upload step sends each prepared Markdown document and its matching metadata sidecar to the configured S3 bucket, under the workflow’s <code>s3Prefix</code>. For <code>page.md</code>, the sidecar must be named <code>page.md.metadata.json</code>, the co…+
<p>Each attribute in the sidecar uses the typed format that both managed and customer-managed knowledge bases accept for Amazon S3 data sources. The following abbreviated sidecar, formatted for readability, shows three of the attributes that the workflow writes:</p> <div class="hide-language"> -
<pre><code class="language-python">response = client.converse(-
modelId="us.anthropic.claude-sonnet-5-5",-
messages=[...],-
requestMetadata={-
"application": "renewal-copilot",-
"workflow": "renewal-risk-summary",-
"tenant": "enterprise-sales",-
"agent": "renewal-agent"+
<pre><code class="language-json">{+
"metadataAttributes": {+
"source_url": {+
"value": {"type": "STRING", "stringValue": "https://www.example.com/services/permits"}+
},+
"crawled_at_epoch": {+
"value": {"type": "NUMBER", "numberValue": 1790640000}+
},+
"content_category": {+
"value": {"type": "STRING", "stringValue": "web"} }-
)</code></pre> +
}+
}</code></pre> </div> -
<p>Amazon Bedrock doesn’t enforce request metadata, and teams drift into inconsistent schemas quickly when every application does its own thing. The platform team should inject mandatory metadata centrally in a shared SDK wrapper or gateway, normalize the key set, and treat metadata coverage a…-
<h3 id="diagnostic-queries">Diagnostic queries</h3> -
<p>With logs and metadata in place, CloudWatch Logs Insights answers the standing questions. <em>Who consumes the most tokens</em>:</p> +
<p>The pipeline uploads only the paths produced by the current crawl, so stale local files are not sent.</p> +
<h3 id="step-3-let-amazon-bedrock-knowledge-bases-chunk-embed-and-index">Step 3: Let Amazon Bedrock Knowledge Bases chunk, embed, and index</h3> +
<p>After content is in S3, an ingestion job hands each document to the data source’s ingestion pipeline. By default, a managed knowledge base parses each file with its managed parser, which automatically selects a parsing approach for your content. You select the chunking strategy when you con…+
<ul> +
<li><strong>Default chunking</strong> uses fixed-size chunks of 300 tokens with 20 percent overlap.</li> +
<li><strong>Fixed-size chunking</strong> splits content by a maximum token count and an overlap percentage that you set.</li> +
<li><strong>No chunking</strong> treats each file as a single chunk, for content that you have already split.</li> +
</ul> +
<p>Chunk boundaries and retrieval results depend on the content and the selected strategy, so compare the options with representative pages and evaluation questions. To change strategies later, create a new data source. For more information, see <a href="https://docs.aws.amazon.com/bedrock/…+
<p>The workflow then starts an ingestion job so the knowledge base processes the new documents. The following simplified excerpt from <code>src/crawl4ai_bedrock_kb/ingest.py</code> shows the call:</p> <div class="hide-language"> -
<pre><code class="language-plaintext">fields @timestamp, identity.arn as caller, modelId,-
input.inputTokenCount as inTokens,-
output.outputTokenCount as outTokens-
| stats sum(inTokens + outTokens) as totalTokens,-
count(*) as calls-
by caller, modelId-
| sort totalTokens desc-
| limit 25</code></pre> +
<pre><code class="language-python">def start_ingestion_job(config: PipelineConfig) -&gt; str:+
# Start an Amazon Bedrock knowledge base ingestion job and return its ID.+
client = boto3.client("bedrock-agent", region_name=config.region)+
response = client.start_ingestion_job(+
knowledgeBaseId=config.knowledge_base_id,+
dataSourceId=config.data_source_id,+
)+
return response["ingestionJob"]["ingestionJobId"]</code></pre> </div> -
<p>The same query grouped by <code>requestMetadata.application</code> and <code>requestMetadata.workflow</code> instead of <code>caller</code> produces the workload-level leaderboard: the view product and engineering teams will ask FinOps for the moment char…-
<p><strong>Note:</strong> Field paths shown in the preceding query are illustrative. Verify against your invocation log schema, which may vary by model, API operation, and logging configuration.</p> -
<h2 id="stage-3-enforce-budgets">Stage 3: Enforce budgets</h2> -
<p>With attribution and observability in place, enforcement becomes an architecture choice rather than a wish. There are three viable tiers. The right one depends on your spend velocity, your risk tolerance for overspend, and how your tools connect to Amazon Bedrock.</p> -
<h3 id="tier-1-monitor-and-alert-no-blocking">Tier 1: Monitor and alert (no blocking)</h3> -
<p>Amazon Bedrock writes principal identity into the billing pipeline automatically. You put AWS Budgets on top for threshold alerts, with the OpenTelemetry dashboards from Stage 2 closing the 24-hour CUR gap for day-to-day visibility. Nothing to deploy, native AWS billing, and no added latenc…-
<h3 id="tier-2-iam-tag-enforcement-deny-on-breach">Tier 2: IAM tag enforcement (deny on breach)</h3> -
<p>An aggregator AWS Lambda function tallies per-user token and dollar counters in Amazon DynamoDB from invocation log data. When a user breaches their threshold, the function adds their identity to a shared IAM deny policy (denying <code>bedrock:*</code>) until a monthly reset or …-
<h3 id="tier-3-gateway-enforcement-synchronous-block">Tier 3: Gateway enforcement (synchronous block)</h3> -
<p>A gateway sits in the inference data path and enforces budgets synchronously, returning HTTP 429 the moment a user or team exceeds their limit. You get sub-second blocking, per-user model allowlists, token rate limits, and centrally enforced routing rules. The trade-off: a component in the …-
<p>For Claude Code and Claude Desktop fleets, the <a href="https://aws.amazon.com/blogs/machine-learning/introducing-claude-apps-gateway-for-aws/" target="_blank" rel="noopener">Claude apps gateway for AWS</a> is the official recommendation. It ships inside the CLI binary and runs …-
<h3 id="choosing-a-tier">Choosing a tier</h3> -
<p>Two factors dominate.</p> -
<p><strong>Connectivity:</strong> If developers call Amazon Bedrock directly with no proxy, only Tiers 1 and 2 are viable.</p> -
<p><strong>Spend velocity:</strong> If spend is already in the millions per month, build enforcement now. If alerts can hold the line for another quarter, invest in Stages 1 and 2 first and add enforcement when the data demands it.</p> -
<p>Most organizations should phase it: observe at Tier 1, then step up. Treat all three as bridges, not destinations: the <a href="https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentcost07.html" target="_blank" rel="noopener">Well-Architected Agentic AI Lens</a>…-
<h2 id="stage-4-optimize-the-defaults">Stage 4: Optimize the defaults</h2> -
<p>Most Amazon Bedrock waste is not individual carelessness. It is platform defaults nobody set. The two highest-impact levers are both platform-controlled.</p> -
<h3 id="model-routing">Model routing</h3> -
<p>Not every task needs a frontier model. Routine completions (boilerplate, linting suggestions, small refactors) can run on smaller, cheaper models with quality reserved for complex reasoning, through <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/intelligent-prompt-routing.…-
<h3 id="prompt-caching">Prompt caching</h3> -
<p>Prompt caching rewards stable repeated prefixes. As of October 2026, the published <a href="https://aws.amazon.com/bedrock/pricing/" target="_blank" rel="noopener">Amazon Bedrock pricing</a> for Anthropic Claude models in US East (N. Virginia) bills a cache write at 1.25x the ba…-
<p>Candidates surface directly from the invocation logs:</p> +
<p>The same <code>StartIngestionJob</code> and <code>GetIngestionJob</code> calls work for both knowledge base types, and you can also monitor progress on the console.</p> +
<h2 id="deploy-the-solution">Deploy the solution</h2> +
<p>In this section, you deploy the workflow into your AWS account with AWS CDK. Only the knowledge base prerequisites differ between the two knowledge base types. When you’re finished, follow the steps in the Clean up section to remove the resources.</p> +
<h3 id="prerequisites">Prerequisites</h3> +
<p>Before you start, make sure that you have:</p> +
<ul> +
<li>An AWS account.</li> +
<li>One of the following knowledge bases: +
<ul> +
<li><strong>A managed knowledge base</strong>, which this post uses, in an AWS Region where <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-regions.html" target="_blank" rel="noopener">managed knowledge bases are available</a>. For instructio…+
<li><strong>An existing customer-managed knowledge base.</strong> To add an Amazon S3 data source, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/s3-data-source-connector.html" target="_blank" rel="noopener">Connect to Amazon S3 for your knowledge base&l…+
</ul> </li> +
<li>An Amazon S3 data source in that knowledge base whose status is <code>AVAILABLE</code>. It can be a new data source or one you already use. It must use the S3 bucket that you configure for this workflow. If the data source filters content by prefix or file pattern, the filters…+
<li>The knowledge base, the S3 bucket, and the stack you deploy all in one Region.</li> +
<li>Authorization to crawl the public websites you configure.</li> +
<li>A deployment workstation with Git, Python 3.10 or later, <code>Node.js</code>, the AWS Command Line Interface (AWS CLI) version 2, the AWS CDK CLI v2, and a running Docker-compatible container engine. The <a href="https://github.com/aws-samples/sample-crawl4ai-bedrock-knowl…+
<li>AWS credentials configured on the deployment workstation with permission to bootstrap the target environment and deploy the CDK stack.</li> +
<li>A bootstrapped CDK environment in the target account and Region. The deployment steps include the bootstrap command.</li> +
</ul> +
<p>You need the workstation only to build and deploy. After deployment, the schedule runs the task in AWS.</p> +
<h3 id="deploy-the-scheduled-workflow">Deploy the scheduled workflow</h3> +
<p>Clone the companion repository and install the deployment dependencies:</p> <div class="hide-language"> -
<pre><code class="language-plaintext">fields requestMetadata.workflow as workflow, modelId,-
input.inputTokenCount as inTokens-
| stats count(*) as calls,-
avg(inTokens) as avgInputTokens,-
sum(inTokens) as totalInputTokens-
by workflow, modelId-
| filter calls &gt; 50 and avgInputTokens &gt; 2000-
| sort totalInputTokens desc-
| limit 25</code></pre> +
<pre><code class="language-bash">git clone https://github.com/aws-samples/sample-crawl4ai-bedrock-knowledge-base.git+
cd sample-crawl4ai-bedrock-knowledge-base/deploy+
python -m venv .venv+
source .venv/bin/activate # Windows PowerShell: .venv\Scripts\Activate.ps1+
python -m pip install -r requirements.txt</code></pre> </div> -
<p><strong>Note:</strong> Field paths shown in the preceding query are illustrative. Verify against your invocation log schema, which may vary by model, API operation, and logging configuration.</p> -
<h3 id="application-level-optimization">Application-level optimization</h3> -
<p>A third class of savings (just-in-time retrieval instead of stuffed context, compaction of long conversations, distilling stable high-volume workflows onto smaller models) belongs to application teams rather than the platform. The platform’s job is to make those savings visible: publish the…-
<h2 id="stage-5-operate-the-loop">Stage 5: Operate the loop</h2> -
<p>The first four stages produce data. This one turns it into decisions. The platform team runs a standing health dashboard with four signals:</p> +
<p>Keep Docker running. Set <code>AWS_DEFAULT_REGION</code> to your knowledge base’s Region first, because <code>cdk deploy</code> uses the Region from your environment or AWS profile, not the one in the bootstrap command. Bootstrap the target account and Region once, i…+
<div class="hide-language"> +
<pre><code class="language-bash">export AWS_DEFAULT_REGION=REGION # Windows PowerShell: $env:AWS_DEFAULT_REGION = "REGION"+
cdk bootstrap aws://ACCOUNT_ID/REGION+
+
cdk deploy \+
-c knowledgeBaseId=ABCDEFGHIJ \+
-c dataSourceId=KLMNOPQRST \+
-c s3Bucket=amzn-s3-demo-bucket \+
-c 'seedUrls=["https://www.example.com/services"]' \+
-c maxDepth=1 \+
-c maxPages=25</code></pre> +
</div> +
<p>Replace every example value, including <code>ACCOUNT_ID</code>, <code>REGION</code>, and the four required context keys, with your own. The example values are syntactically valid, so an unedited command deploys a stack that points at a bucket you don’t own. You can a…+
<p>The <code>maxDepth</code> and <code>maxPages</code> values limit the first crawl to one link level and 25 pages. Raise them after you review the prepared output. To preview which pages a crawl prepares, you can also run the pipeline locally with <code>--dry-run…+
<p>The sample builds an ARM64 container image by default. If Docker on your x86_64 workstation can’t build ARM64 images, add <code>-c cpuArchitecture=X86_64</code> to the <code>cdk deploy</code> command.</p> +
<p>If your account turns on VPC Block Public Access, the task can’t reach the internet from the default public subnet. To check the mode, run <code>aws ec2 describe-vpc-block-public-access-options</code>. In ingress-only mode, deploy with <code>-c egressMode=nat</code>,…+
<p>The stack doesn’t create or change your knowledge base or data source.</p> +
<p>With the default <code>rate(24 hours)</code> schedule, the first run starts when the deployment creates the Amazon EventBridge rule. To follow that run, use the command in the <code>TailLogsCommand</code> output, which <code>cdk deploy</code> prints when …+
<p>The <a href="https://github.com/aws-samples/sample-crawl4ai-bedrock-knowledge-base/blob/main/deploy/README.md" target="_blank" rel="noopener">deployment guide</a> covers the remaining configuration, including crawl boundaries, the schedule, networking, container architecture, mo…+
<h2 id="validate-retrieval-quality">Validate retrieval quality</h2> +
<p>After ingestion, test representative questions with <code>Retrieve</code> before you add generation. A managed knowledge base uses hybrid search. When it uses the default managed embedding model, it also reranks results with its managed reranker. You can narrow results with the …+
<div class="hide-language"> +
<pre><code class="language-python">import time+
+
import boto3+
+
client = boto3.client("bedrock-agent-runtime", region_name="us-east-1")+
week_ago = int(time.time()) - 7 * 24 * 60 * 60+
+
response = client.retrieve(+
knowledgeBaseId="ABCDEFGHIJ",+
retrievalQuery={"text": "How do I apply for a building permit?"},+
retrievalConfiguration={+
"managedSearchConfiguration": {+
"numberOfResults": 5,+
"filter": {+
"andAll": [+
{"equals": {"key": "content_category", "value": "web"}},+
{"greaterThan": {"key": "crawled_at_epoch", "value": week_ago}},+
]+
},+
}+
},+
)+
+
for result in response["retrievalResults"]:+
print(result.get("metadata", {}).get("source_url"), result["content"]["text"][:200])</code></pre> +
</div> +
<p>The example passes an integer timestamp, which both knowledge base types accept in number filters. A managed knowledge base doesn’t support the <code>startsWith</code> and <code>stringContains</code> filters. For a customer-managed knowledge base, pass the same filte…+
<p>Metadata filtering can improve retrieval precision, but it is not an authorization boundary. If different users should see different pages, the Amazon S3 connector for managed knowledge bases supports <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-ds-s3-acl.html…+
<p>To generate responses from a managed knowledge base, use <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-agentic-retrieve.html" target="_blank" rel="noopener">agentic retrieval</a>. The <code>AgenticRetrieveStream</code> operation returns the re…+
<p>Evaluate retrieval relevance, source-page coverage, and citation quality whenever you change content preparation, chunking, or retrieval settings. For the retrieval settings that you can tune, see <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-test-config.html" …+
<h2 id="running-it-repeatedly">Running it repeatedly</h2> +
<p>Each scheduled event launches one standalone Fargate task. The stack doesn’t prevent overlapping runs, so choose an interval longer than the expected crawl and ingestion duration. A failed run isn’t announced, because the sample adds no retry policy, dead-letter queue, or alarm. An ingestio…+
<p>Each run incurs Fargate charges for the configured task size, which defaults to 2 vCPU and 4 GB. The task keeps running while it polls the ingestion job for up to 30 minutes, so a slow ingestion job extends billable task time. If ingestion takes longer, the task exits with a nonzero code, a…+
<p>Uploads are additive: removing a URL or missing a page during a later crawl doesn’t delete its existing S3 objects. To remove stale content, delete the obsolete document and matching sidecar under the configured prefix, then start ingestion again. If you turn on the document deletion safegu…+
<h2 id="responsible-ai-considerations">Responsible AI considerations</h2> +
<p>Retrieved passages become the context that a foundation model uses to generate a response. Apply the following controls before you run this workflow in production:</p> <ul> -
<li><strong>Model mix:</strong> Which models are called and how often, the direct measure of routing discipline.</li> -
<li><strong>Failed request rates:</strong> Errors, throttles, and timeouts. Each retry adds latency, and a client-side timeout on a request that still completes on the service side can be billed twice, once for the abandoned call and once for the retry.</li> -
<li><strong>Cache effectiveness:</strong> The read-versus-write ratio from Stage 4.</li> -
<li><strong>Metadata coverage:</strong> The percentage of production calls with complete metadata. Every uncovered call is an attribution blind spot that quietly degrades every other number.</li> +
<li><strong>Amazon Bedrock Guardrails</strong>: Attach a <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html" target="_blank" rel="noopener">guardrail</a> to your agentic retrieval or <code>RetrieveAndGenerate</code> requests to ap…+
<li><strong>Contextual grounding checks</strong>: Turn on the Guardrails <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-contextual-grounding-check.html" target="_blank" rel="noopener">contextual grounding check</a> to score each response for g…+
<li><strong>Treat crawled content as untrusted input</strong>: Prepared web pages become model context, so text on a public page can attempt prompt injection through retrieval. Don’t rely on guardrails alone to catch it. Restrict seed URLs to sites you trust, and apply the ingesti…+
<li><strong>Citations and human review</strong>: Return citations with every generated response so users can verify claims against the source page. For higher-risk use cases, keep a human reviewer in the loop.</li> +
</ul> +
<h2 id="security-and-compliance-considerations">Security and compliance considerations</h2> +
<p>Review the following considerations before you run this workflow against production content:</p> +
<ul> +
<li><strong>Use trusted public crawl targets</strong>: Seed URLs and crawl settings are deployment configuration, and the Fargate task allows outbound access so it can reach public websites. In the sample’s default networking mode, the task runs in a public subnet with a public IP…+
<li><strong>Respect crawl directives</strong>: The workflow turns on Crawl4AI’s <code>robots.txt</code> check by default, and the CDK path pins it on with no context key to disable it. Verify you have authorization to crawl the target sites, consistent with the <a h…+
<li><strong>Rate limiting</strong>: Configure delays between requests to avoid overwhelming source websites.</li> +
<li><strong>Encryption at rest</strong>: The stack creates an AWS Key Management Service (AWS KMS) customer managed key with automatic key rotation and encrypts its CloudWatch log groups with it. Amazon S3 encrypts new objects with server-side encryption with Amazon S3 managed key…+
<li><strong>Least privilege IAM</strong>: The CDK stack grants the Fargate task role <code>s3:PutObject</code> on the configured bucket prefix. It also grants <code>bedrock:StartIngestionJob</code> and <code>bedrock:GetIngestionJob</code> on the…+
<li><strong>Container and template checks</strong>: The container runs as a non-root user, and the CDK app runs cdk-nag AWS Solutions checks on every synthesis with resource-scoped suppressions.</li> +
<li><strong>Protect deployment configuration</strong>: The CDK app stores non-secret pipeline settings in a plaintext Parameter Store <code>String</code> parameter. Only the parameter name is placed in the task definition. Don’t put passwords, cookies, tokens, or other…+
<li><strong>Ingestion pipeline security</strong>: See <a href="https://aws.amazon.com/blogs/security/securing-the-rag-ingestion-pipeline-filtering-mechanisms/" target="_blank" rel="noopener">Securing the RAG ingestion pipeline: Filtering mechanisms</a> for filtering an…+
</ul> +
<h2 id="clean-up">Clean up</h2> +
<p>To avoid ongoing charges, delete the resources created during this walkthrough:</p> +
<ul> +
<li>From the repository’s <code>deploy/</code> directory, run <code>cdk destroy</code> in the same Region, with the same four required <code>-c</code> values that you passed to <code>cdk deploy</code>. The app checks them on every command. This …+
<li>To remove the ingested web content, delete the prepared objects under <code>s3://&lt;bucket&gt;/&lt;prefix&gt;</code> and sync the data source again. If the document deletion safeguard is on for a managed knowledge base data source, a sync that would delete mor…+
<li>Delete the managed knowledge base only if you created it solely for this walkthrough. A managed knowledge base has no separate vector store to delete.</li> +
<li>For a customer-managed knowledge base, delete the Amazon S3 data source only if you created it for this walkthrough. Its data deletion policy determines whether Amazon Bedrock also deletes the data source’s vector embeddings from your vector store.</li> </ul> -
<p>When a signal drifts, the remediation is a policy change (routing rules, token limits, metadata enforcement), not an email asking teams to be careful.</p> -
<h3 id="the-money-loop">The money loop</h3> -
<p>FinOps runs the money loop on the same data. When attribution is clean, anomaly detection becomes an investigation stream rather than a panicked all-hands: a spike in a tagged team or application triggers a lookup in the Stage 2 leaderboard, not a war room. Chargeback runs from CUR grouping…-
<h3 id="scorecard">Scorecard</h3> -
<p>The following thresholds are starting points, not prescriptions. Calibrate them against your own fleet data and pricing tier before adopting them as policy.</p> -
<table border="1px" width="100%" cellpadding="10px"> -
<tbody> -
<tr> -
<td><strong>Dimension</strong></td> Diff display stops at 400 lines. The line counts above are from the whole diff. 72 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.