Change
e421587
e4215871b187b787aafba2f4c0e798c350b51c7a · commit on GitHub
aws-blog-feed: changed (635817 bytes, HTTP 200)
raw/aws-blog-feed/response.xml modified
- Source
- aws-blog-feed
- Lines added
- +3,996
- Lines removed
- -3,488
- Stored bytes at this commit
- 635,817
- Timestamp
- observed
- Raw artifact at this commit
- raw/aws-blog-feed/response.xml
Recorded headers
| observed_at | 2026-11-04T06:22:00.271Z |
|---|---|
| origin_date | null |
| status | 200 |
| final URL | https://aws.amazon.com/blogs/machine-learning/feed/ |
| etag | null |
| last-modified | Tue, 03 Nov 2026 23:23:13 GMT |
| date | Wed, 04 Nov 2026 06:22:00 GMT |
| age | null |
| cache-control | null |
| cf-cache-status | null |
| content-encoding | null |
| content-length | null |
@
@@ -5,7 +5,7 @@ <atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/> <link>https://aws.amazon.com/blogs/machine-learning/</link> <description>Official Machine Learning Blog of Amazon Web Services</description>-
<lastBuildDate>Mon, 02 Nov 2026 20:28:52 +0000</lastBuildDate>+
<lastBuildDate>Tue, 03 Nov 2026 16:49:29 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -13,313 +13,341 @@ 1 </sy:updateFrequency> <item>-
<title>Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI</title>-
<link>https://aws.amazon.com/blogs/machine-learning/prepare-web-content-for-amazon-bedrock-knowledge-bases-with-crawl4ai/</link>+
<title>Bring structured and unstructured data together with Amazon Quick</title>+
<link>https://aws.amazon.com/blogs/machine-learning/bring-structured-and-unstructured-data-together-with-amazon-quick/</link> -
<dc:creator><![CDATA[Daxeshkumar Patel]]></dc:creator>-
<pubDate>Mon, 02 Nov 2026 17:11:21 +0000</pubDate>-
<category><![CDATA[Advanced (300)]]></category>-
<category><![CDATA[Amazon Bedrock Knowledge Bases]]></category>+
<dc:creator><![CDATA[Puneeth Komaragiri]]></dc:creator>+
<pubDate>Tue, 03 Nov 2026 16:49:29 +0000</pubDate>+
<category><![CDATA[Amazon Quick Suite]]></category>+
<category><![CDATA[Intermediate (200)]]></category> <category><![CDATA[Technical How-to]]></category>-
<guid isPermaLink="false">43ca16f9eb64c74c6edd06ac90851b607a986aa7</guid>+
<guid isPermaLink="false">a3db7b2c3fdf8a48e563f027e982fa25a50c5a02</guid>-
<description>This post shows how to use Crawl4AI, an open source web crawler, as a content preparation layer for Amazon Bedrock Knowledge Bases, handling JavaScript rendering, selective extraction, and custom metadata enrichment before web content reaches an Amazon S3 data source.</description>-
<content:encoded><p>When you build Retrieval Augmented Generation (RAG) applications, answer quality depends on the data that you ingest. <a href="https://aws.amazon.com/bedrock/knowledge-bases/" target="_blank" rel="noopener">Amazon Bedrock Knowledge Bases</a> is the ful…-
<p>For web content, start with the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-ds-webcrawler.html" target="_blank" rel="noopener">Amazon Bedrock Web Crawler</a> data source connector for managed knowledge bases. You give it the seed URLs or sitemaps tha…-
<p>Some sources need more control over what reaches the knowledge base. You might want to index only the main content region of each page, remove navigation and other boilerplate, or attach your own metadata attributes to every document. For that class of source material, you can extend the re…-
<p>In this post, we show you how to choose the appropriate web ingestion path and implement custom preparation for content that needs additional processing. You then start ingestion into a managed knowledge base and validate retrieval quality. The companion <a href="https://github.com/aws-s…-
<p>This is batch ingestion of known web properties, not real-time web access at inference time such as <a href="https://docs.aws.amazon.com/nova/latest/nova2-userguide/web-grounding.html" target="_blank" rel="noopener">Web Grounding</a> with Amazon Nova.</p> -
<h2 id="solution-overview">Solution overview</h2> -
<p>Start with the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-ds-webcrawler.html" target="_blank" rel="noopener">Amazon Bedrock Web Crawler</a> when it can reach and index the pages you need. It respects <code>robots.txt</code>, and after th…-
<p>Use the Crawl4AI preparation path when public source material needs one or more of the following before ingestion:</p> +
<description>Amazon Quick brings structured and unstructured data together in one experience. In this post, you connect Amazon S3 as a knowledge base, enrich a dataset, build a Space, and create a custom chat agent that answers business questions across both data types through a single conversa…+
<content:encoded><p>Organizations collect data in many forms. Structured data lives in databases and spreadsheets, whereas unstructured data sits in documents, reports, and files stored across file systems. Analyzing both together typically requires switching between multiple tools, …+
<p><a href="https://docs.aws.amazon.com/quick/latest/userguide/welcome.html" target="_blank" rel="noopener">Amazon Quick</a> addresses this with a unified experience. You can connect structured data through <a href="https://docs.aws.amazon.com/quick/latest/userguide/working-with…+
<p>In this post, you learn how to:</p> <ul> -
<li><strong>Structured Markdown preparation</strong>: Use <code>DefaultMarkdownGenerator</code> with a pruning content filter to reduce navigation and other boilerplate while preserving headings, lists, and tables. Because pruning is heuristic, review representative ou…-
<li><strong>Bounded, targeted extraction</strong>: Extract only the page region that an optional CSS selector matches. Constrain breadth-first traversal by seed origin, page/path/host scope, depth and page caps, and URL patterns.</li> -
<li><strong>Custom metadata</strong>: Attach source URLs, crawl timestamps, content type, and your own attributes to each document, so that you can filter retrieval results on them.</li> +
<li>Connect Amazon Simple Storage Service (Amazon S3) as a data source and create a knowledge base.</li> +
<li>Build a Space that combines structured and unstructured data.</li> +
<li>Create and configure a custom chat agent that queries across both data types.</li> +
<li>Ask questions that span structured and unstructured sources in a single conversation.</li> </ul> -
<p>Because managed and customer-managed knowledge bases both support Amazon S3 data sources, the Crawl4AI path works with either type. This post uses a managed knowledge base and notes where a customer-managed knowledge base differs.</p> -
<p>The rest of this post follows the companion sample. Treat its architecture and defaults as a starting point, not as requirements of Amazon Bedrock Knowledge Bases.</p> -
<p>By default, the sample runs the AWS Fargate task in a public subnet, where it reaches your configured websites through an internet gateway. Crawl4AI’s role ends once it has prepared the source documents and metadata sidecars. From ingestion onward, Amazon Bedrock Knowledge Bases handles par…+
<h2 id="solution-architecture">Solution architecture</h2> +
<p>The following diagram illustrates how each component connects to deliver a unified data experience through Amazon Quick.</p> <div style="width: 810px" class="wp-caption alignnone">-
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-21438-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-21438-1.png" alt="Architecture diagram sh…-
<p class="wp-caption-text">Figure 1: The Crawl4AI-to-Amazon S3 ingestion path for Amazon Bedrock Knowledge Bases</p>+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-1.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-1.png" alt="Architecture diagram sh…+
<p class="wp-caption-text">Figure 1: Solution architecture for combining structured and unstructured data in Amazon Quick</p></div> -
<p>The numbered steps in the diagram map to the following workflow:</p> +
<p>The architecture has three layers:</p> +
<ul> +
<li><strong>Data layer:</strong> Your data exists in two forms. Unstructured content (documents, PDFs, reports) is stored in Amazon S3 and indexed through a knowledge base. Structured data (metrics, sales figures, and operational key performance indicators) is loaded as enriched d…+
<li><strong>Workspace layer:</strong> A Space brings both data types together into a single workspace. The Space references your knowledge base for unstructured queries and your datasets for structured queries. It can also include dashboards, file uploads, and action connectors. T…+
<li><strong>Agent layer:</strong> The custom chat agent links to the Space and uses it as its knowledge source. When a user asks a question, the agent determines whether to query the knowledge base, the dataset, or both, and returns a combined response with citations. Amazon Quick…+
</ul> +
<p>With this layered approach, you can expand the solution over time. When you add a new data source to the Space, the agent linked to that Space can use it, and responses respect the permissions of the user who asks.</p> +
<h2 id="prerequisites">Prerequisites</h2> +
<p>To follow along with this walkthrough, you need:</p> +
<ul> +
<li>An Amazon Quick account with an Enterprise subscription for the user who follows this walkthrough (creating datasets requires Enterprise). For details, see <a href="https://aws.amazon.com/quick/pricing/" target="_blank" rel="noopener">Amazon Quick pricing</a>.</li> +
<li>An Amazon S3 bucket for your business documents. To create one, see the <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html" target="_blank" rel="noopener">Amazon S3 User Guide</a>.</li> +
<li><a href="https://docs.aws.amazon.com/quick/latest/userguide/s3-admin-setup.html#s3-grant-bucket-access" target="_blank" rel="noopener">Amazon Quick access to your S3 bucket</a>, granted by an Amazon Quick administrator. In the admin console, the administrator chooses Permissio…+
<li>Permissions to create integrations, knowledge bases, datasets, spaces, and chat agents in Amazon Quick.</li> +
</ul> +
<h2 id="prepare-your-sample-data">Prepare your sample data</h2> +
<p>This walkthrough uses two types of data:</p> +
<p><strong>Unstructured documents: </strong>Upload these to your Amazon S3 bucket for the knowledge base:</p> +
<ul> +
<li><a href="https://aws-blogs-artifacts-public.s3.us-east-1.amazonaws.com/BI-7992/AnyCompany_Business_Review.pdf" target="_blank" rel="noopener">AnyCompany_Business_Review.pdf</a></li> +
<li><a href="https://aws-blogs-artifacts-public.s3.us-east-1.amazonaws.com/BI-7992/AnyCompany_Market_Analysis_Report.pdf" target="_blank" rel="noopener">AnyCompany_Market_Analysis_Report.pdf</a></li> +
<li><a href="https://aws-blogs-artifacts-public.s3.us-east-1.amazonaws.com/BI-7992/AnyCompany_Product_Strategy_Memo.pdf" target="_blank" rel="noopener">AnyCompany_Product_Strategy_Memo.pdf</a></li> +
</ul> +
<p><strong>Structured data: </strong>Download this file to your computer. You upload it directly to Amazon Quick in Step 3:</p> +
<ul> +
<li><a href="https://aws-blogs-artifacts-public.s3.us-east-1.amazonaws.com/BI-7992/Product_Sales.csv" target="_blank" rel="noopener">Product_Sales.csv</a></li> +
</ul> +
<h2 id="step-1-create-an-amazon-s3-integration">Step 1: Create an Amazon S3 integration</h2> +
<p>Start by connecting Amazon Quick to your S3 bucket. From the Amazon Quick console, select <strong>Knowledge</strong> in the left navigation panel.</p> +
<p>The knowledge page shows available connector types, including Amazon Q Business, Web Crawler, Amazon S3, and third-party options like Google Drive, OneDrive, and Microsoft SharePoint.</p> +
<p>Under Amazon S3, choose <strong>Add</strong>. In the Create S3 knowledge base dialog, on the Connect S3 bucket page, provide the following details:</p> +
<ul> +
<li><strong>Name:</strong> A descriptive name for your integration (for example, “My Data Access Integration”).</li> +
<li><strong>S3 bucket location:</strong> Choose <strong>Quick Suite instance account</strong> to use a bucket in the same AWS account as Amazon Quick, or <strong>Other AWS account</strong> for a bucket in a different account.</li> +
<li><strong>S3 bucket URL:</strong> The S3 URI of the bucket containing your data. The bucket must be in the same Region as Amazon Quick.</li> +
</ul> +
<p>Choose <strong>Next</strong>.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-2.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-2.png" alt="Create S3 knowledge bas…+
<p class="wp-caption-text">Figure 2: Create integration dialog showing the S3 connection fields</p>+
</div> +
<h2 id="step-2-create-a-knowledge-base">Step 2: Create a knowledge base</h2> +
<p>After the S3 integration is created, Amazon Quick prompts you to create a knowledge base. The knowledge base indexes your S3 content for AI-powered search.</p> +
<p>Configure the knowledge base:</p> +
<ul> +
<li><strong>Name:</strong> “Business Knowledge Base”.</li> +
<li><strong>Description (optional):</strong> Notes about how the knowledge base will be used.</li> +
<li><strong>Content:</strong> Select “Add all content” to index everything in the connected S3 bucket, or “Add specific content” to specify the S3 prefixes for the folders and files to include (prefix filters are case-sensitive).</li> +
</ul> +
<p><strong>Next:</strong> Additional settings opens optional settings for document-level access control lists (ACLs) and a metadata files folder. Decide whether to use ACLs before you create the knowledge base, because you can’t change that choice later. This walkthrough uses sampl…+
<p>Choose <strong>Create</strong>. A success banner confirms the S3 integration, and the knowledge base begins indexing in the background.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-3.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-3.png" alt="Create knowledge base d…+
<p class="wp-caption-text">Figure 3: Create knowledge base dialog with the Business Knowledge Base name and content options</p>+
</div> +
<h2 id="step-3-create-and-enrich-a-dataset">Step 3: Create and enrich a dataset</h2> +
<p>A <a href="https://docs.aws.amazon.com/quick/latest/userguide/working-with-datasets.html" target="_blank" rel="noopener">dataset</a> is your structured data file loaded into Amazon Quick. By enriching the dataset with semantic descriptions, custom instructions, and correct data …+
<h3 id="create-the-dataset">Create the dataset</h3> <ol type="1"> -
<li>Amazon EventBridge starts the Amazon Elastic Container Service (Amazon ECS) task on AWS Fargate on the configured schedule.</li> -
<li>The task crawls the public websites you configured and prepares Markdown with a metadata sidecar for each page.</li> -
<li>The task uploads the prepared files to the Amazon S3 bucket that backs the knowledge base data source.</li> -
<li>The task starts an Amazon Bedrock Knowledge Bases ingestion job.</li> -
<li>Amazon Bedrock Knowledge Bases reads the new and changed objects, parses them, and applies the chunking strategy you selected.</li> -
<li>It creates embeddings and indexes them in the knowledge base’s vector store, which Amazon Bedrock manages for a managed knowledge base.</li> -
<li>Your generative AI application queries the knowledge base for relevant chunks or for a generated response with citations.</li> +
<li>From the Amazon Quick home page, navigate to <strong>Data</strong> in the left navigation.</li> +
<li>Choose <strong>Create dataset</strong>, then select <strong>Upload file</strong>.</li> +
<li>Select the <strong>Product_Sales.csv</strong> file you downloaded earlier and choose <strong>Open</strong>.</li> +
<li>On the confirmation screen, review the column names and data types. Choose <strong>Next</strong> to create the dataset.</li> </ol> -
<p>The task reads its non-secret pipeline configuration from AWS Systems Manager Parameter Store at startup.</p> -
<p>The workflow processes public pages only and doesn’t implement authenticated sessions. For sites that require sign-in, evaluate the <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-ds-webcrawler.html" target="_blank" rel="noopener">Amazon Bedrock Web Crawler au…-
<h2 id="how-the-workflow-works">How the workflow works</h2> -
<p>The AWS Cloud Development Kit (AWS CDK) app packages the implementation in a container image during deployment. When the schedule fires, the container runs <code>scripts/container_entrypoint.py</code> and executes crawl, upload, and ingestion in sequence. The following three ste…-
<h3 id="step-1-crawl-and-extract-content">Step 1: Crawl and extract content</h3> -
<p>The crawl step performs bounded crawling and writes a Markdown file with a metadata sidecar for each successfully prepared page. The following excerpt is simplified. The module <code>src/crawl4ai_bedrock_kb/crawl.py</code> contains the complete seed pacing, URL filtering and sco…+
<p>Amazon Quick ingests the file into SPICE (the in-memory calculation engine) and the dataset appears in your dataset list.</p> +
<p>The dataset takes its name from the uploaded file, so it appears as Product_Sales.csv in your account.</p> +
<h3 id="enrich-the-dataset">Enrich the dataset</h3> +
<p>Enriching your dataset with semantic information helps the AI understand your data in business context. The agent uses this context to translate natural language questions into more accurate SQL queries. From the dataset page, choose <strong>Edit dataset</strong> to enter the da…+
<h3 id="transform-fix-data-types">Transform: Fix data types</h3> +
<p>In the <strong>Transform</strong> tab, update data types for columns that were auto-detected incorrectly. For this dataset:</p> +
<ul> +
<li>Change <strong>Order Date</strong> and <strong>Date Key</strong> from String to Date.</li> +
<li>Change <strong>Customer ID</strong> from String to Integer.</li> +
<li>Change <strong>Sales</strong>, <strong>Discount</strong>, and <strong>Profit</strong> from String to Decimal.</li> +
<li>Change <strong>Quantity</strong> and <strong>Unit Price</strong> from String to Integer.</li> +
</ul> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-4.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-4.png" alt="Transform tab showing d…+
<p class="wp-caption-text">Figure 4: Transform view showing data type changes applied to the Product_Sales.csv dataset</p>+
</div> +
<h3 id="output-add-semantic-context">Output: Add semantic context</h3> +
<p>Select the <strong>Output</strong> tab to access the dataset-level enrichment fields. Here you provide context that helps the AI understand what your data represents and how to use it.</p> +
<p><strong>Description:</strong> Add a dataset-level description:</p> +
<p>“Product sales transaction data for AnyCompany, covering orders from FY2024-FY2025 across North America, EMEA, Asia Pacific, and Latin America. Includes order details, customer information, product lines, and revenue metrics.”</p> +
<p><strong>Custom instructions:</strong> Add guidance for how the AI should query this data:</p> +
<p><strong>USAGE GUIDELINES:</strong></p> +
<ul> +
<li>Revenue values are in USD.</li> +
<li>When comparing performance, group by Region (North America, EMEA, Asia Pacific, Latin America).</li> +
<li>Order Date should be used for time-based trend analysis. The fiscal year matches the calendar year.</li> +
<li>Date Key is a numeric YYYYMMDD representation of Order Date.</li> +
<li>Sales = Quantity x Unit Price x (1 – Discount)</li> +
</ul> +
<p><strong>COMMON USE CASES:</strong></p> +
<ul> +
<li>Regional sales performance comparison.</li> +
<li>Product line revenue analysis.</li> +
<li>Quarterly trend identification.</li> +
<li>Customer segmentation by industry and segment.</li> +
</ul> +
<p><strong>TAGS:</strong> Sales, revenue, orders, regional performance, product analytics</p> +
<p><strong>Column descriptions:</strong> Select each column in the left panel and add a description. For example:</p> +
<ul> +
<li>Order ID: “Unique identifier for each sales transaction (format: <code>REGION-YYYY-NNNNNN</code>)”</li> +
<li>Region: “Geographic sales region: North America, EMEA, Asia Pacific, or Latin America”</li> +
<li>Product: “AnyCompany product name: Analytics Pro, Connect, Automate, AI Assistant, Data Catalog, or Integration Hub”</li> +
<li>Sales: “Order revenue in USD after discount”</li> +
<li>Segment: “Customer size segment: Enterprise, Mid-Market, or SMB”</li> +
</ul> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-5.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-5.png" alt="Output tab showing the …+
<p class="wp-caption-text">Figure 5: Output view showing dataset details with Description and Custom instructions configured</p>+
</div> +
<p>Choose <strong>Save &amp; publish</strong> to save your changes. The dataset summary now shows the columns with their correct data types and the descriptions you added.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-6.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-6.png" alt="Enriched Product_Sales.…+
<p class="wp-caption-text">Figure 6: Enriched Product_Sales.csv dataset summary showing all 16 columns, data types, and descriptions</p>+
</div> +
<p>This enriched dataset provides the structured data layer for your solution. When the agent receives a natural language question about sales data, it uses these descriptions and instructions to generate more accurate SQL queries. Combined with the knowledge base from Step 2, you now have bot…+
<h2 id="step-4-create-a-space">Step 4: Create a Space</h2> +
<p><a href="https://docs.aws.amazon.com/quick/latest/userguide/working-with-spaces.html" target="_blank" rel="noopener">Spaces</a> in Amazon Quick act as unified data workspaces. A Space can hold file uploads, dashboards, knowledge bases, actions, and datasets, giving your chat age…+
<p>From the left navigation, select <strong>Spaces</strong> and create a new Space. Name it “Business Intelligence Hub”.</p> +
<p>Add your knowledge base and dataset to the Space:</p> +
<ol type="1"> +
<li>Choose <strong>Add knowledge</strong> and select <strong>Knowledge bases</strong> from the list. Select “Business Knowledge Base” from the list.</li> +
<li>Choose <strong>Add knowledge</strong> again and select <strong>Datasets</strong> from the list. Select “Product_Sales.csv” from the list.</li> +
</ol> +
<p>The Space now shows both your knowledge base (unstructured documents) and your enriched dataset (structured sales data) linked as data sources. This is the central hub where both data types come together, giving your chat agent a single place to query across everything. Figure 7 shows the S…+
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-7.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-7.png" alt="Business Intelligence H…+
<p class="wp-caption-text">Figure 7: Business Intelligence Hub Space showing the Product_Sales.csv dataset and Business Knowledge Base</p>+
</div> +
<h2 id="step-5-create-a-custom-chat-agent">Step 5: Create a custom chat agent</h2> +
<p>Navigate to <strong>Chat agents</strong> in the left navigation and choose <strong>Create chat agent</strong>. Amazon Quick presents a prompt builder where you can describe your agent in natural language or choose from templates like Training Guide, Project Coordinat…+
<p>For this walkthrough, choose <strong>Skip</strong> to configure the agent manually.</p> +
<div style="width: 702px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-8.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-8.png" alt="Chat agent creation scr…+
<p class="wp-caption-text">Figure 8: Chat agent creation screen with the prompt builder and sample templates</p>+
</div> +
<h3 id="configure-the-agent-persona">Configure the agent persona</h3> +
<p>The configuration panel has several sections. Start with the Agent Persona:</p> +
<ul> +
<li><strong>Agent identity:</strong> “Business Intelligence Assistant”</li> +
<li><strong>Description:</strong> “An AI assistant that breaks down information silos by analyzing business data from your knowledge base”</li> +
</ul> +
<p>For Agent Persona instructions, enter:</p> <div class="hide-language"> -
<pre><code class="language-python">async def crawl_and_prepare(config: PipelineConfig, output_dir: str) -&gt; list[str]:-
written, seen = [], set()-
pages_crawled = 0-
browser_config = BrowserConfig(headless=True, java_script_enabled=True)-
-
async with AsyncWebCrawler(config=browser_config) as crawler:-
for seed in config.seed_urls:-
remaining = config.crawl.max_pages - pages_crawled-
if remaining &lt;= 0:-
break-
-
strategy = BFSDeepCrawlStrategy(-
max_depth=config.crawl.max_depth,-
max_pages=remaining,-
filter_chain=FilterChain([-
SeedScopeFilter(seed, config.crawl.scope),-
# Optional include/exclude URLPatternFilter objects are added here.-
]),-
)-
run_config = CrawlerRunConfig(-
deep_crawl_strategy=strategy,-
markdown_generator=DefaultMarkdownGenerator(-
content_filter=PruningContentFilter()-
),-
css_selector=config.crawl.css_selector, # Optional page region.-
check_robots_txt=config.crawl.check_robots_txt,-
mean_delay=config.crawl.request_delay_seconds,-
max_range=0,-
semaphore_count=1,-
)-
-
results = await crawler.arun(url=seed, config=run_config)-
for result in results:-
if pages_crawled &gt;= config.crawl.max_pages:-
break-
pages_crawled += 1-
requested_url = result.url-
source_url = getattr(result, "redirected_url", None) or requested_url-
if not SeedScopeFilter(seed, config.crawl.scope).apply(source_url):-
continue # Recheck the effective destination after redirects.-
canonical = _canonical_url(source_url)-
if canonical in seen:-
continue-
seen.add(canonical)-
if not result.success or not result.markdown:-
continue+
<pre><code class="language-plaintext">You are a Business Intelligence Assistant designed to help organizations break down information silos. Your primary functions are:-
md_result = result.markdown-
markdown = (-
getattr(md_result, "fit_markdown", None)-
or getattr(md_result, "raw_markdown", None)-
or (md_result if isinstance(md_result, str) else "")-
)-
if not markdown:-
continue+
1. Analyze business data stored in S3+
2. Retrieve relevant findings from the reports and memos in the knowledge base+
3. Provide comprehensive insights that connect internal data with external trends+
4. Generate clear, actionable recommendations based on data analysis-
# The complete implementation adds title, crawl timestamps, content-
# type, and configured custom attributes to this metadata.-
written.extend(-
_write_document(-
output_dir,-
_base_name(source_url),-
markdown,-
{"source_url": source_url},-
)-
)-
return written</code></pre> +
When answering questions, always cite your sources and explain how you arrived at your conclusions. Focus on delivering practical business value and highlighting patterns across different information sources.</code></pre> </div> -
<p>Amazon Bedrock Knowledge Bases supports <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-ds.html" target="_blank" rel="noopener">several document formats</a> for Amazon S3 data sources, including plain text, HTML, Markdown, PDF, and DOCX. The workflow…-
<h3 id="step-2-upload-to-amazon-s3">Step 2: Upload to Amazon S3</h3> -
<p>The upload step sends each prepared Markdown document and its matching metadata sidecar to the configured S3 bucket, under the workflow’s <code>s3Prefix</code>. For <code>page.md</code>, the sidecar must be named <code>page.md.metadata.json</code>, the co…-
<p>Each attribute in the sidecar uses the typed format that both managed and customer-managed knowledge bases accept for Amazon S3 data sources. The following abbreviated sidecar, formatted for readability, shows three of the attributes that the workflow writes:</p> -
<div class="hide-language"> -
<pre><code class="language-json">{-
"metadataAttributes": {-
"source_url": {-
"value": {"type": "STRING", "stringValue": "https://www.example.com/services/permits"}-
},-
"crawled_at_epoch": {-
"value": {"type": "NUMBER", "numberValue": 1790640000}-
},-
"content_category": {-
"value": {"type": "STRING", "stringValue": "web"}-
}-
}-
}</code></pre> +
<p>A note appears confirming that your chat agent comes equipped with enterprise guardrails.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-9.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-9.png" alt="Agent persona configura…+
<p class="wp-caption-text">Figure 9: Agent persona configuration with identity and persona instructions filled in</p></div> -
<p>The pipeline uploads only the paths produced by the current crawl, so stale local files are not sent.</p> -
<h3 id="step-3-let-amazon-bedrock-knowledge-bases-chunk-embed-and-index">Step 3: Let Amazon Bedrock Knowledge Bases chunk, embed, and index</h3> -
<p>After content is in S3, an ingestion job hands each document to the data source’s ingestion pipeline. By default, a managed knowledge base parses each file with its managed parser, which automatically selects a parsing approach for your content. You select the chunking strategy when you con…+
<h3 id="set-communication-style">Set communication style</h3> +
<p>Under Communication style, pick a response style preset. Options include Executive, Technical, and Creative. For a business intelligence use case, select <strong>Executive</strong>.</p> +
<h3 id="link-knowledge-sources">Link knowledge sources</h3> +
<p>Under <strong>Knowledge Sources</strong>, choose <strong>Link spaces</strong> and select the “Business Intelligence Hub” Space you created earlier. This gives the agent access to your S3 knowledge base and other data sources in that Space. When you link a Space, the …+
<p>You can also attach reference documents (PDF, TXT, HTML, MD, CSV, DOC, DOCX) directly to the agent for additional context. Amazon Quick extracts up to 100,000 characters of text from the reference documents that you upload.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-10.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-10.png" alt="Knowledge Sources sec…+
<p class="wp-caption-text">Figure 10: Knowledge Sources section showing the linked Business Intelligence Hub Space</p>+
</div> +
<h3 id="customize-the-agent">Customize the agent</h3> +
<p>Under Customization, configure:</p> <ul> -
<li><strong>Default chunking</strong> uses fixed-size chunks of 300 tokens with 20 percent overlap.</li> -
<li><strong>Fixed-size chunking</strong> splits content by a maximum token count and an overlap percentage that you set.</li> -
<li><strong>No chunking</strong> treats each file as a single chunk, for content that you have already split.</li> +
<li><strong>Welcome message:</strong> “Hello! I’m your Business Intelligence Assistant. I can help you analyze your sales data and business documents and provide insights by comparing internal information with market trends. How can I help you today?”</li> +
<li><strong>Suggested prompts:</strong> “Summarize key insights from our reports” and “Analyze our quarterly sales performance”</li> </ul> -
<p>Chunk boundaries and retrieval results depend on the content and the selected strategy, so compare the options with representative pages and evaluation questions. To change strategies later, create a new data source. For more information, see <a href="https://docs.aws.amazon.com/bedrock/…-
<p>The workflow then starts an ingestion job so the knowledge base processes the new documents. The following simplified excerpt from <code>src/crawl4ai_bedrock_kb/ingest.py</code> shows the call:</p> -
<div class="hide-language"> -
<pre><code class="language-python">def start_ingestion_job(config: PipelineConfig) -&gt; str:-
# Start an Amazon Bedrock knowledge base ingestion job and return its ID.-
client = boto3.client("bedrock-agent", region_name=config.region)-
response = client.start_ingestion_job(-
knowledgeBaseId=config.knowledge_base_id,-
dataSourceId=config.data_source_id,-
)-
return response["ingestionJob"]["ingestionJobId"]</code></pre> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-11.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-11.png" alt="Customization section…+
<p class="wp-caption-text">Figure 11: Customization section with the welcome message and suggested prompts</p></div> -
<p>The same <code>StartIngestionJob</code> and <code>GetIngestionJob</code> calls work for both knowledge base types, and you can also monitor progress on the console.</p> -
<h2 id="deploy-the-solution">Deploy the solution</h2> -
<p>In this section, you deploy the workflow into your AWS account with AWS CDK. Only the knowledge base prerequisites differ between the two knowledge base types. When you’re finished, follow the steps in the Clean up section to remove the resources.</p> -
<h3 id="prerequisites">Prerequisites</h3> -
<p>Before you start, make sure that you have:</p> +
<h3 id="launch-the-agent">Launch the agent</h3> +
<p>Choose <strong>Launch chat agent</strong>. The right panel shows a live preview of your agent with the welcome message and suggested prompts displayed.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-12.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-12.png" alt="Launched Business Int…+
<p class="wp-caption-text">Figure 12: Launched Business Intelligence Assistant with the welcome message and suggested prompts in the chat panel</p>+
</div> +
<h2 id="step-6-query-across-structured-and-unstructured-data">Step 6: Query across structured and unstructured data</h2> +
<p>With the agent running, start with a question about your business results:</p> +
<p>How did sales change by region in 2025?</p> +
<p>For this question, the agent combines results from the Product_Sales.csv dataset with the documents in the S3 knowledge base. It returns a response with regional sales breakdowns, trends, and citations that point back to each source.</p> +
<p>Figure 13 shows the response. Each citation marker links a figure back to its source document.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-13.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-13.png" alt="Chat agent response w…+
<p class="wp-caption-text">Figure 13: Chat agent response showing regional sales analysis from the dataset and the knowledge base documents</p>+
</div> +
<p>When you ask a question about structured data, the agent uses text-to-SQL to translate your natural language into a SQL query. It runs the query against the dataset and returns the results in a readable format. This means you can ask complex analytical questions without writing SQL yourself…+
<p>Next, ask about a specific document:</p> +
<p>What are the key strategic priorities outlined in our product strategy memo?</p> +
<p>The agent summarizes the strategic pillars, the FY2026 roadmap, and the investment requirements from the Product Strategy Memo, as shown in Figure 14.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-14.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-14.png" alt="Chat agent response s…+
<p class="wp-caption-text">Figure 14: Chat agent response summarizing strategic priorities from the Product Strategy Memo</p>+
</div> +
<p>Finally, ask a question that needs both data types:</p> +
<p>How do our 2025 sales results align with the market trends identified in our analysis report?</p> +
<p>The agent compares the trends in the Market Analysis Report with results from your sales data and cites each source, as shown in Figure 15.</p> +
<div style="width: 810px" class="wp-caption alignnone">+
<a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-15.png" target="_blank" rel="noopener"><img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/27/ML-22035-15.png" alt="Chat agent response t…+
<p class="wp-caption-text">Figure 15: Chat agent response comparing 2025 sales results with market trends from the Market Analysis Report</p>+
</div> +
<p>This is the core value of combining data types in a single Space. The agent can pull from structured CSV data and unstructured documents in the same response. You don’t need to know where the data lives.</p> +
<h2 id="step-7-generate-artifacts-and-reports-using-natural-language">Step 7: Generate artifacts and reports using natural language</h2> +
<p>The Business Intelligence Assistant can also generate formatted documents, including Microsoft PowerPoint presentations, Microsoft Excel spreadsheets, and Microsoft Word documents. Try these prompts in the chat agent:</p> <ul> -
<li>An AWS account.</li> -
<li>One of the following knowledge bases: -
<ul> -
<li><strong>A managed knowledge base</strong>, which this post uses, in an AWS Region where <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/kb-managed-regions.html" target="_blank" rel="noopener">managed knowledge bases are available</a>. For instructio…Diff display stops at 400 lines. The line counts above are from the whole diff. 47 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.