llm-catalog-archive

Change

7916599

7916599d793f74f7545a77bf27f2f032bc5a42f9 · commit on GitHub

aws-blog-feed: changed (683172 bytes, HTTP 200)

raw/aws-blog-feed/response.xml modified

Lines added
+1,757
Lines removed
-301
Stored bytes at this commit
683,172
Timestamp
observed
Raw artifact at this commit
raw/aws-blog-feed/response.xml
Recorded headers
observed_at2026-09-16T04:50:06.566Z
origin_datenull
status200
final URLhttps://aws.amazon.com/blogs/machine-learning/feed/
etagnull
last-modifiedTue, 15 Sep 2026 23:52:54 GMT
dateWed, 16 Sep 2026 04:50:06 GMT
agenull
cache-controlnull
cf-cache-statusnull
content-encodingnull
content-lengthnull
@@@ -5,7 +5,7 @@
<atom:link href="https://aws.amazon.com/blogs/machine-learning/feed/" rel="self" type="application/rss+xml"/>
<link>https://aws.amazon.com/blogs/machine-learning/</link>
<description>Official Machine Learning Blog of Amazon Web Services</description>
- <lastBuildDate>Mon, 14 Sep 2026 21:22:45 +0000</lastBuildDate>
+ <lastBuildDate>Tue, 15 Sep 2026 16:42:46 +0000</lastBuildDate>
<language>en-US</language>
<sy:updatePeriod>
hourly </sy:updatePeriod>
@@@ -13,6 +13,1755 @@
1 </sy:updateFrequency>
<item>
+ <title>Optimizing cost and latency with Amazon Bedrock prompt caching</title>
+ <link>https://aws.amazon.com/blogs/machine-learning/optimizing-cost-and-latency-with-amazon-bedrock-prompt-caching/</link>
+
+ <dc:creator><![CDATA[Daniel Abib]]></dc:creator>
+ <pubDate>Tue, 15 Sep 2026 16:18:19 +0000</pubDate>
+ <category><![CDATA[Advanced (300)]]></category>
+ <category><![CDATA[Amazon Bedrock]]></category>
+ <category><![CDATA[Technical How-to]]></category>
+ <guid isPermaLink="false">c9f41d7f970343e8cc614d389738c9638a75056d</guid>
+
+ <description>Prompt caching in Amazon Bedrock can cut input token costs by up to 90% when you repeatedly send the same context to foundation models. This post walks through six practical prompt caching scenarios using the Converse API: message content, system prompt, tool definition, mixed TTL,
+ <content:encoded>&lt;p&gt;Prompt caching in Amazon Bedrock can reduce your input token costs by up to 90 percent when you repeatedly send the same context to foundation models, based on &lt;a href="https://aws.amazon.com/bedrock/pricing/" target="_blank" rel="noopener"&gt;Amazon Bedrock pr
+&lt;p&gt;You can mitigate this issue by shortening prompts, reducing context windows, or implementing application-level caching. Each option involves a trade-off:&lt;/p&gt;
+&lt;ul&gt;
+ &lt;li&gt;&lt;strong&gt;Shortened prompts&lt;/strong&gt; reduce token count but might also reduce context quality.&lt;/li&gt;
+ &lt;li&gt;&lt;strong&gt;Smaller context windows&lt;/strong&gt; lower cost at the expense of the model’s ability to reason over complete information.&lt;/li&gt;
+ &lt;li&gt;&lt;strong&gt;Response caching&lt;/strong&gt; handles identical queries well, yet provides no benefit when the same context is paired with different questions.&lt;/li&gt;
+&lt;/ul&gt;
+&lt;p&gt;Prompt caching in Amazon Bedrock helps reduce this challenge at the infrastructure level. When you cache parts of your conversation context (system prompts, documents, tool definitions), Amazon Bedrock reads the cached tokens on subsequent requests instead of reprocessing them. This can red
+&lt;p&gt;This post walks through six practical prompt caching scenarios using the Converse API in Amazon Bedrock, progressing from basic to advanced patterns:&lt;/p&gt;
+&lt;ol type="1"&gt;
+ &lt;li&gt;&lt;strong&gt;Message content caching&lt;/strong&gt;: Cache long documents for multi-question analysis.&lt;/li&gt;
+ &lt;li&gt;&lt;strong&gt;System prompt caching&lt;/strong&gt;: Cache persona definitions and instructions across conversations.&lt;/li&gt;
+ &lt;li&gt;&lt;strong&gt;Tool definition caching&lt;/strong&gt;: Cache tool schemas for agentic workflows.&lt;/li&gt;
+ &lt;li&gt;&lt;strong&gt;Mixed TTL caching&lt;/strong&gt;: Assign different cache lifetimes to different content tiers.&lt;/li&gt;
+ &lt;li&gt;&lt;strong&gt;Tenant isolation&lt;/strong&gt;: Implement per-tenant cache separation in multi-tenant applications.&lt;/li&gt;
+ &lt;li&gt;&lt;strong&gt;LangChain integration&lt;/strong&gt;: Use prompt caching with the LangChain framework.&lt;/li&gt;
+&lt;/ol&gt;
+&lt;h2 id="how-prompt-caching-works"&gt;How prompt caching works&lt;/h2&gt;
+&lt;p&gt;Prompt caching stores a snapshot of partially processed input so that subsequent requests with the same prefix skip redundant computation. This section covers the request flow, supported models, and pricing.&lt;/p&gt;
+&lt;p&gt;When you include a &lt;code&gt;cachePoint&lt;/code&gt; marker in your request, Amazon Bedrock evaluates whether the content preceding that marker matches an existing cache entry. If it does (a &lt;em&gt;cache hit&lt;/em&gt;), the model can skip reprocessing those tokens and begin generation
+&lt;p&gt;This diagram shows the flow:&lt;/p&gt;
+&lt;div style="width: 810px" class="wp-caption alignnone"&gt;
+ &lt;a href="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/11/ML-20790-1.png" target="_blank" rel="noopener"&gt;&lt;img src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/09/11/ML-20790-1.png" alt="Prompt caching request
+ &lt;p class="wp-caption-text"&gt;Figure 1: Prompt caching request flow, where the first request writes to cache and the second request reads from cache, reducing TTFT and input token cost&lt;/p&gt;
+&lt;/div&gt;
+&lt;p&gt;With this flow in mind, four key concepts determine how caching behaves in practice:&lt;/p&gt;
+&lt;ol type="1"&gt;
+ &lt;li&gt;&lt;strong&gt;Cache scope&lt;/strong&gt;: Cache entries are scoped to individual AWS accounts and AWS Regions.&lt;/li&gt;
+ &lt;li&gt;&lt;strong&gt;Token thresholds&lt;/strong&gt;: Each cache checkpoint must meet a minimum token threshold to activate. For example, Anthropic Claude Sonnet 4.5 and Sonnet 4.6 require at least 1,024 tokens per checkpoint, while Opus models require at least 4,096.&lt;/li&gt;
+ &lt;li&gt;&lt;strong&gt;Time-to-live (TTL)&lt;/strong&gt;: Cache entries expire based on the TTL specified in the request. The default is 5 minutes, with select models supporting up to 1 hour.&lt;/li&gt;
+ &lt;li&gt;&lt;strong&gt;Model-agnostic syntax&lt;/strong&gt;: The Converse API &lt;code&gt;cachePoint&lt;/code&gt; syntax is identical across supported model families, including Anthropic Claude and Amazon Nova.&lt;/li&gt;
+&lt;/ol&gt;
+&lt;p&gt;For the latest model support information, see the &lt;a href="https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html" target="_blank" rel="noopener"&gt;Amazon Bedrock Prompt Caching documentation&lt;/a&gt;.&lt;/p&gt;
+&lt;h3 id="pricing"&gt;Pricing&lt;/h3&gt;
+&lt;p&gt;Prompt caching introduces two token categories in addition to standard input and output tokens:&lt;/p&gt;
+&lt;table border="1px" width="100%" cellpadding="10px"&gt;
+ &lt;tbody&gt;
+ &lt;tr&gt;
+ &lt;td&gt;&lt;strong&gt;Token Type&lt;/strong&gt;&lt;/td&gt;
+ &lt;td&gt;&lt;strong&gt;Description&lt;/strong&gt;&lt;/td&gt;
+ &lt;td&gt;&lt;strong&gt;Cost vs Standard Input&lt;/strong&gt;&lt;/td&gt;
+ &lt;/tr&gt;
+ &lt;tr&gt;
+ &lt;td&gt;&lt;code&gt;cacheWriteInputTokens&lt;/code&gt;&lt;/td&gt;
+ &lt;td&gt;Tokens written to cache (first request)&lt;/td&gt;
+ &lt;td&gt;25% higher than standard input&lt;/td&gt;
+ &lt;/tr&gt;
+ &lt;tr&gt;
+ &lt;td&gt;&lt;code&gt;cacheReadInputTokens&lt;/code&gt;&lt;/td&gt;
+ &lt;td&gt;Tokens read from cache (subsequent requests)&lt;/td&gt;
+ &lt;td&gt;90% lower than standard input&lt;/td&gt;
+ &lt;/tr&gt;
+ &lt;tr&gt;
+ &lt;td&gt;&lt;code&gt;cacheWriteInputTokens (1-hour TTL)&lt;/code&gt;&lt;/td&gt;
+ &lt;td&gt;Tokens written to cache with 1-hour TTL&lt;/td&gt;
+ &lt;td&gt;100% higher than standard input (2x)&lt;/td&gt;
+ &lt;/tr&gt;
+ &lt;/tbody&gt;
+&lt;/table&gt;
+&lt;p&gt;For workloads with repeated context, the savings reach approximately 75 percent on input token costs. For example, if you send a 10,000-token document with 10 different questions, the first request incurs a cache write cost. The remaining nine requests each read from cache at 90 percent red
+&lt;h2 id="prerequisites"&gt;Prerequisites&lt;/h2&gt;
+&lt;p&gt;Before getting started with the scenarios, make sure you have the following:&lt;/p&gt;
+&lt;ol type="1"&gt;
+ &lt;li&gt;An AWS account with Amazon Bedrock access in a supported AWS Region (such as &lt;code&gt;us-west-2&lt;/code&gt;).&lt;/li&gt;
+ &lt;li&gt;Model access enabled for the target model. The examples in this post use Anthropic Claude Sonnet 4.5 (&lt;code&gt;global.anthropic.claude-sonnet-4-5-20250929-v1:0&lt;/code&gt;). See &lt;a href="https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html" target="_blank" rel="no
+ &lt;li&gt;Python 3.10 or later with the following dependencies installed:
+ &lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-bash"&gt;pip install boto3&amp;gt;=1.43.0 langchain-aws&amp;gt;=0.2.12 matplotlib pandas&lt;/code&gt;&lt;/pre&gt;
+ &lt;/div&gt; &lt;/li&gt;
+&lt;/ol&gt;
+&lt;blockquote&gt;
+ &lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; Boto3 1.43.0 or later is required for the &lt;code&gt;ttl&lt;/code&gt; parameter in &lt;code&gt;cachePoint&lt;/code&gt; used in Scenario 4 (Mixed TTL).&lt;/p&gt;
+&lt;/blockquote&gt;
+&lt;ol start="4" type="1"&gt;
+ &lt;li&gt;AWS credentials configured through the default profile or environment variables. See &lt;a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html" target="_blank" rel="noopener"&gt;Configure the AWS Command Line Interface (AWS CLI)&lt;/a&gt; for setup instruc
+&lt;/ol&gt;
+&lt;h2 id="scenario-1-message-content-caching"&gt;Scenario 1: Message content caching&lt;/h2&gt;
+&lt;p&gt;A common use case for prompt caching is caching long documents or reference content that you query repeatedly. For example, in a Retrieval Augmented Generation (RAG) application, you ask multiple questions about the same document, or a coding assistant references a large codebase.&lt;/p&gt;
+&lt;p&gt;In this scenario, you place a &lt;code&gt;cachePoint&lt;/code&gt; marker between the static document and the dynamic question. Amazon Bedrock caches the document on the first call and might reuse it on subsequent calls.&lt;/p&gt;
+&lt;h3 id="how-message-content-caching-works"&gt;How message content caching works&lt;/h3&gt;
+&lt;p&gt;Place a &lt;code&gt;cachePoint&lt;/code&gt; content block after the static content and before the dynamic question. Amazon Bedrock caches everything before the checkpoint and reuses it on subsequent requests:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-python"&gt;content = [
+ {"text": "&amp;lt;static document content&amp;gt;"},
+ {"cachePoint": {"type": "default"}}, # cache everything above
+ {"text": "&amp;lt;user question&amp;gt;"} # dynamic, changes per request
+]&lt;/code&gt;&lt;/pre&gt;
+&lt;/div&gt;
+&lt;p&gt;The following code puts this pattern into practice with a complete working example.&lt;/p&gt;
+&lt;h3 id="implementation"&gt;Implementation&lt;/h3&gt;
+&lt;p&gt;First, set up the Amazon Bedrock runtime client and define a sample document. In a production application, this document can be a PDF, a knowledge base article, or other content exceeding the 1,024-token threshold:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-python"&gt;import boto3
+import json
+import time
+
+MODEL_ID = "global.anthropic.claude-sonnet-4-5-20250929-v1:0" # Minimum token requirement to activate cache: 1,024 tokens
+AWS_REGION = "us-west-2"
+
+bedrock = boto3.client("bedrock-runtime", region_name=AWS_REGION)
+
+# Document LARGE enough (&amp;gt;1024 tokens required for Claude Sonnet 4.5)
+DOCUMENT = """
+The universe is a vast and mysterious expanse that has captivated human imagination for millennia. From the earliest civilizations who looked up at the night sky and wondered about the nature of the stars, to modern astronomers using sophisticated telescopes and spacecraft to explore distant galaxie
+
+Our solar system, located in the Milky Way galaxy, is home to eight planets, numerous dwarf planets, and countless smaller objects including asteroids, comets, and meteoroids. The Sun, a middle-aged G-type main-sequence star, provides the energy that sustains life on Earth and influences the dynamic
+
+Mercury, the innermost planet, experiences extreme temperature variations due to its proximity to the Sun and lack of substantial atmosphere. Venus, often called Earth's twin due to its similar size, has a thick atmosphere composed primarily of carbon dioxide, creating a runaway greenhouse effect th
+
+Mars, the Red Planet, has long been a subject of fascination and speculation about the possibility of extraterrestrial life. Its rusty appearance comes from iron oxide prevalent on its surface. The planet features the largest volcano in the solar system, Olympus Mons, and a canyon system, Valles Mar
+
+The asteroid belt, located between Mars and Jupiter, contains millions of rocky objects ranging from small boulders to the dwarf planet Ceres. These remnants from the early solar system provide valuable insights into planetary formation and the conditions that existed billions of years ago. Scientis
+
+The terrestrial planets share common characteristics: rocky compositions, relatively small sizes compared to gas giants, and solid surfaces. Mercury's heavily cratered surface resembles our Moon, preserving a record of impacts from the early solar system. Venus's dense atmosphere traps heat so effec
+
+Jupiter, the largest planet, is a gas giant composed primarily of hydrogen and helium. Its Great Red Spot, a persistent anticyclonic storm, has been observed for over 400 years. Jupiter's intense magnetic field and numerous moons, including the four Galilean satellites discovered by Galileo Galilei
+
+Saturn, famous for its spectacular ring system, is another gas giant with dozens of moons. Titan, its largest moon, has a thick atmosphere and liquid hydrocarbon lakes, making it one of the most intriguing bodies in the solar system for astrobiological research. The Cassini-Huygens mission provided
+
+Uranus and Neptune, the ice giants, reside in the outer reaches of our solar system. Uranus rotates on its side, likely due to a massive impact early in its history. Neptune, the windiest planet, features storms with wind speeds exceeding 2,000 kilometers per hour. Both planets have ring systems, th
+
+Beyond Neptune lies the Kuiper Belt, a region populated by icy bodies including the dwarf planet Pluto. The New Horizons mission's flyby of Pluto in 2015 revealed a geologically active world with nitrogen glaciers and a hazy atmosphere. Even further out is the Oort Cloud, a hypothetical spherical sh
+
+The outer solar system remains largely unexplored compared to the inner planets. Only Voyager 2 has visited both Uranus and Neptune, conducting brief flybys in the 1980s. Future missions are being planned to study these ice giants in more detail, potentially including orbiters and atmospheric probes
+
+Gas giants and ice giants differ fundamentally in composition. While Jupiter and Saturn are primarily hydrogen and helium, Uranus and Neptune contain significant amounts of water, ammonia, and methane ices. This distinction gives the ice giants their characteristic blue-green colors and different in
+
+Exoplanet research has revolutionized our understanding of planetary systems. The Kepler space telescope discovered thousands of planets orbiting other stars, revealing that planets are common throughout our galaxy. Some of these exoplanets reside in the habitable zone of their stars, where liquid w
+
+The search for extraterrestrial intelligence, known as SETI, uses radio telescopes to listen for signals from advanced civilizations. While no definitive signals have been detected, the Drake Equation provides a framework for estimating the number of communicating civilizations in our galaxy.
+
+Black holes, regions of spacetime where gravity is so strong that nothing can escape, represent some of the most extreme objects in the universe. Stellar black holes form from the collapse of massive stars, while supermassive black holes, containing millions to billions of solar masses, reside at th
+
+The James Webb Space Telescope, launched in 2021, observes in infrared to study the earliest galaxies and probe planetary atmospheres for signs of life. Understanding the universe requires collaboration across disciplines including physics, chemistry, biology, and engineering.
+
+Stellar evolution describes how stars change over their lifetimes. Stars form in molecular clouds when gravity causes dense regions to collapse. Nuclear fusion in the core converts hydrogen to helium, releasing enormous amounts of energy. When stars exhaust their nuclear fuel, their fate depends on
+
+Galaxies, containing billions of stars, come in various shapes including spiral, elliptical, and irregular. The Milky Way is a barred spiral galaxy approximately 100,000 light-years in diameter. Galaxies often cluster together, forming groups and superclusters connected by cosmic filaments of dark m
+
+The cosmic microwave background radiation, discovered in 1965, provides a snapshot of the universe approximately 380,000 years after the Big Bang. Detailed measurements of this radiation have confirmed the Big Bang theory and revealed information about the early universe's composition and geometry.
+
+Space exploration has achieved remarkable milestones since the launch of Sputnik in 1957. Human spaceflight began with Yuri Gagarin's orbit in 1961 and culminated in the Apollo Moon landings. The International Space Station has hosted continuous human presence in space since 2000. Future missions ai
+
+The study of astrobiology examines the origin, evolution, and distribution of life in the universe. Scientists search for biosignatures in planetary atmospheres and analyze extremophiles on Earth to understand the limits of life. The discovery of organic molecules on Mars and in the plumes of Encela
+
+Gravitational wave astronomy represents one of the newest frontiers in space science. The LIGO and Virgo detectors have observed mergers of black holes and neutron stars, confirming predictions from Einstein's general theory of relativity. These observations have opened an entirely new window on the
+"""&lt;/code&gt;&lt;/pre&gt;
+&lt;/div&gt;
+&lt;p&gt;Next, define the caching function. The key elements are the &lt;code&gt;cachePoint&lt;/code&gt; block placed between the static document and the dynamic question:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-python"&gt;def converse_with_cache(document, question):
+ """Query a document with prompt caching enabled."""
+ content = [
+ {"text": document},
+ {"cachePoint": {"type": "default"}},
+ {"text": question}
+ ]
+
+ response = bedrock.converse(
+ modelId=MODEL_ID,
+ messages=[{"role": "user", "content": content}],
+ inferenceConfig={"maxTokens": 512}
+ )
+ return response["usage"]&lt;/code&gt;&lt;/pre&gt;
+&lt;/div&gt;
+&lt;p&gt;Run two requests to observe the caching behavior. The initial call populates the cache, and a subsequent call with a different question reuses it:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-python"&gt;# Request 1: cache write (first time seeing this content)
+usage1 = converse_with_cache(DOCUMENT, "What are the key points?")
+print("Request 1 (cache write expected):")
+print(json.dumps(usage1, indent=2))
+
+time.sleep(1)
+
+# Request 2: cache read (same document, different question)
+usage2 = converse_with_cache(DOCUMENT, "Summarize the conclusions.")
+print("\nRequest 2 (cache read expected):")
+print(json.dumps(usage2, indent=2))&lt;/code&gt;&lt;/pre&gt;
+&lt;/div&gt;
+&lt;h3 id="reading-cache-metrics"&gt;Reading cache metrics&lt;/h3&gt;
+&lt;p&gt;The response &lt;code&gt;usage&lt;/code&gt; object includes two cache-specific fields:&lt;/p&gt;
+&lt;table border="1px" width="100%" cellpadding="10px"&gt;
+ &lt;tbody&gt;
+ &lt;tr&gt;
+ &lt;td&gt;&lt;strong&gt;Field&lt;/strong&gt;&lt;/td&gt;
+ &lt;td&gt;&lt;strong&gt;Description&lt;/strong&gt;&lt;/td&gt;
+ &lt;/tr&gt;
+ &lt;tr&gt;
+ &lt;td&gt;&lt;code&gt;cacheWriteInputTokens&lt;/code&gt;&lt;/td&gt;
+ &lt;td&gt;Tokens written to cache (appears on first request)&lt;/td&gt;
+ &lt;/tr&gt;
+ &lt;tr&gt;
+ &lt;td&gt;&lt;code&gt;cacheReadInputTokens&lt;/code&gt;&lt;/td&gt;
+ &lt;td&gt;Tokens read from cache (appears on subsequent requests)&lt;/td&gt;
+ &lt;/tr&gt;
+ &lt;/tbody&gt;
+&lt;/table&gt;
+&lt;p&gt;In our testing with Anthropic Claude Sonnet 4.5 and a document exceeding 1,024 tokens, the initial response shows a cache write:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-json"&gt;{
+ "inputTokens": 28,
+ "outputTokens": 253,
+ "cacheWriteInputTokens": 1898,
+ "cacheReadInputTokens": 0
+}&lt;/code&gt;&lt;/pre&gt;
+&lt;/div&gt;
+&lt;p&gt;A subsequent request with the same document prefix produces a cache read:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-json"&gt;{
+ "inputTokens": 28,
+ "outputTokens": 294,
+ "cacheWriteInputTokens": 0,
+ "cacheReadInputTokens": 1898
+}&lt;/code&gt;&lt;/pre&gt;
+&lt;/div&gt;
+&lt;p&gt;Notice that &lt;code&gt;cacheReadInputTokens&lt;/code&gt; now reflects the 1,898 tokens read from cache. The entire document prefix was reused without reprocessing. Only 28 tokens (the question itself) were processed as standard input. These cached tokens are billed at the reduced cache-rea
+&lt;h3 id="simplified-cache-management"&gt;Simplified cache management&lt;/h3&gt;
+&lt;p&gt;Claude models on Amazon Bedrock support &lt;em&gt;simplified cache management&lt;/em&gt;. You can place a single &lt;code&gt;cachePoint&lt;/code&gt;, and Amazon Bedrock automatically checks for cache hits on prefixes up to approximately 20 content blocks before that marker. You do not need
+&lt;p&gt;For more granular control, you can place multiple &lt;code&gt;cachePoint&lt;/code&gt; markers after each section of content:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-python"&gt;content = [
+ {"text": SECTION_1},
+ {"cachePoint": {"type": "default"}},
+ {"text": SECTION_2},
+ {"cachePoint": {"type": "default"}},
+ {"text": SECTION_3},
+ {"cachePoint": {"type": "default"}},
+ {"text": QUESTION}
+]&lt;/code&gt;&lt;/pre&gt;
+&lt;/div&gt;
+&lt;p&gt;This approach supports partial cache hits. If only the first two sections match a previous request, the model reuses the cache for those sections and processes the remaining content.&lt;/p&gt;
+&lt;h3 id="streaming-variant"&gt;Streaming variant&lt;/h3&gt;
+&lt;p&gt;The same caching syntax works with &lt;code&gt;converse_stream&lt;/code&gt;. The key difference is that cache metrics arrive in the &lt;code&gt;metadata&lt;/code&gt; event at the end of the stream rather than in the immediate response:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-python"&gt;def converse_stream_with_cache(document, question):
+ content = [
+ {"text": document},
+ {"cachePoint": {"type": "default"}},
+ {"text": question}
+ ]
+
+ response = bedrock.converse_stream(
+ modelId=MODEL_ID,
+ messages=[{"role": "user", "content": content}],
+ inferenceConfig={"maxTokens": 512}
+ )
+
+ text = ""
+ usage = {}
+ for event in response["stream"]:
+ if "contentBlockDelta" in event:
+ text += event["contentBlockDelta"]["delta"].get("text", "")
+ elif "metadata" in event:
+ usage = event["metadata"].get("usage", {})
+
+ return usage, text&lt;/code&gt;&lt;/pre&gt;
+&lt;/div&gt;
+&lt;h3 id="ttft-benchmark"&gt;TTFT benchmark&lt;/h3&gt;
+&lt;p&gt;To quantify the latency improvement, you can measure TTFT with and without caching:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-python"&gt;def measure_ttft(document, question, use_cache=True, iterations=5):
+ """Measure TTFT using ConverseStream."""
+ ttfts = []
+ for i in range(iterations):
+ if use_cache:
+ content = [
+ {"text": document},
+ {"cachePoint": {"type": "default"}},
+ {"text": question}
+ ]
+ else:
+ content = [{"text": document + "\n\n" + question}]
+
+ start = time.time()
+ response = bedrock.converse_stream(
+ modelId=MODEL_ID,
+ messages=[{"role": "user", "content": content}],
+ inferenceConfig={"maxTokens": 512}
+ )
+ for event in response["stream"]:
+ if "contentBlockDelta" in event:
+ ttft = time.time() - start
+ ttfts.append(ttft)
+ for _ in response["stream"]:
+ pass
+ break
+ time.sleep(0.5)
+ return ttfts&lt;/code&gt;&lt;/pre&gt;
+&lt;/div&gt;
+&lt;p&gt;Prompt caching can reduce TTFT, with the benefit growing as the cached prefix size increases. For smaller documents (approximately 2,000–5,000 tokens), the improvement may not be statistically significant across a small number of iterations. The exact improvement varies based on document si
+&lt;h2 id="scenario-2-system-prompt-caching"&gt;Scenario 2: System prompt caching&lt;/h2&gt;
+&lt;p&gt;Many applications use detailed system prompts that define the model’s persona, guidelines, and domain expertise. These system prompts can span thousands of tokens and remain constant across user interactions. With system prompt caching, you pay the full processing cost once and reuse the ca
+&lt;p&gt;The cache point goes inside the &lt;code&gt;system&lt;/code&gt; parameter, separate from user messages.&lt;/p&gt;
+&lt;h3 id="configuration"&gt;Configuration&lt;/h3&gt;
+&lt;p&gt;The Converse API &lt;code&gt;system&lt;/code&gt; parameter accepts an array of content blocks. Place a &lt;code&gt;cachePoint&lt;/code&gt; after the system text:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-python"&gt;system = [
+ {"text": "&amp;lt;detailed system prompt&amp;gt;"},
+ {"cachePoint": {"type": "default"}}
+]&lt;/code&gt;&lt;/pre&gt;
+&lt;/div&gt;
+&lt;p&gt;The next example shows this in a full request with a detailed persona prompt.&lt;/p&gt;
+&lt;h3 id="implementation-1"&gt;Implementation&lt;/h3&gt;
+&lt;p&gt;The following example defines a comprehensive system prompt: an Expert Space Science Advisor persona with detailed response guidelines. The prompt exceeds the 2,048-token threshold required for caching:&lt;/p&gt;
+&lt;div class="hide-language"&gt;
+ &lt;pre&gt;&lt;code class="language-python"&gt;SYSTEM_PROMPT = """You are an Expert Space Science Advisor, a highly knowledgeable
+AI assistant specializing in astronomy, astrophysics, planetary science, and space
+exploration. Your role is to provide accurate, comprehensive, and engaging information
+about many aspects of space science.
+
+## Core Expertise Areas
+
+### Planetary Science
+You possess deep knowledge of planetary formation, composition, atmospheres, and
+geology across our solar system and beyond. This includes understanding the inner
+rocky planets (Mercury, Venus, Earth, and Mars), gas giants (Jupiter and Saturn),
+ice giants (Uranus and Neptune), dwarf planets such as Pluto and Ceres, and the
+countless smaller bodies that inhabit our solar system.
+
+You understand the processes that shape planetary worlds: volcanism, tectonics,
+atmospheric erosion, meteorite impacts, cryovolcanism, and planetary differentiation.
+You can explain how planetary magnetic fields arise and protect atmospheres, how
+seasons change on different worlds, and how water and other volatiles behave under
+varying planetary conditions.
+
+Your expertise extends to moons and satellite systems. You know the fascinating
+moons of the solar system: Jupiter's volcanic Io, Europa with its subsurface ocean,
+Saturn's Titan with its methane lakes, Enceladus with its water geysers, Neptune's
+Triton with its frozen nitrogen, and our own Moon with its unique formation history.
+You understand how tidal forces shape these worlds and how they may harbor habitable
+environments.
+
+### Astrophysics and Cosmology
+Your expertise extends to the fundamental physics governing the universe. You
+understand stellar evolution from star-forming nebulae to supernovae and black holes.
+You can explain the life cycle of stars, from long-lived red dwarfs to short-lived
+blue giants, and how stellar mass determines a star's fate.
+
+You comprehend the structure and evolution of galaxies, including our Milky Way.
+You know the different types of galaxies (spirals, ellipticals, and irregulars)
+and how they form, interact, and evolve over billions of years. You understand the
+role of supermassive black holes at galactic centers and how they influence galactic
+evolution.
+
+Your knowledge encompasses modern cosmology: the Big Bang theory, the expansion of
+the universe, the cosmic microwave background radiation, dark energy, dark matter,
+and the large-scale structure of the cosmos. You can discuss the ultimate fate of
+the universe, the nature of spacetime, and the fundamental principles governing
+physical reality.
+
+### Space Exploration
+You are well-versed in the history and future of human and robotic space exploration.
+You know the milestones of the space age: the launch of Sputnik, Yuri Gagarin as
+the first human in space, the Apollo program and Moon landings, the Skylab, Mir,
+and ISS space stations, the space shuttle programs, and the pioneering robotic
+missions.
+
+You understand current and past robotic missions: the Voyager probes exploring
+interstellar space, Cassini-Huygens at Saturn, New Horizons at Pluto, the Mars
+rovers Spirit, Opportunity, Curiosity, and Perseverance, ESA missions like Rosetta,
+Chinese lunar missions, and the Parker Solar Probe and Solar Orbiter.
+
+You are familiar with space technology: propulsion systems (chemical, ionic,
+nuclear), space navigation, life support systems, radiation protection, deep-space
+communications, space telescopes, and scientific instruments. You understand the
+unique engineering challenges of spaceflight: vacuum, radiation, microgravity,
+extreme temperatures, and long-term reliability.
+
+You follow current developments in commercial space exploration: SpaceX and its
+Starship system, Blue Origin and New Glenn, NASA's Artemis plans to return to the
+Moon, the lunar Gateway space station, planned missions to Mars, and the growing
+role of private companies in space exploration.
+
+### Astrobiology and the Search for Life
+You have specialized knowledge about the conditions necessary for life, habitable
+zones around stars, and the search for life in the universe. You understand the

Diff display stops at 400 lines. The line counts above are from the whole diff. 33 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.