Change
b2be375
b2be3756a05369bb104b9f875641f6265f6c0e27 · commit on GitHub
pytorch-blog-feed: changed (248897 bytes, HTTP 200)
raw/pytorch-blog-feed/response.xml modified
- Source
- pytorch-blog-feed
- Lines added
- +82
- Lines removed
- -43
- Stored bytes at this commit
- 248,897
- Timestamp
- origin
- Raw artifact at this commit
- raw/pytorch-blog-feed/response.xml
Recorded headers
| observed_at | 2026-10-17T04:26:13.219Z |
|---|---|
| origin_date | 2026-10-17T03:05:54.000Z |
| status | 200 |
| final URL | https://pytorch.org/blog/feed/ |
| etag | "6f98b2e09a02c00e3d481990165071f7" |
| last-modified | Fri, 16 Oct 2026 23:37:59 GMT |
| date | Sat, 17 Oct 2026 04:26:13 GMT |
| age | 4819 |
| cache-control | public, max-age=60, s-maxage=43200, stale-while-revalidate=86400, stale-if-error=604800 |
| cf-cache-status | null |
| content-encoding | null |
| content-length | 248897 |
@
@@ -12,7 +12,7 @@ <atom:link href="https://pytorch.org/blog/feed/" rel="self" type="application/rss+xml" /> <link>https://pytorch.org/blog/</link> <description></description>-
<lastBuildDate>Thu, 15 Oct 2026 19:18:47 +0000</lastBuildDate>+
<lastBuildDate>Fri, 16 Oct 2026 16:30:46 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -28,6 +28,86 @@ <height>32</height></image> <item>+
<title>Bringing Elastically Deterministic Data Loading to TorchTitan with Zephon</title>+
<link>https://pytorch.org/blog/bringing-elastically-deterministic-data-loading-to-torchtitan-with-zephon/</link>+
+
<dc:creator><![CDATA[Josh Wills, Maximilian Böther, Ties Robroek and the Zephon team]]></dc:creator>+
<pubDate>Fri, 16 Oct 2026 16:30:46 +0000</pubDate>+
<category><![CDATA[Blog]]></category>+
<guid isPermaLink="false">https://pytorch.org/?p=188117</guid>+
+
<description><![CDATA[<p>Why Zephon? The data that a model sees during training is one of the most powerful levers we have for influencing how well it performs. Much of the day-to-day work...</p>+
<p>The post <a href="https://pytorch.org/blog/bringing-elastically-deterministic-data-loading-to-torchtitan-with-zephon/">Bringing Elastically Deterministic Data Loading to TorchTitan with Zephon</a> appeared first on <a href="https://pytorch.org">PyTorch</a>.</p>+
]]></description>+
<content:encoded><![CDATA[<h3><span style="font-weight: 400;">Why Zephon?</span></h3>+
<p><span style="font-weight: 400;">The data that a model sees during training is one of the most powerful levers we have for influencing how well it performs. Much of the day-to-day work of improving a model happens on the data side: we change the mixture, or the filters that we apply, or how we tok…+
<p><span style="font-weight: 400;">This matters even more now that experiments can be designed, launched, and interpreted by agents instead of people. Agents are great for running lots of experiments in parallel, but they aren’t especially good at noticing when a result actually depends on something…+
<p><span style="font-weight: 400;">To illustrate this, we train a 1B-parameter model in TorchTitan on 16 GPUs, take a checkpoint at step 150, and then resume from that checkpoint on 8, 32, and 64 GPUs alongside the original run, once with a standard data loader and once with Zephon (think of it as t…+
<p style="text-align: center;"><em><span style="font-weight: 400;">[Figure 1: Loss curves after resuming on 8, 32, and 64 GPUs from a 16-GPU checkpoint at step 150, standard loader vs. Zephon, with a barcode comparing the per-step SHA-256 of each global batch against the original run.] </span></em><…+
<p><span style="font-weight: 400;">To get a sense for how much this matters, we train the same 1B-parameter model from scratch for 20B tokens on 8, 16, 32, and 64 GPUs using a standard data loader. We show the scores per GPU count in Figure 2. The model’s DCLM Core v1 score varies by up to 0.82 poin…+
<p style="text-align: center;"><em><span style="font-weight: 400;">[</span><span style="font-weight: 400;">Figure 2: Downstream evaluations at 8–64 GPUs on DCLM Core v1 and FineWeb with non-elastically deterministic (standard) data loading and Zephon</span><span style="font-weight: 400;">. </span><s…+
<p><span style="font-weight: 400;">Note that Zephon does all of this while doing the data prep work on the fly: it tokenizes, packs, shuffles, and mixes the data inside of the training job, taking advantage of all of those CPUs sitting around with nothing to do while the model trains, which means th…+
<h3><span style="font-weight: 400;">A Zephon Pipeline</span></h3>+
<p><span style="font-weight: 400;">Zephon asks you to describe three things: the </span><span style="font-weight: 400;"><code>Datasets</code> you want to train on, a </span><span style="font-weight: 400;"><code>WorkSource</code></span><span style="font-weight: 400;"> that decides how those datasets …+
<pre class="wp-block-code"><code class="language-python">from zephon import Pipeline
+
from zephon.io import Dataset
+
from zephon.work import StaticMixtureWorkSource, TokenEstimation
+
+
source = StaticMixtureWorkSource(
+
datasets=[
+
Dataset.from_path("wiki", "hf://Salesforce/wikitext/wikitext-2-raw-v1/train"),
+
Dataset.from_path("news", "hf://fancyzhx/ag_news/default/train"),
+
],
+
mixture={"wiki": 0.5, "news": 0.5}, # a token ratio, not a sample ratio
+
token_estimation=TokenEstimation(),
+
seed=42,
+
)
+
pipeline = (
+
Pipeline(source)
+
.options(io_options={"cache": {"enabled": True}})
+
.tokenize(tokenizer_id="gpt2", field="text", parallelism=4)
+
.pack_flat(max_length=2048, algorithm="wrap")
+
.shuffle(buffer_size=1024, seed=42)
+
.batch(microbatch_size=8)
+
)
+
for batch in pipeline:
+
train_step(batch.to_training())</code></pre>+
<p>The <code>WorkSource</code> plans the curriculum from each <code>Dataset</code>’s metadata and is kept separate from the operators that actually process those samples so you can change the mixture without touching the processing pipeline and vice-versa. In order to ensure we get the mixture perce…+
<p>Each of the individual <code>Pipeline</code> operators can have its own level of parallelism; here, tokenization gets four workers because it’s the most expensive step. Parallelizing the preprocessing operators is how Zephon keeps the GPUs fed, even on more complex pipelines: on a full vision-lan…+
<p style="text-align: center;"><span style="font-weight: 400;"><iframe class="zephon-embed" style="width: 100%; height: 418px; border: 0;" title="VLM throughput: usable tokens per second per GPU" src="https://www.datologyai.com/embeds/zephon-vlm-throughput/index.html"></iframe></span><span style="fo…+
<h3><span style="font-weight: 400;">How Zephon Keeps the Batches Identical</span></h3>+
<p><span style="font-weight: 400;">Packing and mixing data on the fly is what makes elastic determinism hard. When a data loader tokenizes documents, packs them together into fixed-length sequences, and mixes several datasets together as it goes, the order of the training data is no longer something…+
<p><span style="font-weight: 400;">Zephon allows for elastic determinism without these constraints with three design decisions that build on each other.</span></p>+
<p><b>Accumulators make the decisions; workers do the work.</b><span style="font-weight: 400;"> Every Zephon operator is split into two parts: a single-threaded accumulator that sees every record in order and makes all of the stateful decisions (like which sequences get packed together), and a state…+
<p><b>Lanes instead of ranks.</b><span style="font-weight: 400;"> Instead of dividing the data up among the available GPUs directly, Zephon divides it into a fixed number of logical streams that are called </span><i><span style="font-weight: 400;">lanes</span></i><span style="font-weight: 400;">. Ea…+
<p><b>Flushing and replaying instead of saving state.</b><span style="font-weight: 400;"> A Zephon checkpoint doesn’t contain the contents of any packing bins or shuffle buffers; it only records which chunks of the input were still being worked on when the checkpoint was taken. In order to keep that…+
<h3><span style="font-weight: 400;">Plugging Into TorchTitan</span></h3>+
<p><span style="font-weight: 400;">Our </span><a href="https://github.com/datologyai/torchtitan-zephon"><span style="font-weight: 400;">TorchTitan integration</span></a><span style="font-weight: 400;"> replaces TorchTitan’s default data loader with a Zephon </span><span style="font-weight: 400;"><co…+
<p><span style="font-weight: 400;">The </span><a href="https://github.com/datologyai/torchtitan-zephon"><span style="font-weight: 400;">repo</span></a><span style="font-weight: 400;"> also comes with a quick CPU-only demo that checkpoints a data stream, resumes it on a different number of data-paral…+
<h3><span style="font-weight: 400;">Things to Know Before You Start</span></h3>+
<p><b>Choose your lane count.</b><span style="font-weight: 400;"> Pick a number that’s a common multiple of the data-parallel sizes you expect to run on. If you might run on 8, 16, or 32 GPUs, 32 lanes is a cap that covers all three. Don’t go overboard though (e.g., multiple thousands when your max …+
<p><b>Keep the recipe fixed when you resume on more/fewer ranks.</b><span style="font-weight: 400;"> The data, tokenizer, seed, mixture, context length, and global batch size all need to stay the same. Of course, you may still adjust performance settings such as parallelism.</span></p>+
<p><b>Zephon guarantees the data, not the arithmetic.</b><span style="font-weight: 400;"> The global batches will be identical across GPU counts, and the losses will be very close, but might not be identical since floating-point reduction order in the training loop still depends on how many GPUs are…+
<h3><span style="font-weight: 400;">Looking Ahead</span></h3>+
<p><span style="font-weight: 400;">There are a couple of directions that we’re especially excited about. The first is GIL-free Python: Zephon allows users to pick a </span><i><span style="font-weight: 400;">runner</span></i><span style="font-weight: 400;">, i.e., how to parallelize the operators. On…+
<p><span style="font-weight: 400;">Beyond that, we would love to hear which parts of your data pipeline you would like to move into the training job and how we can help; issues and PRs are most welcome!</span></p>+
<p><b>Learn more: </b><a href="https://www.datologyai.com/blog/zephon"><span style="font-weight: 400;">Launch post</span></a><span style="font-weight: 400;"> · </span><a href="https://arxiv.org/abs/2610.03087"><span style="font-weight: 400;">Paper</span></a><span style="font-weight: 400;"> · </span>…+
<p><script type="text/javascript" nonce="8ab43d9d22c27432e35863e58814753e">+
window.addEventListener('message', function (e) {+
if (e.origin !== 'https://www.datologyai.com' || !e.data ||+
e.data.type !== 'EMBED_RESIZE') return;+
document.querySelectorAll('iframe.zephon-embed').forEach(function (f) {+
if (f.contentWindow === e.source) f.style.height = e.data.height + 'px';+
});+
});+
</script></p>+
<p>The post <a href="https://pytorch.org/blog/bringing-elastically-deterministic-data-loading-to-torchtitan-with-zephon/">Bringing Elastically Deterministic Data Loading to TorchTitan with Zephon</a> appeared first on <a href="https://pytorch.org">PyTorch</a>.</p>+
]]></content:encoded>+
+
+
+
</item>+
<item> <title>Muse Spark Is Now an Open-Weights Model</title> <link>https://pytorch.org/blog/muse-spark-open/</link> @
@@ -1386,49 +1466,8 @@ python scripts/autotune_helion_kernels.py \ -
</item>-
<item>-
<title>New Pathway to PyTorch Certified Associate (PTCA) Certification</title>-
<link>https://pytorch.org/blog/new-pathway-to-pytorch-certified-associate-ptca-certification/</link>-
-
<dc:creator><![CDATA[PyTorch Foundation]]></dc:creator>-
<pubDate>Fri, 02 Oct 2026 19:33:47 +0000</pubDate>-
<category><![CDATA[Announcements]]></category>-
<category><![CDATA[Blog]]></category>-
<guid isPermaLink="false">https://pytorch.org/?p=171978</guid>-
-
<description><![CDATA[<p>We’re excited to bring you the new PyTorch Certified Associate (PTCA) Certification Pathway, combining focused learning modules and the PyTorch Certified Associate (PTCA) certification exam together in one structured experience...</p>-
<p>The post <a href="https://pytorch.org/blog/new-pathway-to-pytorch-certified-associate-ptca-certification/">New Pathway to PyTorch Certified Associate (PTCA) Certification</a> appeared first on <a href="https://pytorch.org">PyTorch</a>.</p>-
]]></description>-
<content:encoded><![CDATA[<p><span style="font-size: 18.72px;"><img decoding="async" class="alignnone size-large wp-image-171984" src="https://pytorch.org/wp-content/uploads/2026/10/PyTorch-Certification-Pathway-1024x536.png" alt="PyTorch Certification Pathway" width="1024" height="536" sr…-
<p>We’re excited to bring you the new <a href="https://training.linuxfoundation.org/certification/ptca-certification-pathway/">PyTorch Certified Associate (PTCA) Certification Pathway</a>, combining focused learning modules and the PyTorch Certified Associate (PTCA) certification exam together…-
<p>The pathway includes four self-paced learning modules plus the PTCA certification exam:</p>-
<ul>-
<li>PyTorch Fundamentals</li>-
<li>Data Handling in PyTorch</li>-
<li>PyTorch Model Development</li>-
<li>Optimizing PyTorch</li>-
</ul>-
<p>Together, the modules cover core skills across the same key areas measured by the PTCA certification, from working with tensors and devices to preparing data, building neural networks, and improving model performance.</p>-
<h4><strong>Learn, Build, Validate, Certify</strong></h4>-
<p>Throughout the pathway, you’ll build skills to:</p>-
<ul>-
<li><strong>Work with tensors and devices</strong> across the model lifecycle, from training through prediction</li>-
<li><strong>Prepare and manage data</strong> using Datasets, DataLoaders, and transforms</li>-
<li><strong>Build neural networks</strong> using PyTorch layers, activations, loss functions, and optimizers</li>-
<li><strong>Measure and optimize performance</strong> using tools and techniques including torch.compile, Automatic Mixed Precision, the PyTorch Profiler, and distributed training concepts</li>-
</ul>-
<p>The full pathway includes 15–17 hours of self-paced learning and hands-on labs designed to help you build a strong foundation. Certification readiness takes more than completing coursework, so additional hands-on practice and real-world experience with PyTorch are recommended as you prepare for t…-
<h4><strong>Put Your PyTorch Skills on a Clear Path</strong></h4>-
<p>PTCA is designed for early-stage practitioners with Python and machine learning experience who are beginning to work with PyTorch. The certification validates your understanding of the core concepts and workflows used by engineers and researchers to design, train, optimize, and work with machine …-
<p>Build your PyTorch skills and get certified today.</p>-
<p>The post <a href="https://pytorch.org/blog/new-pathway-to-pytorch-certified-associate-ptca-certification/">New Pathway to PyTorch Certified Associate (PTCA) Certification</a> appeared first on <a href="https://pytorch.org">PyTorch</a>.</p>-
]]></content:encoded>-
-
-
</item> </channel></rss>-
<!-- plugin=object-cache-pro client=phpredis metric#hits=4630 metric#misses=31 metric#hit-ratio=99.3 metric#bytes=1428000 metric#prefetches=238 metric#store-reads=36 metric#store-writes=9 metric#store-hits=247 metric#store-misses=15 metric#sql-queries=20 metric#ms-total=506.36 metric#ms-cache=18.67 …+
<!-- plugin=object-cache-pro client=phpredis metric#hits=4623 metric#misses=31 metric#hit-ratio=99.3 metric#bytes=1450492 metric#prefetches=237 metric#store-reads=36 metric#store-writes=13 metric#store-hits=246 metric#store-misses=15 metric#sql-queries=25 metric#ms-total=322.70 metric#ms-cache=20.63…27 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.