Change
43e93bf
43e93bf465187c4033a5b27d58dc07a303f869ba · commit on GitHub
pytorch-blog-feed: changed (332025 bytes, HTTP 200)
raw/pytorch-blog-feed/response.xml modified
- Source
- pytorch-blog-feed
- Lines added
- +562
- Lines removed
- -337
- Stored bytes at this commit
- 332,025
- Timestamp
- origin
- Raw artifact at this commit
- raw/pytorch-blog-feed/response.xml
Recorded headers
| observed_at | 2026-10-22T04:45:34.042Z |
|---|---|
| origin_date | 2026-10-22T04:16:20.000Z |
| status | 200 |
| final URL | https://pytorch.org/blog/feed/ |
| etag | "d5d35b0fd9e1c3fbd8aa04423d819019" |
| last-modified | Wed, 21 Oct 2026 20:09:11 GMT |
| date | Thu, 22 Oct 2026 04:45:34 GMT |
| age | 1754 |
| cache-control | public, max-age=60, s-maxage=43200, stale-while-revalidate=86400, stale-if-error=604800 |
| cf-cache-status | null |
| content-encoding | null |
| content-length | 332025 |
@
@@ -12,7 +12,7 @@ <atom:link href="https://pytorch.org/blog/feed/" rel="self" type="application/rss+xml" /> <link>https://pytorch.org/blog/</link> <description></description>-
<lastBuildDate>Tue, 20 Oct 2026 19:32:47 +0000</lastBuildDate>+
<lastBuildDate>Wed, 21 Oct 2026 20:09:11 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod>@
@@ -28,6 +28,561 @@ <height>32</height></image> <item>+
<title>TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL</title>+
<link>https://pytorch.org/blog/torchtitan-update-new-sota-models-performance-at-scale-titanrl/</link>+
+
<dc:creator><![CDATA[The PyTorch Team at Meta]]></dc:creator>+
<pubDate>Wed, 21 Oct 2026 19:45:44 +0000</pubDate>+
<category><![CDATA[Blog]]></category>+
<guid isPermaLink="false">https://pytorch.org/?p=189602</guid>+
+
<description><![CDATA[<p>TorchTitan is a PyTorch-native platform for rapid experimentation and large-scale pre- and post-training of generative AI models. It is designed as a minimal, clean-room implementation of PyTorch-native techniques, with flexible...</p>+
<p>The post <a href="https://pytorch.org/blog/torchtitan-update-new-sota-models-performance-at-scale-titanrl/">TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL</a> appeared first on <a href="https://pytorch.org">PyTorch</a>.</p>+
]]></description>+
<content:encoded><![CDATA[<p><a href="https://github.com/pytorch/torchtitan">TorchTitan</a> is a PyTorch-native platform for rapid experimentation and large-scale pre- and post-training of generative AI models. It is designed as a minimal, clean-room implementation of PyTorch-native techni…+
<p>The TorchTitan team landed massive improvements in the past year, without losing its minimal, extensible ethos. Enhancements include support for SOTA models, dramatic improvements of out-the-box performance at scale, RL support and new communication features.</p>+
<p>TorchTitan is for everyone – easy to use, providing SOTA performance and allowing users to start small and scale up seamlessly. We will present all these new improvements in the following sections, and we look forward to your feedback!</p>+
<h2>New model support</h2>+
<p>SOTA model support is critical for the success of training frameworks, and TorchTitan aims to always provide the latest advancements when it comes to open-weights architecture. <a href="https://github.com/pytorch/torchtitan/tree/main/torchtitan/models">TorchTitan added PyTorch-native support</a> …+
<p>Each frontier model expands TorchTitan’s reusable toolkit, from multimodal data pipelines, sample packing, shared vision components, and Hugging Face checkpoint interoperability to composable 5D parallelism, declarative sharding, optimized Attention Gym and Helion kernels, and rigorous numerical …+
<h2>TitanRL</h2>+
<p>TorchTitan now does reinforcement learning! Many RL stacks keep separate training and inference implementations of the model. Architecture changes have to stay aligned across both, and when a run goes sideways it is hard to tell whether the culprit is your reward, your algorithm, or the two copie…+
<p><!-- IMAGE 1: TitanRL diagram (first image in the doc) - upload via WP media library and insert here --></p>+
<h3><img fetchpriority="high" decoding="async" class="alignnone wp-image-189603 size-full" src="https://pytorch.org/wp-content/uploads/2026/10/one_model_definition-scaled.png" alt="" width="2560" height="952" srcset="https://pytorch.org/wp-content/uploads/2026/10/one_model_definition-scaled.png 2560…+
<h3>Agentic RL at scale</h3>+
<p>TitanRL supports what you need to train agentic workloads. Rollouts run asynchronously – training and generation each scale with multiple forms of parallelism.</p>+
<p>We tested it on several workloads, as seen below, demonstrating that it can perform stable training on MoEs and dense models, for hundreds of steps, on tasks that require long context, code execution, compaction, harness, self-play, etc.</p>+
<p><!-- IMAGE 2: SWE-Rebench train/validation chart - upload via WP media library and insert here --></p>+
<p><img decoding="async" class="alignnone wp-image-189606 size-full" src="https://pytorch.org/wp-content/uploads/2026/10/swe_rebench_train_heldout-scaled.png" alt="" width="2560" height="780" srcset="https://pytorch.org/wp-content/uploads/2026/10/swe_rebench_train_heldout-scaled.png 2560w, https://p…+
<p><em>Figure 1: Train/Validation on SWE-Rebench easy, model Qwen3.5-35B-A3B base, 64K context, 40 GB300</em></p>+
<p><!-- IMAGE 3: Tmax 15k / terminal bench 2.1 chart - upload via WP media library and insert here --></p>+
<p><img decoding="async" class="alignnone wp-image-189608 size-full" src="https://pytorch.org/wp-content/uploads/2026/10/terminal_bench_train_eval-scaled.png" alt="" width="2560" height="780" srcset="https://pytorch.org/wp-content/uploads/2026/10/terminal_bench_train_eval-scaled.png 2560w, https://p…+
<p><em>Figure 2: Train on Tmax 15k, Eval on terminal bench 2.1, model Qwen3.5-35B-A3B base, 128K context, 20 GB300</em></p>+
<h3>Correctness and debugging</h3>+
<p>TitanRL comes with strong correctness guarantees! Say you just enabled a new algorithm or kernel, launched a run, and the rewards differ from your baseline. Is that your change, or noise? Batch-invariant mode takes one source of noise off the table: in supported on-policy configurations, the trai…+
<h2>Out-the-box Performance improvements</h2>+
<p>TorchTitan shipped massive performance improvements for MoE models in the past year – around three focus areas: faster kernels for higher computer utilization, maximizing communication compute overlap, and fine grained control of memory.</p>+
<p><strong>local_compile</strong> brings the benefits of native <code>torch.compile</code> without committing to full-graph or full-block tracing, allowing users to selectively optimize key regions while preserving numerics without maintaining custom kernels.</p>+
<p>For memory management, <strong>torch_remat</strong> offers a modern activation checkpointing API with fine-grained control over recomputed regions, avoiding CPU overhead and custom op wrapper requirements.</p>+
<p>For distributed compute and scaling, <a href="https://github.com/meta-pytorch/dist_moe"><strong>Dist-MoE</strong></a> provides fused communication-compute kernels for NVIDIA SM100+ GPUs with activation packing for dynamic tensors to remain fully CUDA-graphable. Meanwhile, a fully revamped <strong…+
<p>Finally, <strong>TorchTitan overrides</strong> enable clean integration of specialized model-specific kernels—such as fused MLA, fused SwiGLU, and fused Complex RoPE – without sacrificing the simplicity and maintainability of the code.</p>+
<h3>DSV3 671B Results</h3>+
<p>By <a href="https://github.com/pytorch/torchtitan/blob/main/torchtitan_recipes/models/deepseek_v3.py">composing</a> all of the above performance features we evaluated the performance and numerics of Deepseek V3 671b+MTP1 module on the MLPerf training benchmark on our internal GB300 cluster scalin…+
<h4>Results</h4>+
<table>+
<thead>+
<tr>+
<th>Run</th>+
<th>GPUs</th>+
<th>PP / VPP / DP / EP</th>+
<th>LBS / microbatches</th>+
<th>TPS/GPU</th>+
<th>Global TPS</th>+
<th>TFLOPs/GPU</th>+
<th>Peak HBM</th>+
</tr>+
</thead>+
<tbody>+
<tr>+
<td>PP2-256</td>+
<td>256</td>+
<td>2 / 8 / 128 / 64</td>+
<td>120 / 120</td>+
<td><strong>5,324.21</strong></td>+
<td><strong>1,362,998</strong></td>+
<td><strong>1,553.89</strong></td>+
<td>243.86 GiB</td>+
</tr>+
<tr>+
<td>PP2-512</td>+
<td>512</td>+
<td>2 / 8 / 256 / 64</td>+
<td>60 / 60</td>+
<td><strong>5,262.68</strong></td>+
<td><strong>2,694,493</strong></td>+
<td><strong>1,535.93</strong></td>+
<td>224.65 GiB</td>+
</tr>+
<tr>+
<td>PP4-256</td>+
<td>256</td>+
<td>4 / 4 / 64 / 64</td>+
<td>240 / 240</td>+
<td>5,178.22</td>+
<td>1,325,624</td>+
<td>1,511.28</td>+
<td>238.08 GiB</td>+
</tr>+
<tr>+
<td>PP4-512</td>+
<td>512</td>+
<td>4 / 4 / 128 / 64</td>+
<td>120 / 120</td>+
<td>5,087.99</td>+
<td>2,605,049</td>+
<td>1,484.95</td>+
<td><strong>221.14 GiB</strong></td>+
</tr>+
<tr>+
<td>PP8-512</td>+
<td>512</td>+
<td>8 / 2 / 64 / 64</td>+
<td>240 / 240</td>+
<td>4,780.79</td>+
<td>2,447,763</td>+
<td>1,395.29</td>+
<td>233.96 GiB</td>+
</tr>+
</tbody>+
</table>+
<h3>Muse Spark Open</h3>+
<p>For our MuseSpark 1.2 OSS release we <a href="https://github.com/pytorch/torchtitan/blob/main/torchtitan_recipes/models/muse_spark.py">showcase</a> our high precision long context training efficiency demonstrating our TP, EP and CP scaling.</p>+
<h4>Results</h4>+
<table>+
<thead>+
<tr>+
<th>Routing</th>+
<th>GPUs</th>+
<th>Context</th>+
<th>Parallelism</th>+
<th>TPS/GPU</th>+
<th>Global TPS</th>+
<th>TFLOPS/GPU</th>+
<th>MFU</th>+
<th>Peak HBM</th>+
</tr>+
</thead>+
<tbody>+
<tr>+
<td>Learned</td>+
<td>256</td>+
<td>96K</td>+
<td>PP1/DP16/TP4/CP4/EP64</td>+
<td>1,500.61</td>+
<td>384,156</td>+
<td>646.03</td>+
<td>25.84%</td>+
<td>253.66 GiB</td>+
</tr>+
<tr>+
<td>Forced balanced</td>+
<td>256</td>+
<td>96K</td>+
<td>PP1/DP16/TP4/CP4/EP64</td>+
<td>1,839.52</td>+
<td>470,918</td>+
<td>791.95</td>+
<td>31.68%</td>+
<td>253.66 GiB</td>+
</tr>+
<tr>+
<td>Learned</td>+
<td>512</td>+
<td>192K</td>+
<td>PP1/DP16/TP4/CP8/EP64</td>+
<td>1,335.86</td>+
<td>683,961</td>+
<td>781.66</td>+
<td>31.27%</td>+
<td>235.64 GiB</td>+
</tr>+
<tr>+
<td>Forced balanced</td>+
<td>512</td>+
<td>192K</td>+
<td>PP1/DP16/TP4/CP8/EP64</td>+
<td>1,588.22</td>+
<td>813,168</td>+
<td>929.33</td>+
<td>37.17%</td>+
<td>235.65 GiB</td>+
</tr>+
</tbody>+
</table>+
<h2>Ecosystem integrations</h2>+
<p>One of TorchTitan’s strengths is its simplicity and reusability. As a result, several projects in the ecosystem chose to integrate and offer TorchTitan as a trainer. In particular:</p>+
<ul>+
<li>TorchTitan has been integrated to veRL as a training backend (<a href="https://github.com/verl-project/verl/issues/5306">roadmap</a>, <a href="https://github.com/verl-project/verl/tree/main/verl/workers/engine/torchtitan">code</a>)</li>+
<li>TorchTitan has also been added to Miles (<a href="https://github.com/radixark/miles/blob/main/docs/user-guide/training-backend.md#torchtitan">readme</a>, <a href="https://x.com/radixark/status/2108596085472268681?s=20">post</a>)</li>+
<li>TorchTitan is the default training backend for <a href="https://github.com/OpenBMB/Meshy/tree/main">Meshy</a>, and inspires the core implementation of <a href="https://github.com/areal-project/AReaL/tree/main">Areal’s</a> training engine <a href="https://github.com/areal-project/AReaL/blob/main/…+
</ul>+
<h2>FlexShard</h2>+
<p>TorchTitan leverages FSDP2 for row-wise or column-wise sharding model state (parameters, gradients, and optimizer). However, newer features such as distributed Muon optimizer and blockwise quantization of parameters warrant or benefit from more esoteric sharding policies that are not possible wit…+
<p>FlexShard has enabled TorchTitan to support the distributed Muon optimizer in an efficient way where parameters optimized using Muon are sharded using <code>Owned</code> and <code>BlockShard</code> placements – eliminating the need for All Gather collectives before performing the optimizer update…+
<p>FlexShard also enabled TorchTitan to support blockwise-quantized parameter AllGathers: where FP8-quantized parameters are gathered by the collective, reducing communication volume by ~50%. With FSDP2, since the sharding boundaries may not necessarily be aligned with the blockwise quantization bou…+
<p><!-- IMAGE 4: FlexShard chart - upload via WP media library and insert here --></p>+
<h2>GraphTrainer</h2>+
<p><a href="https://github.com/pytorch/torchtitan/tree/main/torchtitan/experiments/graph_trainer">GraphTrainer</a> is the <strong>graph-mode training offering in TorchTitan</strong>, providing an alternative execution model built around an explicit training graph.</p>+
<p>TorchTitan supports both graph-based and eager-based training paths with bitwise-matching numerical results. The graph-based trainer enables advanced graph-level optimizations that are complicated to express in eager execution, including cross-module fusions, explicit graph-level debugging, and t…+
<h2>Distributed infrastructure</h2>+
<p>TorchTitan serves as an important validation stage for new communications features as they ship in PyTorch. Large-scale training stresses every part of the communication stack (initialization, collectives, failure handling and memory registration), which makes TorchTitan a demanding testing groun…+
<p><strong>A new NCCL backend, <a href="https://github.com/pytorch/pytorch/pull/192281">validated on TorchTitan</a>:</strong> In <a href="https://pytorch.org/blog/pytorch-2-14-release-blog/">PyTorch 2.14</a>, the NCCL implementation from torchcomms was ported into c10d as a new, opt-in NCCL backend.…+
<ul>+
<li>One-sided RMA windows</li>+
<li>In-place process-group reconfiguration for fault tolerance</li>+
<li>Eager communicator initialization (in PyTorch 2.15)</li>+
<li>Raw access to NCCL shrink/grow and suspend/resume APIs</li>+
</ul>+
<p><strong>Symmetric memory:</strong> Symmetric memory continues to be an important performance lever for TorchTitan. TorchTitan’s MoE layers can now run on Dist-MoE, a newly open-sourced distributed Mixture-of-Experts library, with ready-to-use DeepSeek V3 recipes for NVIDIA Blackwell GPUs. Dist-Mo…+
<h2>Checkpointing</h2>+
<p>TorchTitan has integrated support for a new distributed checkpoint library, <a href="https://github.com/meta-pytorch/torch_checkpointing">torch_checkpointing</a>. This new library removes per-save synchronization in the default <code>torch.distributed.checkpointing</code> (DCPv1) saves. We do one…+
<p>As pretraining grows past 16k ranks, the previous approach of using GLOO collectives to achieve synchronizations scales poorly with the number of nodes. torch_checkpointing’s approach of only running a lightweight barrier per save wins out at these scales.</p>+
<p>We also improved upon other things within DCPv1, such as saving the whole checkpoint item (e.g. model, optimizer, etc.) in <code>torch.save</code> at once rather than calling <code>torch.save</code> on each individual tensor, and making custom components pluggable (e.g. writing a custom storage b…+
<p>At a scale of 512 H100 GPUs running DSv3 671B with checkpoint interval 50, we measured the following improvements:</p>+
<table>+
<thead>+
<tr>+
<th>Trainer blocking times</th>+
<th>Average</th>+
<th>P99</th>+
</tr>+
</thead>+
<tbody>+
<tr>+
<td>DCPv1</td>+
<td>841.2 ms</td>+
<td>941.46 ms</td>+
</tr>+
<tr>+
<td>torch_checkpointing</td>+
<td>34.1 ms</td>+
<td>36.6 ms</td>+
</tr>+
</tbody>+
</table>+
<table>+
<thead>+
<tr>+
<th>Checkpoint write times</th>+
<th>Average</th>+
<th>P99</th>+
</tr>+
</thead>+
<tbody>+
<tr>+
<td>DCPv1</td>+
<td>384.4 s</td>+
<td>403.1 s</td>+
</tr>+
<tr>+
<td>torch_checkpointing</td>+
<td>13.1 s</td>+
<td>13.2 s</td>+
</tr>+
</tbody>+
</table>+
<table>+
<thead>+
<tr>+
<th>Non-resharding load times</th>+
<th>Average</th>+
<th>P99</th>+
</tr>+
</thead>+
<tbody>+
<tr>+
<td>DCPv1</td>+
<td>194.7 s</td>+
<td>198.2 s</td>+
</tr>+
<tr>+
<td>torch_checkpointing</td>+
<td>79 s</td>+
<td>82.3 s</td>+
</tr>+
</tbody>+
</table>+
<p>These improvements reduce blocking times and checkpoint write times and unlock more frequent checkpoints without risking TFLOP drops for users who adopt torch_checkpointing in their Titan recipes.</p>+
<h2>To get started</h2>+
<p>Clone <a href="https://github.com/pytorch/torchtitan">github.com/pytorch/torchtitan</a>, and follow the instructions listed <a href="https://github.com/pytorch/torchtitan/#quick-start">here</a>. If you have any kind of feedback, we would love to hear about it!</p>+
<p>The post <a href="https://pytorch.org/blog/torchtitan-update-new-sota-models-performance-at-scale-titanrl/">TorchTitan Update: New SOTA Models, Performance at Scale, TitanRL</a> appeared first on <a href="https://pytorch.org">PyTorch</a>.</p>+
]]></content:encoded>+
+
+
+
</item>+
<item>+
<title>Accelerating PyTorch on AMD GPUs with FlyDSL</title>+
<link>https://pytorch.org/blog/accelerating-pytorch-on-amd-gpus-with-flydsl/</link>+
+
<dc:creator><![CDATA[AMD: Xiaobing Zhang, Yutao Xu, Rui Li, Heng Yi, Xinya Zhang, Shijie Feng, Felix Li, Jeff Daily, Jiong Gong, Hattie Wu, Peng Sun, Emad Barsoum; META: Jason Ansel and Simon Layton]]></dc:creator>+
<pubDate>Wed, 21 Oct 2026 17:45:40 +0000</pubDate>+
<category><![CDATA[Blog]]></category>+
<guid isPermaLink="false">https://pytorch.org/?p=189489</guid>+
+
<description><![CDATA[<p>PyTorch users on AMD MI350-series GPUs can now use FlyDSL through existing operator APIs for dense and grouped GEMM, fused multi-head attention (FMHA/Attention), MXFP8/MXFP4 scaled GEMM, RMSNorm, and TopK, without...</p>+
<p>The post <a href="https://pytorch.org/blog/accelerating-pytorch-on-amd-gpus-with-flydsl/">Accelerating PyTorch on AMD GPUs with FlyDSL</a> appeared first on <a href="https://pytorch.org">PyTorch</a>.</p>+
]]></description>+
<content:encoded><![CDATA[<p>PyTorch users on AMD MI350-series GPUs can now use <a href="https://github.com/ROCm/FlyDSL">FlyDSL</a> through existing operator APIs for dense and grouped GEMM, fused multi-head attention (FMHA/Attention), MXFP8/MXFP4 scaled GEMM, RMSNorm, and TopK, without re…+
<p>Across the reported kernel-level operator suites, FlyDSL provides a 1.10x geometric-mean speedup for the 15-shape BF16 dense-GEMM suite over the faster of ATen and Triton at each shape, 1.58x and 1.68x geometric-mean speedups for MXFP8 and MXFP4, respectively, over ATen across 17 shapes per forma…+
<h2>Why FlyDSL as a PyTorch Backend</h2>+
<p>GPU DSLs choose different abstraction levels. The block-tensor programming model in <a href="https://github.com/triton-lang/triton">Triton</a> lets authors describe program-level tiles while the compiler manages much of the thread, wave, and instruction mapping. FlyDSL makes those mappings explic…+
<p>These controls appear directly in the kernels: HTI and staged LDS in Dense/MXFP GEMM, matched operand-scale lifetimes and scaled MFMA in MXFP, and persistent expert scheduling in Grouped GEMM. FlyDSL expresses them in Python and lowers them through MLIR while preserving explicit hardware mapping.…+
<p>Attention shows why this control matters. The FlyDSL attention kernels use a dual-wave software pipeline, which overlaps the softmax VALU work of one KV tile with the MFMA work of another. At the largest head dimensions, they also split the head dimension: in time through LDS, and across waves fo…+
<p>PyTorch turns those specialized templates into an additive, measurable backend. Unsupported workloads retain existing implementations:</p>+
<ul>+
<li>Eager execution: eligible RMSNorm and TopK calls dispatch to FlyDSL when the optional runtime is installed and enabled; other calls retain ATen.</li>+
<li><code>torch.compile</code>: TorchInductor filters eligible Dense, Grouped, and MXFP GEMM candidates, benchmarks FlyDSL beside ATen and Triton where supported, caches the winner for each workload, and runs the fastest measured kernel.</li>+
<li>Multi-head attention: PyTorch’s built-in operator <code>F.scaled_dot_product_attention</code> calls AOTriton, which now picks among FlyDSL, Triton, and AITER ASM kernels for each workload using its offline operator-tuning database.</li>+
</ul>+
<p><!-- IMAGE 1: PyTorch APIs feed three execution paths: eager dispatch chooses FlyDSL for eligible RMSNorm and TopK inputs or ATen otherwise; scaled dot product attention uses AOTriton to select Triton, AITER ASM, or FlyDSL from its tuning database; torch.compile benchmarks eligible dense, grouped…+
<p><em><img decoding="async" class="alignnone wp-image-189499 size-full" src="https://pytorch.org/wp-content/uploads/2026/10/PyTorch-Paths-to-FlyDSL-scaled.png" alt="" width="2560" height="1411" srcset="https://pytorch.org/wp-content/uploads/2026/10/PyTorch-Paths-to-FlyDSL-scaled.png 2560w, https://…+
<p><em>Figure 1. FlyDSL kernels reached through PyTorch APIs. Eager RMSNorm and TopK dispatch by input support; AOTriton selects a tuned SDPA implementation; and TorchInductor benchmarks eligible GEMM candidates.</em></p>+
<h2>Supported Features</h2>+
<p>Current support targets AMD MI350-series GPUs with the <code>gfx950</code> architecture. The following features are available in PyTorch:</p>+
<table>+
<thead>+
<tr>+
<th>Operation</th>+
<th>Mode</th>+
<th>PyTorch API</th>+
<th>Input dtype</th>+
<th>Layout / shape</th>+
<th>Output / options</th>+
</tr>+
</thead>+
<tbody>+
<tr>+
<td>Dense GEMM</td>+
<td>Compile</td>+
<td><code>torch.mm</code></td>+
<td>FP16, BF16</td>+
<td>Static 2D; NN/NT/TN/TT</td>+
<td>Same dtype</td>+
</tr>+
<tr>+
<td>MXFP scaled GEMM</td>+
<td>Compile</td>+
<td><code>F.scaled_mm</code></td>+
<td>MXFP8, MXFP4</td>+
<td>Static 2D; A row-major, B column-major</td>+
<td>FP16/BF16; block scales</td>+
</tr>+
<tr>+
<td>Grouped GEMM</td>+
<td>Compile</td>+
<td><code>F.grouped_mm</code></td>+
<td>FP16, BF16</td>+
<td>Ragged 2D A; grouped 3D B</td>+
<td>Uneven/empty groups</td>+
</tr>+
<tr>+
<td>RMSNorm</td>+
<td>Eager</td>+
<td><code>F.rms_norm</code></td>+
<td>FP16, BF16, FP32</td>+
<td>Contiguous; 1-D norm shape</td>+
<td>Forward</td>+
</tr>+
<tr>+
<td>TopK</td>+
<td>Eager</td>+
<td><code>torch.topk</code></td>+
<td>FP32</td>+
<td>Contiguous; last dimension</td>+
<td>Largest, sorted; functional/<code>out=</code></td>+
</tr>+
<tr>+
<td>Attention (via AOTriton)</td>+
<td>Eager, compile</td>+
<td><code>F.scaled_dot_product_attention</code></td>+
<td>FP16, BF16</td>+
<td>D innermost; head dim 32–512</td>+
<td>Forward and backward</td>+
</tr>+
</tbody>+
</table>+
<p>Dense GEMM’s <a href="https://github.com/pytorch/pytorch/pull/194981">four-layout support</a> accepts row-major and column-major inputs, including eligible transpose views. In the layout names, the first letter describes A and the second describes B; <code>N</code> and <code>T</code> denote…+
<p>MXFP scaled GEMM accepts FP8 E4M3 or packed FP4 E2M1 operands, with one E8M0 scale per 32 values along the reduction dimension (<code>BlockWise1x32</code>). Applications supply the quantized operands and contiguous, unswizzled scale tensors. Both formats use FP32 accumulation and return FP16 or B…+
<p>The FlyDSL attention kernels match the flash and memory-efficient attention capabilities of <code>F.scaled_dot_product_attention</code>, with one current limit: FP32 is not supported. FP32 inputs, and workloads that the tuning database assigns elsewhere, keep AOTriton’s Triton or AITER ASM …+
<p>Each operation has shape and alignment requirements. The detailed <a href="https://github.com/pytorch/pytorch/pull/194981">dense GEMM</a>, <a href="https://github.com/pytorch/pytorch/pull/196719">MXFP scaled GEMM</a>, <a href="https://github.com/pytorch/pytorch/pull/194032">grouped GEMM</a>, <a h…+
<h2>Kernel-Level Performance</h2>+
<p>The following operator-level benchmarks were measured on AMD Instinct MI355X (<code>gfx950</code>) GPUs. The suites use different benchmark setups: dense GEMM uses GPU-event timing of graph replay, grouped GEMM reports steady-state throughput, and RMSNorm and TopK use GPU-event timing of repeated…+
<h3>Dense GEMM: Linear-Layer Workloads</h3>+
<p>Dense matrix multiplication is a building block of transformer projection and feed-forward layers. Its dimensions vary substantially with the number of tokens being processed, making performance across a range of shapes relevant to model developers.</p>+
<p>FlyDSL provides full-tile and half-tile interleaved (HTI) execution paths. The full-tile path uses a configurable <code>STAGES</code>-deep K-tile ring in local data share (LDS). Its prologue primes the ring, the steady-state loop overlaps computation on staged data with asynchronous prefetches in…+
<p>For large GEMMs, the tuned HTI configurations use a 256 × 256 output tile and an eight-wave workgroup (512 threads with Wave64). HTI splits A along M and B along N, maintaining four 128 × 128 output-quadrant accumulators in registers. Within each K tile, <code>A0</code> feeds <code>C00/C01</code>…+
<p>HTI processes consecutive K tiles in pairs using two LDS stages. In steady state, prefetches progressively reuse the stages holding <code>t</code> and <code>t+1</code> for <code>t+2</code> and <code>t+3</code>. Reuse occurs half by half: once the required reads of an operand half’s old LDS conten…+
<p><!-- IMAGE 2: HTI tile decomposition, MFMA wave scope, and phase-staggered loading and computation. Download from the Google Doc, upload to the WP media library, and replace this comment with the image --></p>+
<p><em><img decoding="async" class="alignnone wp-image-189500 size-full" src="https://pytorch.org/wp-content/uploads/2026/10/Dense-GEMM-HTI-Wave-scaled.png" alt="" width="2560" height="1300" srcset="https://pytorch.org/wp-content/uploads/2026/10/Dense-GEMM-HTI-Wave-scaled.png 2560w, https://pytorch.…+
<p><em>Figure 2. HTI wave scope and phase-staggered execution, shown with two groups of four Wave64 waves. Each grid cell is a 16 × 16 output block. Waves per group are tunable with the tile configuration, subject to kernel constraints.</em></p>+
<p>Across 15 BF16 NT shapes, FlyDSL delivers a 1.10x geometric-mean speedup over the faster ATen/Triton baseline at each shape. Gains are strongest in smaller and medium-sized problems: <code>M × N × K = 64 × 4096 × 4096</code>, for example, improves by 1.33x.</p>+
<p><!-- IMAGE 3: FlyDSL dense GEMM speedup for all 15 BF16 NT shapes. Download from the Google Doc, upload to the WP media library, and replace this comment with the image --></p>+
<p><em><img decoding="async" class="alignnone wp-image-189501 size-full" src="https://pytorch.org/wp-content/uploads/2026/10/Dense-GEMM-Performance-across-B15-scaled.png" alt="" width="2560" height="1892" srcset="https://pytorch.org/wp-content/uploads/2026/10/Dense-GEMM-Performance-across-B15-scaled…+
<p><em>Figure 3. BF16 NT GEMM on MI355, relative to the faster ATen/Triton baseline at each shape. All 15 cases are shown: 10 wins, three ties, and two losses using a ±1% tie band.</em></p>+
<p>A later regression check of the current all-layout kernel over the same shapes reported a 1.12x aggregate speedup over the faster original baseline. Figure 3 retains the earlier per-shape measurements because the later check published aggregate results only.</p>+
<p>The two largest square-output problems in this suite slightly favor ATen. Keeping multiple backends enabled lets PyTorch make that choice for each workload. These results characterize NT GEMM; NN, TN, and TT are also supported, with their performance depending on the workload.</p>+
<h3>MXFP Scaled GEMM: Extending to Low-Precision Workloads</h3>+
<p>MXFP extends the shared gfx950 full-tile and HTI GEMM implementations with FP8 operand handling, packed FP4 storage, E8M0 scale staging, and CDNA4 scaled MFMA.</p>+
<p>In HTI, A/B buffers advance by K tile, while separate scale slots hold chunks spanning multiple K tiles. Each MFMA uses register-resident operands and the matching scales for its K position. At chunk boundaries, waits and barriers coordinate reuse of the other scale slot to prefetch the next chun…Diff display stops at 400 lines. The line counts above are from the whole diff. 38 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.