Change
f4fe109
f4fe109d26d5333d46a349acae8ab0a0a5ac2add · commit on GitHub
pytorch-blog-feed: changed (258047 bytes, HTTP 200)
raw/pytorch-blog-feed/response.xml modified
- Source
- pytorch-blog-feed
- Lines added
- +103
- Lines removed
- -361
- Stored bytes at this commit
- 258,047
- Timestamp
- origin
- Raw artifact at this commit
- raw/pytorch-blog-feed/response.xml
Recorded headers
| observed_at | 2026-09-22T04:57:18.159Z |
|---|---|
| origin_date | 2026-09-22T03:12:44.000Z |
| status | 200 |
| final URL | https://pytorch.org/blog/feed/ |
| etag | "f38b9c38428d87a83598d9989a328c8a" |
| last-modified | Mon, 21 Sep 2026 20:41:14 GMT |
| date | Tue, 22 Sep 2026 04:57:18 GMT |
| age | 6274 |
| cache-control | public, max-age=60, s-maxage=43200, stale-while-revalidate=86400, stale-if-error=604800 |
| cf-cache-status | null |
| content-encoding | null |
| content-length | 258047 |
@
@@ -12,13 +12,13 @@ <atom:link href="https://pytorch.org/blog/feed/" rel="self" type="application/rss+xml" /> <link>https://pytorch.org</link> <description></description>-
<lastBuildDate>Fri, 18 Sep 2026 00:27:15 +0000</lastBuildDate>+
<lastBuildDate>Mon, 21 Sep 2026 20:41:14 +0000</lastBuildDate> <language>en-US</language> <sy:updatePeriod> hourly </sy:updatePeriod> <sy:updateFrequency> 1 </sy:updateFrequency>-
<generator>https://wordpress.org/?v=7.1</generator>+
<generator>https://wordpress.org/?v=7.1.1</generator><image> <url>https://pytorch.org/wp-content/uploads/2024/10/cropped-favicon-32x32.webp</url>@
@@ -28,6 +28,105 @@ <height>32</height></image> <item>+
<title>TinyTorch: Don’t Just Import PyTorch. Build It.</title>+
<link>https://pytorch.org/blog/tinytorch-dont-just-import-pytorch-build-it/</link>+
+
<dc:creator><![CDATA[Vijay Janapa Reddi, Harvard University and ETH Zurich · Andrea Mattia Garavagno, ETH Zurich]]></dc:creator>+
<pubDate>Mon, 21 Sep 2026 20:41:14 +0000</pubDate>+
<category><![CDATA[Blog]]></category>+
<guid isPermaLink="false">https://pytorch.org/?p=167810</guid>+
+
<description><![CDATA[A framework you write yourself, tensors through transformers TL;DR Every mature systems project eventually needs a teaching version. TinyTorch is a free, open-source curriculum where you build a working ML...]]></description>+
<content:encoded><![CDATA[<p><em>A framework you write yourself, tensors through transformers</em></p>+
<h2><img fetchpriority="high" decoding="async" class="alignleft wp-image-169836 size-full" src="https://pytorch.org/wp-content/uploads/2026/09/All-PyTorch-Blog-Social-Images-25.png" alt="" width="1920" height="1080" srcset="https://pytorch.org/wp-content/uploads/2026/09/All-PyTorch-Blog-Social-Image…+
<h2>TL;DR</h2>+
<p>Every mature systems project eventually needs a teaching version. TinyTorch is a free, open-source curriculum where you build a working ML framework from scratch, tensors through transformers, in pure Python, using PyTorch’s own API. Twenty modules. Runs on a laptop with 4 GB of RAM and no …+
<p>The rest of this post is why we think it needed to exist, and what six years of running it taught us that might be useful to anyone else doing open curriculum work.</p>+
<h2>Every Systems Field Eventually Builds Its Teaching Version</h2>+
<p>Unix got too big to hold in your head, so Andrew Tanenbaum wrote MINIX. Small enough that a student could actually finish it. It went on to shape a generation of systems engineers and famously inspired Linux.</p>+
<p>Compilers went the same way. LLVM and GCC are decades of excellent engineering and close to unreadable as a first text, so courses teach the Tiger compiler instead. MIT rewrote xv6 from x86 to RISC-V for the same reason, stripping out historical complexity to expose clean abstractions. Before all…+
<p>None of these were trying to replace the production system. They taught what production systems have to hide.</p>+
<p>PyTorch is at that point now, which is a compliment. There is genuinely good writing on its internals, most of it by the people who wrote them, and nearly all of it assumes you already think like a framework engineer. What has been missing is the rung below that. Something you build yourself, wit…+
<h2>Why This Matters to PyTorch, Not Only to Students</h2>+
<p>There is a version of this argument that only educators care about. This is not that version.</p>+
<p>Every framework runs on a small population of people who can reason about it from the inside. The ones who spot a memory leak in tensor caching, who know when gradient checkpointing is worth the recompute, who can look at a slow training run and name the bottleneck before they open a profiler.</p…+
<p>That population does not grow by itself. Right now most people get to PyTorch’s internals by accident, because something broke badly enough to force the trip. They learn the codebase under deadline pressure, from the outside in, in whatever order the bug happened to demand. Anyone who has o…+
<p>Building the thing yourself changes the arrival path, and it changes it permanently. Once you have implemented autograd you cannot unsee the computational graph. Once you have profiled your own memory allocation you cannot unknow the cost.</p>+
<p><!-- IMAGE PLACEHOLDER: Figure 1 - Side-by-side comparison graphic captioned "Building systems creates irreversible understanding." Left panel, "Traditional ML Education": a short torch.nn.Linear snippet with "Problem: You can't debug what you don't understand." Right panel, "TinyTorch: Build → U…+
<p><img decoding="async" class="alignnone wp-image-167831 size-full" src="https://pytorch.org/wp-content/uploads/2026/09/Figure1.png" alt="" width="1656" height="658" srcset="https://pytorch.org/wp-content/uploads/2026/09/Figure1.png 1656w, https://pytorch.org/wp-content/uploads/2026/09/Figure1-300x…+
<p><em><strong>Figure 1</strong>: The same linear layer, two ways. On the left, the framework call that works right up until it does not. On the right, the version you wrote and can open when something breaks. The difference is not syntax, it is whether the abstraction is a wall or a door.</em></p>+
<p>A student who has written <code>backward()</code> themselves, who allocated the momentum and variance buffers and watched Adam’s memory footprint triple, shows up to PyTorch’s real autograd with the mental model already loaded. They read the production code as a more sophisticated ver…+
<p>The part we did not anticipate is that companies want this too. Teams have used TinyTorch for new-hire onboarding as a two to three week intensive, as internal training spread across a quarter, and as targeted debugging workshops where somebody works through Module 06 on autograd or Module 12 on …+
<h2>What We Built</h2>+
<p>Twenty modules in four tiers, driven by a CLI called <code>tito</code>, delivered as Jupyter notebooks with the hard parts cut out for you to fill in. You need Python and to be comfortable with NumPy. You do not need a GPU, a cloud account, or any prior ML systems background.</p>+
<p><!-- IMAGE PLACEHOLDER: Figure 2 - Diagram of the four tiers of TinyTorch modules, each tier depending on the one below it. --></p>+
<p><img decoding="async" class="alignnone wp-image-167832 size-full" src="https://pytorch.org/wp-content/uploads/2026/09/Figure2.png" alt="" width="1656" height="528" srcset="https://pytorch.org/wp-content/uploads/2026/09/Figure2.png 1656w, https://pytorch.org/wp-content/uploads/2026/09/Figure2-300x…+
<p><em><strong>Figure 2</strong>: The four tiers. Each one depends on the tier below it, so you cannot skip ahead to optimization without having built the training loop you are optimizing. Foundation fits a half-semester module, all twenty fit a four-credit course, and self-paced learners take anywh…+
<p>Three design decisions do most of the pedagogical work.</p>+
<p><strong>Systems from day one.</strong> Module 01 ships a <code>memory_footprint()</code> method before it ships matrix multiplication. You learn that one batch of 32 ImageNet images costs 19 MB by computing it, not by reading it somewhere. Later you find out Adam needs roughly 3× the optimizer me…+
<p><strong>Progressive disclosure.</strong> The <code>Tensor</code> class stays clean through Module 05, with no gradient machinery cluttering up data layout and arithmetic. Then in Module 06 you implement <code>enable_autograd()</code>, which bolts <code>requires_grad</code>, <code>.grad</code>, an…+
<p>We went back and forth on this one. The tasteful way to do it is inheritance. What we shipped is runtime monkey-patching, which is going to offend somebody reading this. It won because it keeps one <code>Tensor</code> class across all twenty modules instead of two, and because the moment your ten…+
<p><!-- IMAGE PLACEHOLDER: Figure 3 - Screenshot of a Module 06 notebook cell showing the docstring and numbered steps for implementing the gradient of matrix multiplication, with the implementation left blank for the learner to fill in. --></p>+
<p><img decoding="async" class="alignnone wp-image-167835 size-full" src="https://pytorch.org/wp-content/uploads/2026/09/Figure3.png" alt="" width="1834" height="1284" srcset="https://pytorch.org/wp-content/uploads/2026/09/Figure3.png 1834w, https://pytorch.org/wp-content/uploads/2026/09/Figure3-300…+
<p><em><strong>Figure 3</strong>: The gradient of matrix multiplication, as a learner meets it in Module 06. The docstring gives you the mathematical rule and the numbered approach breaks it into steps, but the implementation is yours. All twenty modules look like this.</em></p>+
<p><strong>Build to validate.</strong> Six historical milestones prove your implementation works. Rosenblatt’s Perceptron in 1958, the XOR crisis in 1969, the backpropagation revival in 1986, the CNN breakthrough in 1998 where your network has to clear 75% on CIFAR-10, the transformer in 2017,…+
<p><!-- IMAGE PLACEHOLDER: Figure 4 - Timeline/ladder graphic of the six historical milestones (Perceptron 1958 through MLPerf-style benchmarking), each milestone unlocking once the modules beneath it work. --></p>+
<p><img decoding="async" class="alignnone wp-image-167836 size-full" src="https://pytorch.org/wp-content/uploads/2026/09/Figure4.png" alt="" width="1060" height="1115" srcset="https://pytorch.org/wp-content/uploads/2026/09/Figure4.png 1060w, https://pytorch.org/wp-content/uploads/2026/09/Figure4-285…+
<p><em><strong>Figure 4</strong>: The milestone ladder. A milestone unlocks only when the modules beneath it produce a working implementation, so the timeline doubles as a progress tracker and a correctness proof. Students recreate 67 years of ML history running nothing but code they wrote.</em></p>+
<p>Throughout, TinyTorch mirrors PyTorch’s API deliberately. The API is the transfer mechanism and it carries both ways. Somebody who builds <code>loss.backward()</code> here can open PyTorch’s version afterward and recognize the shape of it. A PyTorch developer can read our attention mo…+
<p><!-- IMAGE PLACEHOLDER: Figure 5 - Side-by-side code comparison of a TinyTorch training loop and the equivalent PyTorch training loop, showing only the imports differ. --></p>+
<p><img decoding="async" class="alignnone wp-image-167839 size-full" src="https://pytorch.org/wp-content/uploads/2026/09/Figure5.png" alt="" width="2034" height="566" srcset="https://pytorch.org/wp-content/uploads/2026/09/Figure5.png 2034w, https://pytorch.org/wp-content/uploads/2026/09/Figure5-300x…+
<p><em><strong>Figure 5</strong>: A TinyTorch training loop next to the equivalent PyTorch loop. The imports differ and almost nothing else does, which is the entire point. Nothing you learn here has to be unlearned later.</em></p>+
<p>Two things it is not. The resemblance stops at the API surface, so there is no dispatcher, no C++ or CUDA layer, no JIT, nothing distributed. And it is slow. Pure Python runs somewhere between 100 and 10,000 times slower than PyTorch, which we will come back to, because it turned out to matter.</…+
<h2>Where It Came From</h2>+
<p>TinyTorch did not start as a framework. It started as a course with a problem.</p>+
<p>CS 249r launched at Harvard in 2020, a graduate seminar on TinyML, and there was no textbook to assign. So the course notes became one. We put the book up as an open repository, students and educators started fixing examples and proposing chapters, and by 2024 it had outgrown its TinyML origins b…+
<p>Somewhere in there it became obvious a textbook was not enough. Reading about autograd and implementing autograd produce different kinds of knowledge, and only one of them survives contact with a production bug. So the project grew limbs. TinyTorch is the build limb. Around it now sit Marimo labs…+
<p><!-- IMAGE PLACEHOLDER: Figure 6 - Map graphic showing the TinyTorch community's 682 members clustered across 92 institutions worldwide. --></p>+
<p><img decoding="async" class="alignnone wp-image-167840 size-full" src="https://pytorch.org/wp-content/uploads/2026/09/Picture6.png" alt="" width="1783" height="1195" srcset="https://pytorch.org/wp-content/uploads/2026/09/Picture6.png 1783w, https://pytorch.org/wp-content/uploads/2026/09/Picture6-…+
<p><em><strong>Figure 6</strong>: The TinyTorch community map, 682 members across 92 institutions since it launched in December 2025. The clustering is the interesting part. Uptake is heaviest where GPU access is hardest, which is what the accessibility floor was for.</em></p>+
<p>Then there is the part that is slightly embarrassing to write down.</p>+
<p>For five years this was a slow project. Steady, word of mouth, a few hundred stars a year, the kind of thing you keep doing because the students in front of you need it. In August 2025 the repository had about 2,000 stars.</p>+
<p>In October, somebody we had never met posted it on X.</p>+
<p>Two weeks later we were at 5,000. By December we had passed 10,000, and today it is above 27,000, with 95 or more contributors and courses running at 50 or more universities. We would like to tell you we engineered that. We did not. One person with reach decided the work was worth sharing, and fi…+
<p><!-- IMAGE PLACEHOLDER: Figure 7 - Line chart of GitHub stars over time for the Machine Learning Systems repository, showing a flat multi-year plateau followed by a sharp cliff/spike starting around October 2025. --></p>+
<p><img decoding="async" class="alignnone wp-image-167845 size-full" src="https://pytorch.org/wp-content/uploads/2026/09/Figure7.png" alt="" width="2147" height="1193" srcset="https://pytorch.org/wp-content/uploads/2026/09/Figure7.png 2147w, https://pytorch.org/wp-content/uploads/2026/09/Figure7-300…+
<p><em><strong>Figure 7</strong>: Stars on the Machine Learning Systems repository. The cliff is the part everyone looks at. The flat part is where the work happened.</em></p>+
<p>TinyTorch was built at Harvard as part of the <a href="https://mlsysbook.ai">Machine Learning Systems</a> project, and Andrea now maintains it from ETH Zurich, where both of us are currently based. Handing a curriculum to a second institution is the first honest test of whether you wrote it for a…+
<h2>What Transferred to Other Projects</h2>+
<p>These guidelines ask for lessons other institutions can apply, which is the right thing to ask for. Four of them.</p>+
<p><strong>Match the production API exactly.</strong> The highest-leverage decision we made was refusing to invent our own syntax. Every hour a learner spends translating between your teaching API and the real one is an hour that buys them nothing and costs you a fraction of them. Treat API compatib…+
<p><strong>Decide your hardware floor before your feature list.</strong> TinyTorch runs on a dual-core 2 GHz CPU with 4 GB of RAM and no network during training, because we ship two tiny offline datasets (about 1,000 grayscale digits and 350 conversational question-answer pairs, under 50 MB together…+
<p>The slowness we apologized for turned out to be the best accident in the project. When a student’s <code>Conv2d</code> takes 97 seconds on a batch that PyTorch clears in 10 milliseconds, the argument for vectorization stops being something they read and starts being something that happened …+
<p><strong>Instructor infrastructure is the actual bottleneck.</strong> Good content does not get adopted. Gradeable content gets adopted. We shipped NBGrader autograding with locked test cells and point allocations, an <code>INSTRUCTOR.md</code> covering setup and rubrics and the errors students ac…+
<p><strong>Plan succession before you need it.</strong> Academic open source dies when the PI changes focus. We are handling that in the open, with a maintenance commitment through 2027, a two-week pull request review target, and a governance transition across 2026 and 2027 that sets up an educator …+
<h2>What We Still Do Not Know</h2>+
<p>TinyTorch is in preview and aimed at classroom readiness for Fall 2026. The limits are worth more to you than a clean success story.</p>+
<p>We have not measured learning outcomes. The design leans on constructionism, cognitive apprenticeship, productive failure, threshold concepts, and five decades of evidence that build-it-yourself works in systems education. That is a strong prior. It is not a result. We do not have controlled data…+
<p>The scope is also narrower than the ambition. Single-node, CPU-only, so it teaches memory and compute well and teaches nothing about GPU kernels, distributed training, or gradient synchronization. Parallel data loading and GPU memory management are the one competency area we mapped out and then d…+
<h2>Where You Can Help</h2>+
<p>If you would rather just try it, installation is one line and everything runs locally.</p>+
<pre><code class="language-bash">curl -sSL mlsysbook.ai/tinytorch/install.sh | bash</code></pre>+
<p>Five openings, roughly in order of how much each would move things.</p>+
<ol>+
<li><strong>Review a module against real PyTorch semantics.</strong> If you work on PyTorch internals, an hour checking whether our autograd or our optimizer state handling or our KV cache teaches the right mental model is worth an enormous amount. Whatever TinyTorch teaches becomes what a cohort of…+
<li><strong>Pilot a tier and tell us what broke.</strong> Fall 2026 syllabi are mostly locked by now, so realistically that is Spring 2027 or Fall 2027, and shadowing the material this fall is a good way to decide. Foundation fits an undergraduate systems module, all twenty fit a semester, the Optim…+
<li><strong>Write the modules we cannot.</strong> Distributed training, GPU acceleration, parallel data loading. These need somebody who already teaches the material.</li>+
<li><strong>Localize it.</strong> The datasets are small and offline by design, which makes translating the conversational one into another language a weekend project with real reach.</li>+
<li><strong>Adopt it and say so.</strong> Putting your institution on the community map is what tells the next department this is a real option and not an experiment.</li>+
</ol>+
<p>Everything lives in the <a href="https://github.com/harvard-edge/cs249r_book">Machine Learning Systems repository</a>. Code is MIT, curriculum is CC BY-SA 4.0, so forking and adapting for your own institution is explicitly fine, and upstreaming what you fix is appreciated.</p>+
<p>All of it is free and it stays free. If the argument here landed and you want the cheapest way to act on it, star the repository. That number is not a scoreboard for us. It is what a department chair looks at when deciding whether an open curriculum is safe to build a course on, and what a funder…+
<h2>Join the PyTorch Academic OSPO Working Group</h2>+
<p>TinyTorch is one example of universities pushing the PyTorch ecosystem forward through open collaboration, and this post exists because of the PyTorch Academic OSPO Working Group, which has been helping us turn an enthusiastic pile of learners into something with actual governance. Several of the…+
<p>If you are interested in sharing academic projects, developing best practices, or connecting with people working where PyTorch meets academia, consider joining the <a href="https://github.com/pytorch-fdn/wg-ospo-and-academic-outreach">PyTorch Academic OSPO Working Group</a>. It welcomes researche…+
<p><img decoding="async" class="alignnone wp-image-167846 size-full" src="https://pytorch.org/wp-content/uploads/2026/09/ospoOutreach_horizontalColor-scaled.png" alt="" width="2560" height="1440" srcset="https://pytorch.org/wp-content/uploads/2026/09/ospoOutreach_horizontalColor-scaled.png 2560w, ht…+
]]></content:encoded>+
+
+
+
</item>+
<item> <title>PyTorch Day Japan 2026 Comes to Tokyo on December 10</title> <link>https://pytorch.org/blog/pytorch-day-japan-2026-comes-to-tokyo/</link> @
@@ -435,7 +534,7 @@ Tue 20 Oct, 16:20–16:30, LL20AB</p></tr></tbody></table>-
<p><span style="font-weight: 400;">This is the TMEM Alloc:</span><img fetchpriority="high" decoding="async" class="aligncenter wp-image-159911 size-full" src="https://pytorch.org/wp-content/uploads/2026/08/1.png" alt="" width="1788" height="426" srcset="https://pytorch.org/wp-content/uploads/2026/08…+
<p><span style="font-weight: 400;">This is the TMEM Alloc:</span><img decoding="async" class="aligncenter wp-image-159911 size-full" src="https://pytorch.org/wp-content/uploads/2026/08/1.png" alt="" width="1788" height="426" srcset="https://pytorch.org/wp-content/uploads/2026/08/1.png 1788w, https:/…<ul><li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">For the prologue S(i) GEMMs, we can use O(i) for S(i) SFs, as O(i) hasn’t started yet</span></li><li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">SFs for O(i) can live overlapped with S(i) – this is the same strategy used by regular FA which overlaps P(i) with S(i). Thus, there already exists a barrier which ensures that S(i) TMEM is consumed, before we copy O(…@
@@ -1507,365 +1606,8 @@ for n in [1024, 1 << 16, 1 << 20, 1 << 24]: -
</item>-
<item>-
<title>PyTorch 2.14 Release Blog</title>-
<link>https://pytorch.org/blog/pytorch-2-14-release-blog/</link>-
-
<dc:creator><![CDATA[PyTorch Foundation]]></dc:creator>-
<pubDate>Wed, 02 Sep 2026 18:41:48 +0000</pubDate>-
<category><![CDATA[Announcements]]></category>-
<category><![CDATA[Blog]]></category>-
<guid isPermaLink="false">https://pytorch.org/?p=161911</guid>-
-
<description><![CDATA[We are excited to announce the release of PyTorch® 2.14 (release notes)! The PyTorch 2.14 release features the following changes: NVGEMM brings CuTeDSL-generated CUTLASS kernels to Inductor, with epilogue fusion,...]]></description>-
<content:encoded><![CDATA[<p><span style="font-weight: 400;"><img decoding="async" class="alignnone wp-image-161929 size-full" src="https://pytorch.org/wp-content/uploads/2026/09/PyTorch-Release-Highlights-Graphic-3.png" alt="" width="1200" height="628" srcset="https://pytorch.org/wp-conte…-
<p><span style="font-weight: 400;">We are excited to announce the release of PyTorch® 2.14 (</span><a href="https://github.com/pytorch/pytorch/releases/tag/v2.14.0"><span style="font-weight: 400;">release notes</span></a><span style="font-weight: 400;">)!</span><span style="font-weight: 400;"><br />-
</span></p>-
<p><span style="font-weight: 400;">The PyTorch 2.14 release features the following changes:</span></p>-
<ul>-
<li style="font-weight: 400;" aria-level="1"><b>NVGEMM brings CuTeDSL-generated CUTLASS kernels to Inductor,</b><span style="font-weight: 400;"> with epilogue fusion, scaled and NVFP4 GEMM, and grouped-reduction epilogues autotuned alongside Triton and ATen</span></li>-
<li style="font-weight: 400;" aria-level="1"><b>A preview of our rewritten NCCL backend for PyTorch,</b><span style="font-weight: 400;"> ported from torchcomms, implementing the full collective contract with nonblocking communicators and </span><b>advanced features such as fault tolerance and window…-
<li style="font-weight: 400;" aria-level="1"><b>Fault tolerance becomes a first-class c10d concept,</b><span style="font-weight: 400;"> with in-place process-group reconfiguration, one-sided RMA windows, and a Flight Recorder that works for any backend rather than only NCCL</span></li>-
<li style="font-weight: 400;" aria-level="1"><b>Apple Silicon gains native linear algebra,</b><span style="font-weight: 400;"> including Jacobi-kernel SVD, eigh, QR and Cholesky, alongside a five-part reduction rewrite and a further MPSGraph to Metal kernel migration</span></li>-
<li style="font-weight: 400;" aria-level="1"><b>torch.switch generalizes torch.cond to multi-way branching,</b><span style="font-weight: 400;"> and torch.while_loop can now be captured in a CUDA graph</span></li>-
<li style="font-weight: 400;" aria-level="1"><b>Declarative dynamic shapes via @dynamic_spec,</b><span style="font-weight: 400;"> shared across torch.compile, torch.export and make_fx</span></li>-
<li style="font-weight: 400;" aria-level="1"><b>Broader platform support</b><span style="font-weight: 400;">: ROCm 7.14 wheels are produced from the TheRock pip SDK, Intel XPU adds native graph capture, and Inductor targets Rubin (sm_107)</span></li>-
<li style="font-weight: 400;" aria-level="1"><b>Experimental torch.compile support for complex-valued tensors:</b><span style="font-weight: 400;"> Opt-in support decomposes supported complex operations into real and imaginary computations, enabling compiler backends to optimize more complex-number w…-
</ul>-
<p><span style="font-weight: 400;">This release is composed of 2,995 commits from 487 contributors since PyTorch 2.13. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.14. More information ab…-
</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">Bring any questions you may have about this release to our Q&A Webinar. It will be on Thursday, September 17, 2026 and will feature Andrey Talman (Meta), Natalia Gimelshein (Meta), Joe Spisak (Reflection AI), and Chris Gottbrath (Gottbrath Tech, moderator) …-
<p><span style="font-weight: 400;">Connect with the global PyTorch community at the upcoming </span><a href="https://hubs.la/Q04vXWRN0"><span style="font-weight: 400;">PyTorch Conference North America</span></a><span style="font-weight: 400;"> October 20-21, 2026, in San Jose, CA. Explore what’s new…-
<p><span style="font-weight: 400;">Throughout the 2.x series, PyTorch has been evolving from a research-first framework into a unified, hardware-agnostic platform for production training and inference at scale. </span><a href="https://pytorch.org/blog/pytorch-2-12-release-blog/"><span style="font-we…-
<p><span style="font-weight: 400;">PyTorch 2.14 builds directly on those threads. The CuTeDSL path matures into NVGEMM, a full GEMM backend with epilogue fusion and low-precision support. torchcomms lands in-tree as </span><b>a preview of the rewritten NCCL backend</b><span style="font-weight: 400;"…-
<p><span style="font-weight: 400;">PyTorch 2.14 delivers meaningful improvements across performance, reliability, and hardware support. The release introduces NVGEMM, a new GPU math backend that automatically selects the fastest kernel for matrix operations — including support for lower-precision fo…-
<p><span style="font-weight: 400;">Apple Silicon users benefit from native linear-algebra routines (SVD, QR, Cholesky, and more) and a broad migration to hand-tuned Metal kernels that cut overhead on Mac GPUs. On the compiler side, new control-flow primitives (torch.switch, torch.while_loop) give mo…-
<p><span style="font-weight: 400;">Platform support expands to AMD ROCm 7.14, Intel XPU native graph capture, and NVIDIA’s next-generation Rubin architecture. Under the hood, the compiler now overlaps communication with computation by default, batches small GPU kernels more intelligently, and …-
<h2><span style="font-weight: 400;">Performance Improvements</span></h2>-
<h3><span style="font-weight: 400;">Large MPS Op Migration to Native Metal</span></h3>-
<p><span style="font-weight: 400;">Continuing the migration begun in 2.13, a further set of MPS operators moved off Apple’s MPSGraph framework onto hand-written Metal compute kernels, including </span><span style="font-weight: 400;"><code>index_add</code>, <code>index_select</code>, <code>argm…-
<p>Reductions received a dedicated five-part rewrite covering full reductions, inner-dimension reductions, strided and batched outer reductions, small-dimension and narrow kernels, and the <code>argmax</code> and <code>argmin</code> split-K paths. The final part migrates <code>min</code> and <code>m…-
<p><span style="font-weight: 400;">The native Metal path eliminates MPSGraph’s per-op compilation cost and gives PyTorch direct control over thread dispatch and memory access patterns, reducing kernel launch latency across common training and inference workloads on Apple Silicon.</span></p>-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/191101"><span style="font-weight: 400;">#191101</span></a><span style="font-weight: 400;">, </span><a href="https://github.com/pytorch/pytorch/pull/191097"><span style="font-weight: 400;">#191097</span></a…-
<h3><span style="font-weight: 400;">MPS Memory and Copy Paths</span></h3>-
<p><span style="font-weight: 400;">Long-running decode workloads grew the MPS caching allocator’s reserved footprint faster than necessary. The allocator now buckets large allocations to bound reserved memory and uses placement heaps to reduce fragmentation.</span></p>-
<p><span style="font-weight: 400;">Host and device transfers also got shorter paths. CPU to MPS copies blit directly from pinned buffers with event-deferred reclaim, contiguous same-dtype copies use a compute kernel instead of a graph, elementwise ops are vectorized on inner-contiguous sliced views,…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/187441"><span style="font-weight: 400;">#187441</span></a><span style="font-weight: 400;"> and </span><a href="https://github.com/pytorch/pytorch/pull/190438"><span style="font-weight: 400;">#190438</span>…-
<h3><span style="font-weight: 400;">F.linear Decode Path on MPS</span></h3>-
<p><span style="font-weight: 400;">Single-token decode passes a </span><span style="font-weight: 400;"><code>[B, 1, K]</code></span><span style="font-weight: 400;"> activation to </span><span style="font-weight: 400;"><code>F.linear</code></span><span style="font-weight: 400;">, a shape that was fal…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/189855"><span style="font-weight: 400;">#189855</span></a><span style="font-weight: 400;"> by Giovanni Versiglioni, Apple, </span><a href="https://github.com/pytorch/pytorch/pull/186927"><span style="font-…-
<h3><span style="font-weight: 400;">Compute and Communication Overlap On By Default in Inductor</span></h3>-
<p><span style="font-weight: 400;">Inductor’s </span><span style="font-weight: 400;"><code>simple_overlap</code></span><span style="font-weight: 400;"> reordering, which interleaves collectives with independent compute so communication is not left on the critical path, is now enabled by defaul…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/184240"><span style="font-weight: 400;">#184240</span></a><span style="font-weight: 400;">, #</span><a href="https://github.com/pytorch/pytorch/pull/184235"><span style="font-weight: 400;">184235</span></a…-
<h3><span style="font-weight: 400;">reorder_for_locality Opt-In for Training Graphs</span></h3>-
<p><span style="font-weight: 400;"><code>reorder_for_locality</code>,</span><span style="font-weight: 400;"> Inductor’s post-grad locality reordering pass, can now be opted into on training graphs via the new reorder_for_locality_in_training config (default off), where before it only ran on in…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/186643"><span style="font-weight: 400;">#186643</span></a><span style="font-weight: 400;"> by @reger-men)</span></p>-
<h3><span style="font-weight: 400;">Combo Kernels and Reductions in Inductor</span></h3>-
<p><span style="font-weight: 400;">Combo kernels batch many small kernels into one launch, but a single very large reduction in the batch would previously shape the whole kernel. Large reductions are now split out of combo partitioning, combo reductions get dynamic </span><span style="font-weight: 4…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/186668"><span style="font-weight: 400;">#186668</span></a><span style="font-weight: 400;">, </span><a href="https://github.com/pytorch/pytorch/pull/186957"><span style="font-weight: 400;">#186957</span></a…-
<h3><span style="font-weight: 400;">Dynamo Per-Call Overhead</span></h3>-
<p> </p>-
<p>For models with many small compiled regions, fixed per-call cost matters more than graph quality. This release trims that cost in several places. <code>compile_wrapper</code> avoids <code>DispatchKeySet</code> pybind churn on every call, <code>torch._dynamo.disable</code> gets a cheaper path, and…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/190390"><span style="font-weight: 400;">#190390</span></a><span style="font-weight: 400;">, </span><a href="https://github.com/pytorch/pytorch/pull/190392"><span style="font-weight: 400;">#190392</span></a…-
<h3><span style="font-weight: 400;">Eager Dispatch and CPU Kernels</span></h3>-
<p>Several eager-mode hot paths got cheaper. PyObject dispatch is optimized, AOTAutograd avoids an expensive <code>Tensor.detach()</code> when saving graph-input views for backward, autograd stops copying <code>at::Tensor</code> when the profiler is off, <code>addmm</code> avoids a device-to-device …-
<p>These targeted fixes reduce the per-operation tax in eager mode, tightening the performance floor so that common operations like linear layers, autograd bookkeeping, and statistical aggregations don’t carry unnecessary overhead. They keep PyTorch’s default development experience fast …-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/187949"><span style="font-weight: 400;">#187949</span></a> <a href="https://github.com/pytorch/pytorch/pull/189759"><span style="font-weight: 400;">#189759</span></a><span style="font-weight: 400;"> and </…-
<h2><span style="font-weight: 400;">Core Features</span></h2>-
<h3><span style="font-weight: 400;">torch.linalg.polar and torch.linalg.matrix_sqrth</span></h3>-
<p><span style="font-weight: 400;">Two additions to </span><span style="font-weight: 400;"><code>torch.linalg</code>. <code>torch.linalg.polar</code></span><span style="font-weight: 400;"> computes the polar decomposition using cuSOLVER’s QDWH algorithm, with a backward formula on CPU, CUDA an…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/185837"><span style="font-weight: 400;">#185837</span></a><span style="font-weight: 400;"> by Simon Layton, Meta, </span><a href="https://github.com/pytorch/pytorch/pull/189732"><span style="font-weight: 4…-
<h3><span style="font-weight: 400;">Autograd Extension Points</span></h3>-
<p>Three additions give more control over how the autograd graph is built and inspected. <code style="color: #097536; background: none; padding: 0;">torch.autograd.graph.node_creation_hook</code> fires as each autograd node is created, letting tools attach metadata or register hooks at graph-constru…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/189284"> <span style="font-weight: 400;">#189284</span></a><span style="font-weight: 400;"> by Edward Yang, Meta,</span><a href="https://github.com/pytorch/pytorch/pull/189634"> <span style="font-weight…-
<h3><span style="font-weight: 400;">torch.switch Higher-Order Op</span></h3>-
<p><span style="font-weight: 400;"><code>torch.cond</code></span><span style="font-weight: 400;"> expresses a two-way branch, so an n-way dispatch had to be written as nested conditionals, which grows the traced graph and obscures intent. </span><span style="font-weight: 400;"><code>torch.switch</co…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/182902"><span style="font-weight: 400;">#182902</span></a><span style="font-weight: 400;"> and </span><a href="https://github.com/pytorch/pytorch/pull/188374"><span style="font-weight: 400;">#188374</span>…-
<h3><span style="font-weight: 400;">SDPA Fused Backends for Rank-3 Inputs</span></h3>-
<p><span style="font-weight: 400;">Scaled dot-product attention now dispatches to the fused CUDA backends for rank-3 inputs instead of falling back to the math path, so callers who pass unbatched or already-flattened tensors get the fused kernels without reshaping. The fix closes a common performanc…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/192271"><span style="font-weight: 400;">#192271</span></a><span style="font-weight: 400;"> by Driss Guessous, Meta)</span></p>-
<h3><span style="font-weight: 400;">Experimental torch.compile support for complex-valued tensors</span></h3>-
<p><span style="font-weight: 400;">torch.compile support for programs using complex-valued tensors. Supported complex operations are decomposed into real-valued computations that compiler backends can optimize. This enables more complex-number workloads including signal processing, scientific comput…-
<p><span style="font-weight: 400;">(PRs </span><a href="https://github.com/pytorch/pytorch/pull/167621"><span style="font-weight: 400;">#167621</span></a><span style="font-weight: 400;"> and </span><a href="https://github.com/pytorch/pytorch/pull/169832"><span style="font-weight: 400;">#169832</span…-
<h3><span style="font-weight: 400;">Smaller API Additions</span></h3>-
<p><span style="font-weight: 400;">A number of smaller public additions landed this release.</span></p>-
<ul>-
<li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;"><code>torch.utils.checkpoint.checkpoint</code></span><span style="font-weight: 400;"> accepts a decorator and curried calling convention in eager (</span><a href="https://github.com/pytorch/pytorch/pull/189411"><span style=…-
<li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;">Read-only DLPack export and </span><span style="font-weight: 400;"><code>ReadOnlyTensorWrapper</code></span><span style="font-weight: 400;">, so consumers can be handed a tensor they must not mutate (</span><a href="https:/…-
<li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;"><code>Generator.philox_state</code></span><span style="font-weight: 400;"> exposes Philox RNG state reservation to Python (</span><a href="https://github.com/pytorch/pytorch/pull/191019"><span style="font-weight: 400;">#191…-
<li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;"><code>torch.accelerator</code></span><span style="font-weight: 400;"> gains </span><span style="font-weight: 400;"><code>initial_seed</code>, <code>get_rng_state</code></span><span style="font-weight: 400;"> and </span><spa…-
<li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;"><code>LBFGS</code></span><span style="font-weight: 400;"> gains </span><span style="font-weight: 400;"><code>maximize</code></span><span style="font-weight: 400;"> and is a no-op on an empty parameter group (</span><a href=…-
<li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;"><code>linear_cross_entropy</code></span><span style="font-weight: 400;">, introduced in 2.13, supports probability targets on the chunked path (</span><a href="https://github.com/pytorch/pytorch/pull/187053"><span style="fo…-
<li style="font-weight: 400;" aria-level="1"><span style="font-weight: 400;"><code>c10::utils::get_env</code></span><span style="font-weight: 400;"> and </span><span style="font-weight: 400;"><code>set_env</code></span><span style="font-weight: 400;"> are exposed to Python (</span><a href="https://g…-
</ul>-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<h3><span style="font-weight: 400;">Python 3.15 Support and Torchvision ABI Stability – Release Engineering</span></h3>-
<p><span style="font-weight: 400;">PyTorch 2.14 adds binary support for Python 3.15, including the free-threaded (no-GIL) build, 3.15t across all platforms. Wheels are published for Linux on x86_64 and aarch64, Windows, and macOS on Apple silicon, spanning the CPU, CUDA, ROCm, and XPU builds. Also t…-
<p><span style="font-weight: 400;">TorchVision is now ABI stable w.r.t. torch 2.14! This means that torchvision 0.29 will be compatible with future versions of torch: 2.15, 2.16, etc. You won’t need to install a new version of TorchVision when you upgrade torch. As a result, we might stop releasing …-
<h3><span style="font-weight: 400;">Installation</span></h3>-
<p><span style="font-weight: 400;">Python 3.15 and 3.15t wheels are not published to PyPI — they are available to download only via download.pytorch.org, using any of the following commands:</span></p>-
<p><span style="font-weight: 400;"> # CPU</span></p>-
<p><span style="font-weight: 400;"> pip3 install torch –index-url https://download.pytorch.org/whl/cpu</span></p>-
<p><span style="font-weight: 400;"> # CUDA (substitute the CUDA version, e.g. cu126 / cu130)</span></p>-
<p><span style="font-weight: 400;"> pip3 install torch –index-url https://download.pytorch.org/whl/cu130</span></p>-
<p><span style="font-weight: 400;"> # ROCm (substitute the ROCm version)</span></p>-
<p><span style="font-weight: 400;"> pip3 install torch –index-url https://download.pytorch.org/whl/rocm7.14</span></p>-
<p><span style="font-weight: 400;"> # XPU</span></p>-
<p><span style="font-weight: 400;"> pip3 install torch –index-url https://download.pytorch.org/whl/xpu</span></p>-
<p><span style="font-weight: 400;"> The same commands install the free-threaded 3.15t build when run under a free-threaded interpreter.</span></p>-
<p><span style="font-weight: 400;">The same applies to the free-threaded build. Install into a 3.15t interpreter and pip will resolve the cp315t wheels automatically.</span></p>-
<h3><span style="font-weight: 400;">torch.compile is not yet supported on Python 3.15</span></h3>-
<p><span style="font-weight: 400;">Python 3.15 support in 2.14 is eager-only. Calling torch.compile under Python 3.15 raises a RuntimeError rather than falling back silently, so the limitation surfaces immediately rather than as a silent performance loss. If your workload depends on torch.compile, s…-
<p><span style="font-weight: 400;">Dynamo support for 3.15 is in active development, with bytecode and symbolic-conversion handling already landed for the new interpreter. Progress is tracked in </span><a href="https://github.com/pytorch/pytorch/issues/184352"><span style="font-weight: 400;">pytorch…-
<h2><span style="font-weight: 400;">Distributed Training</span></h2>-
<h3>Modern NCCL Backend</h3>-
<p><span style="font-weight: 400;">torchcomms arrived in 2.13 as a communications backend integrated into PyTorch Distributed’s CI and device-mesh paths. In 2.14, the APIs landed in-tree </span><b>as a drop-in replacement “</b><b>nccl2″</b><b> c10d backend, and will be default from…-
<p>API Unstable</p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/188582"><span style="font-weight: 400;">#188582</span></a><span style="font-weight: 400;">, </span><a href="https://github.com/pytorch/pytorch/pull/189359"><span style="font-weight: 400;">#189359</span></a…-
<h3><span style="font-weight: 400;">Fault-Tolerant Collectives in c10d</span></h3>-
<p><span style="font-weight: 400;">When a rank fails in a large job, the usual recovery is to tear down the process group and restart, which discards warm state across the whole cluster. </span><span style="font-weight: 400;">Backend</span><span style="font-weight: 400;"> and </span><span style="fon…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/186298"><span style="font-weight: 400;">#186298</span></a><span style="font-weight: 400;">, </span><a href="https://github.com/pytorch/pytorch/pull/186300"><span style="font-weight: 400;">#186300</span></a…-
<h3><span style="font-weight: 400;">One-Sided (RMA) Window APIs</span></h3>-
<p><span style="font-weight: 400;">Backend</span><span style="font-weight: 400;"> and </span><span style="font-weight: 400;">ProcessGroup</span><span style="font-weight: 400;"> gain one-sided window interfaces, giving remote-memory-access semantics alongside the existing two-sided collectives. One-s…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/186299"><span style="font-weight: 400;">#186299</span></a><span style="font-weight: 400;"> and </span><a href="https://github.com/pytorch/pytorch/pull/189360"><span style="font-weight: 400;">#189360</span>…-
<h3><span style="font-weight: 400;">Backend-Agnostic Flight Recorder</span></h3>-
<p><span style="font-weight: 400;">Flight Recorder, the collective trace buffer used to diagnose hangs and mismatched collectives, was tied to NCCL. </span><span style="font-weight: 400;"><code>FlightRecorderHook</code></span><span style="font-weight: 400;"> records through <code>ProcessGroup</code>…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/189363"><span style="font-weight: 400;">#189363</span></a><span style="font-weight: 400;"> by Tristan Rice, Meta, </span><a href="https://github.com/pytorch/pytorch/pull/185010"><span style="font-weight: 4…-
<h3><span style="font-weight: 400;">Pluggable Distributed Backends</span></h3>-
<p><span style="font-weight: 400;">Adding a communications backend previously meant patching c10d. Backends can now register through Python entry points, backend strings are auto-qualified, and implementation accessors are exposed. We’ve brought the </span><span style="font-weight: 400;"><code…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/187388"><span style="font-weight: 400;">#187388</span></a><span style="font-weight: 400;">, </span><a href="https://github.com/pytorch/pytorch/pull/186853"><span style="font-weight: 400;">#186853</span></a…-
<h3><span style="font-weight: 400;">torch.distributed API improvements: set_timeout, per-op timeouts, get_backend_impl, hooks, weights_only=True, *_single</span></h3>-
<p>We’ve made a whole host of improvements to the torch.distributed API which allow for more control as well as cleaning up some inconsistencies. You can change a process group’s collective timeout after initialization — extending it around a slow checkpoint load, or shortening it so a w…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/187387"><span style="font-weight: 400;">#187387</span></a><span style="font-weight: 400;"> and </span><a href="https://github.com/pytorch/pytorch/pull/187693"><span style="font-weight: 400;">#187693</span>…-
<h3><span style="font-weight: 400;">DTensor Single-Dim Sharding Strategies</span></h3>-
<p>DTensor’s sharding rules were historically written per operator against the whole device mesh, so each rule had to enumerate every combination of placements across all mesh dimensions — long to write and easy to get subtly wrong once the mesh had more than one dimension. This release contin…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/186667"> <span style="font-weight: 400;">#186667</span></a><span style="font-weight: 400;">,</span><a href="https://github.com/pytorch/pytorch/pull/179203"> <span style="font-weight: 400;">#179203</span></a…-
<h3><span style="font-weight: 400;">Symmetric Memory: NCCL backend fixes and allocation layout</span></h3>-
<p> </p>-
<p>Symmetric memory’s NCCL backend had gaps that only surfaced at runtime: <code style="color: #097536; background: none; padding: 0;">barrier()</code> raised a not-implemented error, and the signal pad was never zeroed after allocation, so the signaling protocol had nothing reliable to build …-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">(PRs </span><a href="https://github.com/pytorch/pytorch/pull/188051"><span style="font-weight: 400;">#188051</span></a><span style="font-weight: 400;"> by Kapil Sharma, Meta, </span><a href="https://github.com/pytorch/pytorch/pull/189088"><span style="font-weig…-
<h3><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">Symmetric Memory: reaching NCCL symmetric kernels from ordinary collectives</span></h3>-
<p>Symmetric memory based kernels became available on the NVLink domain in NCCL 2.27. These have been implemented in PyTorch’s symmetric memory in nightly builds since Jan 2026 but weren’t documented in a way that made it easy for users. Responding to feedback we improved the documentation in this r…-
<p><span style="font-weight: 400;">API Unstable</span><i><span style="font-weight: 400;"><br />-
</span></i><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/192515"><span style="font-weight: 400;">#192515</span></a><span style="font-weight: 400;"> by Kapil Sharma, Meta)</span></p>-
<h4><b><strong>Symmetric Memory: one-sided</strong> <strong><code style="color: #097536; background: none; padding: 0;">get</code></strong></b></h4>-
<p><span style="font-weight: 400;">Reading data that lives on another rank has generally meant a collective: every rank participates and synchronizes, even when only one rank actually needs the data. Symmetric memory now exposes </span><code style="color: #097536; background: none; padding: 0;">get<…-
<p><span style="font-weight: 400;">API Unstable</span><i><span style="font-weight: 400;"><br />-
</span></i><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/182378"><span style="font-weight: 400;">#182378</span></a><span style="font-weight: 400;"> by Benjamin Brock, Intel)</span></p>-
<h3><span style="font-weight: 400;">TokenSwitch</span></h3>-
<p>Mixture-of-experts training spends much of its step time sending each token to the ranks holding its chosen experts and bringing the expert outputs back, and teams generally wire that up themselves against a vendor kernel library, backward pass included. <code style="color: #097536; background: n…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/178712"><span style="font-weight: 400;">#178712</span></a><span style="font-weight: 400;"> and </span><a href="https://github.com/pytorch/pytorch/pull/181314"><span style="font-weight: 400;">#181314</span>…-
<h3><span style="font-weight: 400;">Compile-on-One-Rank</span></h3>-
<p> </p>-
<p>Every rank in a distributed job compiles the same model independently, so a single multi-minute compile is paid N times over before training starts. Compile-on-one-rank makes one compiled artifact reusable everywhere: <code style="color: #097536; background: none; padding: 0;">make_fx</code> no l…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/187869"> <span style="font-weight: 400;">#187869</span></a><span style="font-weight: 400;">,</span><a href="https://github.com/pytorch/pytorch/pull/186892"> <span style="font-weight: 400;">#186892</span></a…-
<h2><span style="font-weight: 400;">Compilation and Export</span></h2>-
<h3><span style="font-weight: 400;">Declarative Dynamic Shapes with @dynamic_spec</span></h3>-
<p> </p>-
<p>Telling PyTorch which input dimensions vary has meant a different mechanism per entry point — a <code style="color: #097536; background: none; padding: 0;">dynamic_shapes</code> dict for <code style="color: #097536; background: none; padding: 0;">torch.export</code>, a coarse <code style="color: …-
<p><span style="font-weight: 400;">API Unstable</span><b><br />-
</b><span style="font-weight: 400;"> (PR</span><a href="https://github.com/pytorch/pytorch/pull/187639"> <span style="font-weight: 400;">#187639</span></a><span style="font-weight: 400;">,</span><a href="https://github.com/pytorch/pytorch/pull/185982"> <span style="font-weight: 400;">#185982</span><…-
<h3><span style="font-weight: 400;">AOTInductor External Constants and Zero-Copy Weight Sharing</span></h3>-
<p><span style="font-weight: 400;">Serving several AOTInductor models that share the same weights used to mean every model container allocating and loading its own copy on the GPU. A new C API, </span><span style="font-weight: 400;"><code style="color: #097536; background: none; padding: 0;">AOTIndu…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;"> (PR</span><a href="https://github.com/pytorch/pytorch/pull/188643"> <span style="font-weight: 400;">#188643</span></a><span style="font-weight: 400;"> by @iuliur-meta)</span></p>-
<h3><span style="font-weight: 400;">AOTInductor Compilation</span></h3>-
<p> </p>-
<p>Packaging a model with <code style="color: #097536; background: none; padding: 0;">triton.autotune_at_compile_time=False</code> used to run the whole codegen twice: compile, run to collect kernel metadata, reset state, then recompile for packaging. That path now emits the JIT and AOTI wrapper bod…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;"> (PR</span><a href="https://github.com/pytorch/pytorch/pull/184735"> <span style="font-weight: 400;">#184735</span></a><span style="font-weight: 400;"> and</span><a href="https://github.com/pytorch/pytorch/pull/184736"> <span style="font-weight: 400;">#184736</…-
<h3><span style="font-weight: 400;">AOTInductor Constant Loading</span></h3>-
<p> </p>-
<p>Loading a model’s weights copies them from host memory to the GPU, and a synchronous copy out of pageable memory forces a device-wide synchronization that stalls inference already running on other streams. <code style="color: #097536; background: none; padding: 0;">AOTInductorSetUsePinnedAs…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;"> (PR</span><a href="https://github.com/pytorch/pytorch/pull/186258"> <span style="font-weight: 400;">#186258</span></a><span style="font-weight: 400;"> and</span><a href="https://github.com/pytorch/pytorch/pull/186309"> <span style="font-weight: 400;">#186309</…-
<h3><span style="font-weight: 400;">Helion Backend Integration</span></h3>-
<p>Writing a fast GPU kernel by hand means choosing tile sizes, loop order, and memory access patterns, then re-tuning all of it for every new shape and every new GPU. Helion raises that a level: you write the algorithm in Python, and Helion searches the schedule space and emits Triton for you. PyTo…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR </span><a href="https://github.com/pytorch/pytorch/pull/190636"><span style="font-weight: 400;">#190636</span></a><span style="font-weight: 400;"> by Karthick Panner Selvam, Meta)</span></p>-
<h2><span style="font-weight: 400;">Platform Features and Updates</span></h2>-
<h3><span style="font-weight: 400;">CUDA</span></h3>-
<h4><span style="font-weight: 400;">NVGEMM, a CuTeDSL GEMM Backend for Inductor</span></h4>-
<p>PyTorch 2.13 introduced the NVGEMM CuTeDSL backend for TorchInductor and in this release we are excited to expand support to epilogue fusion — the previous version could emit a standalone kernel, and whatever followed it (a bias add, an activation, a rescale) stayed in a separate kernel that re-r…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">(PRs</span><a href="https://github.com/pytorch/pytorch/pull/186183"> <span style="font-weight: 400;">#186183</span></a><span style="font-weight: 400;">,</span><a href="https://github.com/pytorch/pytorch/pull/187013"> <span style="font-weight: 400;">#187013</spa…-
<h4><span style="font-weight: 400;">CUDA Graph Lifecycle Hooks</span></h4>-
<p>Tools that want to watch CUDA graphs from the outside, like a profiler or a memory tracker, could only register per-graph hooks, which is no help when the graph was constructed by Inductor or NCCL rather than by the tool. This release adds module-level hooks that fire for every graph in the proce…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/190582"> <span style="font-weight: 400;">#190582</span></a><span style="font-weight: 400;"> and</span><a href="https://github.com/pytorch/pytorch/pull/190602"> <span style="font-weight: 400;">#190602</s…-
<h4><span style="font-weight: 400;">Multiple Memory Pools in a Single CUDA Graph</span></h4>-
<p>A <code style="color: #097536; background: none; padding: 0;">CUDAGraph</code> capture previously bound to exactly one memory pool, which meant allocations that need to come from a separate pool — symmetric memory being the case that forced the issue — couldn’t participate in a captured reg…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;"> (PR</span><a href="https://github.com/pytorch/pytorch/pull/187929"> <span style="font-weight: 400;">#187929</span></a><span style="font-weight: 400;"> by @Aidyn-A)</span></p>-
<h4><span style="font-weight: 400;">CUDA Graph Capture for torch.while_loop</span></h4>-
<p>Data-dependent loop counts have been one of the standard reasons a workload can’t be fully CUDA-graph captured, forcing a device-to-host copy to decide how many iterations to run and breaking the capture at exactly the point you’d rather keep it intact. <code style="color: #097536; ba…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;"> (PR</span><a href="https://github.com/pytorch/pytorch/pull/186055"> <span style="font-weight: 400;">#186055</span></a><span style="font-weight: 400;"> by Daniel Galvez, NVIDIA)</span></p>-
<h4><span style="font-weight: 400;">Kernel Annotations for CUDA Graphs</span></h4>-
<p><code style="color: #097536; background: none; padding: 0;">torch.cuda.graph_annotations</code> makes the kernel-annotation API used with CUDA graph capture public: <code style="color: #097536; background: none; padding: 0;">mark_kernels</code> lets you tag GPU work with a name so it shows up lab…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/189417"> <span style="font-weight: 400;">#189417</span></a><span style="font-weight: 400;"> and</span><a href="https://github.com/pytorch/pytorch/pull/191563"> <span style="font-weight: 400;">#191563</s…-
<h4><span style="font-weight: 400;">Post-Facto Memory Snapshot Annotations</span></h4>-
<p> </p>-
<p>Memory snapshots already let you attach metadata to an allocation, but only at the moment it’s created — some information, like whether a tensor ended up retained by the autograd graph, is only known later, after it’s been packed into the backward tape. <code style="color: #097536; ba…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/190575"> <span style="font-weight: 400;">#190575</span></a><span style="font-weight: 400;"> by Edward Yang, Meta)</span></p>-
<h4><span style="font-weight: 400;">TunableOp on CUDA</span></h4>-
<p>TunableOp profiles the available GEMM implementations for each input shape at runtime and caches the fastest, but on CUDA builds it previously had only one candidate to pick from — the cuBLAS default. It now registers cuBLASLt heuristic candidates too, with the candidate count set by <code style=…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;"> (PR</span><a href="https://github.com/pytorch/pytorch/pull/186270"> <span style="font-weight: 400;">#186270</span></a><span style="font-weight: 400;"> by Grayson Derossi, NVIDIA and</span><a href="https://github.com/pytorch/pytorch/pull/189355"> <span style="…-
<h4><span style="font-weight: 400;">cuBLASLt as a grouped GEMM backend</span></h4>-
<p>Grouped GEMM drives MoE layers, where many differently-shaped matmuls are issued together. cuBLASLt joins CUTLASS and the fallback as a backend: default for fp16 on Blackwell with CUDA 13.2+ and Hopper with CUDA 13.3+, opt-in for bf16 on the same combinations via <code style="color: #097536; back…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;"> (PR</span><a href="https://github.com/pytorch/pytorch/pull/177037"> <span style="font-weight: 400;">#177037</span></a><span style="font-weight: 400;"> by Grayson Derossi, NVIDIA)</span></p>-
<h3><span style="font-weight: 400;">ROCm</span></h3>-
<h4><span style="font-weight: 400;">Grouped GEMM, CK Templates and Origami (ROCm GEMM)</span></h4>-
<p><span style="font-weight: 400;">Mixture-of-experts models on AMD GPUs used to miss out on Inductor’s Triton-compiled grouped GEMM, which was previously limited to NVIDIA SM90+ hardware — ROCm fell back to a slower for-loop over hipBLASLt/rocBLAS calls instead. This release brings that Trito…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/188600"> <span style="font-weight: 400;">#188600</span></a><span style="font-weight: 400;"> and</span><a href="https://github.com/pytorch/pytorch/pull/188742"> <span style="font-weight: 400;">#188742</s…-
<h4><span style="font-weight: 400;">FlexAttention Tile Configs for RDNA3</span></h4>-
<p><span style="font-weight: 400;">FlexAttention on AMD’s RDNA3 GPUs (Radeon workstation and consumer cards, not the MI-series datacenter line) was using tile sizes that weren’t tuned for the architecture, leaving performance on the table for short-to-medium sequence lengths. This releas…-
<p><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/177840"> <span style="font-weight: 400;">#177840</span></a><span style="font-weight: 400;"> by Robert Esclapez, AMD)</span></p>-
<h3><span style="font-weight: 400;">MPS (Apple Silicon)</span></h3>-
<h4><span style="font-weight: 400;">Native Linear Algebra</span></h4>-
<p>MPS linear algebra has historically leaned on Apple’s MPSGraph primitives or fallen back to CPU entirely for anything beyond the basics, which made mixed CPU/MPS round-trips a common source of slowdown in numerical code. This release replaces several of those gaps with native Metal kernels.…-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/185954"> <span style="font-weight: 400;">#185954</span></a><span style="font-weight: 400;"> by Darko </span><span style="font-weight: 400;">SImonovski</span><span style="font-weight: 400;">,</span><a href="…-
<h4><span style="font-weight: 400;">FlexAttention Improvements</span></h4>-
<p>Building on FlexAttention’s arrival on MPS in 2.13, this release rounds out several gaps that showed up once people started using it for real models. KV batch broadcasting lets key/value tensors be shared across the query batch instead of requiring an exact match — a prerequisite for paged …-
<p><span style="font-weight: 400;">API Unstable</span></p>-
<p><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/187722"> <span style="font-weight: 400;">#187722</span></a><span style="font-weight: 400;">,</span><a href="https://github.com/pytorch/pytorch/pull/187768"> <span style="font-weight: 400;">#187768</span></a…-
</span></p>-
<h4><span style="font-weight: 400;">MPS Prefill Attention Acceleration</span></h4>-
<p><span style="font-weight: 400;">Apple’s Metal Performance Primitives (MPP), new in macOS 26.2, expose lower-level building blocks for attention-style workloads that weren’t previously available to Metal kernels. This release takes advantage of them with a second prefill attention kern…-
</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/182256"><span style="font-weight: 400;"> #182256</span></a><span style="font-weight: 400;"> by Irakli Salia, Hugging Face)</span></p>-
<h4><span style="font-weight: 400;">MPS acceleration for CTC Loss </span></h4>-
<p><code style="color: #097536; background: none; padding: 0;">ctc_loss</code> — the loss behind alignment-free sequence models like speech recognition and OCR — gets both forward and backward passes on MPS for the first time, closing a gap that previously forced Mac users to fall back to CPU for th…-
</span><span style="font-weight: 400;">API Unstable</span><span style="font-weight: 400;"><br />-
</span><span style="font-weight: 400;">(PR</span><a href="https://github.com/pytorch/pytorch/pull/187716"> <span style="font-weight: 400;">#187716</span></a><span style="font-weight: 400;"> and</span><a href="https://github.com/pytorch/pytorch/pull/188187"> <span style="font-weight: 400;">#188187</s…-
<h3><span style="font-weight: 400;">XPU (Intel GPUs)</span></h3>-
<h4><span style="font-weight: 400;">Enhanced XPU Graph Performance</span></h4>-
<p><span style="font-weight: 400;">Reduced graph capture and replay overhead in XPU Graph, improving execution efficiency and delivering better performance for graph-based training and inference workloads on Intel® Arc<img src="https://s.w.org/images/core/emoji/17.0.2/72x72/2122.png" alt="™" class="…Diff display stops at 400 lines. The line counts above are from the whole diff. 144 lines shown here cut at 300 characters. The raw artifact at this commit is linked above.