Google unveils TorchTPU in a push to break Nvidia’s stranglehold on AI compute

Google unveils TorchTPU in a push to break Nvidia’s stranglehold on AI compute

Google LLC is striving to pry open Nvidia Corp.’s grip on the artificial intelligence computing industry with the launch of a specialized version of the open-source PyTorch framework, called TorchTPU, which is designed to eliminate a key advantage of its rival.

One of the reasons why Nvidia and its graphics processing units remain so dominant in the AI industry is that the vast majority of AI applications are written specifically to run on its GPUs, and it’s this reality that Google is trying to change with today’s release.

The PyTorch framework is a popular open-source toolkit that developers use to create, train and run AI models and neural networks. According to some accounts, more than 85% of all AI applications are created using PyTorch, which was created by Meta Platforms Inc. in 2016, long before anyone realized that large language models would come along to enable the AI boom.

Back then, Nvidia’s GPUs were already the preferred hardware used by developers for AI research, so Meta designed PyTorch specifically to work with that company’s chips. The focus on Nvidia’s hardware has continued, even after Meta open-sourced PyTorch in 2022, handing over stewardship to the nonprofit Linux Foundation.

With TorchTPU, Google has created an alternative version of PyTorch that makes it easier for developers to create models and applications that can run on its Tensor Processing Units or TPUs, which are specialized chips for AI workloads. TorchTPU also provides a simple way for developers to run existing apps and models on TPUs by making a few simple configuration tweaks, without having to rewrite any of the underlying code.

Google isn’t alone in trying to undermine the massive advantage Nvidia had thanks to PyTorch. In fact, it was just earlier this week that Amazon Web Services Inc. announced its own flavor of PyTorch, called TorchNeuron. Similar to TorchTPU, TorchNeuron makes it easy to run AI applications on different hardware, in this case Amazon’s Trainum chips. Amazon went even further, revealing that it has contributed code to the original PyTorch project that will decouple a number of key features from Nvidia’s GPUs, with the aim being to make it more compatible with any kind of hardware.

Google didn’t do that, but it may have gained an advantage over AWS in a couple of other ways. For one thing, the company said it collaborated closely with Meta, Hugging Face and a couple of startups called Inferact Inc. and RadixArk Inc., the developers of vLLM and SGLang, to ensure that various TorchTPU features and code libraries are compatible with their developer tools. With that, it has erased another advantage PyTorch brings.

Google also did something else that AWS didn’t. It revealed a new tool called the TPU agent suite that is meant to help developers move their AI applications to the Google Cloud platform and automatically fix any errors that occur during the migration process. Furthermore, it updated an existing tool called LiteRT, which is used to build AI applications that run directly smartphones, laptops and embedded devices rather than in the cloud. So it should now be compatible with PyTorch-based apps, Google said.

Of course, it will take more than this to convince millions of developers to drop Nvidia’s GPUs and rush to Google’s TPUs. But it does suggest that Google is confident it can chip away at Nvidia’s lead in the longer term.

Image: Google