A new software tool from Amazon aims to make switching from Nvidia to the company’s own AI chips about as simple as changing a setting.
The software, dubbed TorchNeuron, runs code written in PyTorch, the open-source framework used by many AI developers, on Amazon’s Trainium chips. It has been in private beta since late last year, and Amazon said Monday that it’s opening the software to all developers as a public beta.
Until now, developers who wanted to run PyTorch code on Trainium had to modify it to call programming interfaces specific to Amazon’s chips, said Colin Brace, vice president of technology software at Annapurna Labs, Amazon’s chip-design unit. With TorchNeuron, the same code runs after a basic configuration change.
“If you know PyTorch, then you know Trainium,” Brace said in an interview.
Nvidia dominates the market for AI chips, partly because much of the industry’s AI software was built around its CUDA platform. Adapting that code to other hardware can take months of engineering work. That has made many companies reluctant to switch, even as Amazon, Google and others have developed their own chips.
TorchNeuron is part of a broader push by tech giants to loosen Nvidia’s grip on AI computing, even as they remain among its biggest customers.
- Google, which has started selling its TPU chips directly to some outside customers, is working on TorchTPU, a similar project to run PyTorch natively on those chips. Google detailed the effort in April, saying the goal is to let developers move existing PyTorch code to TPUs with minimal changes.
- Microsoft said last week that it plans to run most of its AI services on its homegrown Maia chips rather than Nvidia hardware.
- Amazon is reportedly in talks to acquire Decart, a startup that makes software to help developers move AI models built for Nvidia chips to alternatives such as Trainium, in a deal that could value it at about $7 billion. Decart already runs its own AI models on Trainium, and Nvidia is an investor in the startup.
Asked about Amazon’s relationship with Nvidia, Brace said the company has “always been about choice,” pointing to another high-profile example: AWS offers Intel processors alongside its own Graviton chips.
Amazon plans to release TorchNeuron as open-source software on Tuesday in conjunction with the PyTorch Conference in San Jose, Calif.
Trainium’s biggest users so far are AI labs with close financial ties to Amazon. Anthropic, in which Amazon has invested billions, uses the chips to train and run its Claude models. OpenAI agreed in February to use 2 gigawatts of Trainium capacity as part of a deal in which Amazon is investing $50 billion in the company.
Brace said Amazon’s goal has always been wide use of the chips, calling TorchNeuron “an accelerant” along those lines.
Amazon’s chip business traces back to its 2015 acquisition of Annapurna Labs, an Israeli chipmaker, for a reported $350 million. The Annapurna team went on to design Trainium as well as Amazon’s Graviton processors.
In his annual shareholder letter in April, Amazon CEO Andy Jassy wrote that the company’s chip business, including Trainium and Graviton, had an annual revenue run rate of more than $20 billion. He estimated that figure would be about $50 billion if the chips were sold to AWS and other customers by a standalone chip business.
“There’s so much demand for our chips that it’s quite possible we’ll sell racks of them to third parties in the future,” he wrote.
More than 30 companies and internal Amazon teams have been using TorchNeuron during the private beta. Deepgram, a San Francisco startup that makes speech AI, says its benchmarks showed Trainium processing about 60% more audio per dollar than its existing GPU setup. The company now runs Trainium in production, Brace said.
Inside Amazon, the Prime Video team used TorchNeuron to build Prime 360, an AI system that reconstructs NBA plays in 3D so viewers can watch replays from angles a traditional camera can’t capture. The feature, which runs on Trainium, will appear on select Prime Video broadcasts this NBA season.
To make sure TorchNeuron works with the huge variety of AI models that developers build, Amazon tests it every night against more than 100,000 models from Hugging Face, the popular online hub for open-source AI, Brace said.
Brace said the timing of the project reflects changes in PyTorch itself. The framework has matured, and its maintainers have been working to make it less dependent on any one kind of hardware.
Amazon has contributed to that effort, removing code that tied PyTorch to specific chips. Newer versions of PyTorch can detect which hardware they’re running on, he said, so in some cases developers don’t need to change their code at all.