As Microsoft continues to build-out multi-gigawatt cloud and AI infrastructure deployments, its custom silicon remains a major focus. Today Microsoft reiterated that its Maia 200 is production-proven and deployed across regions in the United States, Europe, and Australia, and that the next-generation Maia 300, which reportedly boosts AI inference performance significantly, is coming in 2027.
For those unfamiliar, the Azure Maia 200 is actively running limited Microsoft Copilot workloads, OpenAI models, internal MAI models, and many open-source frameworks. Microsoft claims Maia 200, which is designed for real-time inference and model development, offers 1.7x higher token throughput per watt for core AI features in Microsoft Teams, Copilot, and Cowork use-cases compared to leading GPUs.
Azure Maia 300 Will Be An Inference Powerhouse
According to Microsoft, the next-generation Azure Maia 300 is a ground-up redesigned, inference-first architecture tailored specifically for memory-intensive, long-context reasoning and latency-critical agentic tasks. The company is claiming Maia 300 will offer 40% more tokens per megawatt than competing GPU solutions, with a full 96-accelerator rack configuration pushing up to 7x the rack-scale token throughput of Maia 200. A full Maia 300 rack will feature 96 accelerators within a 167-kilowatt power envelope, with 27.6 TB of aggregate HBM capacity and 1.7 PB/s of aggregate HBM bandwidth per rack.
Like a number of other architectures announced over the last few months, efficient, optimized data movement was a major design focus of the Maia 300. The architecture combines AI-assisted kernel optimization and on-die SRAM, which will act as a “software-managed scratchpad” to optimize access to memory across compute and the fabric layer, to minimize the need to access the HBM.
The Maia 300: Big Memory Bandwidth And Optimized Data Movement
The Maia 300 features 8 stacks of HBM4 memory to deliver an impressive 18.0 TB/s of peak memory bandwidth within a stout 950-watt Thermal Design Power (TDP). It also leverages a modular chiplet architecture separating compute, memory controllers, and I/O blocks. The Maia 300’s compute chiplets are built on TSMC's advanced N2P process node, but information about the other chiplets wasn’t mentioned in Microsoft’s announcement.
The Maia 300 also features advanced low-precision support for various datatypes, including enhanced MXFP4, to compress memory footprints and lower bandwidth consumption. The chip features high-speed tensor-parallel interconnects as well, with a custom transport protocol over standard Ethernet-based infrastructure. According to Microsoft, this allows seamless scale-up networking from single-racks to multi-gigawatt many-rack datacenter deployments.
By co-designing and optimizing the silicon, software stacks, and cloud infrastructure, Microsoft's believes its Maia architecture will be a highly-efficiency foundation for next-generation frontier AI within Azure.
Marco's interest in computing and technology dates all the way back to his early childhood. Even before being exposed to the Commodore P.E.T. and later the Commodore 64 in the early ‘80s, he was interested in electricity and electronics, and he still has the modded AFX cars and shop-worn soldering irons to prove it. Once he got his hands on his own Commodore 64, however, computing became Marco's passion. Throughout his academic and professional lives, Marco has worked with virtually every major platform from the TRS-80 and Amiga, to today's high end, multi-core servers. Over the years, he has worked in many fields related to technology and computing, including system design, assembly and sales, professional quality assurance testing, and technical writing. In addition to being the Managing Editor here at HotHardware for close to 15 years, Marco is also a freelance writer whose work has been published in a number of PC and technology related print publications and he is a regular fixture on HotHardware’s own Two and a Half Geeks webcast. - Contact: marco(at)hothardware(dot)com