Gridlock

Episode 2 of Power Play, Quartz's six-part investigation into AI and the US power grid.

Missed episode 1? Check it out here.

Every new data center waits on a connection study, and the studies are still run by hand. The turbines and transmission lines that would add capacity take five years or more to arrive. Jackie Snow traces the wait through roughly three thousand utilities, each working the problem alone.

Reported by Jackie Snow, Catherine Baab, and Joseph Zeballos-Roig.

Hosted by Marco Pimentel.

Produced by Simon Litt.

Edited by Georgina Kemp Walker.

Ian Magruder: We're really only using about half the capacity of the grid at any given hour of any given day on average.

Marco Pimentel: Half. We spent a century building this grid and most of the time half of it just sits there. If the country is panicking about not having enough power for AI, why can't we just use the other half?

Jackie Snow: If we could snap our fingers and put more data centers on the grid, there is energy for them, but those days in July in Texas when we need as much energy as possible, that's when the data centers becomes a problem. There is slack in the grid most of the time, but because when the grid is stressed and if it goes down, people die. We want a little bit of slack. We don't want to be running 100% every day because 101 is when things shut off. And that includes like hospitals and homes when it's 110 degrees outside.

Marco Pimentel: I'm Marco Pimentel, and this is Power Play. Over six episodes, we follow the race to rebuild the American grid for AI and the fight over who ends up paying for it. This is episode two, Gridlock.

Jackie Snow: The voice you heard up top is Ian Magruder. I called him for this. He runs the Utilize Coalition, a grid policy group pushing to get more out of the power infrastructure that's already built. Here's how he explains why we built it this way.

Magruder: We have historically built the U.S. grid to peak, and for good reason, because we have to ensure when the power system is the most strained, that we have enough capacity to meet those peak moments and that the lights will stay on. And historically, we haven't had great tools to manage flexibility. And so we've intentionally overbuilt, essentially, to manage those moments. We want some headroom in the system. We're never going to get to 100% utilization or anywhere near there. But given the fact that we're really only using about half the capacity in the grid at any given hour of any given day on average, there's a lot of extra room for growth if we think about unlocking that untapped potential.

Marco Pimentel: Jackie Snow has been reporting this story for Quartz. She covered AI for a decade, which, as she puts it, makes her an energy reporter now. I asked her what exactly slack on the grid means.

Jackie Snow: We build the grid to let everyone take as much energy from it that they could possibly want on the hottest day of the year. And that means we build peaker plants, which are built for the hottest day of the year. And not even just like the hottest day of the year, but the hottest few hours of the hottest day of the year. In some places in the north, the coldest day. Outside of those days, the weather is pretty nice across the country. No one's blasting their AC, and so the grid is just not being used that much. I talked to Paul Denholm, a senior research fellow at the National Renewable Energy Lab. He spent his career, as he puts it, hiding electricity from you so well that almost nobody sees the overbuild.

Paul Denholm: As an electrical engineer and power system engineer, I like to keep it that way, right? I don't want anybody to ever have to worry about their lights going on. That's my mission. However, we've done such a good job for the most part of hiding electricity prices and the realities of how the grid works. We hide things like we built hundreds and hundreds of power plants that only operate a few hours per year. They may operate 5% of the year just so that we can meet the peak demand on that hottest summer day.

Jackie Snow: They are power plants that maybe get turned on for, in some places, 10 hours a year. I think I just read in California it's maybe 40 hours a year. If you're having to build hardened infrastructure that is used for 40 hours a year, that is just inherently not an efficient use of CapEx. This can't be solar because likely the day that you need it is going to be potentially really sunny. But you just happen to have a cloud kind of hanging out over the solar panels that you need, like that can't work. So it is almost always fossil fuels because that is the energy source that we understand the best. An expensive thing to build because, again, since they are so worried about not ever hitting that 101% where bad things start to happen, they build it a little bit bigger than they think they need. And it just becomes an expensive tool that is utilized less than 1% of the year.

Denholm: Instead of saying, gosh, maybe instead of building another one of these peaker power plants, we should figure out if we can take advantage of the demand side, we could absolutely save people money. Because when you look at how much those kilowatt hours actually cost during the peak, it's amazing how expensive that electricity really is. And we just average that out over the entire year. I think there's an extremely strong amount of real evidence that you could actually add quite a bit of data center load to the grid and not cause any additional stresses to the grid most of the time. It's really just these few hundred hours a year where it's super, super hot or super, super cold.

Marco Pimentel: Let's head back to Ian Magruder at the Utilize Coalition.

Magruder: Some parts of the grid are 30% utilized and some are 60%. And there's going to need to be more research done on this, but there's broad consensus. The Secretary of Energy has said our grid is only roughly using half of the capacity at any given time, and that's widely seen to be the case.

Jackie Snow: This is maybe one of the problems of utilities is just a very conservative industry. And once you get used to using this number that your grid is used 30 to 50% of the time and you don't have any brownouts, you don't have any blackouts, the idea of pushing it back up, even if it is up to percentages that we used to see, is maybe something that people are now wary of.

Magruder: I think this is really important in the context of the real demand growth moment that we're in, where we're seeing such a rush of new need for capacity on the grid. We can't build power plants fast enough in this country. And so the question is, how do we get more power?

Marco Pimentel: So if the power is mostly there, why can't a new data center just plug in and pull from it?

Jackie Snow: Before anyone connects anything, the utility has to study exactly what the new load does to the grid. And that study is where the years go. I called Josh Wong, who runs ThinkLabs AI, a company that builds AI models of the grid to speed these studies up. Here's the process today.

Josh Wong: Planning is really under the domain of power system planning, and this includes transmission planning, which is more relevant for large loads like industrial data centers, as well as a distribution planning, which is, for example, more distributed data centers or electric vehicle charging or just load growth from homes and businesses. All these have been really manually driven processes. So an interconnection request comes in and an engineer reviews the application. A lot of time is you setting up these individual, what we call like ad hoc worst case simulations. So the data had to be prepared, and we run through traditional software, and after the analysis, the engineer identifies potential constraints and violations like congestion and voltage issues on the grid incurred by the new interconnection, and also time is used to write reports and repeat studies on feedback.

Jackie Snow: So a person, an actual engineer, sits down and runs the numbers to see what the new load would do to the wires nearby, one at a time.

Wong: So, this process on the distribution side takes about 35 days. Other utilities we have seen take up to months to process these studies. The duration is one thing, but the other is also the volume. Electrification is absolutely driving increased volume in these connections. Traditional engineering tools just can't scale to this volume of analysis. It's like running large-scale climate models on small machines. Engineers can only have time to do a few, like one to a handful of scenarios. So you can't really cover that much grounds.

Jackie Snow: So they run a handful of worst case scenarios and hope those cover it because that's all the old tools had time for. That's the part that AI is trying to change.

Wong: Traditional softwares can only tell you the problems. In the past, we assume everything should be building new lines, new cables, new wires, new transformers, etc. And this takes a lot of time, but also this has its rate impact. Traditional engineering software is scientific. It's first principle, engineering-based, but it's not very scalable. It's like trying to go on a driving app by looking at like calculating formulas between each of your streets and turns. No, I think when we open any driving app, it's a continuous traffic management system. It's already trained. You're basically prompting a route from A to B. Today, the prudent thing to do is to keep human in the loop. The human engineer, given his or her experience, is still important, but also just to validate and fact check the results. Now, I do see a future where, hey, there can be self-service or any pre-screening or complete automation of those processes, but I think there will be a point of diminishing return.

Marco Pimentel: I heard Josh say it takes months to process these studies. Why is that the case?

Jackie Snow: Any land, when you want to have it considered for a data center or energy project, you have to do an energy study on that land. And in many places that goes immediately onto the queue of like, oh, here's an energy need that the utility will have to provide. And that was something that could be done for very little money, I think in the thousands of dollars. And so all of a sudden, you're seeing in these small communities, gigawatts of energy being requested because someone did an energy study on a piece of land that they're hoping to sell to a data center. But now a lot of utilities have wised up and said like, okay, if you're going to do an energy study, it's going to be about 25% of like the overall cost of doing the energy. So it's going from thousands to millions of dollars. And that is getting a lot of people who were kind of prospectors out of the queue. Everyone and their grandma is trying to do an energy study on the piece of land that's not near anything because this does seem like a chance to turn land that maybe isn't that useful or profitable right now into a data center and people think they're going to get rich off that. So I'm really curious if we're going to start to see these numbers start to whittle down now that we're trying to just keep the serious people in this conversation about how much we need.

Marco Pimentel: So with the utilities, with that first come first serve basis, that means that that strip mall lady is treated the same as Microsoft and that the process to actually do this study is not like a day process. This could be months of work and time that essentially if it doesn't go through, that means it's just a bunch of waste.

Jackie Snow: A lot of the numbers that we are seeing that are showing up in the paperwork are people hoping that they have a lottery ticket in the sense that they have a strip mall that is a power line runs behind the back, which in reality, it is unlikely to be the lottery ticket they're hoping for. I wanted to better understand where this was coming from. So I talked to Carl Fleming, an energy and project finance partner at McDermott Will & Emery, who spent years on the data center power deals.

Carl Fleming: You have the federal regulatory oversight, which is one thing, but then you have all the ISOs, RTOs, states, localities, and power producers themselves, all looking to either build projects there to power data centers to interconnect these projects. Each jurisdiction is going to have its own rules and regulations around where you can co-locate, where you can't, what it's like to be in the queue. So it's a bit of, people are feeling their way through the dark at the moment. It was a pre-existing problem that if you were someone in the business of developing energy generation, yeah, we knew this was a problem beforehand. You were out in the market trying to put your letters of credit down to secure your positioning in the queue. With data centers coming online, that began to really push the demand upward. Basically, getting energy has become the biggest CP to turning the lights on for these projects. So I think that's when it's really become a hot-button issue.

Marco Pimentel: Is the wait just bloat that backed everything up, or is there more to that?

Fleming: It's a pretty complicated problem to solve if you're a utility. You don't want to overbuild your grid or overbuild your transmission or interconnection for data centers that may or may not be there. So you're stuck as a utility trying to fund and build out infrastructure for a new technology that is promising to be in your backyard and it may or may not be there. And it takes time to build out that additional capacity. It's not something you just do overnight. So I think it's something that it was an issue and got exacerbated way more quickly than people had anticipated. So it's a process is probably the best way to put it. And the solution for it isn't just to throw more money at it because you have ratepayers who just don't want to fund infrastructure willy-nilly. And then you have maybe hyperscalers or developers that are willing to do so. But that always becomes a negotiation kind of saying who's on the hook for how much and what and what if the data doesn't show up, then what happens? I live not too far from the Virginia data center alley. So that's just like totally all the development is there. It's coming, it's there. So I think, when you try and look at it from that point of view, how do you fund that? And it takes time to build out that additional capacity.

Jackie Snow: Turns out the data itself is a mess too. I talked to Bo Yang, who works on AI and grid software at Hitachi.

Bo Yang: In the past, when one side is passed on some data points to the other side, often there's a long wait. So that delays the overall conversation and decision-making. Quite often, utilities end up with digital dust, is what I call it, because their data coming from different sources, different vendors, at different time resolution. So you end up with multiple versions of truth, right? Any decision made on top of this will not be very efficient. Very lengthy, complex, large-scale studies can be squeezed from days to within half an hour, right? With the overall efficiency improvement in the planning process, you will see the interaction between different stakeholders is much more effective. Now, if you can successfully reduce that wait time to ignorable, then the conversation will flow, right? More efficient decision or better decision will be made. There's no one single magic tool that can optimize the whole thing. It takes the whole village. First, you have to validate the tool, and then you have to make sure that it is well-received, your engineers are well-educated on how to use the tool, how to leverage the tool. And then overall process, it has to be further optimized. So everything has to work together. They identify this business challenge, right, which is the very long generation and connection review cycles. The complexity that planners can study in the past can be 10 times, 20 times more, which means they have more awareness on what's going to happen if this kind of extreme situation happens in my grid. So they can plan their grid more effectively.

Jackie Snow: Let's say we figure out the paperwork issue. Even after a project clears, the physical equipment is its own bottleneck.

Wong: The load growth is definitely increasing faster than the practical pace of infrastructure builds. So to build a new foundation, it might take like five years plus to lay a new line or cable. This is pretty much planned months or years in advance.

Jackie Snow: And the equipment to do this is scarce enough that companies are improvising. Here's Brendan Pierpont, who directs electricity modeling at Energy Innovation, a climate and energy think tank.

Brendan Pierpont: So what a lot of data center companies are doing is they're piecing together turbines and gas engines and I've even heard some like buying jet engines and reconfiguring them to generate power, like all kinds of things that are really not really well thought through strategies to actually provide reliable power on site. This is an example of thinking that that's the easy button, like oh, this is capacity, it's something that I can just turn on and it will run and it will power my facility. And then when you dig into the details it's enormously complex.

Fleming: Now you kind of have this prolonged timeline to get anything interconnected, and that's in addition to all of the supply of turbines going out for gas, which are going out five years from now or longer. So it's kind of a perfect storm of variables that are making kind of bringing generation to meet the data center demand be a bit slower than expected or than hoped, to be honest.

Marco Pimentel: Wow. So years to study, years for the wire, years for the turbine. For the utility on the other end, how much of that can they even fix?

Fleming: In order to get a data center project into the queue and online, you're looking at several years at this point. In order to go and procure, you know, gas turbines to power this stuff, you're looking at a five-year lead time. I don't see a whole lot utilities can do at the moment to change that. You already have your queue, you already have, and you can't change the supply of turbines.

Marco Pimentel: Is the real crisis here a physical shortage of electricity? Or is it that the utility companies just structurally do not possess the organizational capacity to build at this speed anymore?

Jackie Snow: I think it's mostly the latter, but the latter is creating the former. That does mean like there is a shortage of electricity, but that doesn't have to be the case because so much of our energy creating technology, specifically around some renewables like solar, wind, and then batteries, that has improved so much where electrical energy could be really abundant. But as you said, there was 20 years of flat demand and we lost the muscle memory on energy how to build out. Everything is now a financial instrument. Nothing is real. It's just financial instruments all the way down. And there's a deep caution baked in from years of doing it one way. I asked Pierpont where this comes from.

Pierpont: I don't know if that is sort of inherent conservatism in the utilities. They've done things for a certain way for so many years, or if that's like risk aversion, they have a much lower tolerance for trying new things if they're not as proven. I'd argue that they don't have a strong incentive to innovate, to find a lower cost way of doing something, a faster way of doing something, and that's again not a muscle that they've worked.

Jackie Snow: And it does seem like they're both not interested in new tech and the business alignment isn't really there for it. There's so many tweaks that could be done and they're just like, no more gas turbines. But if you built too many power plants outside of the peaker power plants you're expecting to use them most of the year every day, and if you're not, then all of a sudden a lot of the math that you did for these is no longer figuring out on paper. So it's something that they definitely experienced and had to deal with in these years of flat growth. And they're concerned about this happening again. A lot of these utilities are compensated based on infrastructure spending. That was fine when there was no growth. They were able to squeeze out some profit, drip by drip. But when we see this massive need, that level of increase is just going to make bills go crazy. They just didn't have to build big infrastructure. And now some this infrastructure which has been evolving for the last couple decades is even for the people there who do know how to build this is technology that is different than a peaker plant than natural gas plant. This is solar power. This is wind energy. That energy profile looks a little different, and there is just not necessarily enough people within the energy providers who know how to work with that technology. And the speed that we, or the data centers, the AI companies, want them to work with it. There is this other project happening right now, which is the electrification of everything. People's cars are going from internal combustion engines to electrical vehicles. Industry is going from gas and oil to electrifying those plants. And even buildings that also rely on fossil fuels are increasingly electrified. So there is a need for more energy. So building out this infrastructure, which has not really been updated since the 90s, is necessary.

Marco Pimentel: There's about 3,000 utilities, I think, in the United States. So how does the utility companies work in the U.S.?

Jackie Snow: Each one feels almost like a unique butterfly, right? There's ones that are consumer-owned. There are rural ones. There are all sorts of different setups. There are seven major providers, what they call ISOs, that manage the wholesale electricity, which cover about like two-thirds of U.S. consumers. These are very regulated industries. I don't think there's many that are working like the Wild West. So there's a lot of them get to work as monopolies, but they are limited about how much profit they can make since it is monopoly. But the utilities are a highly regulated industry across the United States. So that's one of the reasons that makes this so difficult to talk about in any sort of grand scheme, because what I could say about one community will not hold true for the other.

Pierpont: Our electric grid is this patchwork of state-regulated utilities. Where that demand shows up matters a lot. There's a ton of it in Northern Virginia, there's a ton of it in Texas and Arizona and Georgia, and then if you look to, say, California, electricity prices are high and there's not a lot of people building a enormous amount of data center capacity in California.

Jackie Snow: There's communities where there is slack and they just had a major build out, but then there's other places that don't have any slack and to let a data center come in you'll have to build out more infrastructure. There's places where this is like very slow going. And there's places like Texas where we're seeing a lot of buildout. ERCOT, their energy market, is just better. Texas loves being less regulations. And I think that it's like a more streamlined process. So that's where we're seeing like the giant Stargate project happening outside of Abilene in Texas.

Marco Pimentel: So is regulation the main difference between the states that do this well and the ones that don't?

Jackie Snow: It's part of it, but it isn't the whole story. I talked to Pierpont about this.

Pierpont: There are probably parts of the country where there's a much more proactive approach. I think of some of the recent developments in Minnesota where I think Google worked with Xcel Energy to procure a bunch of clean energy and long-duration energy storage in partnership with the utility. There's some interesting models emerging in states that. But that doesn't necessarily address sort of the scale of data center demand in Northern Virginia and its impacts on the wholesale market and PJM. In the mid-Atlantic region, the grid operator PJM covers Northern Virginia as part of its broad service territory. The biggest issue has just been the scale of the load forecast coming into that region's capacity market combined with the grid operator's slow pace of interconnecting new resources. And so this sort of trifecta of how the market's designed, how the forecasts have boomed, and the inability to get new stuff connected to the grid is just driving up the clearing prices of that market astronomically, and that's impacting everyone across the region.

Jackie Snow: At this point in my research, I found myself wondering, why don't we just rebuild the whole system? So I asked Magruder about it.

Magruder: I think there's a real status quo bias. Typically in America, we built the grid in one way, which is to build large scale generation, large scale transmission. But doing it differently, doing it in a little bit more of a bottom up way, we viewed some of these more distributed technologies that can help us get more out of the grid that we have as in some ways riskier or less proven. And so I do think it's been sort of an uphill battle to get us to think about the grid in new ways. Now, we happen to think we need to do both in this country in this moment.

Jackie Snow: So picture it from the data center side. The grid you want is full of slack, but you can't reach it for years. A lot of them just stopped waiting and started building their own power. What I am seeing is them build energy on campus so they can potentially, yes, stop taking power from the grid for a few hours, but continue their training run. These companies do think about the chips, too, as these depreciating assets that every minute you're not using it, that you are losing money. They do not want to be offline for even one second. So they're pushing a lot of ways to make sure that doesn't happen. We're already seeing them take these risks because there is this rush to build the most powerful model. There's no data center provider that's saying, we don't care if people are going to die, we're going to run this. But we are seeing regulators go in and just have it on paper. If we need you to back off the grid, you'll do it. And I'm not seeing any pushback from these companies.

Fleming: If I'm a utility or if I'm looking to develop a data center, I know I'm facing like a five-year timeframe before I can even commence a lot of stuff. So to find an alternate route to avoid the interconnection issue, you could go and do this co-location arrangement, which is becoming more and more popular.

Jackie Snow: And in fact, some of these data centers that are going to have either batteries or even energy on site. I've talked with Google, it's a possibility of what they could become grid providers in those hours of need and not just like, oh, we won't take anything but we'll give to the grid.

Fleming: In the past the power generators were kind of chasing the data center sites. Recently you've seen a bit of a shift where data center folks are kind of chasing the power sources.

Magruder: Your average American sees this, this need to build more power in America and thinks that's the culprit for my rising utility bills. And the reality is not that simple. The grid doesn't function like most markets. There's so many fixed costs. You and I and all of us in society have paid, anyone who pays a utility bill has paid for the grid. And so, and those fixed costs have actually gone up over time.

Marco Pimentel: So the grid is half empty and still almost impossible to get into. What's standing in the way is time and money. And next time, we follow the money. Who gets to the front of the line and who covers the rest?