A new phase of government AI adoption could bring a very different budget reality

Terry Gerton Gayatri, McKinsey is out with some new research that seems to have a contradiction at its core about artificial intelligence. AI prices are falling rapidly, yet you all are arguing that agency AI bills are likely to rise. So what is happening beneath the surface in the market that makes both of those things true at the same time?

Gayatri Shenai That is correct, Terry. So what we are seeing is that the per token AI costs have fallen 10 fold in the last year alone and about thousand fold over the last three years. So the prices, as we would call it, as it relates to AI costs is coming down. That said, usage is on the rise and we’re seeing it driven by three things. One is we are seeing an increased adoption and usage by just users who are getting way more comfortable with the technology. And as a result, you’re seeing an increase in adoption. The second thing we’re seeing is an increase in agentification. So this is where we stop asking AI questions and it responds to starting to use AI to chain together dozens of steps and decisions and to complete a task end-to-end. So something like identifying a purchase order or a license process, for example. And then the third thing that is happening is pricing is shifting from introductory deals that governments and agencies might have had to much more consumption-based contracting. And so as a result, what is happening then is while the cost per token is coming down, the usage is increasing, and that is likely gonna result in much increased bills and an overall increase in the cost of AI.

Terry Gerton Tim?

Tim Ward Well, Terry, the way that we think about this is that what you really need as a public sector leader is visibility into how usage of AI leads to value and ultimately to the impact on the mission you’re trying to drive. And so what we call out in the article is that agencies that redesign their workflows to really reimagine them and then enable them with AI are going to be the ones who will find a lot of value. And so. Leaders in the public sector are going to have to think about managing AI differently to get the most value as we see the dynamic that Gayatri described play out. And so in the article that we just published, we call out several ways in which this thinking is going to have change. One of them is that AI is no longer basically free. It’s a cost category that has to be managed. And another is that it’s no longer just a different IT line item. You really need to treat AI differently so that decisions about AI value are made thinking about the actual outcomes that the AI can deliver in return for the price or the cost that you’re spending.

Terry Gerton That is a really helpful framing. And Gayatri, one more piece of that framing is that you’ve described this moment and compared it to the federal government’s early migration to cloud services. How does thinking about this in the way we used to think about cloud services help clarify the points that Tim just made?

Gayatri Shenai Yeah, so the cloud journey started very similarly, where almost all assumed initially that migrating will lead to saved costs. Another assumption also made early on in the cloud journey, especially as agencies were moving from on-prem to cloud, was that cost management would be simple, and it wasn’t. So what we are learning and what we’re seeing is agencies, when they moved to cloud, they learned how to manage vendor lock-ins, how to managed complex pricing tiers, how to think about FinOps as a scaffolding needed to track spend, to optimize spend, to manage and govern spend and usage better. And so all those ingredients we see very, very applicable to AI because it’s still very variable consumption. Cloud was variable consumption, AI is variable consumption. There is rapid experimentation happening. There is a rapid adoption happening. And there is a fast moving contracting elements that are evolving that is fast moving. And so what we then are now seeing is what governments learned using cloud, right? That it’s not just, it’s pay-for-use, but it also sounds simple when you call it pay-for-use and you have to build the infrastructure, the muscle, the governance to be able to manage it. All of those things are very applicable to AI as well.

Terry Gerton So Tim, as Gayatri is explaining this, I’m thinking about agentics in the background being like a water faucet that just never gets turned off. How do I know how much water I’m using? How do manage the flow? As agencies are getting comfortable with actually using AI and not just experimenting with it, how does this become a budget problem?

Tim Ward Yeah, you said it. So one pilot is kind of invisible to the budget. But if you’re doing a few thousand of them and they’re all running together, it’s not. That’s real. And so there’s a couple of things that that means for public sector leaders. And so one of them is you’ve got to be really careful not to drown in pilots. Lots of small, disconnected experiments that actually never convert to production. Because then nobody ever builds the forecasting muscle for what production actually costs, and you’re probably not getting great impact from it either. And so the math flips really fast, right? So once you go from hundreds of users who are testing something to maybe hundreds of thousands of users who are depending on it once you’ve scaled it up, you’re crossing over from a budgetary rounding error to a real line item. And so what we tend to recommend to our clients is to focus on. Rewiring workflows end-to-end instead of this generation of thousands of pilots or use cases. In our experience, that’s what helps with ensuring mission impact. And then it becomes tangible. You can both budget for the cost that you’re going to invest in it, and you can track the outcome. Because this isn’t just a cost point, even though the cost is important. You just won’t get the mission impact from thousands of disconnected pilots either. You need to think end to end.

Terry Gerton Tim Ward leads McKinsey’s Global Public Sector Practice and Gayatri Shani is a senior partner at McKinzie. Gayatri, let me come back to you because Tim is describing moving to processes and I think that’s really important because typically somebody owns a process and then somebody can put it in their budget, right? Is that what you’re seeing move, what direction you wanna see people move in?

Gayatri Shenai That is correct, Terry. We would absolutely recommend that for AI, there is a clear owner, because what we are seeing, right, like is AI is not mapping as cleanly or clearly to how one would have bought technology in the past, whether it was hardware or software, we used to have things like number of seats or number of licenses and things like that that would have been used to procure. But that is not the case for AI. For AI, the way costs add up is very much based on tokens, which is how you break down a prompt into small chunks of text. And then you count your input tokens and your output tokens. And then to that, like we talked about, you add in the agents to agentification and AI agents to the mix. And it quickly adds up. So really the questions I would ask people to think about is number one is who owns that number when it comes down to AI costs? Who would you point to as owning that number in your organization? And if you couldn’t point to somebody that itself is a issue to be solved because we all know what happens to unknown spend, it still sits and it actually grows unmanaged. The second question I would ask people to think about would be, can you answer the question, how many tokens are you burning a week? Because if that, answering that question takes you a multi-week, multi-team reconciliation exercise across vendors and across different departments, that’s not going to allow you to create the transparency into your spend to be able to manage it. And then the third question, going back to how AI is similar to cloud would be, how automated are your cloud fin-offs practices and controls because what again we are seeing is the more automated it is, the same thing could apply or something similar could apply to AI and it allows you to manage both your cloud spend and your AI spend simultaneously with more maturity if you could build in those practices.

Terry Gerton Tim, Gayatri has laid out some real challenges there. Help us understand how you really do what she said we need to do.

Tim Ward I’ll give a clear sign in practice that I find really helpful and have used with clients. If you are discussing who’s accountable for a specific AI implementation or use case, and the answer all the time is, oh, it’s the chief AI officer for everything, that’s probably not the right answer. That is probably not accountability. Another tell can be if the answer to the question, Well, why did this cost or this set of costs go up? It’s just a shrug. And then the operating model probably hasn’t caught up to the technology yet. And so the answer to all of this is not centralize everything in one office. And I don’t want to down chief AI officers. They will do an important job. But this needs to be a part of the service leader. The person who owns the outcome needs to own the thinking and the planning and decisions about the AI in that domain. There may be a central team. It’s probably small, but a central team that provides the shared guardrails and best practices, perhaps, that everyone can draw from. But if this isn’t in the line under the service leader who cares about the outcome, it’s unlikely to be a success.

Terry Gerton Gayatri, Tim is talking there about a governance model. Governance comes up every time I have an AI conversation and a lot of people raise an eyebrow and go, yep, we need governance, but it’s hard to go past that. When you’re thinking about governing AI spend, what does a good governance structure look like?

Gayatri Shenai It’s a great question, Terry, and I will say we talk about it at length in our article as well. I would start with, number one, is naming an owner for the AI spend. And to Tim’s point, it better not be the chief AI officer. It needs to be a category of its own, but it needs a joint collaboration with the CFO, the COO, the chief procurement officer, and so on. But naming one owner who owns it is going to be critical. The second, from a governance standpoint, is tagging AI requests, tagging I spend. And not just at the macro level, so not at the agency level, but going down to the program level, the use case level, the model level, so that it is a tagged resource where cost allocation becomes much more transparent and much more clearer. The third would be adding real guardrails. So the ability to add budget gaps, the ability to also catch anomaly alerts at the individual agent level so that it’s not that a misbehaving agent that ran into a whole lot of token costs is caught way later in an invoice, but it’s caught more real time and is again managed. And then the final thing I would add is something that Tim has talked about by now, right? Like a lot around how do we ensure this is outcome-based, this is mission-based. So, don’t measure tokens consumed, but actually cost per outcome. Whether it’s cost per resolved use case or cost per claim processed or cost as a FOIA request answered, whatever is the end-to-end process we imagined, tying the demonstrated value to the mission and to the stakeholders, employees, citizens, customers, vendors that the agency is serving. At that level, you should be able to tell the difference and tie the outcomes to the cost.

Terry Gerton Tim, let me follow up on Gayatri’s conversation with you and bring my faucet metaphor back because as she’s put the concept of spending caps out there, at some point, someone will shut the water off, right? But who knows how much we’ve expended before that. This is going to really change how people talk about their budgets, and it’s one for an agency to kind of predict what theirs will be. How should OMB and Congress be preparing now to start seeing these kind of costs show up in agency budgets?

Tim Ward As we talked about, if you’re going to have control of that faucet, you need to think about it differently, right? And so our research suggests that as AI scales, one of the core management challenges is going to be gaining visibility into where the costs are emerging, and who owns them, and does the result justify it. And today, in practice, the costs may not appear as one obvious AI line item. They probably don’t. They were probably going to show up today across technology, across cloud consumption, vendor arrangements, data infrastructure, and much of the organizational work that goes into just deploying and scaling AI. Over time, I would fully expect that to change. I would expect this to become a more explicit management category, an oversight category. So that rather than being something that is dispersed across multiple existing budgets without a single view of total exposure, you would get and work towards one integrative view. And so from my perspective, the preparation for that lies in creating the management discipline to understand the costs, connect them to the mission outcomes. And so ultimately that’s one of the mission outcomes that we talk about in the article that AI just cannot be managed like a traditional IT line item. A leader in the public sector, also in the private sector, needs to think of it as a dynamic category that needs ongoing visibility and management and ownership as usage scales. So there’s so much potential. That just means we need a different and improved way to think about it.