The first wave of enterprise AI deployments was straightforward: a chat interface calling a model, wrapped around a few clear use cases. The next wave is driven by autonomous agents that maintain state, revisit context, call tools and act under real permissions. This means enterprise AI teams must design inference, compute and deployment so those agents stay affordable, governable and compliant.
CIOs are used to making hard calls whenever the computing paradigm shifts. The move to client/server, web-based services and the cloud all required bets on platforms and architectures that shaped IT for years. The shift to AI feels similar, but the choices are more complicated, harder to see and harder to come back from. And the decision most tech leaders focus on first — which model to use — matters less in the long run than a set of less obvious but critical choices.
More tokens, more authority, more concerns
In the chatbot era, most organizations were token consumers. A user sent a prompt, the system generated a response and the interaction ended. Cost per interaction was bounded and often small enough to vanish into a cloud bill.
Agents do not stop at a single request‑response. They break work into steps, call tools as needed and keep going until a goal is reached. That behavior can drive significantly more inference traffic than a traditional chatbot, especially as the organization moves AI workloads from pilot to production. Over time, that shift turns enterprises from occasional token consumers into regular token producers, making inference cost a first-class strategic concern rather than a background line item.
At the same time, the risk surface is growing, to the point that some are warning of an existential threat and are calling for a slowdown in AI development. Right now, agents have ready access to your data and can act, not just read. They can call internal tools, move money, change records or trigger workflows at machine speed. They can also make mistakes at the same speed, especially when guardrails are not clear (or don’t exist). That combination of more tokens and more authority makes organizations understandably reluctant to stream corporate data into third‑party tools when they are uncertain where that data is processed, how long it is retained or how it might be reused.
Two years ago, those concerns were easy to dismiss as premature or even insignificant. Today, ignoring them creates deployment friction: security reviews that stall, compliance questions that are hard to answer and stakeholders who are not comfortable handing their most sensitive data and systems to black‑box services. AI also accelerates adversarial threats by dramatically shrinking the window between vulnerability discovery and exploitation.
All of this makes inference design critical — and challenging, since organizations may not have paid attention to it before. Inference is the operational phase of AI — the point where a trained model is actually run on real prompts and data to produce answers. It’s also where tokens are consumed and billed and where access controls and logging have to work. The way you build your inference layer determines whether you can see, control and cap token usage. It also determines whether identity and policy are enforced consistently or whether each application bakes its own security into custom code.
Once you have an inference layer that you can observe and control, the next structural question is the model layer itself — or, better yet, the portfolio of models the organization will rely on. The choice of model is the one that many teams started with, assuming the model was the whole story. Now it becomes one layer in a stack you can change rather than a decision you are stuck with.
In fact, many organizations thought — and often still think — that it is best to pick a single “frontier” model and treat it as the standard. However, AI’s future is a mixture of proprietary, open source and domain‑specific models. To maximize cost, performance and control, AI models must map to the right use case (and not vice versa). Just as there is not one single environment powering enterprise IT, there is not one single model that fits every workload. For example, smaller, open-source models are ideal for solving smaller or niche problems, where a frontier model’s power and scale would be wasted.
Maintaining in‑house models “as a service” enables fine‑tuning on proprietary data while minimizing the risk of leaks. A consistent abstraction layer or platform helps shift workloads among models as business needs dictate, without forcing every application team to rewrite prompts and pipelines. Otherwise, a choice made for one proof of concept can lock an entire organization into a single provider’s roadmap.
AI compute needs democratization and customization
For agents, flexibility only pays off if the underlying compute can flex as well; the hardware you choose determines both how far you can scale and what your token economics look like.
GPUs and other AI accelerators are the most visible line item in many AI budgets. It is easy to assume more is always better, but 10 percent utilization means burning money while 100 percent means no room to grow. Flexibility in compute provides resilience against hardware shortages and supply‑chain concerns, without being fully shackled to a single architecture. Put another way, you do not need a wall of top‑shelf GPUs to run a customer chatbot, but you do need serious horsepower for something like drug discovery. The right answer depends on knowing what workloads you are actually running and sizing to that, not buying everything in sight and hoping it fits.
AI must run anywhere
Once you have a handle on accelerators and utilization, the next structural choice is location: which environments you trust to run AI workloads and under what constraints.
“Sovereign AI” can sound like someone else’s problem — European regulators, highly regulated industries and so on. Strip away the labels and it comes down to three very familiar concerns:
- Control over where and how data is processed
- Transparency through audit trails and data lineage, even when models are black boxes
- Flexibility to adapt when, for example, a regulation changes or a provider goes out of business
Successful IT strategies, including AI, need a single, consistent operational experience across the hybrid cloud, from the edge to hyperscaler environments. AI workloads need to reside closest to where the data is born, whether a factory floor, a sovereign data center, or a public cloud instance. Sovereign AI strategies demand regional and country‑specific boundaries to data ingestion and training, meaning location is as important as any other factor in AI.
Some organizations are already building AI “factories” or fabrics: repeatable patterns, shared tooling and governance that let teams create AI applications in a consistent, auditable way, often with sovereign deployment as a first‑class requirement. Without that kind of fabric, you are just turning up the money dial on a single service when you need more scale or new features.
Platforms built on choice
To make choice structural rather than aspirational, it helps to think in terms of two pillars that sit across these decisions: one for inference and one for agents.
- Inference pillar. The inference layer should be the place where you measure and manage token usage. That means visibility into tokens consumed per application and per agent, the ability to forecast and cap spend and the power to route requests across different models without changing how applications talk to the platform. Identity and policy enforcement live here, as well, so that switching models does not mean rebuilding access control and governance from scratch.
- Agent pillar. Agents should be treated like extremely powerful users. Decide up front what agents are allowed to do, what tools and systems they can touch and when they can step out of the sandbox. A controlled runtime defines how long they can run, what they can modify and how their actions are logged. Permissions, time limits and audit trails are not afterthoughts; they are the mechanism that turns “autonomous” from a risk into a feature.
Together, these pillars turn choice into something you can enforce. You choose the models. You choose the environments in which they run. You choose the guardrails and policies that govern how agents behave. And you retain the freedom to run open source and commercial models on infrastructure you control.
Choice, sovereignty and security
In the end, this is about making choice structural: inference you can observe and control, a model portfolio you can change, compute you can rebalance and deployments you can move without rewriting everything. Sovereignty keeps those choices under your control, while security makes sure agents and inference follow the same governance as the rest of your stack, including the freedom to run open-source models on infrastructure you own alongside commercial models where they make sense. Open source strengthens both, because what can be inspected can be secured.
There are a few concrete steps tech leaders can take to surface the AI choices that really matter and then make them in ways that avoid tomorrow’s lock‑in:
- Treat PoCs as architecture, not just experiments. As soon as something works in a demo, assume it might go to production and capture what that implies: how data flows, which inference server you are using, which ports and services are exposed and what pipelines will have to change if this becomes real.
- Be deliberate about accelerators and utilization. Match GPUs and other accelerators to actual workloads so you are not burning money or leaving no room to scale.
- Pay attention to inference and model flexibility. Avoid locking yourself into an inference setup that forces you to rewrite prompts and datasets every time you change models. Understand whether the model weights you rely on are open or closed, because that determines how much control and flexibility you really have over model behavior and where it can run.
- Think in terms of transparency, control and an AI fabric. Design for where data is processed, how workloads are audited and how easily you can adapt to new rules by building repeatable patterns. Think “AI factory” or “AI fabric” rather than one‑off stacks tied to a single service.
The model(s) you start with will change. Agents will evolve. Regulations will tighten and change. What will matter is whether the environment you build around them makes those changes feel like routine upgrades — or full‑scale extractions. The winners in this next phase won’t be the fastest agent adopters; they’ll be the ones who make choices that enable deployment on infrastructure you control, as well as the freedom to pick models, environments and guardrails as conditions change.