Small language models promise savings. Here's what happens later.

The pitch for small language models has become almost reflexive. AI too expensive? Too slow? Too risky? Try a smaller model.

Last year, Gartner predicted that by 2027, organizations will run specialized, task-specific models at roughly three times the volume of general-purpose large language models. That shift is already showing up in enterprise budgets: In July, Gartner projected that spending on domain-specific and specialized AI models will grow 210% in 2026 alone, more than double the growth rate of foundation models.

However, CIOs who are running small language models (SLMs) in production — or trying to — describe a more complex reality. SLMs can offer efficiency, though not to the expected level, and entail costs that don't show up until after go-live.

Where the math actually works

"Enterprise SLM adoption has moved from pilots toward mainstream architecture, and growth is accelerating," said Ameet Ubhayaker, director of enterprise data analytics and AI at Protiviti. He added that he isn't seeing wholesale replacement of large models, but an emerging tiered stack: small models for routing, classification and structured output, with frontier models reserved for ambiguous, multistep work.

Related:Shadow AI has gone agentic. Are organizations ready?

Jason James, previously CIO at Aptos, was running three or four specialized models as prototypes at the retail tech vendor on local hardware rather than in the cloud, specifically to measure whether the cost and effort of running them on-premises beats what they're paying frontier models in token costs.

"It is really a math exercise," James said. "What is the cost to implement? What is the cost to run? And what is the volume of use? If these are all significantly less to run locally, only then should it be considered. If it's only slightly less, then why bother?"

While James recently left Aptos to take on the CIO role at Acumatica, the numbers had been trending in favor of the small models, enough so that Aptos was continuing the pilots. But he noted that these SLMs weren't replacing the frontier tools his Aptos teams were already using. They were giving him more options while avoiding what he calls tokenization bankruptcy — the pattern in which inadequately trained users burn through a year's token budget in weeks because of poor model selection and prompting.

Narrow, repeatable, verifiable tasks such as classification, routing or structured extraction, in environments with real data-boundary or compliance requirements, are where SLMs shine, according to Ubhayaker.

"This does not mean SLMs are replacing frontier models," he said. "The destination is a layered architecture, not wholesale migration."

Related:The 3 organizational barriers blocking AI deployment for CIOs

When the data breaks the business case

That distinction is exactly what Eric Pace learned the hard way. Pace, who recently left an enterprise IT leadership role and now advises companies on AI deployment, describes a small, fine-tuned Llama model his team deployed to curate marketing messaging. It was trained on historical data labeled good or bad. Testing looked strong. The path to production was fast and inexpensive — unexpectedly so. Then the data that had looked reliable in testing didn't hold up at production scale.

"We eventually pulled it out," he said.

Pace added that he still sees SLMs as a time-saver, with drift and versioning manageable within realistic budgets. But he now treats data quality as a materially bigger issue for fine-tuned SLMs than for general AI use.

"Not only is there a need to ensure that the data sets represent what you want as inputs to the model, but you have to find the line," he said. "Overfitting is real." His advice? Keep the task contained and specific, plan for back testing, and don't underestimate the business knowledge and data stewardship required from teams outside IT.

Ubhayaker has seen this play out before, with companies evaluating SLMs against public benchmarks instead of their own production traffic and data.

Related:How CIOs can climb out of the AI governance chasm

"Organizations should also optimize for cost per successful task, not simply cost per token," he said. "A smaller model may appear cheaper, but retries, human review and escalation to larger models can quickly erase those savings."

Who owns the fleet?

Even when an SLM performs, running several of them introduces a different kind of overhead. James describes it in practical terms: when a single workflow passes through more than one specialized model, tracing a failure gets harder, because troubleshooting now means working through multiple model owners to find where — and with whom — the problem originated.

Every specialized model needs its own evaluation and monitoring to catch behavioral drift, Ubhayaker said, and a fleet of them adds complexity around routing, upgrades and lineage. Base model updates and downstream fine-tunes require retraining and re-benchmarking. The bigger risk, in his experience, is a "confidence-maturity gap." Leadership assumes monitoring policies are being followed, while telemetry shows they often aren't, especially as teams deploy specialized models faster than the organization can catalog and govern them.

Rejecting the binary

None of this indicates that SLMs are categorically a bad bet. They just demand more upfront diligence than many expect. Model cost isn't total cost. It's essential to budget for governance, management overhead and training before tallying up any savings, James said.

Whether a task belongs on a small or large language model should be a per-use case call, Ubhayaker said. Deciding factors should include whether the task is bounded and verifiable, whether the organization has the engineering capacity to maintain the model and whether a smaller model can actually achieve the required accuracy — not just which architecture is cheaper on paper. The value of SLMs is conditional, based on the task, the data and who owns what happens to the model after it ships.

What has your organization learned from running SLMs? Let us know at [email protected].

About the Author