IBM-backed llm-d framework adds controls for running AI models in-house

IBM-backed llm-d framework adds controls for running AI models in-house

IBM Corp. and several other contributors to the open-source llm-d project are releasing version 1.0 of their distributed artificial intelligence inference framework, adding operational controls intended to help enterprises run large models reliably on their own infrastructure.

The release, announced ahead of the PyTorch Conference North America in San Jose today and tomorrow, focuses on managing competing workloads, maintaining service during upgrades and improving hardware efficiency as AI agents generate growing volumes of model requests.

Built for Kubernetes, llm-d coordinates model serving across computing infrastructure. IBM, its Red Hat subsidiary and Google LLC are its lead maintainers. They co-founded the project last year with CoreWeave Inc. and Nvidia Corp. Other contributors include Advanced Micro Devices Inc., Cisco Systems Inc., Hugging Face Inc., Intel Corp. and Mistral AI SAS.

The project offers organizations seeking more control over inference costs and infrastructure choices a way to reduce dependence on a single model provider, cloud or hardware platform.

“At IBM Research, we are seeing that open large language models and llm-d can greatly lower the costs of inference in computationally demanding problems such as designing hardware for quantum error correction, helping achieve a 11.8X efficiency advantage,” Ruchir Puri, IBM Research’s chief scientist, vice president of IBM’s Technology & Technical Community and IBM Fellow, wrote in e-mailed comments. “Llm-d 1.0 extends these savings by delivering the operational readiness needed for enterprises to extract more out of their computational capacity.”

The project is focused on what happens after deployment. A production inference service must handle changing traffic, hardware failures, software updates and users competing for resources while maintaining acceptable response times.

Version 1.0 adds support for quality of service across multiple tenants, zero-downtime operations and serving patterns tailored to AI agents. Flow controls and prioritization are designed to keep bursts of agent activity from overwhelming services that require fast responses, such as coding assistants used by people.

The release “adds the maturity to run it like any other critical service: a stable [application programming interface], multiple tenants with competing priorities sharing the same infrastructure and upgrades that don’t interrupt operations,” said Carlos Costa, distinguished engineer at IBM Research and a project co-founder and lead maintainer.

Workloads can behave differently even when they reach models through the same endpoint. An agent may issue many requests, pause to use an external tool and return for additional model calls.

The framework can keep requests directed to appropriate infrastructure across multiple turns, increasing opportunities to reuse previously processed context. That matters because agents repeatedly consult the same code, documents, instructions and conversation history while completing assignments.

A routing system directs requests toward replicas that already hold relevant cached context. Another capability moves cached information among graphics processing unit memory, CPU memory and storage, preserving reusable information when GPU capacity becomes constrained.

These techniques were previously available, but version 1.0 integrates them with operational controls, along with methods that separate prompt processing from response generation, optimize work batches and distribute large mixture-of-experts models across accelerators. IBM cited a recent agentic coding test in which the framework’s router captured 97% of available cache reuse.

In a separate demonstration, IBM and Red Hat said they ran 3,000 software-engineering agents concurrently on self-hosted infrastructure at a cost five to 10 times lower than serving the same open model through a hosted service.

The economics become more significant as agents perform multiple rounds of reasoning, tool use and revision for a single assignment, IBM said. Reprocessing identical material at each step consumes additional computing resources.

Llm-d works with inference engines including vLLM and SGLang, supplying distributed routing and operational capabilities around model servers. It is currently hosted as a sandbox project by the Cloud Native Computing Foundation.

“It delivers high-performance, distributed inference with the reliability and scale enterprises need within an operational model they already know,” said Stephen Watt, vice president in Red Hat’s chief technology officer’s office.

Image: Pixabay