DeepSeek is open-sourcing the software it uses to run its AI models at scale

DeepSeek is open-sourcing DeepInfer, the inference framework that powers its large-scale production services, the company said in the project's GitHub repository.

The release supports the DeepSeek-V4.1-Flash model and runs on Nvidia $NVDA SM100 GPUs and Huawei Ascend 950 NPUs, according to the repository. DeepSeek plans to add support for future systems, according to Bloomberg.

"We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable," the Hangzhou-headquartered company said on Tuesday on X $TWTR, according to Bloomberg.

DeepInfer includes optimized kernels, parallelism methods for attention and mixture-of-experts layers, and a decoding method that uses draft tokens and confidence estimates to schedule verification. It also offers asynchronous scheduling that overlaps CPU batch scheduling with device execution, batch-invariant inference, deterministic sampling and structured outputs. Batch-invariant inference and deterministic sampling work on Nvidia hardware only.

The release provides offline inference and leaves out parts of a complete inference service. Users must supply a KV cache system, prefill and decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode, DeepSeek said. The framework requires Linux, GCC 13.0 or later and Python 3.12, plus CUDA 13 on Nvidia hardware or CANN 9.2 on Ascend.

DeepSeek released DeepInfer under the Apache License 2.0. The company said the framework was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.

The release reveals the methods DeepSeek relies on to hold down inference costs for advanced AI, according to Bloomberg. AI companies tend to guard such production techniques. Firms that host or run DeepSeek's models in the cloud can apply the methods on their own, and the price of serving a given model can vary widely with the serving software, Bloomberg said.

No outside party has yet benchmarked DeepInfer's efficiency gains, according to Bloomberg. The framework applies to DeepSeek's own models, but outside developers might read through the code and borrow its techniques for their own systems, Bloomberg said.

DeepInfer's kernel list includes custom TileLang kernels and covers Huawei's Ascend 950 chips. Last month, DeepSeek announced a partnership with Huawei to build programming tools for Ascend chips. As part of that effort, it open-sourced six software modules for the Ascend platform, including a version of its TileLang chip programming language adapted to the Ascend 950.