- DeepSeek released DeepInfer, the inference framework powering its production models, under the Apache 2.0 license on Oct 20, 2026.
- The release supports DeepSeek-V4.1-Flash and runs on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs.
- DeepSeek says the framework handles 155 trillion tokens per day per cluster with a 96% input cache hit rate.
- DeepInfer was built primarily on vLLM, with additional ideas drawn from SGLang and TensorRT-LLM.
What Happened
DeepSeek has open-sourced DeepInfer, the inference framework it uses to run its large-scale production services, Marktechpost reported (Oct 20, 2026). The release is licensed under Apache 2.0, and its README lists support for both Nvidia SM100 GPUs and Huawei’s Ascend 950 NPUs, though the only model currently supported is DeepSeek-V4.1-Flash.
Why It Matters
By open-sourcing the inference layer rather than just model weights, DeepSeek is giving developers a way to reproduce the serving performance behind its own production deployments. Dual Nvidia and Huawei NPU support also signals DeepSeek’s continued hedge against reliance on a single hardware vendor amid export-control pressure on Chinese AI labs.
Technical Details
DeepInfer includes optimized kernels, parallelism for mixture-of-experts models, speculative decoding with draft tokens, deterministic sampling, and structured outputs. DeepSeek reports throughput of 155 trillion tokens per day per cluster and a 96% input cache hit rate. The framework was built primarily on vLLM, with additional techniques drawn from SGLang and TensorRT-LLM.
Who’s Affected
Some reports noted the initial release lacks pieces of a complete inference service, meaning teams adopting DeepInfer today would need to build supporting infrastructure around it rather than deploy it as a turnkey serving stack. The release is most immediately useful to teams already running or evaluating DeepSeek-V4.1-Flash at scale.
What’s Next
DeepSeek said inference code for future models and ongoing optimizations will continue to land in the DeepInfer repository. The release comes the same week the US barred DeepSeek from federal agency systems and follows DeepSeek’s backing of a $400 million robotics lab, underscoring the company’s push to expand its footprint even as US government access narrows.
