DeepSeek Open-Sources the Engine Behind Its Models

DeepSeek has published DeepInfer, the software it uses to run its own models in production, releasing the framework under a permissive Apache 2.0 licence. The Chinese lab paired it with a smaller tool called deepseek-recipe for token encoding, and said the code already serves its DeepSeek V4.1 Flash model to the public.

Why serving software matters

Open model weights get the attention, but weights alone do not make a model cheap to run. The harder, less glamorous work is inference: feeding requests through the model fast enough, and at low enough cost, to serve millions of users without the bill spiralling. An inference engine squeezes more throughput out of the same hardware, and that is the part companies usually keep to themselves.

DeepInfer bundles the techniques that make large mixture-of-experts models practical, including distributed inference across many chips, speculative decoding, expert parallelism with load balancing, and structured outputs. The project builds on the open vLLM engine and borrows ideas from SGLang and Nvidia's TensorRT-LLM, so it enters a field that is already competitive rather than inventing one.

Running on Huawei as well as Nvidia

The detail drawing the most notice is hardware support. DeepInfer runs on Nvidia's SM100 generation, the silicon behind the B200 and GB200, and on Huawei's Ascend 950 accelerators. Native support for Huawei's chips matters at a moment when American export rules keep the fastest Nvidia parts out of China, and it signals that a serious Chinese model can be served end to end on domestic hardware.

DeepSeek reported throughput and latency figures for V4.1 Flash that have not yet been independently checked, so those numbers are worth treating with care. The release continues the lab's open-first habit, following a fundraising round that swelled toward $14 billion and a wider shift in which Western products are leaning on open Chinese models. Giving away the engine, not just the model, raises the stakes of that trend.