DeepInfer, Explained: How DeepSeek Serves 155 Trillion Tokens a Day

On 16 October 2026, DeepSeek did something AI labs rarely do. It open-sourced the software that runs its own production API.

The project is called DeepInfer. It's the inference framework behind DeepSeek-V4.1-Flash, and according to DeepSeek's own deployment notes, a single cluster running it serves 155 trillion tokens a day. Within weeks, tens of thousands of developers had starred the repository on GitHub.

Most of the AI conversation is about models: who trained the biggest one, who tops which benchmark. DeepInfer is about the other half of the problem, the half that decides what a model costs to use. DeepSeek's documentation puts it in one sentence:

A model defines the boundary of what is possible; serving those capabilities to a massive user base at a reasonable cost is the core problem an inference framework has to solve.

This post explains what DeepInfer is, how it works, and why its design choices are worth understanding even if you'll never run a 128-GPU cluster. I've kept the jargon to a minimum and explained it wherever it's unavoidable.

DeepInfer at a glance

First, some vocabulary

Four ideas come up again and again. Here they are in plain English.

DeepSeek-V4.1-Flash is an MoE model. The benchmark charts in the repo show 384 routed experts. Serving an MoE model well is mostly about getting the right tokens to the right experts on the right GPUs, fast.

The big idea: reading and writing are different jobs

Every request to a language model has two phases, and they want opposite things from the hardware.

• Prefill reads your whole prompt at once. Thousands of tokens go through the model in parallel, so the GPUs are busy doing math. The limit is raw compute. Prefill decides your time to first token.
• Decode writes the answer, one token at a time for each user. Each step does a little math but has to read a lot of model weights from memory. The limit is memory bandwidth, and the fix is to batch many users together. Decode decides your tokens per second.

Because they're so different, DeepSeek runs them on separate pools of machines, a design called prefill/decode disaggregation:

• Prefill runs on ordinary 8-GPU machines, using the smallest expert-parallel group that fits in memory (EP2 to EP8). Prefill batches are already big, so small groups keep load balanced and cut network traffic.
• Decode runs on 64- to 128-GPU supernodes using wide expert parallelism (EP64 to EP128). Spreading the experts across many GPUs means each GPU reads less memory per step, which raises throughput.
• Images are handled by a third pool: a vision encoder service, where a single, cheaper GPU per instance is usually enough.

Once prefill finishes, the KV cache it produced is sent to a decode machine, and decoding continues there.

The architecture

Here's the full system DeepSeek runs in production, from DeepInfer's own documentation:

Follow one request through it:

1. DeepSeek Recipe turns the API request (Chat Completions, Responses or Messages format) into tokens and sampling settings.
2. The Entrypoint checks the KV cache store for a reusable prefix of the conversation.
3. The Prefill Load Balancer sends the request to a prefill engine, and the Vision Encoder processes any images.
4. Prefill computes the KV cache for whatever part of the prompt wasn't already cached.
5. The Decode Load Balancer picks a decode engine, and decode generates the answer.
6. Tokens flow back through the Entrypoint and DeepSeek Recipe, and new KV cache is saved to 3FS storage for next time.

What's actually in the repo

It's important to be precise here, because "DeepSeek open-sourced its inference stack" is only partly true.

The open-source release is the engine: everything that runs on a node and makes one model replica fast. The parts that glue many replicas into a service, namely the KV cache system, the prefill-to-decode wiring and the load balancers, are tied to DeepSeek's internal infrastructure and aren't included. DeepSeek describes their design in its deployment Q&A instead, and says most of the engine-side functionality for disaggregation is already in the code.

So DeepInfer today is best understood as a high-performance engine plus a very detailed blueprint for the rest.

Six techniques worth understanding

1. Wide expert parallelism, with load balancing

In an MoE model, each token is routed to a few experts. With expert parallelism (EP), different experts live on different GPUs. The danger is that some experts are far more popular than others, so a few GPUs end up doing most of the work while the rest wait.

DeepInfer's answer is EPLB (expert-parallel load balancing). Each GPU holds a few redundant copies of popular experts. The system tracks how busy each expert has been over recent steps and periodically reshuffles which GPU hosts what. DeepSeek reports that the slowest GPU ends up only about 4.5% to 9% busier than the average.

Going wider pays off in decode. Here's DeepSeek's comparison of 8-way and 64-way expert parallelism:

Read the chart like this: further right means each user gets their answer faster, and higher means the GPU serves more tokens in total. Every point on the EP64 curve is above and to the right of the matching EP8 point. Spreading experts wider gives you both.

2. A KV cache that lives on SSDs for days

When you continue a conversation, or an agent calls the model again with a long, mostly identical prompt, most of the prompt has been seen before. If its KV cache is still around, prefill can skip it.

DeepSeek takes this further than almost anyone:

• The KV cache is stored on a dedicated 3FS distributed file system with about 100 PiB of SSD, not in GPU memory or RAM.
• It's kept for 3 days for agent traffic and 7 days for chatbot traffic. Three days covers a weekend, so Monday's agent runs can reuse Friday's context.
• Data is protected with 10+2 erasure coding, so each storage group survives two nodes failing at once.
• Because storage is separate from compute, any prefill machine can use any cached prefix, and a crashed GPU server doesn't lose the cache.

Why SSDs? They're far cheaper per gigabyte than memory, and DeepSeek's model compresses its KV cache heavily, so reading it from disk is fast enough. It also helps explain why providers can charge so much less for cached input tokens than for fresh ones.

3. SWA bounded replay: recompute less

DeepSeek-V4.1-Flash uses sliding-window attention (SWA) in many layers, where each token only looks at a fixed number of recent tokens. Normally, rebuilding that state means reprocessing a long stretch of the prompt.

Bounded replay recomputes only the most recent 128 tokens and limits the attention window to that segment. That makes the recomputation cost a constant, no matter how long the conversation is. Applied to the decoder layers, DeepSeek says it nearly halves total prefill computation.

4. DSpark: smarter speculative decoding

Decode produces one token per step, and every step costs a full trip through the model. Speculative decoding cheats, legitimately: a small, fast draft model guesses the next few tokens, and the big model checks all the guesses in a single step. Correct guesses are kept, so one expensive step can yield several tokens.

DeepInfer's version, DSpark, adds a twist. Checking guesses isn't free, and late guesses are usually wrong. So a confidence head estimates how likely each draft token is to be accepted, and the scheduler only verifies the guesses worth checking.

DeepSeek's measurements: verifying 3.2 to 4.4 draft tokens per step instead of always 5, while keeping about the same number accepted (3.20 versus 3.29). At the production load of about 60 sequences per GPU, that's 4% to 7% more throughput, and up to 11% at higher loads.

5. Zero-overhead scheduling

Each decode step has CPU work (deciding what to run next, processing results) and GPU work (running the model). Done one after another, the GPU sits idle while the CPU thinks.

DeepInfer overlaps them. While the GPU runs step N, the CPU is already preparing step N+1 and finishing step N−1. The GPU never waits. It's a simple idea, borrowed from SGLang and NanoFlow, and it's on by default.

6. Correctness you can rely on

Three features exist so that results are predictable, not just fast:

• Structured outputs constrain the model to valid JSON, a regex or a grammar. The constraint is compiled ahead of time on a separate machine, then applied as a token mask at each decode step.
• Batch-invariant inference gives identical results no matter which other requests happen to share the batch. Normally, batch composition can change results in tiny ways.
• Deterministic sampling gives identical output for identical input, settings and seed. That's invaluable for debugging and evaluation.

There's also a security detail in deepseek-recipe, the companion library that converts API requests into tokens. If a user types text that looks like a special control token (such as <｜User｜>), it's encoded as ordinary text, so prompts can't smuggle in control tokens.

The numbers

DeepInfer's README publishes two benchmark charts for DeepSeek-V4.1-Flash at a 131,072-token context.

What these say in plain terms:

• Prefill on just 4 GPUs processes roughly 60,000 to 90,000 new prompt tokens per second per GPU. Bigger batches are more efficient but make each step take longer.
• Decode on 64 GPUs shows the trade-off every inference provider manages. With few users per GPU, each user gets around 325 tokens per second, faster than anyone can read. Pack 256 users per GPU and total output rises tenfold to about 26,000 tokens per second per GPU, while each user still gets about 100 tokens per second.

One caveat from the chart itself: the decode numbers assume three draft tokens with a 75% per-token acceptance rate, so real throughput depends on how predictable the text is.

Could you run it?

You can build it, but this isn't a laptop project.

# Linux, GCC 13+, Python 3.12, and CUDA 13 (or CANN 9.2 for Ascend)
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

# Downloads DeepSeek-V4.1-Flash-1017 from Hugging Face on first run
python examples/offline_inference.py --ep 8

The release supports offline inference: you hand it a batch of prompts and get results back. It's not yet a ready-made API server, because the KV cache system, disaggregation wiring and load balancers aren't included. DeepSeek's own guidance is that the smallest efficient production unit is 64 GPUs for prefill plus 64 for decode.

So who is DeepInfer for?

• Cloud and inference providers that serve DeepSeek models and want to match DeepSeek's own speed and cost.
• Infrastructure engineers studying how frontier-scale serving actually works, with real code and real numbers.
• Everyone else, as one of the most detailed public explanations of production LLM serving. The deployment Q&A alone is worth reading.

What I took away

The model is only half the product. DeepSeek's prices are famously low, and DeepInfer shows where that comes from: disaggregated prefill and decode, wide expert parallelism, days-long SSD caching, smarter speculative decoding and GPUs that never wait. None of that is in the model weights.

Cheap storage beats expensive memory. Keeping KV caches on SSDs for days, instead of in RAM for minutes, is a striking design choice, and it's what makes long agent sessions affordable.

Honest documentation is a feature. The repository is unusually clear about what isn't included, which parts depend on internal infrastructure, and which internal versions are slightly faster than the open ones (FlashMLA's decode kernel, by 5% to 10%). That candor makes it much easier to trust everything else.

Inference used to be the boring part of AI. DeepInfer is a good argument that it's where most of the engineering now lives.

Sources

• DeepInfer on GitHub: README, feature list, build steps and benchmark charts (Apache-2.0)
• DeepSeek V4.1 Flash Inference Deployment Q&A, in the DeepInfer repository: architecture, parallelism, KV cache storage, DSpark, scheduling and production figures
• DeepSeek-V4.1-Flash-1017 on Hugging Face: model card, license and context window

The architecture diagram and the three benchmark charts are reproduced from the DeepInfer repository under its Apache-2.0 license. The repository screenshot was taken in November 2026; star counts change daily.