deepseek-ai/DeepEP — DeepEP: High-Performance GPU Communication for MoE Models
Transcript
QuickFacts
Welcome! Today we're exploring DeepEP, a specialized library built by DeepSeek AI. It's designed to solve one of the hardest problems in modern machine learning: how do you efficiently shuffle data between GPUs when training or serving massive Mixture-of-Experts models? Let's dive in and see what makes this codebase tick.
PlainEnglish
So what does DeepEP actually do? In Mixture-of-Experts models, each input token gets routed to a small subset of specialized expert networks. When those experts live on different GPUs—sometimes hundreds of them—you need lightning-fast communication to shuffle tokens around. DeepEP provides that highway. It uses NVLink for super-fast communication within a single server, and RDMA networks to zip data between servers, all without involving the CPU. The result is a system that works beautifully for both training massive models and serving low-latency inference.
QuickFacts
Let's look at the technology stack. At the application layer, DeepEP hooks into PyTorch and exposes a clean Python API via PyBind eleven. For data movement, it relies heavily on NVSHMEM—NVIDIA's partitioned global address space library—which enables GPUs to initiate RDMA operations directly. The CUDA kernels are hand-tuned for Ampere and Hopper architectures, like the A100 and H800. On the network side, we're talking NVLink for intranode bandwidth—think 160 gigabytes per second—and InfiniBand or RoCE for internode RDMA at around 50 gigabytes per second. The build system ties it all together using setuptools and CMake.
Architecture
Here's the high-level architecture. At the top, your PyTorch model interacts with DeepEP through the Buffer class—a friendly Python interface that hides all the complexity. Buffer talks to a C++ runtime layer via PyBind bindings, which manages memory allocation and coordinates with NVSHMEM. Underneath, we have three families of CUDA kernels. Intranode kernels handle communication within a single node using NVLink. Internode kernels bridge across nodes, combining NVLink and RDMA. And low-latency kernels are a specialized variant optimized for inference, using pure RDMA with minimal overhead. All of these ultimately drive the hardware layer—NVLink switches and RDMA network cards.
Architecture
Let's zoom out and see where the code actually lives. The deep underscore e p directory holds the Python API—buffer dot py is the heart of the user-facing interface. The c src directory contains all the native code. You'll find deep underscore e p dot cpp, which is the PyBind bridge, and then a kernels subdirectory packed with CUDA files. Layout dot c u computes token distributions. Intranode and internode dot c u implement the communication kernels. Internode underscore l l is the low-latency variant. There's also ibgda underscore device, which provides device-side RDMA primitives. Tests live in their own directory, and setup dot py orchestrates the build.
PlainEnglish
Let's talk about the two core operations: dispatch and combine. Think of dispatch like sorting mail at a post office. You have a bag of tokens, and each one has destination addresses—which experts it should visit. Dispatch reads those addresses, bundles tokens by destination, and sends each bundle to the correct GPU. Instead of mail trucks, though, we're using direct GPU links. Combine is the reverse journey. Each GPU sends its processed results back, and you merge everything together, weighing each piece by how important that expert was for that token. It's a beautiful dance of scatter and gather.
Architecture
Here's how normal dispatch works during training or prefilling. First, you call get dispatch layout with your top-k expert indices. The layout kernel calculates exactly which tokens go where and returns routing information. Then you call dispatch with your hidden states. The dispatch kernel writes tokens directly into remote GPU memory—using NVLink for same-node peers, RDMA for cross-node. It returns the tokens this rank received. You feed those through your expert networks, then call combine to gather and merge all the results back. This mode prioritizes throughput—moving thousands of tokens efficiently.
PlainEnglish
DeepEP offers two kernel families tuned for different workloads. Normal kernels are like freight trains—they haul massive cargo, thousands of tokens at a time, efficiently. You wait at the station for confirmation before unloading, but that's okay because the batches are huge. Low-latency kernels, on the other hand, are express delivery. They handle tiny packages—maybe just tens of tokens—and leave immediately without waiting. You can peek at arrivals with hooks while doing other work. For inference decoding where every microsecond counts, this mode is a game-changer.
Architecture
Low-latency mode looks quite different. You call low latency dispatch, which immediately posts RDMA sends and returns a hook—no waiting for the CPU. You invoke the hook to start background receives without blocking any GPU cores. This is critical: the network card pulls data in parallel while your SMs stay free. You can overlap expert computation with the next batch arriving over the network. Then low latency combine sends results back, again using hooks. The entire pipeline can be captured in a CUDA graph for replay, eliminating kernel launch overhead. Measured end-to-end latencies are as low as seventy-seven microseconds.
PlainEnglish
Let's visualize the network topology. Imagine GPUs in a server connected by NVLink—a superhighway running at 160 gigabytes per second. But servers connect via RDMA networks, more like regular roads at 50 gigabytes per second. DeepEP uses asymmetric forwarding to bridge these. When GPU zero needs to send data to GPU two on another server, it first zooms down the NVLink superhighway to GPU one, which is closest to the RDMA exit. GPU one then efficiently transfers onto the RDMA road. This minimizes congestion and keeps the fast links running at full speed.
Architecture
Before you can dispatch tokens, you need to initialize a Buffer object. Here's the journey. First, you create the Buffer, passing in your process group and buffer sizes. The library checks the network topology to detect which GPUs are connected via NVLink. Then it allocates communication buffers—separate pools for NVLink and RDMA traffic. Next comes handle exchange: all ranks share their device IDs and IPC memory handles using an all-gather operation. If you're using multiple nodes, NVSHMEM initializes here, setting up RDMA queue pairs. Finally, the runtime maps all remote peer buffers into the local address space for zero-copy access. Now you're ready to communicate.
Architecture
Let's peek at some actual code. This is from ibgda underscore device dot c u h, the device-side RDMA primitives. The function ibgda post send is called directly from GPU threads—not from the CPU. It manipulates InfiniBand queue pairs to post RDMA send operations. Zero CPU involvement, which is critical for achieving sub-microsecond latencies. Below that, ibgda poll completion lets a GPU thread check the RDMA completion queue. This is how hooks work: the network card does its thing in the background, and you poll when convenient, without tying up streaming multiprocessors. It's a really elegant piece of low-level engineering.
Architecture
DeepEP doesn't work in isolation—it integrates tightly with several external systems. PyTorch provides the process groups for rank coordination, CUDA streams for asynchronous execution, and the tensor API. NVSHMEM is the critical dependency for all RDMA features—it's NVIDIA's PGAS library with InfiniBand GPU Direct Async support. Below that, you need actual InfiniBand or RoCE network hardware with GPUDirect RDMA capabilities and virtual lane support for traffic isolation. Optionally, GDRCopy is a kernel module that provides CPU-assisted IBGDA when driver registry modifications aren't allowed. And if you prefer MPI over PyTorch's distributed layer, the Buffer class accepts MPI4PY communicators as well.
Community
Let's check in on the community health. The repo shows a dormant signal right now—the last commit was about 81 days ago. Over the past year, there were 176 commits, but activity has tapered off recently with just one commit in the last 90 days. The project has 47 contributors all-time, which is pretty healthy for a specialized library. The top contributors are Chenggang Zhao with 88 commits and Shangyan Zhou with 66. There are 13 branches and one tagged release. This looks like a research artifact that had intense development during the DeepSeek V3 project and has since stabilized. It's not abandoned—just mature.
PlainEnglish
Imagine you're a framework developer wanting to add DeepEP to your custom MoE training system. Here's your journey. First, install NVSHMEM following the third-party README—you might need to tweak NVIDIA driver settings or install GDRCopy. Then run setup dot py to compile the CUDA kernels for your target GPU architecture. In your MoE layer code, create a global Buffer once during initialization, sizing it based on your hidden dimension and expert count. During each forward pass, extract the top-k indices and weights from your routing layer. Call dispatch with those tensors and your hidden states—you get back the tokens this rank should process. Run your expert computation on those tokens, then call combine to merge everything back together. For best performance, run the test suite on your cluster to auto-tune the config parameters.
PlainEnglish
So there you have it—DeepEP in depth. This is a library laser-focused on one thing: moving data between GPUs as fast as physically possible for Mixture-of-Experts models. It's tuned for everything from small 8-GPU setups to massive 256-rank clusters. Whether you're training the next generation of foundation models or serving ultra-low-latency inference, DeepEP provides the communication backbone to make it happen. The architecture is clean, the integration points are well-defined, and the performance numbers speak for themselves. Thanks for exploring this with me!
How this was made
Lenzon read deepseek-ai/DeepEP and generated this walkthrough automatically. The narration above is the transcript of what it says.
Explain a pull request from your own repo
Point Lenzon at a repo or a pull request and get a narrated walkthrough like this one.
Try it