Part 2 of 3: turning the Part 1 wire model into tuning decisions.

Tuning decisions from the wire model

Part 2 is the tuning guide: when overlap helps, when kernel choice matters, and when topology dominates everything.

There are three decisions that matter most in practice:

  1. whether DBO can hide the wire at all,
  2. whether DeepEP should be in low-latency or high-throughput mode,
  3. whether the deployment has crossed out of the fast locality domain and therefore entered a different communication regime.

Reminder from Part 1

Part 1 built the wire model. The important quantities were the per-rank token slice \(T = B/P\), the dispatch/combine payload, the scale-out fraction \(\rho\), and the straggler factor \(\gamma\). The takeaway was simple: widening EP only helps when each rank still sees enough work to amortize collective overhead. If \(B/P\) gets too small, the wire shows up before the experts are fully used.

In this part

  • The condition under which DBO turns exposed communication into overlap.
  • The DeepEP low-latency versus high-throughput crossover.
  • The topology boundary where a fast local fabric gives way to slower scale-out networking.

Notation recap

  • \(B\): total tokens in one MoE layer step across the EP group.
  • \(P\): expert-parallel ranks.
  • \(T = B/P\): balanced tokens per rank.
  • \(L\): fixed effective communication overhead.
  • \(BW_{\text{eff}}\): achieved bandwidth on the critical path.
  • \(\gamma\): straggler factor.
  • \(k, d\): routed experts per token and hidden size.
  • \(s_{\text{dispatch}}, s_{\text{combine}}, b_{\Sigma}\): dispatch/combine activation precision and the round-trip routing-sideband term per routed copy.

1) DBO: hiding the wire

The cleanest way to think about DBO is that it tries to turn a sum into a max.

Without overlap, a microbatch effectively pays

$$t_{\text{seq}} \approx t_{\text{comm}} + t_{\text{compute}}$$

With a healthy double-buffered steady state, it moves toward

$$t_{\text{dbo}} \approx \max\left(t_{\text{comm}}, t_{\text{compute}}\right)$$

Use the timeline below to compare sequential execution with a healthy double-buffered schedule. The important thing to watch is whether compute can cover communication, or whether the wire remains exposed.

There are two caveats I always keep attached to that equation:

  • it is a steady-state result, so you still pay fill and drain overhead,
  • it does not create compute cover; it only spends the cover you already have.

So the real question is still

$$t_{\text{compute}} \ge t_{\text{comm}}$$

and under the slightly more honest wire model from Part 1, a rough full-cover threshold is

$$B \gtrsim \frac{P\,L_{\Sigma}}{k\left(\gamma c_{\text{tok}} - \frac{d\,(s_{\text{dispatch}} + s_{\text{combine}}) + b_{\Sigma}}{BW_{\text{eff}}}\right)}$$

where \(L_{\Sigma} = L_{\text{dispatch}} + L_{\text{combine}}\). If you collapse both directions into the old symmetric shorthand and ignore routing sideband, this reduces to the familiar \(L_{\Sigma} \approx 2L\) and \(d(s_{\text{dispatch}} + s_{\text{combine}}) + b_{\Sigma} \approx 2ds\) version.

If the denominator is non-positive, the message is blunt: no amount of batch growth will give you ideal overlap under that model. You need a faster communication path, a smaller routed payload, or faster expert kernels.

This is still a first-order threshold. Any routing skew that inflates communication should be treated as part of the critical payload or absorbed into the measured \(BW_{\text{eff}}\).

The vLLM knobs that actually matter

Current vLLM exposes:

  • --enable-dbo
  • --dbo-prefill-token-threshold, default 512
  • --dbo-decode-token-threshold, default 32
  • VLLM_DBO_COMM_SMS, default 20

Those thresholds decide when it is worth paying the microbatching and overlap machinery. The SM reservation knob is best understood as a resource split, not a speed dial: more communication SMs can help progress on the wire, but they come directly out of the expert-compute budget.

That is also why DBO is not automatically a win on a single node. If communication is already tiny on NVLink or xGMI, there may be little left to hide, and all you are doing is paying pipeline overhead plus a more complicated resource partition.

Use the performance model below to sweep batch size, EP size, communication cost, and compute cost. The useful region is where the DBO curve separates from the no-overlap curve without starving the expert kernels.

The practical lesson is simple: if DBO does not help, first check whether the per-rank work is too small. Many bad DBO debates are really batch-size debates in disguise.

2) DeepEP kernel crossover

DeepEP is useful because it does not pretend one all-to-all kernel is optimal everywhere.

Its README explicitly separates:

  • normal / high-throughput kernels for prefill and training-style traffic, optimized for asymmetric-domain forwarding and supporting SM count control,
  • low-latency kernels for decode-style traffic, using pure RDMA and explicitly described as the latency-sensitive inference path.

The README also makes two practical points that are easy to miss:

  • low-latency mode is more buffer-hungry than normal mode,
  • the decode-style example recommends keeping the actual dispatch batch below 256 tokens per rank for best performance.

The operating rule is: start with low-latency for small decode payloads; move to high-throughput once payload size crosses the measured bandwidth crossover.

The mechanics widget below shows why the two modes behave differently. Low-latency favors direct RDMA. High-throughput pays more setup cost to use topology-aware forwarding.

What the published DeepEP tables are really telling you

DeepEP's own benchmarks are deliberately measuring different operating regions:

  • In a prefill-style benchmark on H800 + CX7 400 Gb/s (\(4096\) tokens, \(d = 7168\), top-8), the published normal kernels reach 43-58 GB/s bottleneck bandwidth on the internode cases.
  • In a decode-style benchmark on the same hardware class (\(128\) tokens, \(d = 7168\), top-8), the published low-latency kernels report 118-194 microseconds dispatch latency over the EP sizes they show.

Those tables are not apples-to-apples, and that is exactly why they are useful. They tell you the intended regimes are different.

The crossover view is still the right one

If you summarize the two families as \((L_{\text{LL}}, BW_{\text{LL}})\) and \((L_{\text{HT}}, BW_{\text{HT}})\), then HT wins when

$$L_{\text{HT}} + \frac{V}{BW_{\text{HT}}} < L_{\text{LL}} + \frac{V}{BW_{\text{LL}}}$$

which gives the crossover payload

$$V^* = \frac{L_{\text{LL}} - L_{\text{HT}}}{\frac{1}{BW_{\text{HT}}} - \frac{1}{BW_{\text{LL}}}}$$

Below \(V^*\), fixed overhead dominates and LL is the natural candidate. Above it, bandwidth dominates and HT becomes the better starting point.

How this maps to vLLM

The main controls are:

  • --all2all-backend deepep_low_latency
  • --all2all-backend deepep_high_throughput
  • VLLM_DEEPEP_BUFFER_SIZE_MB, default 1024
  • VLLM_DEEPEP_HIGH_THROUGHPUT_FORCE_INTRA_NODE, default 0
  • VLLM_DEEPEP_LOW_LATENCY_USE_MNNVL, default 0

Those last two are topology-specific escape hatches for systems with multi-node NVLink-like connectivity. They are not the first place I would look on an ordinary IB or RoCE deployment.

A short illustrative example

Suppose your measurements on one cluster look like this:

  • \(L_{\text{LL}} = 35\ \mu s\)
  • \(L_{\text{HT}} = 120\ \mu s\)
  • \(BW_{\text{LL}} = 22\ \text{GB/s}\)
  • \(BW_{\text{HT}} = 44\ \text{GB/s}\)

Then

$$V^* \approx 3.7\ \text{MB}$$

At \(B = 8192\), \(P = 16\), \(k = 8\), \(d = 7168\), and FP8 dispatch activations, the dispatch activation term alone is about

$$V_{\text{dispatch,act}} \approx \frac{8192}{16} \cdot 8 \cdot 7168 \approx 29\ \text{MB}$$

That already sits far above the toy \(V^*\). The actual dispatch bytes are somewhat higher once routing sideband is included, and a BF16 combine path would be larger again. So you are comfortably above the crossover. That does not prove HT is universally correct; it explains why HT is the more defensible starting hypothesis on that machine for that workload.

Use the analysis widget below to move that crossover point around. The exact answer changes with latency, bandwidth, and skew; the method does not.

3) Hardware locality cliff

The hardware question that matters most is not "which accelerator has the biggest top-line FLOP number?" It is:

How large is the fast locality domain before routed traffic spills onto a materially slower network?

That boundary changes the economics of Wide-EP much more than most marketing summaries do.

StackLargest fast locality domainScale-up anchorScale-out anchor
NVIDIA H100 HGX8 GPUs in one NVLink / NVSwitch domainNVIDIA advertises 900 GB/s GPU-to-GPU interconnect on H100 SXMCross-node traffic typically lands on 400 Gb/s class networking
AWS P58 H100 GPUs in one instanceAWS advertises 900 GB/s NVSwitch inside p5.48xlargeAWS advertises up to 3,200 Gbps EFA per instance
AMD MI300X platform8 GPUs on one platformAMD advertises 896 GB/s aggregate bidirectional peer-to-peer bandwidthExternal networking depends on the platform vendor and deployment
NVIDIA GB200 NVL7272 GPUs in one rack-scale NVLink domainNVIDIA advertises a 72-GPU NVLink domain with 130 TB/s rack-scale NVLink Switch bandwidthCrossing the rack boundary is a different scale-out problem

The units in that table are intentionally not collapsed into one scalar. A per-GPU NVLink number, an aggregate platform bandwidth number, and an instance-level cloud networking number are answering different questions.

The operational interpretation is much simpler:

  • on H100 HGX and MI300X-style systems, the cliff often appears once EP spills past the 8-GPU locality domain,
  • on AWS P5, the same idea applies inside one 8-GPU instance, but the scale-out anchor is instance-level EFA rather than a bare-metal NIC SKU,
  • on GB200 NVL72, the same math holds, but the fast domain is much larger.

So the "hardware cliff" is not a mystical batch threshold. It is the moment your critical routed bytes stop living on the fast fabric and start paying the slower one.

Once transport is no longer the only bottleneck, the next failure mode is imbalance.

Next in the series

In Part 3, I move from transport and topology to operational stability: what actually breaks first in production, when to turn on EPLB, when LPLB is worth thinking about, and why portability is as much a control-plane problem as a kernel problem.

References (for this part)