DeepSeek's DSpark cuts per-user inference latency 85% with speculative batch decoding

DSpark uses a lightweight draft model to propose candidate responses and a larger model to verify them in batches, cutting per-user response time by up to 85%; a confidence-based scheduler then adjusts verification depth based on GPU load. The gain is real: DeepSeek built this explicitly for export-constrained hardware, and it signals that inference efficiency — not raw chip count — is where China's AI push is compounding.

Source: scmp.com ↗

token-by-token output often slowed when responses were lengthy, leading to low utilisation of graphics processing units (GPU) and high user-perceived waiting time, which was a "primary bottleneck in serving AI"

DeepSeek

Why this matters

→ Cuts response latency 85%, directly addressing inference bottleneck that slows real-world AI apps
→ Demonstrates China's efficiency-first AI strategy working around US chip export controls
→ Shows inference optimization now matters more than raw hardware scaling

Efficiency over hardware