An Interactive Reading of

Polestar: Drift-Aware Cache Calibration
and Token Commitment

Efficient Inference of Diffusion Large Language Models
The paper, in plain English

Diffusion language models (dLLMs) promise a way out of the one-token-at-a-time bottleneck of ChatGPT-style autoregressive generation. Instead of writing left to right, a dLLM starts with a strip of masked tokens and iteratively denoises them — many at once — until the answer emerges. The catch: because every token attends to every other token in both directions, the model's internal representations keep shifting with each step. That makes KV-cache reuse unreliable and makes it hard to know when a token is truly "done."

Polestar's insight is that both problems share one root cause: token representation drift. When a new token is unmasked, the attention patterns of nearby tokens shift measurably. Polestar measures this drift with KL-divergence and uses it as a compass. High drift near the active decoding block? Refresh the stale KV cache there. A sharp drift spike on a still-masked token? It is converging toward its final answer — commit it early.

The result: across math and coding benchmarks on LLaDA-8B and Dream-7B, Polestar sets a new state-of-the-art accuracy–throughput Pareto frontier, achieving up to 10.73% accuracy improvement over baselines, 3.7× higher throughput, and decoding parallelism of 3.67 tokens per forward pass — all without any retraining.

I
Polestar-Cache
Uses token representation drift to pinpoint exactly which KV-cache positions have gone stale, then performs sparse, surgical refreshes — no wasted recomputation.
II
Polestar-Commit
Detects sharp drift spikes that signal a token is converging toward its final answer, enabling earlier and more reliable commitment than confidence-only thresholds.
III
System Co-Design
Offloads quantized hidden states to CPU and overlaps drift computation with the forward pass via CUDA streams, hiding overhead entirely.
~ 20 minutes · 8 chapters · 7 interactive simulations
CHAPTER 1

The parallelism dilemma

Autoregressive LLMs write one token at a time — left to right, no exceptions. Diffusion LLMs promise to break that bottleneck by denoising many tokens at once. So why is nobody using them in production yet?

In a diffusion LLM, generation begins with a sequence of [MASK] tokens. At each denoising step $t$, the model predicts the most likely token at every position:

$\hat{y}_i^{(t)} = \arg\max_{v \in \mathcal{V} \setminus [\text{MASK}]} \; m_\theta\!\left(v \mid \mathbf{y}^{(t)},\, i\right), \quad \forall\, i \in \mathcal{J}$

Under semi-autoregressive decoding, the response is split into contiguous blocks of size $B$. Within a block, tokens are denoised in parallel; across blocks, decoding proceeds sequentially. At step $t$, the model commits all positions whose confidence exceeds a threshold $\tau$:

$\mathcal{S}^{(t)} = \bigl\{\, i \in \mathcal{M}^{(t)} \;\big|\; c_i^{(t)} \geq \tau \,\bigr\}$
Sampler set: positions with confidence above the static threshold $\tau$ are unmasked.

The average number of tokens decoded per forward pass — called TPF (tokens per forward) — measures decoding parallelism. A higher $\tau$ means fewer tokens per step (slower but safer); a lower $\tau$ means more tokens per step (faster, but accuracy degrades because uncertain tokens are committed prematurely).

The second challenge is KV-cache reuse. Under block-wise caching, keys and values for positions outside the active block are computed once at block entry and reused across steps. But because of bidirectional attention, these cached states drift out of sync with each new token decoded inside the block — a problem we explore next.

The Confidence–Parallelism Trade-off

Drag the threshold slider. Watch how lower confidence buys parallelism at the cost of accuracy.

0.50 (aggressive)0.99 (safe)
864
Tokens per forward (TPF)
—
Expected accuracy
—
Forward passes / block
—
Errors / block
—
Why this matters
The static confidence threshold $\tau = 0.9$ — the industry default — commits only when the model is already very sure. That leaves substantial parallelism on the table. Many tokens that could be committed safely are held back because their raw confidence is below 0.9, even though their representations have already converged. Polestar's key insight: there is a better signal than confidence alone.
What is token representation drift? →
CHAPTER 2

The invisible earthquake

When a single token is unmasked in a dLLM, every other token's internal representation shifts — not because of a bug, but because of the fundamental mathematics of bidirectional attention. The paper calls this token representation drift.

The paper formalizes drift using the KL-divergence of a token's attention distribution between two consecutive denoising steps. For token position $i$ at layer $\ell$ and step $t$:

$$D_{i,\ell}^{(t)} = \text{KL}\!\left(\, \mathbf{A}_{i,\ell}^{(t)} \;\Big\|\; \mathbf{A}_{i,\ell}^{(t-1)} \,\right)$$
Equation (4): Drift measured as the KL-divergence of attention distributions between consecutive steps.

Why KL-divergence instead of cosine similarity (used by prior work like Elastic-Cache)? Because KL captures how a token's attention has redistributed across the entire sequence, not just how a single proxy vector has shifted. The paper proves this choice matters empirically: KL-based drift achieves 86% selection accuracy for identifying stale KV positions, versus much lower for alternatives (Chapter 8).

Proposition 1: Drift is structural, not accidental

The paper provides a first-order proof that unmasking one position changes the attention output of every other position that can attend to it — even under exact full recomputation. When position $s$ changes its input from $\mathbf{x}_s$ to $\mathbf{x}_s + \boldsymbol{\eta}$, the attention output of position $i$ changes by:

$$\mathbf{o}_i(\mathbf{x}_s + \boldsymbol{\eta}) - \mathbf{o}_i(\mathbf{x}_s) = a_{is}\, W_V \boldsymbol{\eta} \;+\; a_{is}\, \frac{\mathbf{q}_i^\top W_K \boldsymbol{\eta}}{\sqrt{d_h}}\, (\mathbf{v}_s - \mathbf{o}_i) \;+\; O(\|\boldsymbol{\eta}\|^2)$$
Proposition 1: First-order change in attention output when position $s$ is unmasked.

The key observation: $a_{is} > 0$ whenever $s$ is visible to $i$ (which is always true under bidirectional attention). So the effect is generically nonzero. In contrast, under causal (autoregressive) attention, if $s$ is not visible to $i$, then $a_{is} = 0$ structurally, and the effect vanishes.

This is the theoretical foundation for everything Polestar does: drift is not noise to suppress, but a signal to harness.

Simulating Attention Drift

Click "Unmask next token" to step through the denoising process. Watch how each position's attention shifts.

Step
0
Unmasked tokens
0
Avg drift (KL)
—
Max drift position
—
Why this matters
Drift is not a side effect of caching shortcuts — it is baked into the architecture of bidirectional attention. Every token unmasking sends ripples through every other position. The question becomes: can we use these ripples diagnostically? The next two chapters show that drift is an extraordinarily good predictor of both cache staleness and commitment readiness.
How drift reveals cache staleness →
CHAPTER 3

Where the cache rots

KV-cache staleness is not uniform — it concentrates sharply around the active decoding block. The paper shows that token representation drift pinpoints exactly where the cache has gone stale, with a precision that prior proxy signals cannot match.

The paper measures KV-cache staleness as the cosine distance between cached and fully recomputed KV tensors:

$$E_i^{(t,\ell)} \;\triangleq\; 1 - \frac{1}{2}\!\left[\cos\_\text{sim}\!\left(K_i^{(t,\ell)},\, K_{i,\text{ref}}^{(t_b,\ell)}\right) + \cos\_\text{sim}\!\left(V_i^{(t,\ell)},\, V_{i,\text{ref}}^{(t_b,\ell)}\right)\right]$$
KV-cache staleness metric (§C.2): distance between cached KV (from block entry $t_b$) and recomputed reference.

Corollary 1 of Proposition 1 shows that hidden-state drift directly induces KV-state drift. The change in key and value at position $i$ is $\Delta K_i = W_K \Delta H_i$ and $\Delta V_i = W_V \Delta H_i$. If the combined projection $[W_K; W_V]$ has a restricted minimum singular value $\sigma_{\min} > 0$ on the drift subspace, then:

$$\sqrt{\|\Delta K_i\|_2^2 + \|\Delta V_i\|_2^2} \;\geq\; \sigma_{\min}\, \|\Delta H_i\|_2$$
Corollary 1: Hidden-state drift lower-bounds KV-state drift.

This means: wherever hidden states drift, KV states drift too. The paper's Figure 3(a) confirms this empirically — the top-25% highest-drift tokens in the local window around the active block accumulate cache error far faster than low-drift tokens or tokens far from the block.

Drift vs. Cache Staleness Correlation

Each dot is a token position. Drag the step slider to see how the correlation evolves during decoding.

Step 1Step 20
1B5B
Drift–staleness correlation
—
High-drift share of error
—
Tokens needing refresh
—
Why this matters
If you know where the cache is stale, you can refresh only those positions — a sparse, surgical update instead of a wasteful full-sequence recomputation. This is the architectural insight behind Polestar-Cache, which we build in the next chapter. The correlation between drift and staleness is so strong that Polestar achieves 86% selection accuracy using drift, versus 52% for the cosine-similarity proxy used by Elastic-Cache.
Building Polestar-Cache →
CHAPTER 4

Sparse, surgical refresh

Polestar-Cache clusters the local window's hidden states, projects cluster centroids through attention to compute "proxy drift," and refreshes only the top-$k$ most stale positions — layer by layer, step by step.

Polestar-Cache maintains a local window $\Omega_b = \Omega_b^p \cup \Omega_b^s$ around the active block, consisting of a prefix window ($|\Omega_b^p| = 2B$) and a suffix window ($|\Omega_b^s| = B$). At block entry, it caches the input hidden states $H_{\Omega_b}$ for all layers.

During decoding, at each step $t$ and layer $\ell$, cluster centroids are projected into query tokens to compute a proxy attention distribution $P^{(t,\ell)}$:

$$D^{(t,\ell)} = \text{KL}\!\left(\, P^{(t,\ell)} \;\Big\|\; P^{(t-1,\ell)} \,\right)$$
Proxy drift: KL-divergence of centroid-level attention distributions.

The top-$k$ clusters by drift magnitude are selected, and a sparse forward pass over just those clusters' hidden states produces an update packet $U^{(t,\ell+1)}$. This packet refreshes stale cached hidden states and KV entries within the local window for the next layer. The centroids are updated incrementally to reflect the refreshed states.

The update schedule is also drift-aware: Polestar-Cache refreshes when either (a) a single step decodes more than $\tau_\text{upd}$ tokens, or (b) the cumulative decoded tokens since the last refresh exceeds $\tau_\text{upd}$ (default: $\tau_\text{upd} = 3$). This reacts to bursts of contextual change rather than refreshing on a rigid timer.

Polestar-Cache Refresh Simulator

Adjust the clustering parameters and watch how the refresh budget affects cache accuracy.

416
1 (minimal)8 (maximal)
1 (frequent)10 (rare)
Cache freshness
—
Compute saved
—
Effective accuracy
—
Refreshes / block
—
Why this matters
Polestar-Cache's drift-guided refresh means the cache stays accurate without paying for full-sequence recomputation at every step. This directly improves throughput: fewer wasted FLOPs per step, and higher prediction confidence (because the cache the model attends to is fresher). The paper reports that Polestar-Cache alone improves accuracy by +1.01% and throughput by 1.26× over Fast-dLLM — before token commitment even enters the picture.
How drift enables early token commitment →
CHAPTER 5

The convergence spike

A token's journey toward its final answer is not gradual — it happens in a sharp jump. Polestar-Commit detects these drift spikes to commit tokens earlier and more reliably than any static confidence threshold.

Polestar-Commit maintains a drift history window of length $h$ (default $h = 5$). The drift delta for masked position $j$ is the deviation of current drift from its historical mean:

$$\Delta_j^{(t)} = D_j^{(t)} - \text{mean}(\Theta)$$
Drift delta: how much the current drift exceeds the recent rolling average.

This delta is then compared against a confidence-conditioned dynamic threshold that adapts to each token's certainty level:

$$\tau_{d,j}^{(t)} = \alpha\,(\tau_s - c_j^{(t)})^2$$
Dynamic drift threshold (§4.2): lower-confidence tokens require stronger drift spikes to be committed.

A token at position $j$ in the current block is committed when $\Delta_j^{(t)} \geq \tau_{d,j}^{(t)}$. This elegantly interpolates between two regimes:

For the suffix block, Polestar uses a conservative rule: drift identifies which suffix positions are eligible (high-drift suffix clusters from the update packet), but commitment requires $c_j \geq \tau_s$. So drift determines where to look, confidence determines whether to act.

The Dynamic Commitment Threshold

Adjust the scaling factor α and confidence to see how the drift threshold curve shifts. The green zone is where tokens get committed.

2 (aggressive)40 (conservative)
0.700.99
0 (raw drift)10
Tokens per forward (TPF)
—
Expected accuracy
—
Confidence-only TPF
—
Speedup from drift
—
Why this matters
Confidence alone is a blunt instrument: lowering $\tau_s$ from 0.9 to 0.7 increases parallelism but crashes accuracy. Polestar-Commit's drift-aware threshold is fundamentally different — it commits tokens when their representations have converged, not when their raw logits happen to cross a line. The paper shows this lifts TPF from 2.87 to 3.67 while maintaining accuracy — something no static threshold can achieve.
How system design makes it practical →
CHAPTER 6

Overlapping the overhead

Polestar's drift computation adds work to every decoding step. Without system-level optimization, the gains from smarter caching and commitment would be eaten by overhead. The paper solves this with CPU offloading and CUDA-stream-based latency hiding.

CPU Offloading

Cached hidden-state clusters are quantized using NVFP4 residual quantization (1 sign bit, 2 exponent bits, 1 mantissa bit, with FP8 group scales). This reduces memory by 4× compared to FP16 and slashes CPU–GPU transfer bandwidth. Only the top-$k$ drifting clusters are fetched asynchronously from pinned CPU memory during decoding — and their dequantization is fused with the update-packet computation to eliminate separate overhead.

CUDA-Stream Latency Hiding

Polestar uses separate CUDA streams for its operations:

Optimized vs. Unoptimized Execution

Toggle between execution modes to see how CUDA-stream overlap eliminates critical-path overhead.

Critical-path overhead
—
Per-layer latency
—
Overhead saved
—
Quantization Matters
The paper ablates quantization formats: direct NVFP4 without residuals collapses accuracy to 54.98%, but residual NVFP4 recovers to 78.33% — within 0.57% of FP16. The residual trick (quantizing the difference between each hidden state and its cluster centroid) is what makes aggressive 4-bit compression viable without accuracy loss.
Results: the new Pareto frontier →
CHAPTER 7

A new efficiency frontier

Across four benchmarks, two model families, and two generation lengths, Polestar dominates the accuracy–throughput Pareto frontier. It is simultaneously more accurate and faster than every prior baseline.

The main results table (Table 1 in the paper) compares Polestar against five baselines across GSM8K, MATH, and HumanEval on both LLaDA-8B-Instruct and Dream-7B-Instruct. Key headline numbers:

Accuracy–Throughput Pareto Frontier

Select a benchmark and model to explore the trade-off space. Each dot is a method.

GSM8K
Math · 5-shot
MATH
Math · 4-shot
HumanEval
Code · 0-shot
Polestar accuracy
—
Polestar throughput
—
Speedup vs baseline
—
Best competitor acc.
—
Why this matters
Polestar does not just win on one axis — it dominates the frontier. Every other method forces you to pick a point on the accuracy–speed curve. Polestar moves the curve itself. This is what algorithm–system co-design looks like when it works: the drift signal is good enough to guide both cache management and commitment, and the system engineering is tight enough that the signal's computation is effectively free.
What the ablations reveal →
CHAPTER 8

What actually matters

The paper's ablation studies dissect every design choice: drift measurement policy, commitment rule, cluster count, update schedule, and more. Here is what the data says about what drives Polestar's gains.

Component Ablation

Starting from Fast-dLLM as the base, each Polestar component adds measurable improvement:

The component stack tells a clean story: Polestar-Cache improves accuracy by +1.01% and TPF by +0.15 over Fast-dLLM. Adding current-block commitment lifts TPF to 3.48. Adding suffix-block commitment reaches the final 3.67 TPF — with negligible accuracy loss.

Drift Measurement Policy

Polestar's KL-divergence achieves 86% selection accuracy for identifying the top-25% most stale KV positions. Random selection gets 25% (by definition). The cosine-similarity proxy used by Elastic-Cache gets only 52% — barely better than random for the hardest positions.

Commitment Rule Comparison

Static $\tau_s = 0.9$ commits 2.87 TPF safely. Lowering to $\tau_s = 0.7$ increases TPF to 3.71 but drops accuracy by 5%. Raw drift magnitude recovers some of that loss. But the drift delta with dynamic threshold achieves 3.48 TPF while improving accuracy to 78.6% — strictly better than the static threshold.

Hyperparameter Stability

Across the full sweep range, Polestar remains stable near its defaults. Block size $B = 32$ is the sweet spot ($B = 64$ degrades accuracy to 75.1% from premature commitment). History length $h = 5$ balances responsiveness and stability. The scaling factor $\alpha = 10$ sits in a wide plateau where both accuracy and throughput are near-optimal. This robustness means Polestar does not require per-model or per-dataset tuning — the same defaults work across LLaDA, Dream, and LLaDA-V.

The big picture
Polestar's contribution is not a single trick but a unified framework: one signal (token representation drift) solves two problems (cache staleness and commitment readiness) simultaneously. The ablations confirm that every piece earns its keep, and the defaults are robust. The method is training-free, works across model families and modalities, and sets a new Pareto frontier that prior work cannot match. That is what a well-designed inference framework looks like.

"Polestar leverages token representation drift to selectively refresh stale KV-cache positions and to commit stable tokens earlier in both current and suffix blocks, while system optimizations reduce the overhead of Polestar-specific operations. Across mathematics, coding, and hard-to-parallelize benchmarks, Polestar establishes a new state of the art on the accuracy–throughput Pareto frontier."

— Lee et al., 2026