Almost every chatbot writes one word at a time, left to right. Diffusion language models write a whole chunk at once and refine it over a few passes, the way a photo sharpens out of static. The diffusion text models that work best today are discrete: every slot holds either a definite word or a blank "mask" that gets filled in, and once a word is in place it stays. That makes it hard to change course partway, and hard to get good results with only a few refinement passes.
This paper works in a continuous space instead. Each of the 131,072 tokens in the vocabulary becomes a point on a 16-dimensional sphere. Generating text means starting from pure Gaussian noise and sliding these points step by step until each lands on a word. Because the path is smooth, you can steer it with tools borrowed from image diffusion: classifier-free guidance (lean towards what the prompt implies) and score temperature (commit harder). The NVIDIA team scales this up to 3B and 8B parameters by warm-starting from an existing autoregressive model, and they call the result Sigma.
On math and coding, Sigma-3B in diffusion mode averages 64.34 across six benchmarks. That is level with NVIDIA's own masked-diffusion model at 64.74, and it posts the best base-model scores on GSM8K (84.5) and HumanEval (59.5). Turn the knobs towards more variety and it pulls ahead when you sample many times: 91.8% vs 87.6% pass@256 on MBPP, and +10 points pass@256 on AIME 2025. Text diffusion can work in a continuous space at billion-parameter scale, and that space gives you real levers to steer it.
Words are learned unit vectors in 16-D. Gaussian noise blurs them, and a transformer predicts a probability-weighted average of embeddings to undo the blur.
Guidance (w) and score temperature (τ) bend the denoising path. Together they set where the model sits between careful single answers and diverse exploration.
Deterministic DDIM, a sphere-spreading regulariser (SUSReg) and trajectory distillation keep quality reasonable at 16, 8 and even 4 steps per block.
You can add Gaussian noise to a pixel. How do you add noise to the word "def"?
Following Plaid, a variational diffusion model over token embeddings, a sequence $\mathbf{x}\in\{0,1\}^{L\times V}$ of one-hot rows is mapped through a learnable embedding matrix $\mathbf{E}\in\mathbb{R}^{V\times d_e}$, whose rows lie on the unit sphere, with $d_e = 16 \ll V = 131{,}072$. The forward process is variance-preserving Gaussian noise on $\mathbf{e} := \mathbf{x}\mathbf{E}$:
The signal-to-noise ratio $\mathrm{SNR}(t)=\alpha_t^2/\sigma_t^2$ falls monotonically with $t$. Sigma learns the schedule too. Its endpoints and its interior shape are both trainable:
Here $\varsigma$ is the sigmoid and $\tilde\gamma\in[0,1]$ is a monotone shape. The NELBO's diffusion term depends only on the endpoints $\gamma_0<\gamma_1$. The interior shape only changes the Monte-Carlo variance of the loss, so the two parts are learned for different reasons. Sigma-3B's learned endpoints are $\gamma_0=-8.188$ and $\gamma_1=5.000$ (App. G, Fig. 12). We use those as the slider defaults.
One design choice gets special mention. Because $\mathbf{E}$ appears inside eq. (1), the embeddings become likelihood-trained parameters. The model is graded on how well it can recover tokens from noise, so it learns to spread words apart where that helps. This is what separates Plaid-style models from continuous models built on fixed encodings such as one-hot vectors, and not learning $\mathbf{E}$ makes Plaid noticeably worse.
Discrete diffusion lives on a lattice with more than 100,000 categories per position. Sigma lives in a smooth 16-D space where small moves are possible. That smoothness is what lets the later chapters steer, shortcut and distil the generation path.
The network never outputs a point. It outputs odds, and the odds become a point.
A denoiser $\mathbf{x}_\theta(\mathbf{z}_t,t)\in(\Delta^V)^L$ outputs a categorical distribution over the vocabulary at every position. The categorical reparameterisation turns it into a predicted clean embedding:
Training minimises the negative ELBO, which splits into a prior KL, a reconstruction cross-entropy at $t=0$, and an SNR-weighted regression of $\mathbf{e}_\theta$ onto the true embedding:
In the toy below, the "network" is the exact Bayes posterior over 8 tokens with a uniform prior: $x_\theta(v)\propto\exp(\alpha_t\langle\mathbf{z}_t,\mathbf{e}_v\rangle/\sigma_t^2)$. It is the best answer any trained network could give, so you can see what perfect training would aim for. The readout shows the per-position integrand of $\mathcal{L}_\text{diff}$, $-\tfrac12\mathrm{SNR}'(t)\|\mathbf{e}_\theta-\mathbf{e}\|^2$.
Sigma also lets the denoiser see its own previous guess: $\mathbf{x}_\theta(\mathbf{z}_t,t;\mathbf{e}_\text{sc})$. During training, 25% of samples first run a pass with no gradient to produce $\mathbf{e}_\text{sc}=\mathbf{x}_\theta(\mathbf{z}_t,t;\mathbf{0})\mathbf{E}$, and only the second pass is trained. The other samples get $\mathbf{0}$, so both branches are learned. This adds only about 8% to non-embedding FLOPs. At inference the previous step's prediction is fed back for free.
The output is a mixture of real token embeddings, so the model never invents an off-vocabulary vector. Its uncertainty shows up geometrically as how far inside the sphere the guess falls. That is the quantity the steering knobs in Chapter 5 reshape.
Diffusion likes to see everything at once. Long documents and KV caches like to go left to right. Sigma does both.
The sequence is split into $B$ contiguous blocks $\{\mathbf{x}^{(b)}\}_{b=1}^B$ of size $L'=L/B$. The likelihood factorises autoregressively across blocks, and each factor is bounded by a conditional NELBO:
Reconstruction (at $t=0$) and diffusion (at $t\in(0,1)$) cannot share one forward pass. So, as in RePlaid, each block draws $R\sim\mathrm{Bern}(p_\text{rec})$ and evaluates only one term, in a Horvitz–Thompson estimator:
Clean tokens x attend causally to earlier clean tokens (the AR stream, also used for L_ar).
A noisy block z(b) attends bidirectionally within itself.
Noisy block b attends to the clean blocks b′ < b, its context.
The variance has a closed form. With deterministic $\mathcal{L}_\text{rec}=a$ and $\mathcal{L}_\text{diff}$ of mean $b$ and Monte-Carlo spread $s$: $\mathrm{Var}=a^2/p+(b^2+s^2)/(1-p)-(a+b)^2$, minimised at $p^\star=a/(a+\sqrt{b^2+s^2})$. The paper's $p_\text{rec}=0.1$ was found empirically to minimise RePlaid's training-objective variance. That fits a regime where reconstruction is cheap relative to the diffusion term. The toy losses here are illustrative; the paper does not report $a$, $b$ or $s$.
Training from scratch is slow, so Sigma warm-starts from an AR checkpoint pretrained on 1T tokens. The 16-D diffusion embedding is initialised randomly because the dimensions don't match. Continuous diffusion also suffers from high gradient variance, so the AR head stays in the objective as a regulariser:
The flagship 3B run decays $w_\text{diff}$ from 1.0 to 0.3 at 200B tokens. Without the AR loss, the AR-stream perplexity explodes early in training (Fig. 3c). Everything Sigma adds on top of the shared transformer is a 16-D unit-norm embedding table, a linear up-projection, a self-conditioning projection, Fourier time conditioning and a learnable schedule. Together that is under 1% of parameters. Training runs 150K steps at batch 512 and $L=4096$, over 300B tokens, and took about 9,600 / 19,200 GB200 GPU-hours for 3B / 8B.
The block-causal design keeps what makes AR models cheap to serve, a reusable KV cache. The diffusion runs only inside a 32-token window. The Horvitz–Thompson trick makes each block cost one forward pass without biasing the ELBO.
The same denoiser, two ways to take a step: one adds fresh dice rolls, the other follows the slope.
To generate block $b$, the clean context $\mathbf{c}:=(\mathbf{p},\mathbf{r}^{(<b)})$ sits in the KV cache and $\mathbf{z}_1^{(b)}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. At each step, the denoiser gives $\hat{\mathbf{e}}_\theta=\mathbf{x}_\theta(\mathbf{z}_t^{(b)},t;\mathbf{c})\mathbf{E}$ and the implied noise $\hat{\boldsymbol\epsilon}_\theta=(\mathbf{z}_t^{(b)}-\alpha_t\hat{\mathbf{e}}_\theta)/\sigma_t$. The two samplers then step from $t$ to $s<t$:
After the last step, tokens are read out as $\arg\max\,\mathbf{x}_\theta(\mathbf{z}_0^{(b)},0;\mathbf{c})$ and appended to the context. In the toy below, the "language" has 16 tokens in $d_e=16$ and two-token blocks. The model's conditional distribution puts a chosen mass on its favourite pair and spreads the rest over five others. The denoiser is the exact Bayes posterior, so every error comes from the sampler.
Note the failure mode. At very few steps, both samplers over-produce the favourite pair: a single jump lands on the posterior mean, which rounds to the mode. In the real model, the same discretisation error shows up as falling benchmark accuracy below roughly 16 steps per block (Fig. 7). The paper also tried higher-order ODE solvers from image diffusion (DEIS, DPM-Solver++). Surprisingly, they did worse than DDIM. A uniform-γ time grid, equally spaced in log-SNR, works best across a wide range of budgets.
In the paper, DDIM substantially outperforms DDPM at low NFEs while matching it at high NFEs. Sigma-3B's six-task average is 64.34 with 256 DDPM steps, 60.45 with 32 DDIM steps and 55.17 with 16. An 8× cut in compute costs only about 4 points.
Image diffusion has had a "make it more like the prompt" dial for years. Discrete text diffusion struggles to support one. Sigma gets it for free.
Classifier-free guidance. The clean context $\mathbf{c}$ is the condition. Dropping it ($\varnothing$) means the noisy block attends only to itself:
This is a pure inference-time trick. Sequence packing means the first block of a pretraining document often starts mid-document with no context, so the model already learned the unconditional branch, and no train-time context dropout is needed. Both branches are packed into one forward pass with a special mask, which costs +1.8% at $L'=32$.
Score temperature. Rescaling the score by $1/\tau$ is equivalent to extrapolating away from the naive guess $\mathbf{z}_t/\alpha_t$ (since $\mathbb{E}[\mathbf{z}_t\mid\mathbf{x}]=\alpha_t\mathbf{e}$):
Proposition 1 (they commute). Applying ST then CFG equals CFG then ST, and the result is the score of a tilted distribution:
The proof uses Tweedie's formula, $\mathbf{s}=\tfrac{\alpha_t}{\sigma_t^2}(\mathbf{e}-\mu_t)$ with $\mu_t=\mathbf{z}_t/\alpha_t$: CFG and ST are both affine maps fixing $\mu_t$, so they commute. Self-conditioning is fed only the unsteered conditional prediction.
Using both knobs beats using either alone (Fig. 5), and both run at almost no latency cost. The heat-map also shows the edge: at τ = 0.05, guidance of 3 or more drives every benchmark into single digits. The extrapolation overshoots the sphere and decoding falls apart.
For one answer you want a sharp model. For 256 attempts, as in reinforcement learning or test-time search, you want a curious one.
With $n$ samples per problem of which $c$ are correct, the unbiased estimator is $\text{pass@}k = 1-\binom{n-c}{k}\big/\binom{n}{k}$, averaged over problems. The toy below runs 16 "problems", each with six valid two-token answers. The model's preferred answer is right on the easy problems and wrong on the hard ones, and 48 samples are drawn per problem with the steered DDIM sampler from Chapter 5.
The paper's evidence, base models first. On MBPP, as k grows, the best Sigma setting moves from τ = 0.1 to 0.4, 0.7 and finally 1.0. Sigma overtakes the masked-diffusion baseline NLD at k = 32 and reaches 91.8% vs 87.6% at k = 256. Sigma repeats only 10.4% of its 256 output strings, against 53.1% for NLD. NLD gets more correct samples per problem (139.5 vs 91.5), but Sigma finds more distinct correct programs (71.4 vs 46.9). On GSM8K (right chart, Table 19), NLD leads at small k, the curves cross near k = 16, and Sigma is ahead from k = 32. It solves 1,314 of 1,319 problems at least once, against 1,305 for NLD. On Minerva Geometry at matched pass@1, Sigma ties NLD's best-tuned sampler. It beats only NLD's lower-temperature settings, by 1.9 to 6.6 points at pass@256.
For the instruct models on AIME 2025, Sigma-3B-Instruct (w = 3, τ = 0.3) leads NLD's best sampler at every k, by +1.7 points at k = 1 and +10.0 at k = 256 (0.633 vs 0.533). With only 30 problems, the authors note that just 1 of 27 paired comparisons has a confidence interval that excludes zero.
Pass@k is a proxy for how much room reinforcement learning has to find better answers. Sigma has two continuous dials that slide it along the exploration–exploitation frontier, with no retraining. That suggests continuous diffusion could be a good starting point for RL. The paper does not run RL itself.
Sigma was handed 16 dimensions for its vocabulary and, left alone, used about four.
SUSReg compares the empirical characteristic function of 1-D projections of the embeddings along random directions $\mathbf{a}_m\sim\mathrm{Unif}(\mathbb{S}^{d_e-1})$ with the closed-form target for a uniform sphere, under an Epps–Pulley Gaussian-weighted $L_2$ metric:
The factor $V$ makes the expectation independent of vocabulary size as $V\to\infty$. Sigma-SUSReg-3B uses $\lambda_\text{susreg}=0.005$, $M=512$ directions, and $P=17$ trapezoid points on $[-5,5]$. The cost, $O(VM(d_e+P))$, is negligible next to a transformer pass. The toy below computes $s(\mathbf{E})$ live with $M=48$.
Below 16 NFEs even DDIM drops off sharply, so the many-step trajectory of Sigma-3B is distilled into a few-step student with PDD. The student's parallel heads cannot use recurrent computation, so self-conditioning is disabled in both teacher and student. CFG is also amortised: a single conditional student call learns to match the two-call guided teacher. The paper's numbers (Table 8, six-task average, block size 32):
SUSReg costs almost nothing in validation perplexity, yet adds +3.3 points at 16 steps and +5.7 at 8. Combined with distillation, it makes 4 and 8 steps per 32-token block usable, up to 8 tokens per network call. A masked model that commits tokens in discrete chunks has no smooth trajectory to distil.
Matched data, matched architecture, matched FLOPs: how does continuous diffusion stack up?
A few observations beyond the averages. In AR mode, Sigma-3B gets the best six-task average of any 3B model (65.47), so the diffusion training did not damage the AR head. Diffusion-mode Sigma-3B beats LLaDA-8B by 22.6 points and Dream-7B by 5.6 on average. Inside a block, Sigma commits early positions later and late positions earlier than a confidence-based masked sampler, so it shows less left-to-right bias (Fig. 20). The CFG/ST settings tuned on base models carried over unchanged to the SFT models, which suggests they are robust to fine-tuning and scale.
Evaluations are in non-thinking mode, with modest SFT and no RL. Self-conditioning cannot be used by the parallel PDD student. AIME has only 30 problems, so pass@k gaps there are mostly not statistically resolved. Sigma still trails the AR-mode baselines on harder instruct tasks such as AIME and LiveCodeBench. Our toy simulations reproduce the mechanisms (sampler error, CFG/ST tilting, collapse), not the benchmark numbers.
Before this paper, continuous text diffusion was stuck below 2B parameters and mostly evaluated unconditionally. Sigma shows it can match the best masked diffusion model at 3B and 8B on reasoning and code. It also brings along the image-diffusion toolbox: guidance, temperature, ODE samplers and distillation. That is why the authors consider continuous diffusion a promising route to fast language generation.