After Autoregression
Parallel and any-order generation is the first credible bid against left-to-right — simultaneously a training story, a transparency story, and a serving story.
§1 · The bet
Every article in this series has treated one fact as fixed: language models emit text one token at a time, left to right. That factorization was a modeling choice that hardened into infrastructure — and in mid-2026 the alternative stopped being a curiosity. This closing essay argues the bet in three separable evidence lines from this collection: a training line (diffusion language models now train from scratch at serious scale and close gaps on hard benchmarks), a transparency line (their interpretability question has been asked and partially answered), and a systems line (they already pay rent in production serving). Three independent lines rising together is what a paradigm bid looks like from the inside.
SCOPE, per this series' design review: the argument runs entirely on the mid-2026 wave in this collection. The earlier diffusion-LM canon is not re-treated here; where a paper in this corpus names its own predecessor as a baseline, it is reported as that paper's comparison, nothing more. The ten-year framing of paradigm bets in general belongs to the published essays [← C2]; latent-compute recursion stays with [← A11].
The case against left-to-right is now three independent evidence lines — training, transparency, systems — all sourced from a single month's literature. No single line is decisive; their simultaneity is the signal.
§2 · The training story
The center of the training line is iLLaDA, from Renmin University and ByteDance Seed: an 8B masked diffusion language model trained from scratch with fully bidirectional attention, keeping the masked-diffusion objective through both pre-training and supervised fine-tuning — no autoregressive stage anywhere in the recipe (Nie et al., 2026). The scale is no longer toy: 12T pre-training tokens, then a 25B-token instruction corpus for 12 epochs, with variable-length generation for efficiency. Against LLaDA — the predecessor the paper names as its baseline — the improvements are large and specific: iLLaDA-Base gains 21.6 points on BBH and 14.9 on ARC-Challenge; iLLaDA-Instruct gains 14.5 on MATH. A diffusion-native training pipeline, at trillion-token scale, improving on reasoning benchmarks: that is the existence proof the paradigm needed.
Post-training has followed the modality. On-policy self-distillation — the workhorse treated at recipe level in [← C4] — was built autoregressive-first: privileged information enters as a left-to-right prefix, supervised token by token. Luo, Chen and colleagues show that design "fundamentally conflicts with the arbitrary-order generation of dLLMs," and rebuild it as d-OPSD: the self-teacher is constructed from self-generated answers used as suffix conditioning — the student learns from its own future experience rather than a privileged prefix — and supervision moves from token level to step level, matching the iterative denoising process (Luo, Chen, et al., 2026). It is a quietly important signal: when a paradigm gets its own post-training stack rather than borrowing one, it has stopped being a demo.
And between the camps sits a pragmatic middle path. Mila's AC-GPT observes that the thing everyone actually wants from any-order models — evaluating and sampling arbitrary conditionals, a block of text conditioned on past and future — can be had with a simple modification to a standard causal transformer, in a single forward pass, while preserving the left-to-right objective that makes training efficient; existing LLMs can even be fine-tuned into it (Lu, Elmoznino, et al., 2026). The middle path matters for the bet's honest odds: if arbitrary conditioning is the killer feature, autoregression may absorb it without ceding the throne.
The training line matured on three fronts at once: from-scratch masked diffusion at 12T tokens with double-digit reasoning gains over its named baseline, a diffusion-native post-training stack (suffix-conditioned, step-level d-OPSD), and an AR-compatible route to arbitrary conditioning that could let the incumbent absorb the challenger's best feature.
§3 · The transparency story
A new generation paradigm inherits an old obligation: can anyone see what it is doing? Google DeepMind's interpretability team — Engels, McDougall and Chughtai as primary contributors, with Neel Nanda advising — put the question to DiffusionGemma with unusual precision, splitting transparency into variable transparency (do we understand intermediate snapshots of the computation?) and algorithmic transparency (can those snapshots reconstruct how the output was reached?) (Engels, McDougall, Chughtai, et al., 2026).
The naive audit is alarming: DiffusionGemma does more of its work in continuous latent space, and its opaque serial depth — serial computation between interpretable states — appears at first 28.6× higher than the corresponding autoregressive Gemma 4 model. The recovery is the finding: the information flowing between denoising steps can be routed through an interpretable token bottleneck with no decrease in downstream performance in the pulled results. The transparency tax of leaving left-to-right is real, measurable — and, at least for this decomposition, payable. For a series that spent three articles on evaluators and verification, the meta-point writes itself: the interpretability community engaged the new paradigm before it shipped at scale, which is the correct order of operations for once.
Off the AR path, interpretability had to be redefined — and was: variable vs algorithmic transparency, a measured 28.6× naive opaque-serial-depth gap, and a token-bottleneck construction that restores inspectability without costing performance. The paradigm's transparency question is open, quantified, and being answered.
§4 · The systems story
The third line is the least romantic and the most predictive: parallel generation is already earning production money, in two shapes.
The hybrid shape is NVIDIA's TwoTower. Existing diffusion LMs force one network to serve two roles — represent context and iteratively denoise. TwoTower decouples them: a frozen autoregressive context tower processes clean tokens causally, and a trainable diffusion denoiser tower with bidirectional block attention refines noisy blocks by cross-attending to it (Reda, Kamalu, et al., 2026). Built on an open 30B hybrid Mamba-Transformer MoE and trained on ~2.1T tokens, it retains 98.7% of the autoregressive baseline's quality at 2.42× higher wall-clock generation throughput, with code and weights released. That is not a research promise; that is a price-performance clause.
The infiltration shape was covered from the serving side in [← 11] and needs only its paradigm reading here: block-diffusion draft models slot into speculative decoding precisely because they emit whole blocks in one forward pass, and confidence-scheduled verification exists precisely because parallel proposals decay. The challenger's first production job is drafting for the incumbent — history's usual arrangement between paradigms, and the cheapest way for the new factorization to accumulate deployment evidence while the from-scratch line matures.
The systems line pays today: a frozen-AR-context + diffusion-denoiser hybrid holds 98.7% of baseline quality at 2.42× throughput with open weights, and pure-parallel drafters already work inside the incumbent's serving stack. Paradigms that earn revenue while still "losing" tend not to stay losers.
§5 · The wave, scoped — and the falsifiers
Assemble the timeline (see Figure 2) and the shape of mid-2026 is unmistakable: within roughly five months, this one collection contains a from-scratch 12T-token dLLM, a diffusion-native post-training method, a frontier-lab transparency audit, a released production hybrid, and two serving systems exploiting parallel drafts. The cautionary sibling rides along: autoregressive video diffusion's exposure-bias problem — models trained on ground-truth context but deployed on their own imperfect outputs — needed Self Forcing's train-on-your-own-rollouts correction (Huang et al., 2026), the [← 7] lesson in another costume. New factorizations do not repeal train-under-deployment-distribution; they re-encounter it.
An essay-grade bet owes its readers falsifiers, and this one has three, one per evidence line. Training: if by the 2028 review no from-scratch diffusion LM matches a same-compute autoregressive model on broad reasoning benchmarks — not curated subsets — the training line failed. Transparency: if token-bottleneck-style constructions cannot be maintained at production scale without measurable performance cost, the paradigm carries a standing interpretability tax that agent deployments [← 8] will refuse to pay. Systems: if hybrids like TwoTower remain the ceiling — diffusion as the incumbent's accelerator, never the model of record — then "after autoregression" was really "beside autoregression," and the AR-compatible middle path of AC-GPT absorbs the rest. The dated review is committed below.
CLAIM — parallel/any-order generation becomes a peer factorization for language models, not an accessory.
EVIDENCE NOW — from-scratch 12T training with double-digit reasoning gains over the named baseline; a diffusion-native post-training stack; a quantified-and-answered transparency audit; 98.7% quality at 2.42× throughput, weights released.
FALSIFIERS — no same-compute reasoning parity by review date; interpretability constructions that do not scale; hybrids as a permanent ceiling.
REVIEW — July 2028, against this article's three lines, in public, whatever the verdict.
The bet is filed with its falsifiers: reasoning parity at matched compute, transparency that scales, and at least one deployment where the diffusion model is the model of record. Review scheduled for July 2028 — and the exposure-bias lesson says whichever paradigm wins will still have to train under the distribution it deploys under.
What comes next
This closes the reading order — twelve articles, four substrates, one loop. If the series has a single sentence, it is D1's: self-improvement stopped being a training-time event and became an operating property. Where the loop goes next is not a section; it is whichever of these substrates your production system touches first. Start at the beginning, and bring a Loop Card.
References
- Nie, S., Min, Q., Xu, S., Huang, Z., Song, Y., Shan, Y., Lin, Y., Zhao, W. X., Li, C., & Wen, J.-R. (2026). Improved Large Language Diffusion Models. arXiv preprint. arXiv:2606.25331.
- Engels, J., McDougall, C., Chughtai, B., Kramar, J., Rajamanoharan, S., Wu, C., Conmy, A., Chen, A. Q., Tarbouriech, J., Ma, M., O'Donoghue, B., Lopes de Oliveira, J. G., Shah, R., & Nanda, N. (2026). How Transparent is DiffusionGemma? arXiv preprint. arXiv:2606.20560.
- Luo, Y., Chen, Z., Wang, H., Hu, X., Zhang, Y., Sha, Z., & Liu, S. (2026). Learning from the Self-future: On-policy Self-distillation for dLLMs. arXiv preprint. arXiv:2606.18195.
- Lu, Y., Elmoznino, E., Gagnon, L., Mittal, S., Kasetty, T., & Lajoie, G. (2026). Simplifying the Modeling of Arbitrary Conditionals in Natural Language. arXiv preprint. arXiv:2606.14943.
- Reda, F., Kamalu, J., Waleffe, R., Patwary, M., Shoeybi, M., & Catanzaro, B. (2026). Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context. arXiv preprint. arXiv:2606.26493.
- Chen, J., Liang, Y., & Liu, Z. (2026). DFlash: Block Diffusion for Flash Speculative Decoding. arXiv preprint. arXiv:2602.06036.
- Cheng, X., Yu, X., Shao, C., Li, J., Xiong, Y., et al. (2026). DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv preprint. arXiv:2607.05147.
- Huang, X., Li, Z., He, G., Zhou, M., & Shechtman, E. (2026). Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. arXiv preprint. arXiv:2506.08009.