Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models

來源原標題

Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models

原始連結

https://machinelearning.apple.com/research/diffusion-autoregressive-performance

摘要原文

research area Speech and Natural Language Processing content type paper published August 2026 Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models Authors Minseo Kim†, Coleman Hooper‡, Aditya Tomar‡, Chenfeng Xu‡††, Mehrdad Farajtabar, Michael W. Mahoney‡§¶, Kurt Keutzer‡, Amir Gholami‡§* View publication Copy Bibtex Large Language Models (LLMs) have achieved state-of-the-art performance on a broad range of Natural Language Processing (NLP) tasks, including document processing and code generation. Autoregressive Language Models (ARMs), which generate tokens sequentially conditioned on all previous tokens, have been the predominant paradigm for LLMs. While these models have achieved high accuracy across a range of downstream tasks, they exhibit low arithmetic intensity due to the inherent sequential dependency in next-token prediction. Recently, Diffusion Language Models (DLMs) have emerged as a promising alternative architecture. DLMs generate output tokens in parallel, mitigating the limitations of sequential decoding. However, the performance implications of DLMs relative to commonly deployed ARMs are not fully understood. In this work, we present a comprehensive study of the performance characteristics of ARMs and DLMs, combining theoretical analysis with empirical profiling to characterize the trade-offs between these approaches. We show that although DLMs can achieve higher arithmetic intensity than ARMs by leveraging parallelism across token positions, they fail to scale effectively with longer contexts. We then explore block-wise decoding for DLMs, which decouples arithmetic intensity from sequence length and enables better scaling to long contexts (similar to ARMs). We also examine batched inference and find that ARMs exhibit superior throughput as they benefit more from parallelism across sequences in the batch. Finally, we highlight opportunities for accelerating DLM inference, emphasizing that reducing the number of sampling steps is key for open-source DLMs to achieve lower latency relative to ARMs. † Seoul National University ‡ University of California, Berkeley § ICSI ¶ LBNL †† University of Texas at Austin * Advisory role Related readings and updates. Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential August 8, 2025 research area Methods and Algorithms , research area Speech and Natural Language Processing Autoregressive language models are constrained by their inherently sequential nature, generating one token at a time. This paradigm limits inference speed and parallelism, especially during later stages of generation when the direction and semantics of text are relatively certain. In this work, we propose a novel framework that leverages the inherent knowledge of vanilla autoregressive language models about future tokens, combining techniques to… Read more Scaling Diffusion Language Models via Adaptation from Autoregressive Models April 16, 2025 research area Speech and Natural Language Processing conference ICLR Diffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling, potentially addressing limitations of autoregressive (AR) models. However, current DLMs have been studied at a smaller scale compared to their AR counterparts and lack fair comparison on language modeling benchmarks. Additionally, training diffusion models from scratch at scale remains challenging. Given the prevalence of open-source AR… Read more Discover opportunities in Machine Learning. Our research in machine learning breaks new ground every day. Work with us