Arbitrage: Efficient Reasoning via Advantage-Aware Speculation

來源原標題

Arbitrage: Efficient Reasoning via Advantage-Aware Speculation

原始連結

https://machinelearning.apple.com/research/arbitrage-efficient-reasoning

摘要原文

research area Speech and Natural Language Processing content type paper published August 2026 Arbitrage: Efficient Reasoning via Advantage-Aware Speculation Authors Monishwaran Maheswaran†*, Rishabh Tiwari†*, Yuezhou Hu†*, Kerem Dilmen†, Coleman Hooper†, Haocheng Xi†, Nicholas Lee†, Mehrdad Farajtabar, Michael W. Mahoney†‡§, Kurt Keutzer†, Amir Gholami†‡ View publication Copy Bibtex Modern Large Language Models achieve impressive reasoning capabilities with long Chain of Thoughts, but they incur substantial computational cost during inference, and this motivates techniques to improve the performance-cost ratio. Among these techniques, Speculative Decoding accelerates inference by employing a fast but inaccurate draft model to auto-regressively propose tokens, which are then verified in parallel by a more capable target model. However, due to unnecessary rejections caused by token mismatches in semantically equivalent steps, traditional token-level Speculative Decoding struggles in reasoning tasks. Although recent works have shifted to step-level semantic verification, which improve efficiency by accepting or rejecting entire reasoning steps, existing step-level methods still regenerate many rejected steps with little improvement, wasting valuable target compute. To address this challenge, we propose ARBITRAGE, a novel step-level speculative generation framework that routes generation dynamically based on the relative advantage between draft and target models. Instead of applying a fixed acceptance threshold, ARBITRAGE uses a lightweight router trained to predict when the target model is likely to produce a meaningfully better step. This routing approximates an ideal ARBITRAGE ORACLE that always chooses the higher-quality step, achieving near-optimal efficiency–accuracy trade-offs. Across multiple mathematical reasoning benchmarks, ARBITRAGE consistently surpasses prior step-level SD baselines, reducing inference latency by up to ∼ 2× at matched accuracy. † UC Berkeley ‡ ICSI § LBNL * Equal contribution Related readings and updates. Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference December 11, 2025 research area Methods and Algorithms , research area Speech and Natural Language Processing Speculative decoding accelerates LLM inference by using a draft model to look ahead, but gains are capped by the cost of autoregressive draft generation: increasing draft size elevates acceptance rates but introduces additional latency overhead exacerbating the speed-accuracy tradeoff. Prior methods (Medusa, Hydra, EAGLE) partially reduce draft cost but either degrade acceptance or introduce overheads that limit scaling. We present Mirror… Read more Speculative Streaming: Fast LLM Inference Without Auxiliary Models November 19, 2024 research area Methods and Algorithms , research area Speech and Natural Language Processing conference EMNLP , Workshop at NeurIPS This paper was accepted at the Efficient Natural Language and Speech Processing (ENLSP) Workshop at NeurIPS 2024. Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks… Read more Discover opportunities in Machine Learning. Our research in machine learning breaks new ground every day. Work with us