Alibaba Qwen Introduces QwenGyre: Elastic Reinforcement Learning RL for XLong-Horizon Agents
https://www.marktechpost.com/2026/10/12/alibaba-qwen-introduces-qwengyre-elastic-reinforcement-learning-rl-for-xlong-horizon-agents/📌 Qwen推出QwenGyre：讓訓練agent的GPU不再閒置
TL;DR：阿里Qwen團隊用彈性排程與前綴共享軌跡樹,讓小時級、百萬token的agent訓練速度翻倍。
訓練一個能跑好幾個小時、處理近百萬token的coding agent,聽起來就是GPU資源調度的夢魘。阿里Qwen團隊推出的QwenGyre,打的就是這個主意。
🤔 什麼是「XLong-horizon」任務
研究團隊把「XLong-horizon」定義為橫跨數小時、涉及數百次模型與工具互動的agent任務,例子包括依規格生成整個程式碼庫(如NL2Repo-Bench)。這類任務的rollout(推演)時間長、分支多,傳統的同步或固定資源分配訓練方式,很難把GPU用好用滿。QwenGyre是一套線上強化學習系統,透過不修改、黑箱式的harness來訓練coding agent,目標正是這類長達數小時、接近百萬token的rollout。
🧩 兩個核心設計:GPU彈性搬移 + 把分支歷史變成有界訓練樣本
QwenGyre把GPU分成共享同一套訓練佈局的「cell」,每個cell可以選擇服務rollout,也可以作為完整的訓練副本。其中有一個core cell持有權威的模型權重與optimizer狀態,其他satellite cell會在之後加入訓練。排程器會追蹤一個「rollout waterlevel」指標(排隊中與正在執行的任務數),當剩餘cell仍能撐住所有正在跑的工作時,其他cell就會切換去做訓練。
為了讓底層那個不能修改的harness持續運作,模型呼叫會經由一個proxy重新導向:這個proxy用token-in、token-out(TITO)的方式,記錄精確的token ID與行為log機率,並把這些呼叫存成「前綴共享的軌跡樹(prefix-shared trajectory trees)」。團隊發現把分支上限設為5,訓練分數就能追平不設上限的版本,同時把前向反向傳播時間縮短25.2%。訓練本身採用經過調整的GSPO,每組16個rollout。
📊 48步跑了75小時,評估通過率從52.48%升到58.54%
所有對照實驗都用Claude Code 2.1.220作為harness,並維持相同的GPU預算與一致的staleness設定。在Qwen自家旗艦Qwen3.8 2.4T模型上,QwenGyre完成48個訓練步驟耗時75.42小時;相較之下,Async方式要134.55小時,Colocate方式要91.47小時。訓練後的評估通過率也從52.48%提升到58.54%。
⚠️ 數字背後的保留
文章特別提醒,這些加速數字是各自專案在自身設定下量出來的,並不能直接跨專案比較——換句話說,QwenGyre相對Async、Colocate的優勢,是在同一套實驗條件下的內部對照,不代表放到別的系統上也會有同樣倍數的提升。
🎯 實務啟示
對於要訓練長流程、重度呼叫工具的coding agent的團隊,QwenGyre示範了兩個可借鏡的思路:一是讓GPU在rollout與訓練之間彈性流動,而不是靜態分配;二是用前綴共享的軌跡樹管理分支歷史,把本來難以利用的發散rollout轉成有界、可訓練的樣本。這對任何想壓榨長horizon RL訓練效率的工程團隊,都是值得研究的系統設計範本。
🔗 來源
- 標題：Alibaba Qwen Introduces QwenGyre: Elastic Reinforcement Learning RL for XLong-Horizon Agents
- 作者／機構：Asif Razzaq，MarkTechPost
- 連結：https://www.marktechpost.com/2026/10/12/alibaba-qwen-introduces-qwengyre-elastic-reinforcement-learning-rl-for-xlong-horizon-agents/
#QwenGyre #Qwen #ReinforcementLearning #Alibaba #CodingAgent #LLMTraining #GSPO #LongHorizonRL #GPUScheduling #OnlineRL
原始資料 MarkTechPost · 收集於 2026-10-12
摘要原文
Alibaba’s Qwen team has introduced QwenGyre , an elastic reinforcement learning RL framework for extreme-long-horizon agent training. QwenGyre is an online reinforcement learning system from Alibaba that trains coding agents through unmodified, black-box harnesses. It targets rollouts that run for hours and process close to 1M tokens. The QwenGyre elastic RL framework was used on Qwen’s flagship Qwen 3.8 2.4T model. It rests on 2 core ideas. It moves idle GPUs between rollout and training. It also turns branching agent histories into bounded training samples. The research team defines ‘XLong-horizon’ tasks as agent runs spanning hours and hundreds of model and tool interactions. Examples include generating full repositories from specs, as in NL2Repo-Bench . Two problems appear at this scale: QwenGyre splits GPUs into cells that share a training layout. Each cell can serve rollouts or host a full training replica. A core cell owns the authoritative weights and optimizer state. Satellite cells join training later. The scheduler tracks a rollout waterlevel: queued and running executions. As executions finish, cells switch to training once the remaining cells can still serve all live work. Model calls are rerouted through a proxy, so the harness keeps running. A proxy records exact token IDs and behavior log-probabilities using token-in, token-out (TITO) . Calls are stored as prefix-shared trajectory trees . A cap of 5 matched uncapped training scores while cutting forward-backward time by 25.2%. Training uses an adapted GSPO with 16 rollouts per group. All runs used Claude Code 2.1.220 as the harness, with equal GPU budgets and matched staleness. On the 2.4T model, QwenGyre finished 48 steps in 75.42 hours. Async took 134.55 hours and Colocate took 91.47 hours. Evaluation pass rate rose from 52.48% to 58.54%. Speedups come from each project’s own setup and are not directly comparable. Check out the full QwenGyre paper on arXiv and Qwen’s announcement post . All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter . Wait! are you on telegram? now you can join us on telegram as well. The post Alibaba Qwen Introduces QwenGyre: Elastic Reinforcement Learning RL for XLong-Horizon Agents appeared first on MarkTechPost .
由 claude-code/sonnet 自動生成