Google Research Introduces Retrieve-for-Train (R4T): An RL-Compiled Diffusion Retriever for 12× to 20× Faster Query Fan-Out
https://www.marktechpost.com/2026/09/16/google-research-introduces-retrieve-for-train-r4t-an-rl-compiled-diffusion-retriever-for-12x-to-20x-faster-query-fan-out/📌 【Google Research】RL 訓練一次，Diffusion Retriever 讓查詢擴展快 12 到 20 倍
TL;DR：R4T 用 RL 離線學會多樣化 query fan-out，蒸餾進 diffusion 模型後單次前向就能生成全部檢索方向。
搜尋一個「露營裝備」，你期待看到帳篷、睡袋、爐具、頭燈,而不是十頂長得幾乎一樣的帳篷。這正是 query fan-out（把一個籠統查詢拆成多個子查詢）要解決的問題,但 Google Research 發現,讓 LLM 在推論時即時做這件事,會撞上兩個結構性麻煩。
🤔 paraphrastic collapse 與延遲,兩個推論時的陷阱
研究團隊觀察到,zero-shot 的 Qwen3-4B 面對「Bohemian festival style」時,寫出的子查詢是「bohemian festival fashion」和「festival bohemian clothes」,這類近義詞彼此高度相似,檢索回來的結果自然也高度同質(paraphrastic collapse)。另一個問題是延遲:自迴歸生成加上多次檢索呼叫本身就慢,而 Best-of-N 取樣雖能提升品質,卻等比例放大推論成本。
🧩 RL 練一次,蒸餾進 diffusion 模型跑一次
R4T(Retrieve-for-Train)的做法是:用強化學習離線訓練一次,學出「好的 query fan-out」該長什麼樣子,再把這個行為蒸餾進一個小型 diffusion 模型,讓它一次前向就產生全部檢索方向,不必再迭代生成。
針對開放式抽象檢索(OAR),獎勵函數由三個加權項組成;針對弱監督組合式檢索(WSCR),獎勵則是 fan-out 檢索到參考集合中項目的比例。消融實驗說明了三項 OAR 獎勵為何都不可或缺:只用 groundedness 時,Gemma3-4B 收斂成「line ending line ending line ending」這種無意義字串;加入 alignment 反而讓 collapse 更快,策略開始不斷複述查詢本身的同義詞;只有同時加入 diversity,兩種走捷徑的行為才被堵住。
訓練採用 GRPO 搭配 soft PPO 正則化(加入前向與反向 KL 懲罰),關鍵設定為 group size 8、learning rate 1×10⁻⁷、global batch size 512。實驗分別在 Polyvore 時尚穿搭資料集(搭配 CLIP-based matryoshka encoder,128 維)以及一組專有的專家歌單音樂資料集(MuLan embeddings)上進行。每種 fan-out 方法都產生 k=10 個子查詢,Best-of-N 則設 N=5,OAR 品質由 LLM 裁判以 5 分制 Likert 量表評分。
📊 多樣性提升,速度快一個數量級
在 Polyvore 上,Gemma3-4B 搭配 R4T-FOLM 平均分數為 49.1,優於 Best-of-N 的 40.9 與 zero-shot 的 38.5;多樣性指標從 zero-shot 的 56.0 提升到 76.8,R4T-Diffusion 仍保留了大部分,達 74.3(diffusion 版不產生文字子查詢,因此 groundedness 未被回報)。在音樂資料集上,R4T-FOLM 平均 58.1,優於 Best-of-N 的 49.2。
WSCR 在 Polyvore 上則呈現覆蓋率與多樣性的取捨:R4T-FOLM(Qwen)拿下 Recall@5K 20.9、Hit@5K 64.6,超過 Gemini-2.5-Flash 的 15.7 與 52.1,但其 Vendi Score 掉到 27.5,作者將此歸因於高強度 RL 最佳化下輸出熵下降;R4T-Diffusion(Qwen)則保留較高的 Vendi Score 34.7,Recall@5K 為 16.5。
速度差異最直觀:batch size 8 時,自迴歸 fan-out 約需 1.46 秒,diffusion 只要 0.07 秒;batch size 1024 時,自迴歸逼近 50 秒,diffusion 僅 4.21 秒。作者總結為一致的 12 倍到 20 倍加速。
🎯 實務啟示
如果你的檢索或推薦系統需要輸出「一組」而非「一個」結果,而目前用 LLM 即時生成子查詢,R4T 的思路值得參考:把昂貴的探索行為(RL)放到離線訓練階段,線上服務用小而快的模型執行,是用推論成本換訓練成本的典型範例,也是 diffusion 模型在非影像生成任務上的一個有趣應用。
🔗 來源
- 標題:Google Research Introduces Retrieve-for-Train (R4T): An RL-Compiled Diffusion Retriever for 12× to 20× Faster Query Fan-Out
- 作者/機構:Asif Razzaq, MarkTechPost
- 連結:https://www.marktechpost.com/2026/09/16/google-research-introduces-retrieve-for-train-r4t-an-rl-compiled-diffusion-retriever-for-12x-to-20x-faster-query-fan-out/
#GoogleResearch #InformationRetrieval #DiffusionModels #ReinforcementLearning #QueryFanOut #GRPO #RecommenderSystems #MachineLearning #RAG #LLM
原始資料 MarkTechPost · 收集於 2026-09-18
摘要原文
Search and recommendation systems increasingly need to return a set of results, not one best match. A query like ‘camping gear’ should return a tent, a sleeping bag, a stove, and a headlamp. It should not return 10 near-identical tents. Google Research has introduced Retrieve-for-Train (R4T) for this problem. The framework uses reinforcement learning (RL) once, offline, to learn good query fan-out. It then distills that behavior into a small diffusion model that produces all retrieval directions in 1 pass. Query fan-out splits one broad prompt into several sub-queries. The research team identifies 2 problems when a generic LLM does this at inference time. The first is paraphrastic collapse. For ‘Bohemian festival style,’ zero-shot Qwen3-4B wrote ‘bohemian festival fashion’ and ‘festival bohemian clothes.’ These near-synonyms retrieve a homogeneous slate. The second is latency. Autoregressive generation plus repeated retrieval calls is slow. Best-of-N sampling improves quality but multiplies inference cost. For open-ended abstract retrieval (OAR), the reward combines 3 weighted terms: For weakly supervised compositional retrieval (WSCR), the reward is the fraction of reference-set items the fan-out retrieves. The ablation explains why all 3 OAR terms are important. With groundedness alone, Gemma3-4B converged to strings like ‘line ending line ending line ending.’ Adding alignment made collapse even faster, as the policy repeated paraphrases of the query. Adding diversity closed both shortcuts. Training uses GRPO with soft PPO regularization, which adds forward and reverse KL penalties. Key settings include group size 8, learning rate 1×10⁻⁷, and global batch size 512. Experiments used the Polyvore fashion outfit dataset with a CLIP-based matryoshka encoder at 128 dimensions. They also used a proprietary expert-playlist music dataset with MuLan embeddings. Every fan-out method produced k = 10 sub-queries, and Best-of-N used N = 5. OAR quality was scored by an LLM judge on 5-point Likert scales. On Polyvore, Gemma3-4B R4T-FOLM averaged 49.1, versus 40.9 for Best-of-N and 38.5 zero-shot. Diversity rose from 56.0 zero-shot to 76.8. R4T-Diffusion retained most of it at 74.3. On Music, Gemma3-4B R4T-FOLM averaged 58.1 versus 49.2 for Best-of-N. Groundedness is not reported for R4T-Diffusion, since it produces no text sub-queries. WSCR results on Polyvore show a coverage and diversity trade-off. R4T-FOLM (Qwen) reached 20.9 Recall@5K and 64.6 Hit@5K, versus 15.7 and 52.1 for Gemini-2.5-Flash. Its Vendi Score, however, dropped to 27.5. The authors link this to reduced output entropy under strong RL optimization. R4T-Diffusion (Qwen) kept a higher Vendi Score of 34.7 with 16.5 Recall@5K. At batch size 8, autoregressive fan-out took about 1.46 seconds. The diffusion model took 0.07 seconds. At batch size 1024, autoregressive fan-out reached nearly 50 seconds, versus 4.21 seconds for diffusion. The authors report a consistent 12× to 20× speedup. Check out the Paper and Technical details . All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter . Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Google Research Introduces Retrieve-for-Train (R4T): An RL-Compiled Diffusion Retriever for 12× to 20× Faster Query Fan-Out appeared first on MarkTechPost .
由 claude-code/sonnet 自動生成