RLTL;DR: Self-Improvement by Internalizing Self-Generated Feedback
https://machinelearning.apple.com/research/rltl-dr-self-improvement📌 Apple新作RLTL;DR：自產回饋如何打破RLVR瓶頸
TL;DR：Apple提出RLTL;DR，讓模型內化自己寫的失敗筆記，在Pass@128=0的難題上從0%衝到14-31%。
強化學習與可驗證獎勵（RLVR）的標準做法，是讓 agent 對一項任務多次嘗試，再把訓練訊號集中在成功的那幾次上。問題是，當任務難到 agent 幾乎不可能成功、又沒有教師模型或範例解答可以蒸餾時，這套機制根本無從啟動。Apple 的研究團隊在 NeurIPS 發表的 RLTL;DR，試圖解決這個自我提升（self-improvement）的死結。
🤔 當任務難到連 128 次嘗試都全軍覆沒
論文鎖定的場景很極端：在困難的 tool-calling 與程式設計資料集上，先篩選出 Pass@128 = 0 的題目，也就是模型嘗試 128 次都無法答對的任務。在這種情境下，標準 GRPO 訓練 Qwen 3.5 9B Thinking 策略模型，Pass@1 一直停滯在 0% 到 1%，完全學不動——因為沒有任何一次成功的 rollout 可以作為正向訊號。
🧩 讓模型自己寫一句 TL;DR，再把它寫進權重裡
RLTL;DR 的做法是：每次嘗試失敗後，把驗證器（verifier）的輸出展示給策略模型看，讓它自己寫一條 TL;DR 式的洞見（insight）作為回饋；下一次 rollout 會以所有先前累積的洞見為條件（conditioned），依序取樣 rollout 直到找到解答為止。更關鍵的一步，是對這些「上下文中的洞見」做反向傳播（backpropagation），讓模型把「任務 → 洞見」這個映射直接內化進參數裡，而不只是停留在 prompt 層面的提示。
為了進一步拆解這個效果從何而來，團隊還做了一個簡化版 SFTL;DR：完全不展示也不對任何 rollout 做反向傳播，只用（任務, 洞見）這種配對資料做監督微調。
📊 內化之後，不用提示也能維持 12-13% Pass@1
結果顯示，RLTL;DR 突破了標準 GRPO 卡住的學習障礙：在訓練時上下文中帶著洞見，Pass@1 達到 14% 到 31%；更重要的是，到了評估階段，即使上下文中完全不放洞見，Pass@1 仍能維持在 12% 到 13%——證明模型確實把「任務 → 洞見」的映射學進了自己的權重，而不是單純依賴提示工程。
團隊進一步確認，關鍵就在這個內化過程本身：簡化版的 SFTL;DR 只用 4,000 筆（任務, 洞見）配對資料訓練，完全不碰任何 rollout，就幾乎追平了 RLTL;DR 以及在完整 rollout 上做傳統 SFT 的效果。
💡 一種更精簡的訓練範式
這個結果指出一種值得延伸的訓練思路：與其蒐集大量完整 rollout 軌跡去蒸餾，不如蒐集「這類任務該記住這種重點」式的精簡洞見配對，就能達到接近的效果。對計算成本敏感的自我提升訓練場景而言，這代表一條更經濟的路徑。
🎯 實務啟示
如果你正在為難度極高、幾乎沒有成功範例可蒸餾的任務設計 RL 訓練流程，這篇論文提示了一個可嘗試的方向：先讓模型對自己的失敗寫洞見、把洞見放進下一輪的上下文，再對這些洞見做反向傳播以求內化；若計算資源有限，甚至可以跳過完整 rollout，只蒐集（任務, 洞見）配對做監督微調，用更低成本逼近類似效果。
🔗 來源
- 標題：RLTL;DR: Self-Improvement by Internalizing Self-Generated Feedback
- 作者／機構：Michael Kirchhof, Eleonora Gualdoni, Andrew Szot, Khashayar Gatmiry, Aryo Lotfi, Abbas Kazerouni, Omar Attia, Sanjoy Chowdhury, Alexander Toshev／Apple
- 連結：https://machinelearning.apple.com/research/rltl-dr-self-improvement
#ReinforcementLearning #RLVR #SelfImprovement #Apple #NeurIPS #LLM #GRPO #SelfSupervised #AIResearch #MachineLearning
原始資料 Apple ML · 收集於 2026-10-01
摘要原文
research area Methods and Algorithms conference NeurIPS content type paper published October 2026 RLTL;DR: Self-Improvement by Internalizing Self-Generated Feedback Authors Michael Kirchhof, Eleonora Gualdoni, Andrew Szot, Khashayar Gatmiry, Aryo Lotfi, Abbas Kazerouni, Omar Attia, Sanjoy Chowdhury, Alexander Toshev View publication Copy Bibtex The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTL;DR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task → insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128 = 0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTL;DR breaks through this learning barrier, achieving a Pass@1 of 14–31% with insights in context during training and, crucially, 12–13% when no insight is in context at eval time. We identify that the key is the task → insight internalization. To study this further, we reduce our approach to SFTL;DR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form “on this sort of task, keep this sort of thing in mind”, which we hope to inspire future research on. Related readings and updates. DR-MPC: Deep Residual Model Predictive Control for Real-World Social Navigation March 6, 2025 research area Computer Vision , research area Methods and Algorithms conference IEEE Robotics and Automation Letters How can a robot safely navigate around people with complex motion patterns? Deep Reinforcement Learning (DRL) in simulation holds some promise, but much prior work relies on simulators that fail to capture the nuances of real human motion. Thus, we propose Deep Residual Model Predictive Control (DR-MPC) to enable robots to quickly and safely perform DRL from real-world crowd navigation data. By blending MPC with model-free DRL, DR-MPC overcomes… Read more Apple Workshop on Machine Learning for Health 2023 November 8, 2023 research area Health Earlier this year, Apple hosted the Workshop on Machine Learning for Health. This two-day hybrid event brought together Apple and the academic research community and clinicians to discuss state-of-the-art machine learning (ML) research in health. Read more Discover opportunities in Machine Learning. Our research in machine learning breaks new ground every day. Work with us
由 claude-code/sonnet 自動生成