REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff
https://machinelearning.apple.com/research/reversal-bench-rl-cliff📌 【Apple最新研究】打翻的杯子救不回來：reset-free RL的可逆性懸崖
TL;DR：Apple提出REVERSAL-BENCH，用可控可逆性參數證明reset-free強化學習在不可逆情境下會直接卡死。
把積木推下桌子、把顆粒狀物質灑出來，這些動作一旦發生就無法復原。但大多數標榜「不需要外部重置」的強化學習演算法，設計時卻悄悄假設了一個永遠能被復原的世界。Apple 的新研究，把這個隱藏假設攤在陽光下。
🤔 自主強化學習的核心目標，卡在一個現實假設上
自主強化學習（autonomous RL）的一大目標，是讓 policy 能夠持續訓練、不需要仰賴外部重置（reset）來把環境「還原」到初始狀態。然而論文指出，現有的 reset-free RL 典範，大多依賴底層環境具備「可逆性（reversibility）」這項特質，但在真實世界的操作任務中，這項特質常常並不成立：把物品推下桌子、灑出顆粒狀物質等事件，都是無法復原的。
🧩 用連續參數ρ控制可逆程度，再配上reset oracle驗證
為了系統性測量這個問題，研究團隊提出 REVERSAL-BENCH，透過一個連續參數 ρ∈[0, 1] 來控制環境的可逆程度，並提供一套 reset oracle：一個 ground-truth 驗證機制，用來檢驗某個狀態是否仍然可以被復原。這個基準涵蓋五種物理引擎中的八種操作情境，並用來評估一系列不同的 policy 架構，包括標準的 actor-critic 演算法、safe RL，以及專為 reset-free 場景設計的特化框架。
📊 隨著不可逆程度上升，reset-free agent出現明顯懸崖
實驗結果顯示一個清楚的「可逆性懸崖（reversibility cliff）」：隨著 ρ（不可逆程度）提高，reset-free agent 會持續被吸入不可復原的狀態，而採用傳統重置機制的 episodic agent，則能維持穩定的學習表現。這個失效模式不只出現在自主的 reset-free baseline，也同樣出現在受限強化學習（constrained RL）中。原因在於 reset-free agent 沒有外部重置可用，一旦轉移進入不可復原狀態，就會被永久困在原地，後續學習就此停滯；研究團隊也證實，這種「吸收現象（absorption）」在使用學到的操作 policy、於完整物理模擬環境中依然存在。透過與幾何結構完全相同、但屬性可逆的對照環境比較，團隊確認這種崩潰是由不可逆性本身直接造成，而非來自障礙物複雜度等其他因素。
💡 能預測得到危險，不代表躲得掉
研究團隊也評估了一種安全防護盾（safety shield），會嘗試在不可逆的失敗發生前主動介入。結果顯示，可復原性這件事本身是可以被準確預測的；但主動介入是否真的能成功挽回局面，主要仍取決於 agent 在物理上是否還來得及躲開那個陷阱。換句話說，「預知風險」與「成功避開風險」是兩回事，安全防護盾並非萬能解方。
🎯 實務啟示
對於打算把 reset-free RL 用在真實世界機器人操作任務的團隊，這篇論文提醒了一件容易被忽略的事：在投入訓練資源之前，應該先評估任務本身的可逆性程度，並在 agent 可能踏入不可逆狀態之前設計好介入或避讓機制，而不是假設「訓練得夠久」就能自動解決卡死問題。研究團隊也釋出了 REVERSAL-BENCH 基準套件、標記了可復原性的大型多模擬器資料集，以及 reset oracle 本身，可以作為檢驗自家 policy 是否會掉入這個懸崖的現成工具。
🔗 來源
- 標題：REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff
- 作者／機構：Riyaaz Shaik, Chandru Venkataraman（Apple Machine Learning Research）
- 連結：https://machinelearning.apple.com/research/reversal-bench-rl-cliff
#ReinforcementLearning #ResetFreeRL #Robotics #Apple #MachineLearning #AIResearch #Benchmark #AutonomousRL #SafeRL #RoboticManipulation
原始資料 Apple ML · 收集於 2026-09-17
摘要原文
research area Methods and Algorithms content type paper published September 2026 REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff Authors Riyaaz Shaik, Chandru Venkataraman View publication Copy Bibtex A central goal of autonomous reinforcement learning is continuous policy training without external resets. However, existing paradigms largely depend on underlying environmental reversibility, a property absent in real world manipulation, where events such as pushing objects off tables or spilling granular substances cannot be undone. We introduce REVERSAL-BENCH, a benchmark that controls reversibility via a continuous parameter ρ∈ [0, 1] and provides a reset oracle, a ground-truth verification mechanism to test state recoverability across eight manipulation settings in five physics engines. Evaluating a broad spectrum of policy architectures—including standard actor-critic algorithms, safe RL, and specialized reset-free frameworks reveals a sharp reversibility cliff: reset-free agents are consistently absorbed into irrecoverable states as ρ increases, whereas episodic agents maintain steady learning. We see this failure mode across autonomous reset-free baselines and constrained RL. Because reset-free agents lack external resets, any transition into an irrecoverable state results in permanent absorption, leaving the agent trapped where further learning halts. We show that this absorption phenomenon persists in full physics simulations under learned manipulation policies. By evaluating against geometrically identical reversible counterparts, we confirm that this breakdown is causally driven by irreversibility rather than obstacle complexity. We release the benchmark suite, a large multi-simulator dataset labeled with recoverability and a reset oracle. We also evaluate a safety shield that intervenes before irreversible failures occur, showing that while recoverability can be predicted accurately, active recovery primarily succeeds only when the agent can physically steer clear of the trap. Related readings and updates. AgentBuilder: Exploring Scaffolds for Prototyping User Experiences of Interface Agents January 9, 2026 research area Human-Computer Interaction Interface agents powered by generative AI models (referred to as “agents”) can automate actions based on user commands. An important aspect of developing agents is their user experience (i.e., agent experience). There is a growing need to provide scaffolds for a broader set of individuals beyond AI engineers to prototype agent experiences, since they can contribute valuable perspectives to designing agent experiences. In this work, we explore the… Read more Towards Learning Multi-Agent Negotiations via Self-Play January 28, 2019 research area Computer Vision Workshop at ICCV Making sophisticated, robust, and safe sequential decisions is at the heart of intelligent systems. This is especially critical for planning in complex multi-agent environments, where agents need to anticipate other agents’ intentions and possible future actions. Traditional methods formulate the problem as a Markov Decision Process, but the solutions often rely on various assumptions and become brittle when presented with corner cases. In… Read more Discover opportunities in Machine Learning. Our research in machine learning breaks new ground every day. Work with us
由 claude-code/sonnet 自動生成