NVIDIA AI Releases Molt: A PyTorch-Native Agentic Reinforcement Learning Framework
https://www.marktechpost.com/2026/08/01/nvidia-ai-releases-molt-a-pytorch-native-agentic-reinforcement-learning-framework/📌 【NVIDIA NeMo 新作】輕量級 PyTorch 原生框架 Molt：專為研究者設計的 Agentic RL 工具
TL;DR：NVIDIA 發布 Molt，以極簡程式碼量與高度整合性，解決 Agentic RL 研究中複雜的 pipeline 修改成本。
🧪 追求極致精簡：程式碼量僅 8.6K 行
在 Agentic Reinforcement Learning (RL) 研究中，開發者常需不斷修改演算法、估計器 (estimators) 或 Rollout 方案。在主流框架中，這些更動往往會連動到訓練器、分散式後端與 Rollout 膠水程式碼，增加研究者的負擔。
NVIDIA NeMo 團隊推出的 Molt 旨在降低這種成本。其設計目標是讓程式碼精簡到研究者能完全理解，且 AI 程式碼助手也能輕易閱讀與推理。
📊 與同類框架的規模對比
根據對程式碼導入圖 (import graph) 的追蹤，Molt 的 RL 核心程式碼量僅約 8.6K 行，遠比其他框架精簡：
| 框架 | RL 程式碼行數 (估計) |
|---|---|
| Molt | 8.6K |
| OpenRLHF | 7.2K |
| Slime | 25K |
| verl | 62K |
🧩 核心架構：解耦合的異步訓練流程
Molt 採用了高度整合且不需 Fork 原始碼的設計，透過整合現有工具來實現高效能：
- Ray：負責資源配置與非同步隊列。
- vLLM：負責 Rollout (採樣)。
- NVIDIA AutoModel (搭配 FSDP2)：負責訓練。
其執行時 (Runtime) 由三個部分組成：一個 Agent 池、一組位於請求路由器 (Request Router) 後方的 vLLM 引擎，以及單一的可訓練策略 Actor。
為了確保效能，Molt 採用了「部分 Rollout 暫停 (Partial Rollout Pauses)」機制：在 Actor 訓練時會暫停引擎，直接透過 NCCL 將 Actor 的分片 (shards) 廣播至各引擎，並恢復保留的請求，而非直接丟棄。
⚠️ 硬體門檻與適用場景
雖然 Molt 提供了 Apache 2.0 授權、Slurm 腳本與預建容器，但該研究定位於「研究基礎設施」而非生產級訓練服務。其提供的範例配置假設使用 2 個節點（共 16 顆 H100 GPU），其中 8 顆用於訓練，8 顆用於 Rollout。
這使得 Molt 主要適用於：
- 前沿實驗室與具備充足資金的 AI 新創公司（進行 Post-training）。
- 金融、醫療與機器人領域的企業 AI 研究團隊（針對私有環境訓練 Agent）。
- 擁有多節點 H100/H200 存取權限的學術實驗室。
💡 確保正確性的三大不變量 (Invariants)
為了處理複雜的 Agent 任務（如多輪工具使用、程式碼執行、視覺語言環境），Molt 依據以下原則設計：
- Token 身份一致性：定義軌跡的是採樣的 Token ID，而非重新 Tokenize 後的文本。
- 策略版本語義：可訓練的 Token 會保留其行為策略的對數機率 (log-probabilities)，非同步使用時會進行 Token 級別的修正。
- 前向一致性：Rollout 與 Actor 必須在模型語義上保持一致。針對 Mixture-of-Experts (MoE) 策略，Molt 採用了 Rollout Routing Replay 技術，讓訓練時的 Forward Pass 能重現 vLLM 採樣時選擇的專家 (experts) ID，避免數值差異導致的路由錯誤。
🎯 實務啟示
對於需要快速迭代 Agent 演算法的研究者而言，Molt 提供了一個「低摩擦」的環境。它不試圖重寫現有的優質工具（如 vLLM），而是透過精簡的介面將它們串聯，讓研究者的注意力能集中在演算法邏輯，而非維護複雜的訓練管線。
🔗 來源
- 標題：NVIDIA AI Releases Molt: A PyTorch-Native Agentic Reinforcement Learning Framework
- 作者／機構：Asif Razzaq @ MarkTechPost
- 連結：https://www.marktechpost.com/2026/08/01/nvidia-ai-releases-molt-a-pytorch-native-agentic-reinforcement-learning-framework/
#NVIDIA #NeMo #ReinforcementLearning #PyTorch #AgenticAI #MachineLearning #DeepLearning #vLLM #DistributedTraining #AIResearch
原始資料 MarkTechPost · 收集於 2026-08-02
摘要原文
Agentic reinforcement learning research is constant algorithm modification. New estimators, new pipeline stages, new rollout schemes. In mainstream frameworks each change threads through layers of trainer, distributed backend, and rollout glue. That cost lands on the researcher at every iteration. Molt , from NVIDIA’s NeMo team, targets that cost directly. Its a PyTorch-native agentic RL framework with an unusual design target. The codebase should be compact enough for a researcher to hold in their head, and for an AI coding assistant to read and reason about in its entirety. The stated footprint is roughly 8.6K lines of RL code, measured by tracing the import graph from each framework’s RL entry point. The same method counts about 62K lines for verl, 25K for slime, and 7.2K for OpenRLHF. Yes. Molt ships under Apache 2.0 with launch codes, Slurm scripts, and a prebuilt container. But the research paper positions it as research infrastructure, not a production training service, and hardware is the real gate. The shipped recipes assume 2 nodes of 8 H100 GPUs, split 8 for training and 8 for rollout. That puts Molt in reach of frontier and frontier-adjacent labs, well-funded AI startups doing post-training, enterprise AI research groups in finance, healthcare, and robotics that train agents against proprietary environments, and academic labs with multi-node H100/H200 access. Applications include multi-turn tool-use agents, code-execution agents, vision-language environments (the shipped geo3k recipe), LLM-as-judge reward loops, and on-policy distillation onto a smaller student. Molt composes Ray for placement and asynchronous queues, vLLM for rollout, and NVIDIA AutoModel with FSDP2 for training. None of the three is forked, so upstream improvements arrive as a container pin rather than a rebase. The runtime is an agent pool, a set of vLLM engines behind a request router, and a single trainable policy actor. A streaming pool keeps prompt groups in flight so engines never drain while the actor trains. Partial rollout pauses the engines, broadcasts actor shards over NCCL directly to each engine, and resumes retained requests instead of discarding them. An RL run names one Python module that exports an AgentRunner . Everything else is ordinary code, including the reward. Two forms are supported. With Env , the framework owns the LLM loop in a Gymnasium-aligned step() . With ChatAgent , the user owns the loop through a stock OpenAI or Anthropic SDK. Molt launches a loopback server that speaks both wire protocols, and every request decodes server-side into one token-exact accumulation. When a long-horizon agent compacts its context and rewrites the prefix, the server seals the current segment and opens a fresh one automatically. Three correctness invariants organize the design. Token identity: sampled token ids define the trajectory, not a retokenized transcript. Policy-version semantics: trainable tokens keep their behavior-policy log-probabilities, and asynchronous use is corrected per token behind a sequence-level gate. Forward consistency: rollout and actor must agree on model semantics. For mixture-of-experts policies, the last invariant matters most. The rollout and training routers select experts independently, and small numerical differences can flip top-k choices. Molt applies rollout routing replay , where vLLM returns its per-token expert ids and the training forward replays them. Check out the Pa p er and Repo here . Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter . Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post NVIDIA AI Releases Molt: A PyTorch-Native Agentic Reinforcement Learning Framework appeared first on MarkTechPost .
由 tencent/hy3:free 自動生成