Beyond VLAs: How World Action Models Reshape Robot Manipulation

來源原標題

Beyond VLAs: How World Action Models Reshape Robot Manipulation

作者

Michelle Horton

原始標籤

Agentic AI / Generative AI · Robotics · Simulation / Modeling / Design · Physical AI · Robot Manipulation · Thor

原始連結

https://developer.nvidia.com/blog/beyond-vlas-how-world-action-models-reshape-robot-manipulation/

摘要原文

A central challenge in robotics is building policies that generalize beyond the demonstrations they’re trained on. A policy that succeeds in a training scene often fails when object shapes, positions, or lighting change. Generalizing to these new conditions requires the policy to understand the tasks underlying physics, not just mimic the demonstrations. This ability comes from the backbone it’s built on. The standard way to build a language-conditioned robot policy is to add an action module to a pretrained vision-language model (VLM) , producing a vision-language-action (VLA) model. This approach has carried generalist manipulation a long way. But a VLM backbone learns to describe the world, not predict how it evolves . That missing dynamics model is exactly what a robot needs when a task depends on anticipating how a scene will change. A growing line of research replaces the language backbone with a video world model , producing world action model (WAM) . NVIDIA researcher Jim Fan explored this shift in his Robotics’ End Game talk—an idea later summarized as “VLAs are dead, long live World Action Models.” This post explores how post-training can turn WAMs into specialized robot policy, how WAMs compare to VLAs, and why the open NVIDIA Cosmos 3 world model provides a strong foundation for building WAMs. How post-trained policies are built today In the VLA paradigm, a pretrained VLM provides semantic understanding of scenes and language instructions, while post-training learns to map that understanding to robot actions. Modern generalist robot policies increasingly build on this approach. However, a VLM is optimized to produce text about images, not to model how a scene will evolve. It does not learn what happens to a mug when the gripper closes, how a towel folds, where an object lands when released. VLAs generalize well semantically but are less effective at physical generalization to unseen behaviors and environments, as their backbones model language and perception rather than world dynamics. What a WAM changes A WAM overcomes that dynamics-modeling limit by building the policy on a video world model. Because the backbone models how the world evolves, post-training does not have to teach dynamics from scratch, it specializes a model that already has a physics prior. The NVIDIA research paper World Action Models are Zero-shot Policies shows that jointly predicting video and action gives a policy properties a VLA cannot easily acquire: It learns from diverse data. A VLA learns to map instructions to trajectories, and often requires near-identical demonstrations of the same task. A WAM learns physics: how objects move when pushed, grasped, or dropped. Any interaction data teaches it something. A varied dataset that would be wasted on a VLA becomes a training signal, and data collection gets cheaper. It generalizes in the open world. Physics is more general than semantics. The way an object falls or slides doesn’t change with the object, so a model that has learned dynamics carries that knowledge into scenes and motions it was never trained on. It adapts to new robots with few demonstrations. A model that already understands physical interaction needs far less task-specific data to specialize to a new arm or gripper. For a team building a policy, these are the practical wins: less data to reach a given capability, better behavior outside the training distribution, and a shorter path to a new embodiment. They are properties of the pretraining backbone , so they show up in every policy post-trained from it. Cosmos 3 is a strong WAM foundation Cosmos 3 is an omni-model world foundation model built on a Mixture-of-Transformers (MoT) architecture. Multimodal input flows through an autoregressive transformer for reasoning producing discrete tokens such as text. This guides a diffusion transformer for continuous modalities, including image, video, audio, and action. Text is generated by next-token decoding; everything else, including actions, is synthesized through iterative denoising. A single model spans these modalities while keeping the generation mechanism best suited to each. Cosmos 3 comes in three sizes: 4B NVIDIA Cosmos Edge, 16B NVIDIA Cosmos Nano, and 64B NVIDIA Cosmos 3 Super. What makes Cosmos 3 a strong foundation for post-training is the breadth of its physical-world data. The dataset includes roughly 767M images,348M videos of real-world dynamics, 8M action samples spanning robot manipulation, autonomous driving, camera motion, and egocentric motion. Figure 1. A VLA generates robot actions from semantic reasoning and robot state, while a WAM jointly predicts actions and future world states From world model to robot policy: Cosmos 3 Policy DROID models Cosmos 3 is the starting point for specialization. Cosmos3-Nano-Policy-DROID is a 16B-parameter policy post-trained from Cosmos 3 Nano for the DROID platform, which is a Franka Panda arm with a Robotiq gripper.A 4B version, Cosmos3-Edge-Policy-DROID , is post-trained the same way, which can be used for on-device deployment. Given a language instruction and multi-camera observations, it generates robot action trajectories. Three properties follow from its omni foundation: It imagines while it acts. When the model outputs actions, it can also output a video: what the robot’s cameras will see if those actions are executed. The action and the predicted outcome come from the same model, at the same time. It keeps the full omni architecture. Post-training removes nothing. The policy checkpoint can still reason and generate video, not just output joint positions. The prior is measurable. The Cosmos 3 technical report compares two DROID policies trained with the same recipe, data, and compute. One started from the base checkpoint, while the other started from an omni checkpoint trained on multi-domain action data. The omni checkpoint raised RoboLab success from 28.1% to 36.8%. This is clear evidence that the improvement comes from the architecture, not just from scale. Figure 2. A robot observes a banana on the desk (left); the model’s predicted action plan is shown as text Deployment considerations A WAM carries the full generative world model, not just an action head. It’s larger than compact VLAs, but Cosmos 3 map to different deployment tiers rather than forcing one trade-off: Workstation serving (Nano, 16B ). Cosmos3-Nano-Policy runs beside the robot rather than on board. Real-world DROID deployment serves it on a single NVIDIA RTX PRO 6000, with the robot streaming observations over the network and receiving action chunks back. On-device (Edge, 4B). Cosmos 3 Edge runs the same policy workload directly on embedded hardware. It operates at robot-control resolution (640×360 observations) and generates 32 actions per inference on NVIDIA Jetson Thor while achieving real-time control at 15 Hz. It’s supported across NVIDIA edge computers including RTX PRO GPUs, DGX, GeForce RTX GPUs, and Jetson, including the new Jetson T2000 and T3000 modules. Why build robot policies with Cosmos 3? WAMs represent a shift from learning to act to learning how the world evolves. Cosmos 3 makes this approach practical: Open foundation, SOTA starting point. The base model, datasets, post-training recipe, trained weights, evaluation tools, and serving stack are all released under a license that permits commercial use. Faster adaptation. Strong physical priors cut the task-specific data a new policy needs. Convert your data to the LeRobotDataset format the robot-learning ecosystem already records into, run the published recipe. One foundation, many robots. Each new embodiment, such as Franka, dual-arm setups, UR, WidowX, still requires its own post-training run. But all of them start from the same pretrained foundation rather than pretraining a world model from scratch, reducing the demonstrations needed for each.