Generate Trajectories, Reasoning Traces, and Auto-Labels with NVIDIA Alpamayo 2 Super

來源原標題

Generate Trajectories, Reasoning Traces, and Auto-Labels with NVIDIA Alpamayo 2 Super

作者

Elizabeth Goodman

原始標籤

Developer Tools & Techniques · Robotics · Simulation / Modeling / Design · Top Stories · Automotive / Transportation · Cosmos · DRIVE · Robot Navigation

原始連結

https://developer.nvidia.com/blog/generate-trajectories-reasoning-traces-and-auto-labels-with-nvidia-alpamayo-2-super/

摘要原文

Autonomous vehicle (AV) development often relies on separate models for trajectory generation, high-level intent prediction, scene understanding, and data labeling. This separation makes it hard to compare related outputs, investigate model behavior, and reuse the same representations across the development workflow. NVIDIA Alpamayo 2 Super is an open 34-billion-parameter reasoning vision-language-action (VLA) model designed to accelerate autonomous vehicle (AV) development. It combines the 32-billion-parameter NVIDIA Cosmos 3 Super Reasoner with a 2-billion-parameter diffusion-based Action Expert and is post-trained with reinforcement learning. The reasoner interprets multi-camera video, language context, and prior motion history, while the Action Expert converts the model’s resulting internal representation into a future ego-vehicle trajectory. Alpamayo 2 Super’s perception expands to 360-degree coverage across up to seven cameras and can return multiple complementary outputs: future trajectories, Chain-of-Causation (CoC) reasoning traces, high-level meta-actions, grounded answers to questions about the scene, and reasoning auto-labels. This multi-task design gives AV developers a common foundation across multiple stages of the development workflow. The same foundation model can be used as an offline policy teacher, an evaluation critic, a data engine, or a starting point for new task customization, instead of maintaining a separate model for each stage of the workflow. This post provides a hands-on introduction to four Alpamayo 2 Super-enabled workflows: Generate trajectories and CoC reasoning traces, evaluating the results in open-loop and closed-loop benchmarks. Predict meta-actions such as yield, change lanes, and stop alongside a trajectory. Ask natural-language questions about a multi-camera driving scene. Generate CoC auto-labels with 2D grounding on your own clips. The model weights are available on Hugging Face and the inference notebooks on GitHub . The model is released under OpenMDW-1.1, the Linux Foundation permissive license for open model distributions, which covers fine-tuning, derivative models, and commercial redistribution. Distilled models can be deployed commercially without further permission from NVIDIA, and model outputs carry no license conditions. Figure 1. Alpamayo 2 Super processes multimodal inputs through a 32B Cosmos 3 Super Reasoner and a 2B Action Expert Planning and reasoning Reasoning through new scenarios is a fundamental problem in autonomous driving. Navigating construction zones, partially occluded pedestrians, unusual right-of-way interactions, and objects entering the roadway requires more than matching a common trajectory pattern. A useful driving model must identify scene context that matters, connect it to the appropriate driving decision, and produce an action consistent with that decision. Trajectories and CoC traces: What and why Alpamayo 2 Super, like its predecessors, jointly produces output trajectories and CoC reasoning traces. The trajectory expresses what the ego vehicle could do next. The CoC reasoning trace provides insights into why a driving decision was made from observed scene context. Returning both outputs makes it easier to understand the model’s decision-making, curate difficult cases, compare a deployed policy with a larger teacher, and diagnose whether a failure originated in perception, reasoning, or action generation. CoC traces also feed into the NVIDIA Halos safety validation workflows by enabling introspection into the model’s understanding of the scene. The Alpamayo 2 Super repository’s inference notebook loads a surround-view clip, prepares ego-motion history, and samples a trajectory with its associated CoC trace. The core inference step and outputs are shown below. from alpamayo2_super import helper from alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset from alpamayo2_super.models.alpamayo2_super import Alpamayo2Super from alpamayo2_super.visualization import plot_inference_result data = load_physical_aiavdataset( "030c760c-ae38-49aa-9ad8-f5650a545d26", t0_us=2000000, ) model = Alpamayo2Super.from_pretrained("nvidia/Alpamayo2-Super", dtype=torch.bfloat16, device_map="cuda:0") model_inputs = helper.prepare_model_inputs(data, model.config, model.tokenizer) model_inputs = helper.to_device(model_inputs, "cuda") torch.cuda.manual_seed_all(42) with torch.autocast("cuda", dtype=torch.bfloat16): pred_xyz, pred_rot, logprob, extra = model.sample_trajectories_from_data( data=model_inputs, top_p=0.98, temperature=0.6, num_traj_samples=1, diffusion_kwargs={"inference_step": 10}, return_extra=True, ) fig, metadata = plot_inference_result( data=data, pred_xyz=pred_xyz, extra=extra, ) Figure 2. Alpamayo 2 Super planning in a challenging scene Evaluation methods To evaluate the model’s output reasoning and trajectory quality, we can use open-loop and closed-loop evaluation methods. Open-loop evaluation measures trajectory and reasoning quality on recorded scenes by comparing them to ground-truth labels. Alpamayo 2 Super achieves the following results: Trajectory prediction: Across 1,434 challenging samples from the Physical AI AV Dataset , it records a 6.4-second minADE_6 of 0.911 m. AV reasoning: It scores 0.433 on the Physical AI AV Reasoning Benchmark . LingoQA: Alpamayo 2 Super achieves 79.2 on the LingoQA benchmark , ranking first among 37 evaluated models. With 34 billion parameters, it leads Qwen2.5-VL (72B) by 17.0 points, Qwen3-VL (32B) by 7.0 points, Gemini 2.5 Pro by 15.1 points, and GPT-4o by 23.2 points. Lower minADE_6 values indicate better trajectory predictions; higher reasoning scores indicate better performance. The main challenge with open-loop metrics, however, is that they evaluate predictions against a fixed, prerecorded future and therefore don’t capture what happens after the model’s first action, which may affect the rest of the scene. For example, if the ego vehicle changes lanes, an open-loop replay may continue moving an adjacent vehicle along its recorded trajectory rather than accounting for how it would respond to the ego vehicle. Closed-loop simulation executes each predicted action within the scene and, when the simulator includes reactive behavior models, captures how surrounding agents may react. NVIDIA AlpaSim enables this by repeatedly rendering observations, querying the policy, and applying its actions so developers can measure these closed-loop effects over time. To run Alpamayo 2 Super on an AlpaSim evaluation suite, use the corresponding shell command: uv run alpasim_wizard deploy=local topology=2gpu driver=alpamayo2 wizard.log_dir=$PWD/tutorial eval.video.video_layouts=[REASONING_OVERLAY] with the following AlpaSim wizard configuration: # Should be used in defaults list, e.g. # - /driver: alpamayo2 # Type validation happens at driver runtime via OmegaConf.structured merge defaults: - alpamayo_configs # Camera and simulation configs for 4-cam 10Hz - _self_ # YAML values override schema defaults # Alpamayo 2 Super Driver Configuration for Alpasim # Logging level (uses wizard's global setting) log_level: ${wizard.log_level} # Model configuration model: model_type: alpamayo2 # Entry-point name in alpasim.models registry # HuggingFace model ID (requires cached download or hf authentication in the driver container) checkpoint_path: "nvidia/Alpamayo2-Super" # # Alternative local path to a pre-downloaded model # checkpoint_path: "/mnt/drivers/alpamayo2/Alpamayo2-Super" device: "cuda" # Enable classifier-free guidance navigation sampling (NOTE: this requires 2 GPUs with at least 70 GB VRAM). # Set to true only when sufficient GPU memory is available. use_classifier_free_guidance_nav: false # Server configuration host: "0.0.0.0" port: ???