How to Use AI Agents to Prepare 3D Scenes for Simulation

來源原標題

How to Use AI Agents to Prepare 3D Scenes for Simulation

作者

Tanya Lenz

原始標籤

Agentic AI / Generative AI · Simulation / Modeling / Design · NemoClaw · OpenUSD · Physical AI

原始連結

https://developer.nvidia.com/blog/how-to-use-ai-agents-to-prepare-3d-scenes-for-simulation/

摘要原文

Agentic AI workflows can be used to prepare and validate digital twins for physical AI systems. Agents can inspect 3D scenes, author simulation-relevant data in OpenUSD , add physics properties, render preflight views, and validate the result against simulation-ready (SimReady) requirements. This workflow follows that process from a scene in Blender to a simulation-ready OpenUSD handoff for NVIDIA Isaac Sim or NVIDIA Isaac Lab . In practice, however, if you’re building agents for robotics, the workflow can often get stuck. It’s tempting to blame the hard part on the policy, the model, or the training loop, but the bottleneck often happens earlier: the robot does not have a simulation-ready world to train in. The extra work required to get a 3D scene into that state is laborious, time consuming, and frequently out of scope for the robotics simulation engineer. This post walks through an agent workflow for preparing a Blender scene for robotics simulation using NVIDIA Omniverse Libraries . Codex, powered by OpenAI GPT-6 Astra, coordinates the overall task, interprets results, and guides iteration. Specialized subagents built with the Hermes agent harness and deployed through NVIDIA NemoClaw use Omniverse Libraries to inspect the scene, author simulation metadata, configure physics, and render visual preflight views. Together, these components connect reasoning, tool execution, and validation into a repeatable process for delivering a simulation-ready OpenUSD world. Why is preparing a 3D scene for simulation challenging? The scene exists—the assets are there, created by a 3D artist in Blender—but is it usable for simulation? Has all of the following prep work been done? Are objects labeled? Are collision meshes correct? Are materials meaningful for simulation? Are sensors placed and configured? Does the scene export cleanly to USD? Can the robot perceive the target objects? Will the scene pass validation before you burn time debugging it in Isaac Sim or Isaac Lab? This prep work is tedious, repetitive, and easy to get wrong. It is also exactly the kind of work agentic systems should help with, if they have the right tools. The point of using NVIDIA Omniverse Libraries in an agent workflow is to integrate the tools agents need to build SimReady worlds. How can agents help prepare a 3D scene for simulation? A general-purpose agent such as Codex by ChatGPT or Claude Cowork by Anthropic can look at a Blender scene and recognize that it needs to be made simulation-ready. Recognition is useful, but it’s not enough. To help a robotics developer, the agent must be able to act on the scene: Inspect the Blender scene Identify missing simulation metadata Add semantic labels Configure sensors Author collision shapes and physics properties Render visual preflight views Run SimReady validation Automatically repair safe issues Escalate ambiguous decisions to a human A broad request to make a 3D scene ready for simulation becomes a multi-agent engineering workflow. Codex or Claude serves as the main agent, coordinating the overall task of preparing the scene for simulation. NVIDIA NemoClaw provides a reference architecture for building the specialized subagents that perform each job. These subagents can use open source agent harnesses such as Hermes, OpenClaw, or LangChain, configured with different NVIDIA Nemotron models for vision, reasoning, and tool use. In a configuration using Astra and Hermes, Codex uses Astra to translate the developer’s objective into tasks, identify dependencies, and review results from specialized Hermes subagents deployed through NemoClaw. For example, making an object grabbable requires coordinated updates to its semantic label, rigid-body configuration, and collision geometry. Astra helps connect those requirements across subagents and determine which checks are needed before the workflow proceeds. NVIDIA Omniverse Libraries provide the tools the subagents call to act on the scene. OpenUSD operations establish the shared scene structure, ovphysx authors and checks physics properties, ovrtx renders visual preflight views, and SimReady validation evaluates the resulting assets against a target simulation profile. Each subagent owns a specific job and its acceptance criteria. Safe, mechanical issues can be fixed automatically. Decisions that depend on developer intent, such as an uncertain semantic label or physical behavior, are escalated to a human with the relevant context and a proposed next step. The pattern is: Codex or Claude coordinates NemoClaw agents reason Omniverse Libraries act Together, these layers turn the prompt, “Make this scene simulation-ready” into a tool-driven workflow with specialized jobs, a persistent scene state, validation gates, and human review where judgment matters. Figure 1. Reference workflow demonstrating how to use Omniverse Libraries to activate a Blender scene with agents How to use agents to prepare a 3D scene for simulation Begin by specifying the main objective for the orchestration agent, including the input, desired output, destination, and validation criteria. This gives Codex or Claude enough structure to coordinate the overall task, route work across specialized NemoClaw subagents, and decide when the job is actually complete. Input: Blender scene Goal: Prepare it for robotics simulation Output: USD-based simulation-ready world Destination: Isaac Sim or Isaac Lab Validation: Visual preflight + SimReady validation By specifying the main objective, the task changes from simply “make this scene better” to a coordinated agent workflow. Codex or Claude manages the overall request, NemoClaw subagents reason through specialized jobs, and Omniverse Libraries provide the tools that modify, render, validate, and prepare the world. After setting your goal, follow the steps below. Step 1: Inspect the scene through Blender MCP The first subagent connects to Blender through a Model Context Protocol (MCP) server and uses it to inventory the scene. MCP gives the agent a controlled tool interface into Blender: instead of guessing from screenshots or relying on manual exports, the agent can call tools to inspect objects, collections, transforms, materials, cameras, lights, and scene metadata. That scene inventory becomes the shared context the rest of the subagents use. This subagent should answer: What objects exist? What collections and hierarchy exist? What materials are assigned? What cameras and lights are present? What looks like a robot target, obstacle, floor, shelf, or bin? What simulation data is missing? The output should be structured as follows: { "objects": 142, "materials": 37, "missing": [ "semantic_labels", "collision_meshes", "camera_sensors", "physics_materials" ] } This provides a shared starting point for the other subagents. The Hermes inspection subagent returns its structured findings to Codex. Astra uses this inventory and the developer’s robotics objective to identify missing information and plan the next tasks. For example, identifying the robot’s target objects helps determine which assets need movable-body properties and which sensor viewpoints require review. Codex then delegates those tasks to the relevant Hermes subagents deployed through NemoClaw, with clear acceptance criteria and unresolved assumptions flagged for developer input. The workflow in this post uses The Junk Shop by Alex Trevino (original concept by Anais Maamar) for the demo (Figure 2). Codex coordinates NemoClaw to orchestrate which subagent is needed for the requested task. In this instance, NemoClaw is working through Blender MCP to run The Junk Shop scene and inventory the objects, materials, scene structure, and more. Figure 2. The Blender add-on bridges scene data to Omniverse Libraries through gRPC, enabling rendering and physics capabilities to feed directly back into the Blender Viewport Step 2: Move toward USD as the contract Blender is the authoring environment.