Optimizing Agent Reliability and Tool Calling Efficiency for Chip Design Using NVIDIA NeMo Gym

來源原標題

Optimizing Agent Reliability and Tool Calling Efficiency for Chip Design Using NVIDIA NeMo Gym

作者

Tanya Lenz

原始標籤

Agentic AI / Generative AI · Simulation / Modeling / Design · AI Agent · Build AI Agents

原始連結

https://developer.nvidia.com/blog/optimizing-agent-reliability-and-tool-calling-efficiency-for-chip-design-using-nvidia-nemo-gym/

摘要原文

AI agents offer remarkable productivity gains in complex chip design workflows, but their tendency to generate plausible yet incorrect artifacts makes reliability a major barrier to production adoption. In multistage engineering workflows, an undetected error early in setup can silently propagate through downstream stages and final artifact generation. Evaluating and refining these agents against verified final outputs, rather than isolated intermediate steps, is essential for building trustworthy automation. NVIDIA NeMo Gym , an open library for evaluating and improving models and agents using environments, addresses this challenge. The NeMo Gym agentic evaluation environment is designed to diagnose failure modes and track error propagation across complex, multistage workflows. By establishing a domain-specific verifier to inspect final outputs against verified reference artifacts, NeMo Gym enables engineers to trace complete rollout trajectories and identify root causes behind subtle errors. This post walks through how NeMo Gym was used to evaluate and improve the reliability of the Siemens Fuse EDA AI Agent when calling characterization tools. The Siemens Fuse team used NeMo Gym to scale up evaluations and verify the resulting Liberty (.lib) files against known references and tracing—both common and rare failures through each rollout. Those findings guided changes to the agent’s skill and Model Context Protocol (MCP) tool descriptions, increasing the measured benchmark composite score from approximately 55% to ~95% across a benchmark of representative standard-cell characterization tasks, while reducing turns and output tokens. Siemens’ work provides a practical example for chip design teams and EDA developers on how to leverage the NVIDIA AI software stack and open libraries to improve agent reliability and chip design workflows. Automating standard-cell characterization Standard-cell characterization is the process of generating accurate timing, power, and noise models for logic cells across operating conditions. It is foundational to chip design and required whenever a semiconductor foundry issues a new process design kit (PDK). Each new PDK carries SPICE transistor models; design rules; and process, voltage, and temperature (PVT) definitions that connect the manufacturing process to design tools. When a foundry updates a PDK, existing characterization datasets may no longer accurately describe the updated devices. Timing and power models must then be regenerated and validated for every affected cell across combinations of input slew, output load, and PVT conditions. The Solido Characterization Suite agent automates this process. The agent orchestrates tools from the broader Solido Characterization Suite through Siemens Fuse EDA AI Agent, which exposes MCP servers and skills to common coding harnesses including Claude Code, Codex, and Cursor. MCP interfaces make these accelerated tools accessible to engineering agents, while skills guide their use in characterization workflows. Faster simulation helps agents complete engineering iterations sooner, while NVIDIA NeMo Gym helps developers measure and improve how reliably and efficiently agents orchestrate those tools. The Solido Characterization Suite agent must perform tool setup, large-scale SPICE simulation, distributed compute, Liberty file generation, debugging, and final validation (Figure 1). Errors often produce plausible outputs instead of clean failures, leaving incorrect thresholds, missing corners, or incomplete cells hidden. Because these stages depended on one another, an early setup error could propagate into every step that followed. Defining a rigorous evaluation protocol was a key step in preparing the Solido Characterization Suite agent for production workflows. Figure 1. Agentic characterization workflow from SPICE inputs through pilot validation, scale-out, Liberty generation, and waveform comparison. Early setup errors propagate to the final library Measuring and improving agent reliability with NeMo Gym Part of the NVIDIA NeMo collection of open libraries for agent specialization and optimization, NeMo Gym integrates with the standard-cell characterization flow inside a controlled evaluation loop (Figure 2). It defines the task, tools, state, and verifier for each rollout so that attempts started from a clean state and produced comparable results. It also records thinking traces, input and output messages, and tool actions. This enabled engineers to run the following optimization loop: Run baseline rollouts with a versioned task and verifier in a custom NeMo Gym environment Inspect trajectories, failed tool calls, and verifier diagnostics Identify ambiguous instructions, unproductive actions, and incorrect tool selection Refine the agent’s skills and MCP tool descriptions Repeat the same tasks under the same evaluation conditions Compare verified outcomes and efficiency per successful result Figure 2. NeMo Gym checks the Liberty files produced by the Solido Characterization Suite agent and Solido tools, then feeds failure analysis into the next skill and tool-description revision. The entire process is driven by a coding agent with a human in the loop Reward design and artifact verification As the team analyzed agent trajectories, end-to-end evaluation uncovered failures that intermediate checks missed. For example, final Liberty verification exposed missing cells and operating conditions, incorrect slew thresholds, and timing or power values outside tolerance. These findings show why engineering agents must be evaluated against final artifacts rather than individual tool calls. To make the work repeatable, the team implemented verifiers that provide a final composite reward score based on the following metrics: Coverage : Whether the agent produced every expected cell and PVT corner. Validity : Whether the delivered Liberty files are well-formed and usable by the validation tools. Relative validation: Whether the output introduced failures beyond those already present in the reference. Accuracy : The fraction of expected timing and power values that falls within the defined tolerance. Integrity : Whether the characterization workflow produced the complete result rather than reusing, truncating, or reformatting existing values. The reward score is defined as: \(R=\mathbf{1}\left[\text{Coverage}\right]\times\mathbf{1}\left[\text{Validity}\right]\times\mathbf{1}\left[\text{Relative Validation}\right]\times\mathbf{1}\left[\text{Integrity}\right]\times\frac{X_{\text{tol}}}{X_{\text{total}}}\) Here, \(X_{\mathrm{tol}}\) is the number of values in the output that are within tolerance, and \(X_{\mathrm{total}}\) is the total number of output values. Their ratio represents the fraction of output values within tolerance. Each indicator \(\mathbf{1}[\cdot]\) equals \(1\) when its corresponding gate passes and \(0\) when it fails. The reward is zero if any gate fails. Using this verifier, the team systematically evaluated and tuned the agent’s performance across multiple iterations. Optimizing the agent The team used the open source GF180MCU PDK to benchmark the Solido Characterization Suite agent on Liberty-file generation. The dataset includes 229 standard cells across 36 PVT corners, using Solido AFS for SPICE simulation. A custom agentic environment was built using NeMo Gym. For each run, the characterization flow was restricted to two cells across 36 PVT corners, as this smaller subset was sufficient for tuning the agentic workflow. Each run took approximately 1.5 to 2 hours, and the team conducted approximately 25 rollouts using Claude Opus 4.8 running in Claude Code as the fixed model and agent harness. Across successive evaluation rounds, the team refined the agent’s characterization skills and MCP tool descriptions.