NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning

來源原標題

NVIDIA Nemotron 3.5 Super VL Powers Specialized Agents with Multimodal Reasoning

作者

Michelle Horton

原始標籤

Agentic AI / Generative AI · Developer Tools & Techniques · Top Stories · AI Agent · featured · NVFP4

原始連結

https://developer.nvidia.com/blog/nvidia-nemotron-3-5-super-vl-powers-specialized-agents-with-multimodal-reasoning/

摘要原文

Always-on agents are taking on longer, more complex enterprise workflows. A single request can involve reading a document, checking information across sources, following company policy, and calling several tools. NVIDIA Nemotron 3.5 Super VL is an open, customizable multimodal model for these specialized workflows. Its 120B-parameter mixture-of-experts (MoE) architecture activates 12B parameters per token, balancing model capacity with inference efficiency. On a single GPU, this model delivers up to 5x higher throughput for agentic tasks compared to other open models in its size category. Figure 1. Nemotron 3.5 Super VL delivers over 600 TPS on Artificial Analysis Intelligence Index agentic tasks on the Baseten endpoint The model combines reasoning, instruction following, and tool use with text, image, document, and video understanding. By combining these capabilities into one model, teams can handle text and visual tasks without deploying separate text and vision-language models or passing task context between them. Developers can post-train Nemotron 3.5 Super VL with NVIDIA NeMo using their own data, tools, policies, and workflows. This post explains how to use that foundation to build specialized enterprise agents and describes the training and inference techniques that power it. Why Nemotron 3.5 Super VL for specialized agents? Agents need to understand both their tasks and operating environments. A customer-service agent must follow company return policies. Research agents connect evidence across sources, while document agents interpret tables, charts, and page layouts. Nemotron 3.5 Super VL combines reasoning and visual understanding across a range of agent workflows, including: Coding: Test and validate code, analyze errors, and recommend fixes. Web research: Search across websites, connect information, and verify answers. Document intelligence: Extract information from tables, charts, and page layouts to produce summaries and reports. Cybersecurity: Identify security vulnerabilities and triage threats to prioritize remediation. Video understanding: Extract information for search, content review, and incident analysis. Within a system of models, Nemotron 3.5 Super VL handles specialized, multistep work that needs reasoning, tool use, and text and visual understanding together. Smaller models, such as NVIDIA Nemotron 3.5 Lightning , handle high-volume text tasks, while frontier models such as NVIDIA Nemotron 3 Ultra tackle complex planning and orchestration. Customize Nemotron 3.5 Super VL for domain data, tools, and policies Specialized agents must learn how an organization works. Domain terminology, tool interfaces, operating procedures, and policy requirements shape how agents complete tasks successfully. Organizations in industries ranging from cybersecurity and finance to healthcare are post-training Nemotron 3.5 Super VL to outperform frontier models in accuracy and efficiency across their production workflows. For example: Scale AI, an AI data and technology provider, post-trained Nemotron 3.5 Super VL for automated vendor credit processing at Skydance Corporation, a media and entertainment company. Inference cost for this workload will decrease by 40x and latency by 5x. Ramp, a financial operations platform provider, is converting receipt images into structured JSON containing merchant information, dates, and payment data. Ramp expects up to 50% higher throughput and 10x lower latency. AI21 and Alice post-trained Nemotron 3.5 Super VL for threat intelligence and vulnerability analysis, improving accuracy by 41%. Figure 2. Post-trained Nemotron 3.5 Super VL achieves leading performance for specialized tasks across industries Teams can use NVIDIA NeMo open libraries to post-train models to meet these requirements. Developers can use NVIDIA NeMo AutoModel for supervised fine-tuning (SFT) and low-rank adaptation (LoRA) to teach domain-specific behavior. Reinforcement learning (RL) improves performance in agentic environments that reflect organizational workflows. Alongside Nemotron 3.5 Super VL, NVIDIA is releasing open datasets on Hugging Face. These include agentic tool-use and conversational data, finance reasoning and tool-calling datasets, instruction-following and chat data, long-context transduction and translation data, fairness data, and 40 vision-language datasets. These datasets can supplement proprietary data for SFT and RL. For multimodal data, start with NeMo Data Designer resources for chart question answering and long-document understanding . Explore NVIDIA NeMo RL and NVIDIA NeMo Gym for RL and environment-based evaluation. Organizations can adapt and deploy the model’s open weights in supported local, data center, or cloud environments to meet their infrastructure and data requirements. For customization resources, see the Nemotron 3.5 Super VL usage cookbook on GitHub. Combine text and visual understanding in one workflow Nemotron 3.5 Super VL combines text, image, document, and video understanding with reasoning and instruction following to answer questions across these inputs. Its visual training covers charts, tables, optical character recognition (OCR), multipage documents, and video understanding. A two-stage SFT pipeline trains the language and vision components, combining visual evidence with reasoning, chat, safety, long-context data, and agent trajectories. For document intelligence, the same model can interpret a chart, connect it with a table elsewhere in a document, and summarize both. Developers can tailor the workflow to their own documents and evaluation criteria. Diverse training data generated with NeMo Data Designer helped the model better understand complex charts. On CharXiv, a chart-reasoning benchmark, the model scored 80.8%, compared with 63% for Nemotron 3 Nano Omni . With Python code interpretation, it reached 84%. For the approach and lessons learned, read the chart-understanding deep dive and the open-source chart question-answering pipeline . This model was built with support and expertise from Nemotron Coalition organizations. For instance, high-quality datasets from Perplexity helped improve and streamline the model’s mid-training. Environments from Prime Intellect enabled building better teacher models. Nemotron 3.5 Super VL is also one of the top open models on the OCRBench v2 leaderboard . Accuracy and efficiency across agent tasks Evaluations of Nemotron 3.5 Super VL span Hermes, LangChain, OpenClaw, OpenCode, Pi, Codex, and Claude. In addition to coding tasks, agent harness providers use their own evaluations to test a wide range of agent tasks. On Hermes Bench and LangChain behavioral evaluations, Nemotron 3.5 Super VL scores higher than Nemotron 3 Ultra. Model Hermes Bench LangChain Evals Nemotron 3 Ultra 56.33% 84.3% Nemotron 3.5 Super VL 63.44% 87.05% Table 1. Nemotron 3.5 Super VL scores higher than Nemotron 3 Ultra on Hermes Bench and LangChain evaluations For always-on agents, efficiency determines how much work a deployment can complete. Throughput, token usage, tool calls, and the steps needed to reach a correct result all affect task completion time. Figure 3. Nemotron 3.5 Super VL uses higher throughput and token efficiency to deliver 4x faster task completion This model was also optimized to run locally on NVIDIA DGX Spark . An independent evaluation by EXO Labs placed Nemotron 3.5 Super VL on the Pareto frontier for accuracy and task completion time. Figure 4. Nemotron 3.5 Super VL balances accuracy and speed for agent tasks on DGX Spark Nemotron 3.5 Super VL under the hood The model builds on a strong foundation and uses a hybrid architecture, speculative decoding, context reuse, and mixed precision to reduce inference work across repeated agent calls. Building a strong base model Nemotron 3.5 Super VL was distilled from Nemotron 3 Ultra. Ultra’s coding capabilities, combined with the SkewReverse loss used during distillation, helped improve coding accuracy.