Introducing NV-Reason-CT Open 3D CT VLM for Radiologist Chain-of-Thought Reasoning

來源原標題

Introducing NV-Reason-CT Open 3D CT VLM for Radiologist Chain-of-Thought Reasoning

作者

Tanya Lenz

原始標籤

Agentic AI / Generative AI · Developer Tools & Techniques · Simulation / Modeling / Design · AI Foundation Models · Medical Imaging · Open Source · Training AI Models · VLMs

原始連結

https://developer.nvidia.com/blog/introducing-nv-reason-ct-open-3d-ct-vlm-for-radiologist-chain-of-thought-reasoning/

摘要原文

Radiology AI has made remarkable strides in detecting abnormalities across chest X-rays, pathology slides, and 2D scans. Yet one of the most clinically rich and data-dense modalities—the 3D computed tomography (CT) scan—remains largely underserved by modern vision language models (VLMs) . Frontier general-purpose models perform poorly on volumetric imaging, and most open medical AI models lack the multistep conversational depth that radiologists need to trust and verify AI-generated findings. NVIDIA is addressing this gap with NV-Reason-CT, a VLM purpose-built for 3D CT analysis. NV-Reason-CT extends chain-of-thought reasoning to full volumetric CT, generating structured diagnostic reports, emulating radiologist internal thinking, and supporting multistep follow-up conversation across chest and abdomen. It builds on the reasoning methodology pioneered by NV-Reason-CXR , validated in a multireader clinical study accepted at RSNA 2026 confirming radiologist time savings while maintaining diagnostic accuracy. NV-Reason-CT is an open research and development foundation; not an autonomous diagnostic system or a cleared clinical product. It is an AI foundation model designed for researchers and developers building specialized CT analysis applications to post-train for their use case. Why 3D CT reasoning demands a different approach A single abdominal CT study can comprise 300–600 axial slices, encoding anatomical context across three spatial dimensions that a standard 2D encoder simply cannot reconstruct from independent slices. This volumetric complexity creates a series of compounding challenges for medical AI to do with perception, reasoning, and conversational depth: Perception : Standard VLMs treat image input as a 2D token grid. Processing a CT volume as a stack of independent 2D frames discards the spatial relationships between slices that define structures like masses, effusions, and infiltrates—structures whose shape, extent, and density only become clinically meaningful in three dimensions. Reasoning : Even models that correctly perceive an abnormality often output a diagnostic label without articulating why . Radiologists don’t think in labels—they think in systematic anatomical reviews, differential diagnoses, and degrees of confidence. An AI that cannot reproduce that reasoning process cannot be audited, taught from, or safely integrated into clinical workflows. Conversational depth : A radiologist reviewing a suspicious finding doesn’t close the case at first glance. They ask follow-up questions, reconsider differentials, and correlate findings across anatomical regions. Most existing models lack the multiturn dialogue capability to support this kind of iterative clinical reasoning. Providing full 3D reasoning for CT analysis NV-Reason-CT combines a dedicated full 3D vision transformer (ViT) encoder with a language model trained to generate chain-of-thought reasoning that mirrors how radiologists systematically analyze CT volumes. Unlike approaches that adapt 2D encoders to CT by treating slices independently, NV-Reason-CT processes the CT volume as a true 3D input. This preserves through-plane anatomical continuity and enables the model to reason about structures holistically, the way a radiologist would when scrolling through a study. Core model capabilities include the following: Structured report generation : NV-Reason-CT generates detailed structured reports. The NVIDIA team curated a CT ontology covering 30 chest and 29 abdominal abnormalities to guide and evaluate the model—such lung nodules, pneumothorax, hepatic lesions, renal cysts, and more—in a format that maps naturally to clinical documentation workflows. Radiologist-emulating chain-of-thought : NV-Reason-CT can also generate reasoning emulating radiologist thought chains. The model produces step-by-step internal thinking—examining anatomical regions systematically, surfacing relevant findings, considering differential diagnoses, and articulating uncertainty—in the style of an experienced radiologist working through a study. Multistep conversational follow-up : Clinicians and researchers can ask follow-up questions about specific findings, request clarifications on differential diagnoses, or probe the model’s reasoning at any stage. This multiturn capability transforms NV-Reason-CT from a report generator into an interactive diagnostic partner. Full 3D ViT encoder : A purpose-built 3D vision encoder processes CT volumes natively, extracting volumetric features that 2D-based approaches cannot recover. In addition, 3D vision token grid coordinates are passed to LLM to account for spatial inter-token relationship throughout the LLM layers through 3D MRoPE. This architectural choice enables the model to reason about spatial extent, cross-sectional morphology, and inter-slice relationships—the perceptual foundations of accurate CT interpretation. How is NV-Reason-CT architecture purpose-built for volumetric reasoning? The model architecture combines Qwen3.5-4B LLM with 3D ViT (Primus/Colipri). All weights are retrained end-to-end on large cohort or CT data with structured report, reasoning traces, multistep VQA (designed internally). Standard transformer-based VLMs are designed for 2D images. Adapting these to CT by flattening a volume into a sequence of 2D slices loses the spatial structure that defines volumetric pathology. The encoder architecture is adapted from Primus 3D ViT, initialized with Colipri weights prior to training; It processes CT volumes resampled to 192³ voxels at 2 mm isotropic resolution, using non-overlapping 8x8x8 patch tokens—resulting in 24x24x24 = 13,824 vision token context. Instead of merging (or downsizing), all vision tokens are passed to LLM (together with their 3D grid coordinates). The LLM includes 3D MRoPE to account for the 3D spatial relationship of vision tokens. The language model component is trained to reason in the style of a radiologist: systematically reviewing anatomical regions, noting normal findings alongside abnormal ones, expressing calibrated uncertainty, and arriving at a structured conclusion. The model is designed to respond not as a classifier, but as a teacher: explaining the problem, walking through the evidence, and arriving at a diagnosis through visible logical steps. What is the NV-Reason-CT training methodology? Building on the approach introduced with NV-Reason-CXR, NV-Reason-CT follows a two-stage training pipeline: supervised fine-tuning followed by reinforcement learning (RL) . Stage 1: Supervised fine-tuning on radiologist reasoning data The initial stage trains the model on the mixture of data, including structured report, expert radiologist reasoning annotations, and general VQA. Radiologists contributed detailed chain-of-thought dictations for CT studies that capture their internal review process, including what they examine in each anatomical region, which findings they consider significant, which differentials they weigh, and how they arrive at their final assessment. The resulting curriculum spans approximately 550,000 structured QA examples across chest and abdominal regions covering section-level anatomy QA, laterality-specific and localized finding QA, severity-level QA, and binary abnormality identification. Refusal examples for invalid prompts and mismatched image-text pairs were included to improve robustness. Training data includes CT-RATE, NIH CT datasets, and CancerVerse. This dataset is supplemented with high-quality synthetic reasoning data distilled from large language models, using expert radiologist annotations as grounding examples. The combined dataset provides the model with a rich signal for what structured radiological reasoning looks like across a wide range of CT findings. Stage 2: RL for reasoning quality The second stage uses Group Relative Policy Optimization (GRPO) to refine reasoning quality.