Depth-Grounded Vision Language Models: Conversational AI in Spatial Interfaces for Robot-Supported Mixed Reality

Large language models (LLMs) and vision-language models (VLMs) are increasingly used in mixed-reality (MR) interfaces, yet most operate on text or 2D imagery and remain unreliable for metric spatial reasoning. We present an exploratory robot-supported MR prototype that combines RGB-D sensing, YOLO object detection, and a VLM-based conversational layer to communicate sensor-grounded object distances to users e.g., the chair is 3.02 meters away. The purpose is not to make the VLM infer metric depth from monocular images, but to investigate whether robot-measured 3D information can be made available through natural-language spatial feedback. In a toxic-smoke MR scenario targeting the emergency first response (EFR) domain, participants estimated distances to a simulated endangered person and an exit window under three conditions: video-only, depth-agnostic VLM support, and depth-augmented VLM support. The primary behavioral result shows that depth augmentation significantly reduced baseline-adjusted distance-estimation error and variability compared with depth-agnostic VLM support. A system-output validity analysis further showed that depth-agnostic VLM distance reports were highly inaccurate, whereas depth-augmented reports remained close to ground truth. Subjective measures showed a favorable but exploratory awareness–workload pattern and should not be interpreted as confirmatory evidence. We contribute an early spatial interface prototype showing how robot-measured 3D information can transform VLM interaction from plausible scene description into metrically grounded spatial communication for mixed-reality human-robot support.