Robbyant/lingbot-map

LingBot-Map: Geometric Context Transformer for Streaming 3D Reconstruction Robbyant Team teaser.mp4 🗺️ Meet LingBot-Map! We've built a feed-forward 3D foundation model for streaming 3D reconstruction! 🏗️🌍 LingBot-Map has focused on: Geometric Context Transformer : Architecturally unifies coordinate grounding, dense geometric cues, and long-range drift correction within a single streaming framework through anchor context, pose-reference window, and trajectory memory. High-Efficiency Streaming Inference : A feed-forward architecture with paged KV cache attention, enabling stable inference at ~20 FPS on 518×378 resolution over long sequences exceeding 10,000 frames. State-of-the-Art Reconstruction : Superior performance on diverse benchmarks compared to both existing streaming and iterative optimization-based approaches. 📑 Table of Contents Click to expand 📰 News 📋 TODO ⚙️ Installation 📦 Model Download 🚀 Quick Start 🎬 Interactive Demo ( demo.py ) Try the Example Scenes Streaming with Keyframe Interval Windowed Inference (for long sequences, >3000 frames) Sky Masking Visualization Options Performance & Memory 🎥 Offline Rendering Pipeline ( demo_render/batch_demo.py ) 📜 License 📖 Citation ✨ Acknowledgments 📰 News 2026-06-28 — Fixed an SDPA KV cache bug. The SDPA backend now performs better on long sequences . We still recommend the FlashInfer backend for the best performance. 2026-05-25 — 📊 Evaluation benchmark released . We released the evaluation scripts for KITTI and Oxford Spires — see benchmark/ for the pipeline, and run preprocess/oxford.py to prepare Oxford Spires data before evaluation. 2026-04-29 — 📹 Long-video demo released . We released a very-long-video example (~25 000 frames, 13-minute indoor walkthrough) rendered with the offline pipeline — see Worked Example for the command, flag rationale, and rendered output. 2026-04-27 — 🚀 LingBot-Map accelerated . Pull the latest main and run python demo.py --compile ... or python gct_profile.py --backend flashinfer --dtype bf16 --compile to verify on your hardware. 2026-04-24 — Fixed a FlashInfer KV cache bug where --keyframe_interval > 1 silently cached non-keyframes. You should now see better pose and reconstruction quality when running with more than 320 frames . 📋 TODO ✅ Release evaluation benchmark ✅ Oxford Spires dataset ✅ KITTI dataset ✅ VBR dataset ✅ Droid-W dataset ✅ TUM-D dataset ✅ 7-scenes dataset ✅ ETH3D dataset ✅ Tanks and Temples dataset ✅ NRGBD dataset ✅ Release demo scripts ✅ Indoor long-video demo ( Featured indoor walkthrough ) ✅ Outdoor long-video demo ✅ LingBot-World demo ( Worked example ) ✅ Aerial long-video demo ⚙️ Installation 1. Create conda environment conda create -n lingbot-map python=3.10 -y conda activate lingbot-map 2. Install PyTorch (CUDA 12.8) pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128 PyTorch 2.8.0 is the recommended version because NVIDIA Kaolin (required by the batch rendering pipeline) has prebuilt wheels for torch-2.8.0_cu128 . If you only need demo.py you may use a newer PyTorch, but the batch renderer then requires building Kaolin from source. For other CUDA versions, see PyTorch Get Started . 3. Install lingbot-map pip install -e . 4. Install FlashInfer (recommended) FlashInfer provides paged KV cache attention for efficient streaming inference. It is a pure-Python package that JIT-compiles CUDA kernels on first use, so a single wheel works across CUDA/PyTorch versions: pip install --index-url https://pypi.org/simple flashinfer-python --index-url https://pypi.org/simple is only needed if your default pip index is an internal mirror that doesn't have flashinfer-python . (Optional) For faster first-use, you can additionally install a CUDA-specific JIT cache: pip install flashinfer-jit-cache -f https://flashinfer.ai/whl/cu128/flashinfer-jit-cache/ . See FlashInfer installation for details. If FlashInfer is not installed, the model falls back to SDPA (PyTorch native attention) via --use_sdpa . 5. Visualization dependencies (optional) pip install -e " .[vis] " 📦 Model Download Model Name Huggingface Repository ModelScope Repository Description lingbot-map robbyant/lingbot-map Robbyant/lingbot-map Balanced checkpoint (used in paper, benchmark and offline demo) — trade off all-around performance across short and long sequences. lingbot-map-stage1 robbyant/lingbot-map Robbyant/lingbot-map Stage-1 training checkpoint of lingbot-map — can be loaded into the VGGT model for bidirectional inference (c2w). 🚧 Coming soon: we're training an stronger model that supports longer sequences — stay tuned. 🚀 Quick Start After installation, run your first scene with one command: python demo.py --model_path /path/to/lingbot-map.pt \ --image_folder example/courthouse --mask_sky This launches an interactive viser viewer at http://localhost:8080 . See Interactive Demo below for the full set of scenes and flags, or jump to Offline Rendering Pipeline for long-sequence batch rendering. 🎬 Interactive Demo ( demo.py ) Run demo.py for interactive 3D visualization via a browser-based viser viewer (default http://localhost:8080 ). Try the Example Scenes We provide three example scenes in example/ that you can run out of the box: # courthouse scene python demo.py --model_path /path/to/lingbot-map.pt \ --image_folder example/courthouse --mask_sky output_pointcloud_side_by_side.mp4 # University scene python demo.py --model_path /path/to/lingbot-map.pt \ --image_folder example/university --mask_sky output_pointcloud_side_by_side.mp4 # Loop scene (loop closure trajectory) python demo.py --model_path /path/to/lingbot-map.pt \ --image_folder example/loop output_pointcloud_side_by_side.mp4 🎯 Featured: indoor walkthrough (~25 000 frames, 13 minutes) Sequence is too long for the interactive viser viewer — this clip was rendered with the Offline Rendering Pipeline . See that section for the full command. We will provide more examples in the follow-up. Dynamic Demo (From Droid-W) Dataset: Download the demo sequences from robbyant/lingbot-map-demo on Hugging Face. Example run on the dynamic sequence from the dataset above (sky masking on, 4 camera optimization iterations, keyframe every 2 frames): Run the dynamic sequence with sky masking, 4 camera optimization iterations, and an input stride of 2: python demo.py \ --image_folder /path/to/dynamic\ --model_path ../../Lingbot-Map/lingbot-map.pt \ --camera_num_iterations 4 \ --mask_sky \ --stride 2 output_pointcloud_side_by_side_under_10MB.mp4 Streaming with Keyframe Interval Use --keyframe_interval to reduce KV cache memory by only keeping every N-th frame as a keyframe. Non-keyframe frames still produce predictions but are not stored in the cache. This is useful for long sequences which exceed 320 frames (We train with video RoPE on 320 views, so performance degrades when the KV cache stores more than 320 views. Using a keyframe strategy allows inference over longer sequences.). In demo.py, the keyframe interval is calculated automatically. Note on inference range. Our method does not perform state resetting by default, so the maximum inference range is bounded by the longest distance seen during training on the dataset. Beyond that distance, state resetting becomes necessary. If you observe pose collapse, switch to windowed mode ( --mode windowed ) — in most cases tuning --keyframe_interval alone is enough and the rest of the windowed parameters can stay at their defaults. Windowed Inference (for long sequences, >3000 frames) python demo.py --model_path /path/to/lingbot-map.pt \ --video_path video.mp4 --fps 10 \ --mode windowed --window_size 128 --overlap_keyframes 16 --keyframe_interval 2 Sky Masking Sky masking uses an ONNX sky segmentation model to filter out sky points from the reconstructed point cloud, which improves visualization quality for outdoor scenes.