ai-dynamo/dynamo

| Docs | Roadmap | Recipes | Examples | Containers | Digest | Design Proposals | How to Contribute | Slack | Dynamo The open-source, datacenter-scale inference stack. Dynamo is the orchestration layer above inference engines — it doesn't replace SGLang, TensorRT-LLM, or vLLM, it turns them into a coordinated multi-node inference system. Disaggregated serving, intelligent routing, multi-tier KV caching, and automatic scaling work together to maximize throughput and minimize latency for LLM, reasoning, multimodal, and video generation workloads. Built in Rust for performance, Python for extensibility. Community Events Date Event Location Thu, Oct 15, 2026 CoreWeave x NVIDIA Dynamo Meetup San Francisco Tue, Oct 13, 2026 Dynamo Engineering Sync – Wed, Sep 23, 2026 Dynamo community meeting Online Events are updated automatically. Subscribe to our public calendar . When to use Dynamo You're serving LLMs across multiple GPUs or nodes and need to coordinate them You want KV-aware routing to avoid redundant prefill computation You need to independently scale prefill and decode (disaggregated serving) You want automatic scaling that meets latency SLAs at minimum total cost of ownership (TCO) You need fast cold-starts when spinning up new replicas If you're running a single model on a single GPU, your inference engine alone is probably sufficient. Feature support at a glance: SGLang TensorRT-LLM vLLM Disaggregated Serving ✅ ✅ ✅ KV-Aware Routing ✅ ✅ ✅ SLA-Based Planner ✅ ✅ ✅ KVBM 🚧 ✅ ✅ Multimodal ✅ ✅ ✅ Tool Calling ✅ ✅ ✅ Full Feature Matrix → — LoRA, request migration, speculative decoding, and feature interactions. Key Results Result Context 7x higher throughput per GPU DeepSeek R1 on GB200 NVL72 w/ Dynamo vs B200 without ( InferenceX ) 7x faster model startup ModelExpress weight streaming (DeepSeek-V3 on H200) 2x faster time to first token KV-aware routing, Qwen3-Coder 480B ( Baseten benchmark ) 80% fewer SLA breaches Planner autoscaling at 5% lower TCO ( Alibaba APSARA 2025 @ 2:50:00 ) 750x higher throughput DeepSeek-R1 on GB300 NVL72 ( InferenceXv2 ) What Dynamo Does Most inference engines optimize a single GPU or a single node. Dynamo is the orchestration layer above them — it turns a cluster of GPUs into a coordinated inference system. Architecture Deep Dive → Core Capabilities Capability What it does Why it matters Disaggregated Prefill/Decode Separates prefill and decode into independently scalable GPU pools Maximizes GPU utilization; each phase runs on hardware tuned for its workload KV-Aware Routing Routes requests based on worker load and KV cache overlap Eliminates redundant prefill computation — 2x faster TTFT KV Block Manager (KVBM) Offloads KV cache across GPU → CPU → SSD → remote storage Extends effective context length beyond GPU memory ModelExpress Streams model weights GPU-to-GPU via NIXL/NVLink 7x faster cold-start for new replicas Planner SLA-driven autoscaler that profiles workloads and right-sizes pools Meets latency targets at minimum total cost of ownership (TCO) Grove K8s operator for topology-aware gang scheduling (NVL72) Places workloads optimally across racks, hosts, and NUMA nodes AISimulate Predicts serving behavior and searches deployment configurations offline Finds a strong serving configuration without bringing up a GPU cluster Fault Tolerance Canary health checks + in-flight request migration Workers fail; user requests don't New in 1.0 Zero-config deploy ( DGDR ) (beta): Specify model, HW, and SLA in one YAML — AISimulate performance models estimate candidate configurations, Planner optimizes the topology, and Dynamo deploys Agentic inference: Per-request hints for priority, expected output length, and speculative prefill, plus session metadata for tracing and SGLang subagent KV isolation. LangChain + NeMo Agent Toolkit integrations Multimodal E/P/D: Disaggregated encode/prefill/decode with embedding cache — 30% faster TTFT on image workloads Video generation: Native FastVideo + SGLang Diffusion support — real-time 1080p on single B200 K8s Inference Gateway plugin: KV-aware routing inside the standard Kubernetes gateway Storage-tier KV offload: S3/Azure blob support + global KV events for cluster-wide cache visibility Request Routing Topologies Dynamo can expose traffic through two Kubernetes request routing topologies. Both expose an OpenAI-compatible API and support the same backends, disaggregated serving, and KV-aware routing. Topology What it is When to use Dynamo-native Frontend routing The Dynamo Frontend serves HTTP and the integrated Dynamo Router makes worker-selection decisions. No external gateway is required. Local development, single-cluster deployments, and environments where Dynamo should own the request entry point end to end. Gateway API routing with GAIE A Kubernetes Gateway API Inference Extension gateway calls the Dynamo Endpoint Picker Plugin (EPP) before forwarding to the selected worker's Frontend sidecar in --router-mode direct . Kubernetes platforms that standardize on Gateway API, or deployments where gateway-level policy, auth, rate limiting, and observability should sit at the cluster edge. Request flow for the Dynamo-native path is client → Frontend → Router → workers . Request flow for the Gateway API path is client → Gateway → EPP → Frontend sidecar (direct) → workers . See the Gateway API Inference Extension (GAIE) guide for the Gateway API setup, supported features, and configuration. Quick Start This repo ships agent skills: if you work with an AI coding agent (Claude Code, Codex, Cursor), clone the repo and ask it to deploy, troubleshoot, benchmark, or optimize a Dynamo deployment. The skills activate automatically; no setup required. Option A: Container (fastest) Choose either vLLM or SGLang: vLLM # Pull a prebuilt container docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1 # Inside the container — start frontend and worker python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 & python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --discovery-backend file & # Once the worker is ready, send a request curl -s localhost:8000/v1/chat/completions -H " Content-Type: application/json " -d ' { "model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 100 } ' | jq SGLang # Pull a prebuilt container docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.5.1 # Inside the container — start frontend and worker python3 -m dynamo.frontend --http-port 8000 --discovery-backend file > /dev/null 2>&1 & python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --discovery-backend file & # Once the worker is ready, send a request curl -s localhost:8000/v1/chat/completions -H " Content-Type: application/json " -d ' { "model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 100 } ' | jq Also available: tensorrtllm-runtime:1.5.1 . Option B: Install from PyPI Install uv ( curl -LsSf https://astral.sh/uv/install.sh | sh ), then choose your backend: vLLM uv pip install --prerelease=allow " ai-dynamo[vllm] " SGLang uv pip install --prerelease=allow " ai-dynamo[sglang] " Note: TensorRT-LLM requires pip with --extra-index-url https://pypi.nvidia.com . See the install guide for TRT-LLM-specific instructions. Then start the frontend and the matching worker as shown above. See the full installation guide for system dependencies and backend-specific notes. Option C: Kubernetes (recommended) For production multi-node clusters, install the Dynamo Platform and deploy with a single manifest: # Zero-config deploy: specify model + SLA, Dynamo handles the rest apiVersion : nvidia.com/v1beta1 kind : DynamoGraphDeploymentRequest metadata : name : my-model spec : model : Qwen/Qwen3-0.6B backend : vllm sla : ttft : 200.0 # ms itl : 20.