How distributed inference keeps large AI models moving

來源原標題

How distributed inference keeps large AI models moving

機構

IBM

原始標籤

AI · AI for Code · Explainer · Generative AI · Open Source

原始連結

https://research.ibm.com/blog/distributed-inference?utm_medium=rss&utm_source=rss

摘要原文

When an AI model answers a question or completes a task, it performs inference, i.e., using a trained model to generate an output in response to some input. Doing that at scale, however, is more challenging than it sounds. Running a large language model on a GPU is one thing. Serving that model to thousands or even millions of people and AI agents at the same time is quite another. Requests arrive with different priorities and lengths, which have unique computing needs. Some users expect an immediate response to simple queries, while others submit large jobs that can run in the background. AI agent traffic adds a new pattern: Agents may send bursts of related requests, pause to call a tool, and then return with nearly the same context. Meanwhile, the system must make efficient use of expensive accelerators without creating long queues or repeatedly performing the same computations. This challenge resembles air traffic control. A busy airport cannot simply send each arriving aircraft to the first available runway. Controllers in the tower must account for the aircraft’s location, destination, size, and fuel needs, as well as the traffic already moving through the system. The challenge might be simple if all air traffic controllers had to do was keep every runway busy, but an airport isn’t a parking lot, and not every airplane can be treated the same way. Distributed inference plays a similar coordinating role for LLM inference, working a layer above model servers like vLLM and SGLang , where it orchestrates and optimizes real-world traffic. While those serving engines run models efficiently on individual servers, a distributed inference serving framework coordinates groups of model servers across a cluster. Its router can consider queue depth, cache availability, and server roles before deciding where a request should go. To understand how distributed inference works, we’ll look at six notable technical innovations—prefix-cache-aware routing, KV-cache offloading, flow control, batch inference optimization, prefill/decode disaggregation, and wide expert parallelism. We’ll also see how each comes together in llm-d, an open-source, Kubernetes-native inference framework. Prefix-cache-aware routing: Sending work where context already lives KV-cache offloading: Extending a model’s working memory Flow control: Deciding what moves and when Batch inference optimization: Filling spare capacity without clogging the system Prefill/decode disaggregation: Separating two different types of work Wide expert parallelism: Spreading a giant model across the cluster Coordinating the whole inference system Prefix-cache-aware routing: Sending work where context already lives When a language model reads a prompt, it produces an internal representation of the words and their relationships, storing much of this intermediate state in a key-value cache, or KV cache. If a later request begins with the same text, or even portions of the same text, the model may be able to reuse that stored computation rather than re-process the shared text. For example, if an existing prompt is “What is the capital of Kansas?” then later on you ask, “What is the capital of California?” the KV cache for the "what is the capital of" portion is reused. This opportunity arises frequently in real workloads. A chatbot receives the same system instructions at the beginning of every conversation. A coding assistant repeatedly examines the same repository. An AI agent may revisit a long record of previous steps while testing different solutions. In each case, much of the prompt remains unchanged from turn to turn. A conventional load balancer may send the next request to whichever server appears least busy, whether or not its cache contains the relevant context. As a result, the model must reread material that another server has already processed. Prefix-cache-aware routing instead asks two questions: Which server already holds the greatest portion of this prompt’s prefix, and which server has enough capacity to handle the request? The llm-d router uses cache information and operating conditions to score the available model servers and select a suitable destination. There is both an approximate implementation that includes a lightweight index of recent routing decisions, and a precise implementation based on actual token data and cache-state updates from model servers. If an aircraft already has its passengers, baggage, and ground crew waiting at one gate, sending the plane to another gate would introduce needless work. In the same way, a distributed inference system tries to send a request to the model server that already contains the required context. The payoff is less redundant prefill computation, lower latency, and greater throughput. In a recent demonstration from the team behind llm-d, prefix-aware routing captured 97% of the available reuse in the tested workload. More broadly, the technique is especially useful for agentic systems because their requests often share long instructions, source material, conversation histories, and tool outputs. llm-d was launched in May 2025 as a collaborative effort by founding members IBM Research, Red Hat, Google Cloud, NVIDIA, and CoreWeave. Several founding partners also took part from the start, including industry leaders like AMD, Cisco, Hugging Face, Intel, Lambda, and Mistral AI, and academic supporters at the University of California, Berkeley, and the University of Chicago. IBM Research, Red Hat, and Google Cloud continue to lead llm-d's development today, and its community of partners and contributors has grown considerably. In March 2026, the founders donated llm-d to the Cloud Native Computing Foundation as a sandbox project, to further the mission of deploying inference stacks for any model, on any accelerator, in any cloud. KV-cache offloading: Extending a model’s working memory Cache-aware routing is most valuable when cached information remains available. The problem is that although accelerator memory is fast, its space can be quickly filled by long prompts, many concurrent users, and extended agent sessions. When space runs short, a model server may evict older KV-cache entries. If a later request needs that material, the model must calculate it again, wasting accelerator cycles rebuilding context that existed moments earlier. Distributed inference addresses this problem through KV-cache offloading . Rather than treating accelerator memory as the only place where cached context can live, it chooses an offloading target based on workload requirements. The most immediately useful entries can remain in high-bandwidth accelerator memory, while other entries can move to larger but slower tiers such as CPU memory, local solid-state storage, or a shared file system. A cache index tracks what is stored and where, allowing the routing layer to incorporate offloaded data into its decisions. Retrieving cached information from CPU memory or storage is slower than reading it directly from accelerator memory, but it is much faster and less expensive than recomputing the entire prompt. Offloading therefore expands the system’s effective cache capacity beyond the physical memory available on its accelerators. This matters most for workloads with long, reusable contexts and many simultaneous sessions. In the same llm-d demonstration mentioned in the previous section, GPU memory supplied about 90% to 92% of reusable prefill prefixes for 128 concurrent requests, with CPU offloading helping to preserve prefixes beyond GPU capacity. In another demonstration , CPU offloading helped the system sustain high throughput and low time to first token across a long, agent-driven benchmark. Because offloading runs during prefill and writes to slower tiers happen asynchronously, these gains come without slowing token generation. Flow control: Deciding what moves and when Routing answers where a request should go.