Smaller, faster, safer: running Kimi and GLM at scale

來源原標題

Smaller, faster, safer: running Kimi and GLM at scale

作者

ascorbic

原始標籤

hackernews

來源訊號

HN points 207　HN 留言 50

原始連結

https://blog.cloudflare.com/smaller-faster-safer-models/

摘要原文

Workers AI runs inference for some of the best open models in the world on GPUs in Cloudflare data centers close to your users. Two of the most capable, and most demanding, are Moonshot's Kimi K-series and Z.ai 's GLM. They are large, long-context, mixture-of-experts models, and they are wonderful to use. They are also very hard to serve efficiently because of memory constraints. We've written before about how we serve large models on Workers AI and about separating the prefill and decode phases of inference to get more out of each GPU. This post looks at three techniques we layer on top of that to fit these models into memory and keep them fast: quantizing the KV cache, compressing the model weights, and, because both of those pack more requests onto shared hardware, protecting the cache those requests share. These optimizations enable us to support more customers at lower costs, with no change in model accuracy. All our experiments and production traffic are running and benchmarked with SGLang , an open-source inference serving framework. We found that SGLang offers the best performance in the market, and we work closely with the SGLang team to upstream patches and new features to make our work available to the open-source community. Quantizing the KV cache As a model generates text, it stores the attention keys (K) and values (V) for every token it has already processed in a structure called the KV cache. The cache is what lets the model extend a long conversation without re-reading the entire context on every new token. For a long-context model, it grows quickly, and it is usually the KV cache, not the model's weights, that fills up GPU memory first. By default, the cache is stored in 16-bit precision (BF16). We store it in 8-bit floating point instead (FP8, e4m3), which halves its size. On Kimi K2.6, that raises the amount of context we can hold in memory from roughly 686,000 tokens to about 1.37 million, twice as much. It's worth being precise about where the benefit comes from, because it isn't raw speed. Quantizing the cache adds a small amount of work per token, since the FP8 attention kernel has to convert values as it reads them. What it changes is how many requests we can keep resident at once. The following measurements are for Kimi K2.6 decoding on a disaggregated H200 deployment, comparing the attention kernels directly: Concurrent requests BF16 KV cache (tok/s) FP8 KV cache (tok/s) 1 137 125 8 731 689 16 1,106 1,028 32 1,558 1,489 64 Out of memory 2,192 At any single concurrency level, BF16 is a few percent faster per token. But BF16 runs out of cache at 32 concurrent requests and can't admit a 33rd, while FP8 keeps going to 64 and reaches 2,192 tokens per second, about 41% higher than BF16's peak, for roughly 30% less cost per token. Because we run prefill and decode as separate pools, we can apply this where it helps most: prefill is compute-bound rather than memory-bound, so there we leave the cache in BF16 and keep its slightly higher throughput. None of this would matter if it changed the model's answers, so we checked. Across our evaluation suite, FP8 and BF16 caches are indistinguishable: Benchmark BF16 KV FP8 KV GSM8K 94.24 94.09 ARC-Easy 89.06 89.14 ARC-Challenge 66.72 67.49 MMLU 89.11 89.04 MMLU-Pro 80.29 79.29 mcxams (internal benchmark) 61 / 63 61 / 63 Tool-call validity 92.2% 92.6% Compressing the model weights The KV cache is one demand on GPU memory; the model's weights are the other. For GLM 5.2, we compress the weights from 8-bit floating point down to 4-bit integers (INT4) with no loss in accuracy. The checkpoint shrinks from 705 GB to 421 GB, about 40%, and per-GPU memory across an 8-way tensor-parallel deployment drops from roughly 88 GB to 52 GB, which leaves room for around 1.18 million tokens of KV cache on the same hardware. Across our evaluation suite, INT4 and FP8 weights are indistinguishable: Benchmark / Capability Metric FP8 INT4 GSM8K Exact match 94.39% 93.56% GSM8K Flexible 94.24% 93.48% ARC-Easy Accuracy 86.62% 86.15% ARC-Easy Acc (norm) 84.51% 85.19% ARC-Challenge Accuracy 64.93% 64.85% ARC-Challenge Acc (norm) 67.24% 66.64% MMLU Average 86.60% 86.54% MMLU-Pro Exact 80.80% 80.47% mcxams (internal benchmark) Passed 62 / 63 62 / 63 Smaller weights make the decode phase faster, and for a clear reason: generating each token means streaming the model's weights out of GPU memory, so decode speed is limited by memory bandwidth. Move less data and every token arrives sooner. The effect is largest at low concurrency, where per-request latency matters most: Concurrent requests GLM FP8 (tok/s) GLM INT4 (tok/s) INT4 gain 1 60 92 +55% 8 425 513 +21% 16 683 825 +21% 32 994 1,267 +27% 64 1,672 1,933 +16% Prefill behaves differently. It is compute-bound, and INT4 weights have to be expanded back out before the model can multiply with them, so that extra step makes prefill slower rather than faster, GLM sustains about 10,160 tokens per second of prefill in FP8 versus 8,660 in INT4. As with the KV cache, the disaggregated design turns this into a choice rather than a compromise: we run INT4 for decode, where it wins, and FP8 for prefill, where it wins. Model accuracy stays within 0.8 points of the FP8 model across every benchmark we run, making its quality indistinguishable. Protecting a shared KV cache Both techniques above have the same effect: they let many more requests share one GPU's memory at the same time. That efficiency is the whole point, but it also means hundreds of requests are reading and writing pages of the same physical KV cache. The mechanisms that make this fast, paged attention, continuous batching, cache reuse, all rely on getting the bookkeeping exactly right, and at our request volumes, even a one-in-a-billion mistake would show up regularly. So we built KV cache integrity checking as a layer of defense. The idea is straightforward: every physical cache page gets a tag that changes whenever the page is reallocated, and the server records which pages and tags each request expects to use. Before supported decode operations read from the cache, those mappings are checked. If anything doesn't match, the affected request is aborted rather than allowed to return data from the wrong page. The question that decides whether a safety check ships is what it costs. We measured it on a mid-sized production model in a two-prefill, two-decode configuration, with 8,192-token inputs and 1,000-token outputs: Concurrency Throughput change p95 latency change 1 −0.53% +0.42% 2 −0.38% +0.54% 4 −0.79% +0.63% 8 −0.43% +0.80% The cost is under 1% on both throughput and tail latency, and even the upper bound of the 95% confidence interval stays near 1%. We kept it computationally cheap by running the validation as a separate batch check rather than fusing it into the attention kernel, which would have introduced a race between GPU thread groups. It's enabled per deployment, and the default path uses a no-op tracker with no measurable overhead, so deployments that don't need it pay nothing. What's next Serving frontier models efficiently is a moving target, and this is the ongoing work behind it. We're expanding FP8 KV caches across more of the fleet, validating NVFP4 weights on Blackwell (NVIDIA’s GPU architecture), and working toward making integrity checks something we can leave on everywhere at negligible cost. These optimizations will allow us to continue to support more customers at a lower cost and at the same accuracy. If squeezing the best open models onto GPUs and serving them to millions of developers sounds like your kind of problem, come work with us . (207 points, 50 comments on Hacker News)