The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

來源原標題

The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

作者

Latent Space

原始標籤

blog

原始連結

https://www.latent.space/p/inference-eng

摘要原文

We first covered Baseten last year when DeepSeek mania was at peak hype. Now they have raised a monster $13B round and become one of the new cohort of AI Infra decacorns that are (with Nvidia, Intel, and the semis complex) chief beneficiaries of the Inference Inflection . We return to Baseten at the peak of the 2026 edition of Open Weights debate . Ali has published a viral breakdown of Kimi K3 : And since you last saw him, Philip has spoken at AI Engineer and written the definitive book on Inference Engineering spotted all over SF: Three years ago, inference engineering barely existed as a category. Today, it is one of the most critical disciplines in AI. Inference engineering inherently tackles a different question than standard model training: “How do you turn those weights from training into a product that is fast, reliable, and affordable at scale?” Focusing on these creates an entirely new optimization problem. In one recent GLM-5.2 experiment, quantizing more of the model actually preserved its benchmark quality while increasing throughput by 20% , because the errors introduced in different layers could cancel each other out. Inference is no longer just the final step after training. It is becoming its own engineering discipline, with its own research problems, infrastructure, and increasingly specialized roles. In this episode, Baseten’s Philip Kiely and Ali Taha join swyx and Vibhu to explain what actually happens after a new open model is released and what it takes to turn “we generated a token” into a fast, reliable, production-ready API. We go deep on cache-aware routing, disaggregated prefill and decode, quantization, speculative decoding, KV-cache movement , model parallelism, GPU kernels, and the race to make frontier models up to 10× faster. Philip and Ali explain why inference optimizations can still produce gains of 20%, 100%, or even 200% ; how quantization errors can cancel one another out; why identical weights can behave differently across clusters; and how Baseten grafted a Kimi vision encoder onto GLM-5.2 without changing the underlying language model. The conversation then expands beyond LLMs into NVIDIA Dynamo , mega kernels, Rubin, AI-specific chips, local inference, video generation, diffusion versus autoregressive models , and the enormous compute barrier to generating coherent long-form video. Finally, we explore the convergence of training and inference , continual learning through persistent KV cache, and the emerging loop where models help optimize the infrastructure that runs them. What happens when a 200,000-token request enters an inference system Cache-aware routing and reusing previously computed KV cache Why prefill and decode are increasingly handled by different GPUs When dedicated deployments become cheaper and more reliable than shared APIs How speculative decoding uses a smaller model to accelerate a larger one Tool calling , structured outputs, and what LLMs actually do What it takes to support a new open model on day zero Grafting Kimi’s vision encoder onto GLM-5.2 Retrofitting inefficient model layers with components from other architectures Why models sometimes collapse into repeating the same token How hardware, kernels, and race conditions create nondeterministic failures Preserving model fidelity while making inference faster How quantization errors can cancel each other out Why inference optimizations still deliver gains of 20%, 100%, and 200% How optimized serving can make a model up to 10× faster NVIDIA Dynamo , KV-aware routing, and distributed model serving Speculative decoding the speculative decoder Why local AI is about making models less dumb while data-center AI is about making them less slow Tensor, expert, and pipeline parallelism across GPUs Hardware-aware model design, auto-tuning, and the case against mega kernels Rubin and why inference is becoming a systems problem Whether modern GPUs are evolving into programmable AI ASICs Why enormous models like Kimi K3 require GB300-class hardware Why open-source video generation still trails Veo, Kling, and other closed models The quadratic attention bottleneck behind long-form AI video Autoregressive video, real-time generation, and compounding quality drift Why future video systems may combine autoregressive and diffusion architectures Training for inference and inference for training Continuous post-training, deployment, evaluation, and improvement loops How GLM-5.2 helped optimize the kernels serving GLM-5.2 itself Why faster networking could unlock dramatically faster decoding Continual learning, KV-cache compaction , and persistent model memory How to build a day-0 API for Kimi K3 22580: From GPT2 to Kimi3, Explained LinkedIn: https://www.linkedin.com/in/philipkiely Inference Engineering: https://www.baseten.co/inference-engineering/ LinkedIn: https://www.linkedin.com/in/aliestaha/ X: https://x.com/waterloointern 00:00:00 Introduction and the 200K-Token Prompt 00:03:18 Dedicated Deployments, Speculative Decoding, and Tool Calling 00:11:26 Launching Production-Ready Open Models 00:19:06 Model Retrofits, Failure Modes, and Nondeterminism 00:28:22 Quantization and Canceling Errors 00:32:15 The Race to 10× Faster Inference 00:40:48 Dynamo, Speculation, and Local vs. Data-Center AI 00:50:18 Model Parallelism, Auto-Tuning, and Mega Kernels 01:00:55 Rubin, GPUs vs. ASICs, and Custom AI Chips 01:10:03 Giant Models and the Limits of GPU Memory 01:12:42 AI Video, Quadratic Attention, and Autoregressive Generation 01:21:47 Audio, Images, and Diffusion Models 01:27:32 Training, Self-Optimizing Models, and Continual Learning Swyx [00:00:00]: Okay, we’re here in the studio with Philip, old friend from Inference Engineering, the book, as well as Baseten and everything that you’ve done, you and I have done before, as well as Ali. Welcome. Ali [00:00:15]: Pleasure to meet you. Swyx [00:00:15]: Waterloo intern. Ali [00:00:16]: Waterloo intern, always. Swyx [00:00:17]: When did you get “Waterloo intern” as a handle? Ali [00:00:19]: As a handle? Oh. Ali [00:00:20]: I think the rebranding happened mid-March. When I saw it was open, I was like, “I have to take it. Up for grabs.” Philip [00:00:26]: The problem is that Ali is really good at his job and is not gonna be an intern much longer. Philip [00:00:30]: So we have to figure out who’s gonna get the handle. Ali [00:00:33]: Well, I’ll pass the torch over to the next intern. Swyx [00:00:34]: Oh, okay. It can be, like, you just pass it to another Waterloo grad. Ali [00:00:37]: To another Waterloo intern. No, bruh. Swyx [00:00:40]: Intern, yeah. Philip [00:00:41]: You gotta get an intern from Waterloo. Ali [00:00:42]: Yeah, I’ve gotta get an intern from Waterloo. Ali [00:00:44]: But they have to follow the path. Swyx [00:00:45]: Oh, it could, but it could come from Baseten, so it’s like whoever Baseten gets from Waterloo. Swyx [00:00:49]: Has the title of Waterloo. Ali [00:00:50]: It stays in the ecosystem. Ali [00:00:52]: Halfway through the internship, you either get it or you’re out. Philip [00:00:55]: You should also do, like, a big graduation ceremony where you change the handle. Philip [00:00:59]: For everybody. Swyx [00:01:00]: You guys are good at ceremonies, clearly. We had a nice launch of the book, very successful. But before we get into all that, I wanna start off with a fun question for you. Okay, you’re an expert inference engineer. What happens when I send a long query, say two hundred thousand tokens into Baseten’s inference? What’s the process of query through GPU model routing, balancing, all that? What is all the stuff that we don’t think about? Philip [00:01:26]: With a long query specifically, the first thing that I’m gonna ask is, “Have you sent me this query before, or at least part of it?” and I really hope you have, because it’s gonna be a lot easier for me and a lot cheaper for you.