[AINews] Megakernels are so dead and so back

來源原標題

[AINews] Megakernels are so dead and so back

作者

Latent Space

原始標籤

blog

原始連結

https://www.latent.space/p/ainews-megakernels-are-so-dead-and

摘要原文

Part of our Inference Engineering Masterclass pod yesterday involved a spicy discussion about Megakernels: megakernels are dead why are megakernels useful? you spend two months writing a kernel to save time on launch overhead and poor inter-kernel overlap. you had PDL but then people said it wasn't perfect, that you could still get some marginal gains due to straggler CTAs and therefore- wait, sorry, I forgot, Rubin fixes that (kernel two needs 10 CTAs and kernel one has seven finished and three straggling, kernel two launches seven of its CTAs). given a long enough timeline, it all evens out. no serious inference provider is using a 67k loc hand-fused forward pass kernel in production, and the teams doing that are doing so out of pure research. The full discussion, for those who care to listen through: Ali: A fused kernel can’t save you . Like here with tensor parallelism, half the matrix is on one GPU and the other half is on another, and if I need the entire matrix in order to do like a nonlinear operation in the next step, which is, for instance, like if I’m doing attention, I need the softmax, or I need to do like exponentiation, I need to have the entire row. So I need to know what the partial result was from GPU 2 and what the partial result was from GPU 1 in order to be able to do the softmax in the next stage. So I have to make them communicate with each other, even if I had a fused kernel, because of the nonlinearities within each one . Also with like mega kernels, like honestly, I’m very bearish. It was a good research direction, and it seems like intuitively, theoretically, it’s nice. You have a lot of launch overhead from launching- Just- one kernel- Yeah, just keep fusing it and moving the data. Just fuse everything together. But the kernel complexity itself is very difficult to write a very optimized mega kernel. It’s very difficult to do so. And not to name any companies , but like even the companies that have worked or people that I’ve spoken to who work at companies that do fused mega kernels, they very often don’t end up running those in production because the TensorRT-LLM and modular kernels that launch are faster because you can optimize each individual component, and you can just have them parallelize with each other. One of the tech leads at NVIDIA launched a Twitter post said like, “We’re pulling the curtain on Rubin, and here’s the specs.” And the third tweet showed, like not to get too technical into it, I and I need to read it much more, but the GPU is designed in such a way that it kills mega kernels. So it seems like that entire research field won’t be continued. He was quoting ( friend of the show! ) Kyle Kranen announcing dependency triggers - one part of the pipeline blockage that previously justified kernel fusion: As voiced on the show, there are still physical constraints that are unanswered, but it makes complete sense that Nvidia is updating Rubin design to better fit macabre things that are being done in kernel-land. One of Ben Spector’s megakernel coauthors , Stuart Sul, is now leading the team that released Mixture of Kittens (a reference to Ben’s delightfully named ThunderKittens , and part of Dan Fu’s group ), Cursor’s open source megakernel today: Headline results are compelling - a 41% increase in overall tokens per second. At scale, this translates to billions of dollars worth of savings. AI News for 8/3/2026-8/4/2026. We checked 12 subreddits, 544 Twitters and no further Discords. AINews' website lets you search all past issues. As a reminder, AINews is now a section of Latent Space . You can opt in/out of email frequencies! Frontier Model Releases: Qwen 3.8-Max, Alpamayo 2 Super, Pokee-Isaac, Maple-Preview, and Shieldstral Qwen’s release cadence continues across modalities : @Alibaba_Qwen launched Qwen3.8-Max as “better and cheaper,” and quickly pushed it into agent ecosystems via Hermes Agent , Nous Research , and ClinePass . On the vision side, @skalskip92 highlighted Qwen3.8-Max’s box-conditioned detection behavior, reporting 60% mAP with a single box and 80% with multiple boxes for hard-to-describe concepts; Qwen’s image stack also moved up, with @arena and @Alibaba_Qwen noting Qwen-Image-3.0-Pro reached #5 in the Text-to-Image Arena. NVIDIA and Mistral both leaned into deployable specialization : @JensenHuang introduced Alpamayo 2 Super for AV reasoning with commercial-use open release terms, while @MistralAI launched Shieldstral , a 3B open-weights safety model designed for on-device moderation/classification. @vllm_project shipped day-0 serving support and highlighted one-forward-pass safety scoring, multimodal input, 12 languages , and 32k context . Long-context and efficient-weight experimentation accelerated : @Pokee_AI released Pokee-Isaac 28B , claiming a 10M-token context , 93.3% RULER at 10M , and single-GPU deployability starting from an RTX 4090; the model is also said to have day-0 support in vLLM and SGLang . Meanwhile @deepgrove_ai introduced Maple-Preview , an open-source 20B-A1B ternary-weight reasoning model said to run at 200+ tok/s on a Mac Mini M4 and outperform others in its weight class. Both releases point to a growing split: not just bigger frontier models, but aggressive exploration of context architecture and low-bit/ternary efficiency. Inference Economics, Routing, and Kernel/Serving Infrastructure Pricing pressure is now changing product design : The permanent Luna repricing from @thsottiaux triggered immediate discussion about always-on helper workloads; @theo described Luna as cheap enough to spin up on nearly every prompt for metadata/status generation. In parallel, several posts stressed just how dominant DeepSeek-V4-Flash is on price: @kimmonismus , @AndrewCurran_ , @ollama , and @EpochAIResearch all reinforced the idea that open(-weight) or quasi-open serving economics are now competitive enough to shape stack choices, especially for high-volume agent workflows. Routing is becoming a first-class systems problem : @tomas_hk launched Not Diamond Code , a router for long-horizon coding agents that selects both model and reasoning effort per step, claiming 20–65% cost reduction without quality loss. Similar themes showed up in @cognition , where Devin Fusion became 4% more intelligent and 27% cheaper on FrontierCode 1.1 thanks to harness/model improvements, and in @togethercompute , which reported that a Kimi-first cascade with test-suite verification outperformed Sol alone at lower cost on DeepSWE. The infra layer got meaningfully deeper : @cursor_ai open-sourced MoK , its NVL72 MoE training megakernel, with the most concrete performance claim of the day in training systems. @ArtificialAnlys added a new Endpoint Accuracy Index , benchmarking how much accuracy serverless endpoints preserve relative to self-hosted reference deployments; one practical takeaway was that output-token limits and tool-call formatting differences materially degrade endpoint quality. On the serving side, @kimmonismus highlighted Celeris-1 as topping Artificial Analysis speed rankings at roughly 2,086 tok/s while staying in the 75.9% MMLU-Pro range on commodity GPUs, and @vllm_project reminded engineers that native Transformers models can now load into vLLM without custom integrations. Agent Harnesses, Self-Improvement Loops, and Tooling for Production Agents Training inside the harness is becoming normal rather than novel : @liquidai described LFM2.5-2.6B as being post-trained through real agent harnesses—SFT, expert specialization, multi-domain on-policy distillation, and agentic RL using Pi , Hermes Agent , and OpenClaw , with per-rollout sandboxing and outcome rewards. The model was then positioned by @maximelabonne , @nicodotdev , @OsaurusAI , and others as a genuinely usable small agentic model for local/background workflows.