Why I'm still bearish on LLMs after Navier-Stokes

來源原標題

Why I'm still bearish on LLMs after Navier-Stokes

作者

jaykru

原始標籤

hackernews

來源訊號

HN points 437　HN 留言 569

原始連結

https://dank.systems/posts/2026-09-15-ai-bear.html

摘要原文

why i'm still bearish on LLMs after navier-stokes Why I'm still bearish on LLMs after Navier-Stokes orthography mode: based | cringe cringe mode is provided with AI assistance, switch to based mode if you prefer artisanal tokens. [ thank you to claude fable 5.1, holden saberhagen, gabriel kammer, andres erbsen, alice mckean, and tristan wylde-larue for comments on this post ] i'll begin with a few theses for the reader to chew on: the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers, but current frontier models need laborious oversight and guardrails on even the simplest tasks. one misled by the headline shows of force (navier-stokes, freebsd RCEs, the huggingface incident) and frontier lab rhetoric into believing meaningful autonomy has been achieved need only look at the software firms continuing to employ and hire bottom quartile software engineers who would score far below the models they supervise on the benchmarks du jour. the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data; but even small perturbations within a covered class of task result in outright failure or reward hacking. the present problem of reward hacking can be solved only by rigorous specification by domain experts. the time of domain experts is expensive. rigorous specification is itself a skill, demanding its own expertise outside of a given problem domain. even many skilled software engineers are bad at it. for the vast majority of domains, the intersection of domain experts and specification experts is ludicrously small. the labor costs of rigorous specification can greatly exceed that of direct implementation of an informal specification. the hardware engineering world presents a great case study on this, where a typical CPU project anecdotally has about three times as many specification and validation engineers as design engineers and a 5:1 ratio is not unheard of . even worse, many tasks don't admit a convenient spec-and-forget regime where you write a specification once and continuously implement against it: rigorous formal specifications frequently evolve in conversation with insights derived from discoveries made while implementing according to the informal specification. for tasks that enjoy high level one-and-done specifications (say an executable ISA specification for a family of CPU architectures) the costs of verification against such high level specifications are insurmountable with current technology, necessitating the use of lower level specifications that are both more expensive to construct and far more fragile to design flux. navier-stokes and statements in pure mathematics like it are the absolute best case scenario for agentic work against rigorous specification. the theorem statement itself is already a rigorous specification. it has undergone decades of auditing by the mathematical community and its rendering in lean is a straightforward translation defined in terms of battle-tested mathematical objects from mathlib. the verifier, the lean theorem prover, has been extensively audited and specifically designed to avoid the types of unsoundness that would make it vulnerable to reward hacks. even lean and theorem provers like it are not invulnerable: soundness bugs have allowed LLMs to launder bogus proofs through the proof kernel before and it is not improbable that more such bugs exist. this is the rosiest setup; the vast majority of human knowledge work does not look like this. i'll comment below on the few areas of knowledge work that do resemble pure mathematics in this respect. the best alternative to rigorous specification is human review. human review doesn't scale well to the volumes of output produced by language models. to make matters worse, even expert human review is extremely vulnerable to reward hacking: consider the xz backdoor and the infamous UMN hypocrite commits that landed in linux. if human review remains a critical part of the agentic production loop, the pace of production is necessarily bottlenecked by factors like the limits of human time and attention; it is a total non-starter for the country full of geniuses in a datacenter frontier lab CEOs would have you believe is perpetually just a few more months out. taken together, it appears that for most domains LLMs will continue to look like a cracked intern: quick and effective in the hands of an adult but not given run of the place. most firms will not be able to adopt fully autonomous AI, not for problems of skill issue or lagging technology diffusion but rather for structural reasons seemingly endemic to current architectures. the classes of firms that can accept the use of fully autonomous LLMs are few, by my count just three: those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc. those who need done a small set of narrowly defined tasks with existing clear guardrails: repetitive physical labor in a controlled environment, call center and customer service chat work, etc. those that can accept or already do by nature the costs of rigorous specification and validation: chip design, drug discovery, and other domains where failure on deployment is an existential concern. the first two classes are price sensitive and arguably don't need the jump in reasoning quality you see going from cheap to frontier models. most of these firms will be best served by open models running on cheap hardware, perhaps even locally at the site of use. for the first and third classes, the type of fuzzy combinatorial search that has produced headline results in mathematics and security research seems more sensitive to agentic swarm width than reasoning capacity: see small open models reproducing the mythos CVEs that drove the spring 2026 hype cycle. if that is indeed true, there is even greater reason to use cheap open models that enable you to run the same workload with wider swarms. the third class of firms might still use frontier models, though it's not totally clear that their work couldn't be done with cheap models like deepseek v4.1 flash, and the swarm width advantage i hypothesized above gives them all the more reason to push for cheaper models. another interesting property of firms of this class is that they are generally very secretive about their IP and probably aren't overjoyed about shipping it all to anthropic and openai even with supposed agreements to not train on user data . now, you might propose that even if the frontier labs are cooked, the data center full of brainlets scenario drives just as much AI compute as an artificial superintelligence scenario. the difference is that the data center full of geniuses is self-driving and limited only by how much compute it can consume while the brainlet swarms will be heavily bottlenecked by their human orchestrators. my personal bet is that the blast radius will go far beyond the frontier labs. [ Thank you to Claude Fable 5.1, Holden Saberhagen, Gabriel Kammer, Andres Erbsen, Alice McKean, and Tristan Wylde-LaRue for comments on this post ] I'll begin with a few theses for the reader to chew on: The frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers, but current frontier models need laborious oversight and guardrails on even the simplest tasks. (437 points, 569 comments on Hacker News)