Jev: System One models for Prod, not God — with Diogo Almeida, CEO, TypeSafe AI
https://www.latent.space/p/jev📌 InstructGPT 共同作者推 Jev：拋棄 RLHF、轉向 RLCD，為讓 AI 真正能在 Production 跑起來
TL;DR：TypeSafe AI 推出 System One 模型 Jev，用 RLCD 取代 RLHF，針對「校準決策」而非「人類偏好」最佳化，要讓 AI 像 regex 一樣消失在軟體背景裡。
Diogo Almeida 在發布影片 4,000 萬觀看、佔據 AI timeline 的那週，對 Latent Space 說：「情緒上從未這麼糟，像具殘軀；但理智上覺得整個 AI 領域終於有人看清現實了。」這位 InstructGPT (Ouyang et al., 2022) 共同作者、前 OpenAI 後訓練團隊成員，花了 2023–2024 年試圖修正他眼中的核心缺陷——可靠性，最後決定另起爐灶，創立 TypeSafe AI 並推出 Jev。
🎣 聊天模型贏了世界，卻輸了自動化
ChatGPT 的壓倒性成功，讓業界把所有資源壓在「自迴歸、聊天調教、序列到序列」這條單一路徑上。Diogo 指出：從 Function Calling、Structured Outputs 到 Reasoning，全都是在字串預測典範上打補釘的 hack。結果是：模型能解決納維-斯托克斯方程，卻無法可靠地自動化最基礎的軟體工作——幻覺、阿諛奉承、拒答、校準失效，一旦把模型埋進軟體依賴鏈裡，這些問題就會被放大成系統性風險。
🤔 RLHF 的三條支流與「校準危機」
Diogo 追溯 RLHF 三大源頭：Christiano et al. 2017（機械手臂後空翻）、Stiennon et al. 2020（學習摘要）、Ouyang et al. 2022（InstructGPT，他的作品）。他認為：為了訓練「討人類喜歡的助手」，我們犧牲了「可組合、可程式化、可自動化」的特質。RLHF 最佳化的是人類偏好評分，導致模型學會迎合評分者而非輸出認識論上誠實的機率；RLVR（以程式驗證獎勵）雖能解數學難題，卻加劇「鋸齒智能」，且難與其他軟體整合。
🧩 RLCD：為 System One 任務學「認識論誠實的機率」
Jev 的核心創新是 RLCD（Reinforcement Learning for Calibrated Decisions），一種未公開的新技術。目標很單純：在 System One（快速、直覺、可組合）任務上，輸出「校準過的決策」與「認識論誠實的機率」。
- 不追求人類偏好分數 → 避免幻覺與阿諛奉承。
- 不追求程式可驗證的正確性 → 避免鋸齒智能與整合困境。
- 追求 Calibration（校準）：模型輸出的信心度＝實際正確率，讓下游程式能直接信任並分支。
Diogo 在 AIE 講座中詳細說明：為何一代訓練「有用助手」的過程，反而損害了模型作為自動化元件的能力。
📊 拒絕公開 Benchmark，只看「智能 per Dollar」
TypeSafe 公開拒絕建立 JevBench，也拒絕參與通用排行榜。Diogo 的理由：「You get what you optimize for.」若為了榜單分數調整資料與目標，就會偏離 Production 需求的北極星。他們自稱 Data Lab 而非 Model Lab——挑選正確任務、正確資料，比堆算力更重要（這就是他稱之為「最苦澀的教訓」）。甚至宣稱：「給我 10 億美元我也不會拿去預訓練。」
產品面上，Jev 提供：
- 長期支援（LTS）版本控管，不搞滾動更新驚嚇下游。
- Programming Primitives：Guided Responses、Structured State、Computer Use、Structured Output——用結構化狀態取代巨大 System Prompt。
- Intelligence per Dollar 作為核心 KPI，追求速度與成本效益。
💡 深入分析：從「巨大 Prompt」到「微小、可測決策」
Diogo 主張開發者應將 AI 工作流拆解為 小而可測量的決策單元，再串接成控制流。Jev 的設計哲學是「消失在背景裡」——像 regex 一樣平淡無奇、極度可靠、不需時刻監控。
- Coding Agents：官方指南展示如何用 Jev 重構單一模型架構的 Coding Agent，引入共享狀態、子代理、多代理協作，跳出 KV Cache 瓶頸。
- Dark Data & Computer Use：真實世界的非結構化操作紀錄（Dark Data）是下一波資料金礦，Jev 的 Computer Use 能力正是針對此場景。
- Inverse SaaS-pocalypse：AI 不會取代 SaaS，而是超級充電既有軟體，讓現有應用獲得可程式化的智能層。
⚠️ 限制與現狀
- RLCD 技術細節尚未公開發表，外部無法驗證理論優勢。
- Jev 目前以 API 形式提供，模型權重不開放，生態系仍在早期建構階段（Cookbook、Patterns 剛釋出）。
- 「拒絕 Benchmark」雖有哲學一致性，但也讓外部難以橫向比較基礎能力。
🎯 實務啟示：給工程師的三個行動重點
- 重新審視評估指標：若你的系統把 LLM 埋在關鍵路徑，請測量 Calibration Error（ECE） 而非單看 Accuracy/F1。模型敢不敢說「我不確定」，比答對多少題更重要。
- 拆解 Prompt，建立 State Machine：別再寫 2,000 token 的 System Prompt。把任務拆成「分類 → 擷取 → 決策 → 行動」的微小步驟，每步用 Jev 這類 System One 模型處理，並用 Structured State 傳遞上下文。
- 把「拒答」當作 Bug 處理：在自動化流程裡，模型拒答等於拋出未處理例外。選擇支援「零拒答、可控制輸出分布」的模型，或在上游加上 Guardrail 而非仰賴模型內建對齊。
🔗 來源
- 標題：Jev: System One models for Prod, not God — with Diogo Almeida, CEO, TypeSafe AI
- 作者／機構：Latent Space
- 連結：https://www.latent.space/p/jev
#Jev #TypeSafeAI #RLCD #SystemOne #LLM #AIEngineering #Calibration #InstructGPT #DiogoAlmeida #LatentSpace
原始資料 Latent Space · 收集於 2026-09-22
摘要原文
Tickets for AIE NYC now open, and apply for the invite-only AIE CODE . Join us ! We have an unusual relationship with today’s guest: for years since coauthoring the InstructGPT paper , Diogo Almeida had been saying that API-available frontier models have been going down the wrong path, everything from the alignment to refusals to reliability perspectives, that we have dropped every mode other than autoregressive chat-tuned LLMs because of the overwhelming success of ChatGPT. In a launch video now viewed ~40M times (by comparison, GPT4o was 22M , Fable 5 was 15M , Navier Stokes was 74M , and 6 Astra was 137M ), Diogo introduced Jev and it immediately took over the AI timeline — we’ll skip full Jev explainers because your favorite AI influencer/educator has probably already done one. We also collected: the official patterns and cookbooks you should see first, from Allie speed based - games and computer use the voice + computer use example we discuss at 1h34 mins guided responses in text messages Jev for coding agents has an official guide reasonable pushback from Theo - Diogo has published a note on the Tyranny of the KV Cache that you should read as a followup after the pod for Jev + coding agents, because of his belief that Cache Rules Everything Programming Languages built atop Jev (Diogo’s fave) Jev for analytics replay and user journey review a core goal of Jev is to “disappear into the background” - eg as unremarkable as regex blending transformers and classifiers Jev vs GLiNER (note difference/pushback , agreed , agreed , agreed ) Instead we’ll focus on what we can uniquely offer — a broader philosophical and mission-based understanding of how and why Jev was created , and what you should expect next in terms of future models from TypeSafe ( ReasoningJev ?) and what usecases and ideas you should work on vs the 55th low effort clone of Jev’s API or doing a generic JevBench benchmark - something Diogo has rejected publicly . Diogo knows a good deal about RLHF, given that he was on the team that pioneered post-training at OpenAI — and traces the three branches to Christiano et al 2017 (the robot backflip demo), Stiennon et al 2020 (learning to summarize) and his baby, Ouyang et al 2022 (InstructGPT). From there on, every innovation from Function Calling to Structured Outputs to Reasoning felt like a hack on top of the string based, sequence to sequence prediction paradigm. As he mentions on the pod, from 2023-2024 he struggled unsuccessfully, due to both personal and organization underestimation, to train a model that accurately addressed what he saw as the core problem with making LLMs the heart of software: reliability . Jev’s core innovation is " Reinforcement Learning for Calibrated Decisions ”, a novel, unpublished technique that optimizes for “answers with epistemically honest probabilities on System One tasks” rather than human rated feedback (RLHF) — which causes hallucinations, sycophancy, and permanent reliance on humans — or programmatically verifiable outputs with rubrics (RLVR) — which solves Navier Stokes but exacerbates jagged intelligence and doesn’t integrate well with other software. We’ve talked about the calibration problem before on the pod, but probably the single best place to understand why RLCD became necessary is Diogo’s AIE talk , which discusses why a generation of training helpful AI assistants for humans has impaired them for training models for composable, programmable AI for automation . At the end he also teases his contrarian opinion on scaling laws - which teases how to build a modern neolab without the billions of dollars the major labs have… We spend a good amount of time discussing Diogo’s essay on the Bitterest Lesson : His point is that “You get what you optimize for and the bitterest lesson in ML is that the most important part of it isn’t ML at all.” - and picking the right north star, eg upvoting for user preference vs being integrated into tool calls - makes everything else fall in line. We’re excited to catch up with a freshly dyed Diogo to discuss: Why AI can solve extraordinarily hard problems but still fail to automate basic work What System One Models are and why Jev is built for software rather than chat RLHF, mode collapse , calibration, and the hidden costs of optimizing for human preferences Why refusals become a problem when AI is buried inside software dependencies Why TypeSafe rejects public benchmarks and optimizes for intelligence per dollar The “bitterest lesson”: why the right task and the right data can matter more than compute Why TypeSafe thinks of itself as a data lab rather than a model lab RLCD vs. RLHF and RLVR as fundamentally different North Stars for AI Why reliability and robustness matter more than simple determinism Jev’s programming primitives and how intelligence maps into software control flow Why developers should decompose AI workflows into small, measurable decisions How structured state replaces giant prompts and system messages Why Diogo thinks AI should eventually disappear into the background of software The “inverse SaaS-pocalypse” and how AI could supercharge existing software System One vs. System Two intelligence and the limits of reasoning models Dark data, computer use, real-time intelligence, and Jev’s biggest early use cases Why Jev could reshape coding agents built around a single-model architecture Why Diogo says he wouldn’t pre-train with $1 billion The OpenAI journey that led to TypeSafe and why he thinks many neo-labs are approaching AI incorrectly Coding agents beyond the KV cache , shared state, sub-agents, and the multi-agent future LinkedIn: https://www.linkedin.com/in/diogomda X: https://x.com/CompleteSkeptic TypeSafe AI: https://typesafe.ai/ 00:00:00 Jev Launch Week and the AI Economic Revolution 00:02:50 What Is Jev? System One Models and Programmable AI 00:05:54 RLHF, Mode Collapse, Calibration, and Yann LeCun 00:10:29 Programmatic AI, Refusals, and Safety Alignment 00:17:21 Why TypeSafe Rejects Public Benchmarks 00:20:43 The Bitterest Lesson: Data, Compute, and the Right Task 00:24:59 RLCD vs. RLHF and RLVR 00:28:42 Why Powerful AI Still Hasn’t Automated the Economy 00:39:55 Reliability, Robustness, and Determinism 00:48:11 Model Versioning, LTS, Speed, and Intelligence per Dollar 00:54:04 Inside Jev’s API and Programming Primitives 00:58:28 How to Build with Jev: Structure, Decomposition, and Small Decisions 01:18:28 The Inverse SaaS-pocalypse and AI Disappearing into Software 01:33:21 Computer Use, Dark Data, and Jev’s Biggest Use Cases 01:38:48 How Jev Could Reshape Coding Agents 01:41:00 AI Safety, Frontier Pacing, and the Limits of RLVR 01:48:03 Why Diogo Wouldn’t Pre-Train with $1 Billion 01:55:19 The OpenAI Story Behind TypeSafe 02:01:41 Why Diogo Thinks Most Neo-Labs Are Getting AI Wrong 02:08:00 Coding Agents Beyond the KV Cache and the Multi-Agent Future Swyx [00:00:00]: Okay, we’re in the studio. A special occasion because this week, Diogo, my good buddy, launched Jev, and it’s been taking over the complete timeline. How do you feel? What’s it like to be you right now? Diogo Almeida [00:00:16]: Emotionally? Diogo Almeida [00:00:17]: Never been worse. Like, I’m a ragged corpse of a person right now because there’s so much going on, and I’m like a technical CEO, so I have, like, a lot of fires to fight. Diogo Almeida [00:00:29]: But mentally, I feel—I say this all the time, and I’ve been saying this kind of for years in my over-under events. Like, I feel like the entire AI field is like one of those, like, carnival house of mirrors, and everyone is just insane and saying the weirdest stuff that doesn’t make sense. And it feels like for just this week, like, I’m on a better in sync with reality and like, oh, people see it now. AI can be so much more than what was once thought. Diogo Almeida [00:01:06]: And like, yes, we are going to make. Like, an AI-based economic revolution is back on the table, and this is fucking awesome.
由 nvidia/nemotron-3-ultra-550b-a55b:free 自動生成