Perplexity Trains Its Computer Agent on Real Mistakes With Hint-Guided Self-Distillation
https://www.marktechpost.com/2026/09/25/perplexity-trains-its-computer-agent-on-real-mistakes-with-hint-guided-self-distillation/📌 Perplexity 讓模型從自己的失敗紀錄中學糾錯
TL;DR：Perplexity 用「提示引導自我蒸餾」訓練 agent，把工具呼叫失敗率壓低了 21.2%,但方法沒開源。
一個 agent 明明答對了任務，卻可能是因為它在中途犯了錯又僥倖救回來。如果只模仿「成功的完整軌跡」，等於連那個錯誤動作也一起學進去了。Perplexity Research 這篇最新的後訓練研究，就是想解決這個矛盾。
🤔 成功的軌跡裡藏著該丟掉的錯誤
Perplexity Research 發表一份新的後訓練研究，直接用 Perplexity Computer 裡真實使用者的 session（包含失敗的 session）訓練模型。標準的拒絕採樣微調（rejection sampling fine-tuning，RFT）做法是評判每個 session，只模仿成功的那些。但「成功」不代表每一步都正確,agent 完全可能在某次工具呼叫犯錯後又自行修正,最終給出正確答案,若把整段軌跡照單模仿,反而會把那個錯誤動作也一併強化。反過來，直接丟掉失敗的 session，又等於白白浪費了「這裡有個明確可避免的錯誤」這個清楚的訓練訊號。
🧩 拆成兩個決策：模仿什麼、糾正什麼
Perplexity 團隊把問題拆成兩個獨立決策：哪些 session 裡的行為值得模仿，哪些 turn（對話輪次）裡的錯誤值得糾正。成功的 session 可以同時提供模仿目標與糾正目標，失敗的 session 則只提供糾正目標。
糾正的核心是一種「提示（hint）」機制：hint 是一段簡短的糾正指示，且必須根基於模型當下已經擁有的資訊。文中舉例，一次搜尋呼叫把 recency_filter 設成 ‘year’，但 schema 只允許 ‘day’、‘week’ 或 ‘month’，hint 會指出是哪次呼叫失敗、附上驗證錯誤訊息，並建議一個允許值或直接省略該可選欄位。
糾正的訓練機制稱為 On-Policy Self-Distillation（OPSD）：訓練器對同一筆記錄的 turn，用同一個 GLM 5.2 checkpoint 跑兩次，teacher 那次看得到 hint，student 那次看不到，兩者都使用 teacher forcing，因此不會生成替代答案。teacher 的下一個 token 機率分布被 detach（脫離梯度）後，透過前向 KL 散度作為 student 的軟目標。合併後的損失函式是 (CE + λ × KL) 除以被模仿的 token 數，當 λ 設為 0 就退回標準的 SFT。CE 項不可或缺,因為若只靠糾正項訓練,teacher 與 student 有可能靠忽略上下文本身就達成一致,反而學不到真正有用的訊號。
📊 A/B 測試：工具呼叫失敗率降了 21.2%
在一次上線的 A/B 測試中，兩個訓練 checkpoint 之間，工具呼叫失敗率從 2.24% 降到 1.77%，Perplexity 團隊表示這是統計上顯著的 21.2% 相對降幅。
資料管線只取自 GLM 5.2 服務、且符合訓練資格的 Computer session，含個資（PII）與已選擇退出的使用者一律排除。一個 LLM judge 先篩出難度落在 5 分量表中 4 或 5 分的任務；判斷一個 session 是否算「成功」，須經過兩個 LLM judge 都認可最終交付結果；針對使用者回饋，則由三個 LLM judge 定位出真正該負責的 turn，至少兩個判斷一致才算數。這個機制的重要性在於，文中提到使用者抱怨前的最後一個 assistant turn，只有大約一半的機率才是真正的問題根源。每一條 hint 也會反向檢查，確保它只使用「錯誤發生之前」模型就已掌握的資訊，藉此降低後見之明偏誤（hindsight bias）。文中另舉一例：使用者在 Paychex 上詢問自己的「w3」，模型誤判為 W-2 的筆誤而搜錯表單，對應的 hint 針對的是「更早的理解錯誤」，而不只是修正最終答案。
⚠️ 目前無法直接部署
Perplexity 尚未釋出後訓練後的權重或訓練程式碼，這個模型目前只作為 Perplexity Computer 裡的一個模型選項存在；底層的基礎模型 GLM 5.2 本身則是在 Hugging Face 上公開可取得的。
🎯 實務啟示
這套「模仿與糾正目標分離」的思路，對自建 agent 訓練 pipeline 的團隊有直接參考價值：與其把失敗 session 整批丟棄，不如區分「哪個 turn 出錯」與「哪個 session 的整體行為值得模仿」，再用根基於既有資訊的 hint 做 teacher forcing 式的自我蒸餾。可惜方法雖具體，訓練程式碼與權重都未開源，想要複現目前只能依賴論文本身描述的損失函式與資料篩選流程自行實作。
🔗 來源
- 標題：Perplexity Trains Its Computer Agent on Real Mistakes With Hint-Guided Self-Distillation
- 作者／機構：Michal Sutter，MarkTechPost
- 連結：https://www.marktechpost.com/2026/09/25/perplexity-trains-its-computer-agent-on-real-mistakes-with-hint-guided-self-distillation/
#Perplexity #SelfDistillation #PostTraining #AgentTraining #RejectionSampling #LLM #GLM #AIAlignment #ToolUse #MachineLearning
原始資料 MarkTechPost · 收集於 2026-09-25
摘要原文
Perplexity Research published a new post-training study . It trains a model inside Perplexity Computer on real user sessions, including failed ones. The method pairs rejection sampling fine-tuning with hint-guided self-distillation. In a live A/B test, tool-call failures fell from 2.24% to 1.77% between 2 trained checkpoints. Perplexity team reports this as a statistically significant 21.2% relative reduction. Is it deployable? Not directly. Perplexity has not released the post-trained weights or training code. The model runs only as a model option inside Perplexity Computer. The base model, GLM 5.2 , is openly available on Hugging Face. Standard rejection sampling fine-tuning (RFT) judges each session and imitates only the successful ones. A successful outcome does not mean every step was correct. An agent can recover from a bad tool call and still deliver the right answer. Imitating that full trajectory can reinforce the error. Discarding failed sessions also throws away clear evidence of avoidable mistakes. Perplexity team separates 2 decisions: which sessions hold behavior worth imitating, and which turns hold mistakes worth correcting. Each assistant turn gets 1 of 3 treatments: Successful sessions can supply both imitation and correction targets. Unsuccessful sessions supply only correction targets. A hint is a short corrective instruction grounded in information the model already had. In one example, a search call set recency_filter to ‘year.’ The schema allowed only ‘day,’ ‘week,’ or ‘month.’ The hint names the failed call, includes the validation error, and suggests an allowed value or omitting the optional field. The corrective part uses On-Policy Self-Distillation (OPSD). The trainer runs the same GLM 5.2 checkpoint twice on the recorded turn. The teacher pass sees the hint; the student pass does not. Both use teacher forcing, so no replacement answer is generated. The teacher’s next-token probabilities are detached and act as a soft target through forward KL. The combined loss is (CE + λ × KL), divided by the number of imitated tokens. Setting λ to 0 recovers standard SFT. The CE term matters. Correction-only training can let teacher and student agree by ignoring context. The pipeline draws from training-eligible Computer sessions served by GLM 5.2. Sessions with personally identifiable information and users who opted out are excluded. An LLM judge keeps tasks rated 4 or 5 on a 5-point difficulty scale. Two LLM judges must both approve the final delivery for a session to count as successful. For user feedback, threes LLM judges locate the responsible turn, and at least 2 must agree. This is important because the last assistant turn before a complaint is the root cause only about half the time. Each hint is also checked against information available before the mistake. That check reduces hindsight bias. One example: a user asked for their ‘w3’ on Paychex. The model assumed a W-2 typo and searched for the wrong form. The hint targets that earlier interpretation, not just the final answer. Check out the Technical Details . All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter . Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Perplexity Trains Its Computer Agent on Real Mistakes With Hint-Guided Self-Distillation appeared first on MarkTechPost .
由 claude-code/sonnet 自動生成