DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
https://machinelearning.apple.com/research/deepambigqa-multihop-questions📌 【Apple ML 研究】LLM 難以處理歧義問題，連 GPT-5 的回答完整度也令人擔憂
TL;DR：新數據集 DeepAmbigQA 指出，LLM 在處理需解決「名稱歧義」與「多跳推理」的複雜問題時，回答完整度極低。
🤔 複雜問題對 LLM 的雙重挑戰
當問題變得複雜，例如：「哪位《熱血拼搏》(Heat) 的演員曾獲得至少一座奧斯卡獎？」這類問題對 LLM 提出了兩個嚴苛要求：
- 辨識歧義：必須從多部同名電影中區分出正確的那一部。
- 多跳推理 (Multi-hop reasoning)：必須跨越大量演員資料，整合證據以找出正確答案。
目前的問答 (QA) 基準測試（Benchmarks）鮮少能同時對這兩項挑戰進行評估。
🧩 DEEPAMBIGQAGEN：自動化生成流水線
為了應對此問題，研究團隊提出了 DEEPAMBIGQAGEN，這是一個自動化數據生成流水線。該流程以文本語料庫與關聯知識圖譜 (Linked Knowledge Graph) 為基礎，能生成既自然又可驗證的問題，並系統性地將「名稱歧義」與「多步推理」嵌入其中。
📊 DeepAmbigQA 數據集與實驗結果
研究團隊基於此流水線構建了 DeepAmbigQA 數據集，其中包含 3,600 個問題，其中一半的問題需要解決顯性的名稱歧義。
實驗發現，即使是目前最頂尖的模型，在處理這類問題時表現依然不佳：
| 問題類型 | 精確匹配 (Exact Match) 分數 |
|---|---|
| 具備歧義的問題 | 0.13 |
| 非歧義的問題 | 0.21 |
實驗結果顯示，即便是 GPT-5 在處理具備歧義的問題時，其精確匹配分數僅有 0.13，顯示出模型在資訊蒐集與回答完整性方面的不足。
⚠️ 對資訊蒐集與完整性的需求
這些研究結果強調，未來的問答系統需要更強大的能力，以確保在面對複雜情境時，能夠進行穩健的資訊搜集並提供完整的回答。
🎯 實務啟示
對於開發者而言，這提醒我們在設計整合搜尋工具的 LLM 應用時，不能僅依賴模型內建知識，必須加強模型在面對歧義實體時的檢索與推理能力，以確保回答的精準度與完整性。
🔗 來源
- 標題：DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness
- 作者／機構：Jiabao Ji, Min Li, Priyanshu Kumar, Shiyu Chang, Saloni Potdar @ Apple / University of California, Santa Barbara
- 連結：https://machinelearning.apple.com/research/deepambigqa-multihop-questions
#AI #LLM #DeepAmbigQA #AppleML #MachineLearning #QuestionAnswering #MultiHopReasoning #NLP #KnowledgeGraph #AIResearch
原始資料 Apple ML · 收集於 2026-08-07
摘要原文
research area Knowledge Bases and Search , research area Speech and Natural Language Processing content type paper published August 2026 DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness Authors Jiabao Ji†**, Min Li, Priyanshu Kumar, Shiyu Chang†, Saloni Potdar View publication Copy Bibtex Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such as “Which actor from the film Heat won at least one Academy Award?”, which requires (1) distinguishing between multiple films sharing the same title and (2) reasoning across a large set of actors to gather and integrate evidence. Existing QA benchmarks rarely evaluate both challenges jointly. To address this, we introduce DEEPAMBIGQAGEN, an automatic data generation pipeline that constructs QA tasks grounded in text corpora and linked knowledge graph, generating natural and verifiable questions that systematically embed name ambiguity and multi-step reasoning. Based on this, we build DEEPAMBIGQA, a dataset of 3,600 questions requiring multi-hop reasoning and half of them explicit name ambiguity resolving. Experiments reveal that, even state-of-the-art GPT-5 show incomplete answers, achieving only 0.13 exact match on ambiguous questions and 0.21 on non-ambiguous questions. These findings highlight the need for more robust QA systems aimed at information gathering and answer completeness. † University of California, Santa Barbara ** Work done while at Apple Related readings and updates. Can Open Domain Question Answering Models Answer Visual Knowledge Questions? February 28, 2022 research area Speech and Natural Language Processing The task of Outside Knowledge Visual Question Answering (OKVQA) requires an automatic system to answer natural language questions about pictures and images using external knowledge. We observe that many visual questions, which contain deictic referential phrases referring to entities in the image, can be rewritten as “non-grounded” questions and can be answered by existing text-based question answering systems. This allows for the reuse of… Read more Question Rewriting for End to End Conversational Question Answering February 21, 2021 research area Knowledge Bases and Search , research area Speech and Natural Language Processing conference WSDM Conversational question answering (QA) requires the ability to correctly interpret a question in the context of previous conversation turns. We address the conversational QA task by decomposing it into question rewriting and question answering subtasks. The question rewriting (QR) subtask is specifically designed to reformulate ambiguous questions, which depend on the conversational context, into unambiguous questions that can be correctly… Read more Discover opportunities in Machine Learning. Our research in machine learning breaks new ground every day. Work with us
由 tencent/hy3:free 自動生成