Linkup Research Releases SPARSEUP: A 149M-Parameter Open-Source Sparse Embedding Model
https://www.marktechpost.com/2026/09/19/linkup-research-releases-sparseup/📌 149M參數稀疏嵌入模型SPARSEUP，補上開源檢索三缺一
TL;DR：Linkup 發布149M參數的開源稀疏嵌入模型SPARSEUP，BEIR-13平均nDCG@10達56.4，權重與訓練細節全公開。
多數開源檢索模型都是稠密向量：一段文字對應一個向量。Linkup 這次補的，是業界最近一次開源浪潮裡缺的那一塊。
🤔 LightOn留下的空缺：稀疏編碼器
觸發這次發布的是 LightOn 稍早的 DenseOn 與 LateOn 專案，該專案公開了訓練資料、訓練流程、一個稠密模型與一個 late-interaction 模型。Linkup research 團隊發布的 SPARSEUP，補上了其中缺少的稀疏（sparse）編碼器，並沿用同一個骨幹模型家族與微調資料，讓三種檢索風格可以直接放在一起比較。稀疏模型輸出的是對整個詞彙表的權重分布，而非單一向量：每個維度對應一個真實的 token，因此向量能直接放進 inverted index，人類也能直接讀懂內容，且通常對罕見詞的匹配效果較好。
🧩 從LateOn-unsupervised出發，接回MLM head
SPARSEUP 建立在一個149M參數的 ModernBERT 骨幹之上，以 Apache 2.0 授權開源，可透過 Transformers 或 Sentence Transformers 載入，並需要開啟 trust_remote_code=True。訓練起點是 LateOn-unsupervised 檢查點，但該檢查點原本沒有 MLM head，Linkup 團隊把 ModernBERT 原生的 MLM head 重新接回去。微調階段使用 LightOn 的微調資料組合，僅採用對比學習（contrastive learning），沒有經過 cross-encoder 蒸餾；每個查詢會從50個候選中取樣7個困難負例（hard negatives），再搭配 batch 內負例，整個訓練過程可在單張 H100 上完成。
團隊也提到，直接在這個骨幹上訓練標準的 SPLADE 會產生塞滿停用詞（stopwords）的龐大向量，因此做了調整：查詢與文件分別加上 [Q]、[D] 前綴，並改用內積（dot product）計算分數；評估時查詢最大長度設為128個 token，文件為512個 token。
📊 BEIR-13平均56.4分，控制變因後仍略遜同源模型
在 BEIR-13（不含 MS MARCO）的評測中，Linkup 團隊表示 SPARSEUP 是他們所知150M參數以下、基於詞彙表的稀疏編碼器中表現最好的一個，平均 nDCG@10 為56.4分。但在骨幹與資料都固定的控制比較下，數字沒有那麼漂亮：LateOn 得58.9分、DenseOn 得57.9分，SPARSEUP 為56.4分——且 SPARSEUP 使用近似的 Seismic 搜尋，LightOn 報告的則是精確搜尋結果。細看子項目，SPARSEUP 在 ArguAna 與 Touché 上勝出，也在 HotpotQA 上贏過 DenseOn，但在語意性較強的資料集上落後，其中 FiQA 差距最大，DBPedia 也是弱項。若改看去污染（decontaminated）後的 BEIR，與 DenseOn 的差距縮小到0.17分，不過團隊也提醒，去污染後的 NQ 與 MS MARCO 分別只剩21與46筆查詢，結果雜訊較大，不宜過度解讀。
在 MS MARCO 上，SPARSEUP 平均每個查詢產生47個非零詞項、每份文件190個，相較之下 SPLADE-v3 分別是25與170個。搭配 Seismic 反向索引，SPARSEUP 在單執行緒、平均每次查詢約380微秒的條件下，能達到超過97%相對於精確搜尋的召回率。
⚠️ 刻意維持稀疏，犧牲一點分數
Linkup 表示，如果放大向量維度，理論上可以再拉高1到2個 BEIR 分數，但團隊選擇維持稀疏設計，以保留稀疏模型在索引效率與可解釋性上的優勢。
🎯 實務啟示
對於已經在用 inverted index 做檢索、又想要引入語意能力的團隊，SPARSEUP 提供了一個權重、資料組合與訓練細節都公開的可複現基準；搭配 Seismic 這類近似索引，能在維持毫秒級查詢延遲的同時取得不錯的召回率，是評估「要不要導入稀疏檢索」時值得放進候選清單的選項。
🔗 來源
- 標題：Linkup Research Releases SPARSEUP: A 149M-Parameter Open-Source Sparse Embedding Model
- 作者／機構：Asif Razzaq, MarkTechPost
- 連結：https://www.marktechpost.com/2026/09/19/linkup-research-releases-sparseup/
#SPARSEUP #SparseEmbedding #InformationRetrieval #OpenSourceAI #ModernBERT #BEIR #SemanticSearch #Apache2 #EmbeddingModel #NeuralIR
原始資料 MarkTechPost · 收集於 2026-09-20
摘要原文
Linkup research team releases SPARSEUP , an open-source learned sparse embedding model. The model runs on a 149M-parameter ModernBERT backbone and ships under Apache 2.0. Linkup team reports 56.4 average nDCG@10 on BEIR-13. It calls this the strongest public vocabulary-based sparse encoder it knows of under 150M parameters. Is it deployable? Yes. The weights are on Hugging Face under Apache 2.0. The model loads through Transformers or Sentence Transformers with trust_remote_code=True . Most open retrieval models are dense: 1 vector per text. Sparse models output weights over a vocabulary instead. Each dimension maps to a real token, so vectors fit inverted indexes and humans can read them. They also tend to match rare words well. The trigger was LightOn’s DenseOn and LateOn release . LightOn published open data, a training recipe, a dense model and a late-interaction model. SPARSEUP fills the missing sparse slot. It uses the same backbone family and fine-tuning data, so all 3 retrieval styles can be compared side by side. Training starts from LateOn-unsupervised . That checkpoint had no MLM head, so the team grafted back ModernBERT’s original one. Fine-tuning used LightOn’s fine-tuning mixture with contrastive learning only. Each query gets 7 hard negatives sampled from a pool of 50, and in-batch negatives. There is no cross-encoder distillation, and training fits on a single H100. A vanilla SPLADE on this backbone produced huge bags full of stopwords. Linkup fixed this with 3 changes: Queries and documents take [Q] and [D] prefixes, and scoring is a dot product. Evaluation max lengths are 128 tokens for queries and 512 for documents. Against other sparse encoders on BEIR-13 (nDCG@10, without MS MARCO), per the model card : The controlled comparison is less flattering. With backbone and data fixed, LateOn scores 58.9, DenseOn 57.9 and SPARSEUP 56.4. SPARSEUP uses approximate Seismic search, while LightOn reports exact search. SPARSEUP wins ArguAna and Touché and beats DenseOn on HotpotQA. It lags on more semantic sets, with FiQA showing the largest gap. DBPedia is another weak spot. On decontaminated BEIR, the gap to DenseOn shrinks to 0.17 points. Linkup warns that decontaminated NQ and MS MARCO have only 21 and 46 queries, so those results are noisy. On MS MARCO, SPARSEUP averages 47 non-zero terms per query and 190 per document. SPLADE-v3 averages 25 and 170. With the Seismic inverted index, it reaches over 97% recall against exact search in about 380 microseconds per query, single-threaded. Linkup says inflating vector size could add 1 to 2 BEIR points, but it chose to stay sparse. Check out the Model Weights and Technical Details . All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter . Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us The post Linkup Research Releases SPARSEUP: A 149M-Parameter Open-Source Sparse Embedding Model appeared first on MarkTechPost .
由 claude-code/sonnet 自動生成