AMD Releases Instella-MoE-16B-A3B: A Fully Open Mixture-of-Experts LLM With 2.8B Active Parameters Trained On Instinct GPUs
https://www.marktechpost.com/2026/08/01/amd-instella-moe-16b-a3b-fully-open-mixture-of-experts-llm/📌 【AMD 全新開源】Instella-MoE-16B-A3B：用 2.8B 參數實現高效能 MoE 架構
TL;DR：AMD 發布全開源 MoE 模型，透過系統級設計實現 12.7% 訓練加速與顯著推理效能提升。
AMD 釋出了全新開發的 Instella-MoE-16B-A3B 模型，這是一個完全開源的 Mixture-of-Experts (MoE) 語言模型。該模型不僅從頭開始訓練，更展現了在 Instinct GPU 上的強大實作能力，對需要高性能且高透明度的研究人員來說，這是一個極具價值的開源資產。
🧩 16B 總參數僅需 2.8B 運算：高效能 MoE 架構設計
Instella-MoE 採用 Decoder-only 架構，具備 27 層、2048 隱藏層維度（hidden size）以及 128,896 個 token 的詞彙表。其核心設計如下：
- MoE 結構：每個 MoE 層包含 2 個共享專家（shared experts）以及從 64 個專家中選出的 6 個路由專家（routed experts）。
- 運算效率：雖然總參數達 16B，但每個 token 僅需啟動 2.8B 參數，大幅降低運算成本。
- 關鍵技術創新：
- Gated MLA (Multi-head Latent Attention)：在 MLA 基礎上加入輕量級的學習輸出閘（learned output gate），透過線性投影（linear projection）得出輸入條件閘，並在輸出投影前進行乘法運算。
- FarSkip-Collective 連結技術：將過時或部分的 activations 傳遞至 MoE 與 attention 層，藉此將專家並行（expert-parallel）的通訊與運算進行重疊（overlap）。
📊 訓練與評估：在多項指標中領先全開源模型
該模型在 Instinct MI300X 與 MI325X GPU 上進行訓練，並在多個階段進行優化：
- 訓練流程：
- 預訓練：使用 7.1T tokens（包含 Nemotron-CC-v2、MegaMath 等資料集）。
- 中段訓練：使用 Dolma3 資料集進行權重平均（weight averaging）。
- 長文本擴展：透過 YaRN 與增加 RoPE theta，將上下文窗口從 4K 擴展至 64K。
- 後訓練（Post-training）：經歷 SFT、DPO 以及基於 Miles 框架的強化學習（RL），並透過 Multi-Teacher On-Policy Distillation 整合增益。
- 效能數據：
- 訓練加速：預訓練速度提升 12.7%。
- 推理效能：在使用專家並行時，首字延遲（time to first token）最高可減少 39.2%。
- 基準測試：Base 模型平均分數為 76.7，在全開源模型中表現最強（優於 Moonlight-16B-A3B 與 OLMo 系列），但在 Qwen3.5-4B-Base (79.5) 之後。
- 程式碼與邏輯能力：在 HumanEval+ 獲得 65.7 分；在 WinoGrande 獲得 86.5 分。
⚠️ 使用限制與授權
需要注意的是，雖然 AMD 釋出了訓練各階段的權重、資料組合、訓練配置與推理程式碼，但模型權重僅根據 ResearchRAIL 授權，僅限學術與研究用途，並非直接可用的商業模型。不過，其訓練程式碼採用 MIT 授權，對工程師而言是極具重用價值的技術資源。
🎯 實務啟示
對於研究者而言，Instella-MoE 的價值不在於直接部署商業產品，而於其「透明度」。AMD 提供了從預訓練到強化學習的完整技術細節與程式碼，這對於想要優化 MoE 架構、研究系統級通訊（如 FarSkip-Collective）或探索長文本擴展技術的工程師來說，是極佳的參考範本。
🔗 來源
- 標題：AMD Releases Instella-MoE-16B-A3B: A Fully Open Mixture-of-Experts LLM With 2.8B Active Parameters Trained On Instinct GPUs
- 作者／機構：Asif Razzaq @ MarkTechPost
- 連結：https://www.marktechpost.com/2026/08/01/amd-instella-moe-16b-a3b-fully-open-mixture-of-experts-llm/
#AMD #MoE #LLM #MachineLearning #OpenSource #InstinctGPU #AIResearch #DeepLearning #Transformer #LargeLanguageModels
原始資料 MarkTechPost · 收集於 2026-08-02
摘要原文
AMD released Instella-MoE-16B-A3B , a fully open Mixture-of-Experts language model trained from scratch on Instinct MI300X and MI325X GPUs. The model holds 16B total parameters but activates only 2.8B per token. AMD is publishing weights from every training stage, along with data mixtures, training configs, and inference code. Two systems-level choices carry the release: Gated Multi-head Latent Attention and FarSkip-Collective connectivity. Partly. The weights ship under a ResearchRAIL license for academic and research purposes only , so this is not a drop-in commercial model. The training codebase is MIT licensed, and that is the more reusable asset here. Instella-MoE is a decoder-only MoE with 27 layers, hidden size 2048, 16 attention heads, and a 128,896-token vocabulary. Each MoE layer uses 2 shared experts plus 6 routed experts selected from 64. That yields 2.8B active parameters against 16B total. A Multi-Token Prediction objective is used during pre-training and mid-training. There are two structural choices that are important to know. Gated MLA adds a lightweight learned output gate to Multi-head Latent Attention. A dedicated linear projection derives an input-conditioned gate, applied multiplicatively before the output projection. FarSkip-Collective passes outdated and partial activations into the MoE and attention layers, overlapping expert-parallel communication with computation. AMD reports a 12.7% pre-training speedup and up to a 39.2% reduction in time to first token when serving with expert parallelism. Pre-training covers 7.1T tokens from open corpora including Nemotron-CC-v2, MegaMath, FineMath, RefineCode, and TxT360. Mid-training uses Dolma3 Dolmino 100B across three data variants, merged by weight averaging. A long-context stage extends the window from 4K to 64K using YaRN, an increased RoPE theta, and document masking. Post-training runs SFT on Dolci-Think-SFT-7B plus Nemotron mixtures, ending on a feedback-driven 512K-example set targeting measured weaknesses. DPO follows, with router bias updates and the auxiliary load-balancing loss disabled to prevent degradation. RL runs in the Miles framework: 1,400 steps of instruction-following RLVR, then Multi-Teacher On-Policy Distillation to fold that gain back without losing math or code. The base checkpoint averages 76.7, the strongest among fully open models, ahead of Moonlight-16B-A3B (76.2), SmolLM3-3B-Base (70.5), OLMo-3-7B (70.1), and OLMoE-1B-7B (61.9). It trails Qwen3.5-4B-Base (79.5). It leads on WinoGrande (86.5) and scores 65.7 on HumanEval+. Long-context averages are 41.5 on HELMET and 79.4 on RULER. Post-training climbs from SFT (71.58) to DPO (72.67) to Think (73.22), above Olmo3-7B-Think (71.97), Gemma-4-E4B think (70.47), and Qwen3.5-4B (69.73). IFEval rises from 77.08 to 83.70. Check out the ROCm blog , Hugging Face collection and GitHub . Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter . Wait! are you on telegram? now you can join us on telegram as well. Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us Sources: ROCm blog · Hugging Face collection · GitHub The post AMD Releases Instella-MoE-16B-A3B: A Fully Open Mixture-of-Experts LLM With 2.8B Active Parameters Trained On Instinct GPUs appeared first on MarkTechPost .
由 tencent/hy3:free 自動生成