OpenTPU – An open-source AI accelerator, developed by AI
https://github.com/FeSens/openTPU📌 openTPU:讓 AI agent 自己設計、也自己跑在上面的開源推理晶片
TL;DR:一個由 AI agent 設計的開源 AI 加速器,從 SystemVerilog 硬體到編譯器全棧開源,實機跑出與模擬器逐 bit 一致的結果。
AI 能不能設計硬體,這件事討論很久了。但 openTPU 問的問題更進一步:AI agent 設計出來的晶片,能不能拿來跑它自己的 inference?這個專案在 Hacker News 上拿下 334 點、388 則留言,顯然戳到了不少工程師的好奇心。
🤔 從 auto-arch-tournament 延伸到真實晶片
openTPU 延續 auto-arch-tournament 的精神,把範圍從軟體架構擴展到 AI 加速器本身。它同時是一個可供學習的專案:整個加速器放在一個小型 monorepo 裡,從頭到尾可讀完,包含硬體設計(SystemVerilog)、指令集(ISA)、逐 bit 精確的模擬器、一套 kernel 語言與其編譯器,以及驅動真實 PCIe 卡的 host 軟體。如果想搞懂一個 AI 加速器怎麼從 Python 裡的一次 matmul 落到實體電路的線路,README 指出這是個不錯的起點。
🧩 沒有 cache、沒有隱藏排程的簡單機器
這臺機器刻意做得很簡單:一個 sequencer 每個 cycle 發一個指令給少數幾個單元,DMA 負責搬資料,matrix unit 把從 DRAM 串流進來的 int8 權重做乘法,vector unit 做 fp32 運算,quantizer 把結果轉回 int8。沒有 cache、沒有隱藏排程,每一次資料搬動都是一條指令,因此追蹤執行軌跡能精確看到每個 cycle 花在哪裡。
Kernel 層用一套叫 ol 的語言搭配 @ol.jit 裝飾器撰寫,README 給出的範例示範了如何用它寫一個簡化版的 MLP kernel,透過 load、quantize、dot、all_gather 等算子描述資料流。整個軌跡是:ol 語言與編譯器 → ISA(每條指令 8 個 32-bit word)→ ISA 模擬器與 RTL(逐 bit 一致,由測試驗證)→ Vivado bitstream → FPGA 卡,再透過 PCIe 連到 host,由 otpu-chat、otpu-smi、otpu-lens 等工具操作。
📊 十個真實模型跑在 Kintex-7 FPGA 卡上,逐 token 比對模擬器
硬體用的是 Inspur YPCB-00338 卡(Xilinx Kintex-7 xc7k480t,兩個 DDR3 channel),實機輸出與模擬器逐 bit 相同。README 列出的部分效能數據(decode 為device 端量測,DRAM 頻寬為卡上計數器讀值,DDR3-1066 理論峰值 17.1 GB/s):
| 模型 | 權重精度 | Decode(device) | Prefill(device) | DRAM 使用率 |
|---|---|---|---|---|
| LFM2.5-230M | int8 | 59.0 tok/s | 295.6 tok/s | 85% |
| LFM2.5-230M | 4-bit,int8 head | 85.8 tok/s | 335.4 tok/s | 82% |
| Qwen3.5-0.8B | int8 | 17.6 tok/s | 61.4 tok/s | 85% |
| LFM2-2.6B | 4-bit,int8 head | 10.96 tok/s | 20.6 tok/s | 93% |
| SmolLM3-3B | 4-bit,int8 head | 8.74 tok/s | 22.8 tok/s | 92% |
README 說明這是以單一 bitstream(133.33 MHz)搭配四欄 systolic matrix unit 跑完所有模型,解碼速度受限於 DRAM 頻寬,讀取速度落在 DDR3 峰值的 82% 到 94% 之間。與前一代採用 Xilinx MIG、兩欄 matrix unit、120.755 MHz 時脈的 se-cand3 影像相比,decode 效能相差在 2.3% 以內,但 prefill 快了 1.3 到 2.0 倍。
對於超過 4 GiB 卡上記憶體的 MoE 模型,README 描述了一種把專家權重從 host 儲存裝置串流進卡的方案:卡負責路由每個 token 並計算每個專家,host 只在缺少對應專家時把它從 pool file 複製進卡上的插槽。像 Qwen3.5-35B-A3B 這種 34.7B 參數、3.0B active 的模型,實測可達 3.95 tok/s,專家命中率 62%。
💡 量化換速度,但有可量測的精度代價
4-bit 權重採用 FP4 數值搭配兩層 block scale,平均每個權重 4.25 bits,LM head 保留 int8 以維持準確度。README 指出這能把每個 token 的傳輸量減少約三分之一,decode 速度提升 40%(Qwen3.5)到 45%(Qwen3、LFM2),但也帶來可量測的 perplexity 代價,具體數字依模型列在文件中。
⚠️ 侵限與門檻
這終究是一塊 FPGA 卡而非量產晶片,時脈僅 133.33 MHz,效能被 DDR3 記憶體頻寬牢牢卡住;Qwen3.5-4B 的 int8 image 已經超過 4 GiB,更大模型得靠 MoE offload 的方式迂迴處理。對於想直接拿來生產部署的讀者,這仍是一個研究與教學導向的專案。
🎯 對工程師的意義
比起效能數字本身,這個專案真正的價值在於它把「一個 matmul 怎麼從 Python 落到電路」的全部中間層都攤開可讀:指令集、模擬器、編譯器、RTL 全部在同一個小型 monorepo 裡逐 bit 對應。對想理解 AI 加速器底層運作,或想評估 agent 在硬體設計上能走多遠的工程師,這是一份值得逐行讀的素材。
🔗 來源
- 標題:OpenTPU – An open-source AI accelerator, developed by AI
- 作者／機構:fsbonetto(Hacker News)
- 連結:https://github.com/FeSens/openTPU
#OpenTPU #AIAccelerator #FPGA #HardwareDesign #SystemVerilog #Quantization #Inference #OpenSource #AIAgents #ChipDesign
原始資料 Hacker News · 收集於 2026-10-07
摘要原文
openTPU An open-source AI accelerator, developed by AI. openTPU brings the lessons of auto-arch-tournament to AI accelerators. It asks two questions: how far can AI agents go at hardware design, and can they build the chip that runs their own inference? otpu-chat running LFM2.5-230M on the FPGA card (left), with otpu-smi showing the card's utilization and DRAM bandwidth (right). A place to learn openTPU is also a learning project. The whole accelerator lives in one small monorepo that you can read end to end: the hardware design (SystemVerilog), the instruction set, a bit-exact simulator, a kernel language and its compiler, and the host software that drives a real PCIe card. If you want to understand how an AI accelerator works, from a matmul in Python down to the wires, this is a good place to start. Results The design runs ten modern models with their real weights on an Inspur YPCB-00338 card (Xilinx Kintex-7 xc7k480t, two DDR3 channels), and the card produces the same tokens as the simulator, bit for bit. Model Weights Decode, device Decode, wall Prefill, device DRAM while decoding LFM2.5-230M int8 59.0 tok/s 52.3 tok/s 295.6 tok/s 14.5 GB/s (85% of peak) LFM2.5-230M 4-bit, int8 head 85.8 tok/s 82.1 tok/s 335.4 tok/s 14.1 GB/s (82%) Qwen3-0.6B int8 21.6 tok/s 21.3 tok/s 92.1 tok/s 14.4 GB/s (84%) Qwen3-0.6B 4-bit, int8 head 31.3 tok/s 30.7 tok/s 103.4 tok/s 13.9 GB/s (82%) Qwen3.5-0.8B int8 17.6 tok/s 16.3 tok/s 61.4 tok/s 14.5 GB/s (85%) Qwen3.5-0.8B 4-bit, int8 head 24.5 tok/s 23.3 tok/s 66.7 tok/s 14.1 GB/s (83%) Gemma 4 E2B 4-bit, int8 head 10.57 tok/s 10.53 tok/s 32.1 tok/s 15.6 GB/s (92%) Gemma 4 E2B 4-bit, 4-bit head 12.14 tok/s 12.09 tok/s 29.9 tok/s 15.5 GB/s (91%) LFM2-2.6B int8 6.05 tok/s 6.03 tok/s 21.4 tok/s 16.1 GB/s (94%) LFM2-2.6B 4-bit, int8 head 10.96 tok/s 10.93 tok/s 20.6 tok/s 15.8 GB/s (93%) SmolLM3-3B int8 5.00 tok/s 4.99 tok/s 21.1 tok/s 16.0 GB/s (94%) SmolLM3-3B 4-bit, int8 head 8.74 tok/s 8.72 tok/s 22.8 tok/s 15.7 GB/s (92%) Phi-4-mini (3.8B) int8 3.99 tok/s 3.98 tok/s 13.8 tok/s 16.0 GB/s (94%) Phi-4-mini (3.8B) 4-bit, int8 head 6.56 tok/s 6.55 tok/s 15.0 tok/s 15.8 GB/s (92%) Qwen3.5-2B int8 8.02 tok/s 8.00 tok/s 38.2 tok/s 16.0 GB/s (94%) Qwen3.5-2B 4-bit, int8 head 12.09 tok/s 12.03 tok/s 41.7 tok/s 15.8 GB/s (92%) Qwen3.5-4B 4-bit, int8 head 5.88 tok/s 5.87 tok/s 12.9 tok/s 15.7 GB/s (92%) Gemma 4 E4B int8, 4-bit head and down 0-23 3.78 tok/s 3.75 tok/s 14.8 tok/s 16.0 GB/s (94%) Measured on the card: the first three models on 2026-09-29 with the production image deploy_champ_e698dcd7 . LFM2-2.6B, SmolLM3-3B and Phi-4-mini on 2026-09-30, and Qwen3.5-2B and 4B and Gemma 4 on 2026-10-01, with build B, deploy_fused133c_79c5707a , production since then. Build B decodes LFM2-2.6B, SmolLM3 and Phi-4-mini 8-9% faster than e698dcd7 (Gemma 4 E2B 10%), at 91-94% of the DRAM peak instead of 82-87%. Qwen3.5-4B's int8 image is over 4 GiB. The image: main e698dcd at 133.33 MHz, one bitstream for all models. It has LiteDRAM controllers calibrated by a small CPU inside the memory core, a four-column systolic matrix unit and the stream engine ( docs/stream.md ). DDR3-1066, with a 17.1 GB/s peak. The host: the card sits in opentpu (Intel Core i7-4790). Method, tools/qual/perf.py : decode is 64 greedy tokens after a 512-token prompt, with the host's argmax in the loop (not streamed). "Device" counts only the cycles the accelerator runs; "wall" adds the host. Prefill is the 512-token prompt, on the device. DRAM traffic comes from the card's own counters while it runs. Gemma 4 E2B keeps its per-layer embedding tables on the card (3.5-3.6 GiB images; docs/gemma4.md ); in int8 it does not fit. It matches Hugging Face's greedy tokens on three prompts with either head. E4B's table (2.95 GB) stays on the host, which copies one 11 KB row into the card per token; its image is 3.96 GiB, int8 with the head and the first 24 layers' down projections in 4-bit ( docs/gemma4_e4b.md ). In the card's own decode loop (the card picking every token) Gemma 4 decodes faster: E2B 11.01 / 12.73 tok/s (int8 / 4-bit head), E4B 3.83 tok/s, on the device. Every configuration matches the simulator token for token, per-position and with the resident decode program. More detail in docs/board.md . With the logits streamed back while the card runs ( tools/decode_profile.py , 96 tokens), 4-bit decode is faster, in device / wall tok/s: LFM2: 89.5 / 84.5; Qwen3: 33.7 / 33.3; Qwen3.5: 24.6 / 24.2; LFM2-2.6B: 11.07 / 11.02 (build B); SmolLM3-3B: 8.92 / 8.89 (build B); Phi-4-mini: 6.69 / 6.67 (build B). The previous production image, se-cand3, was built with the Xilinx MIG, a two-column matrix unit and a 120.755 MHz clock. Measured the same way, the new image: decode: within 2.3% of se-cand3's in every configuration. Decode is bound by DRAM, and LiteDRAM reads at 82-85% of the DDR3 peak, as the MIG did. prefill: 1.3x (Qwen3.5) to 2.0x (LFM2 4-bit) faster. calibration: when the image starts, the core's CPU calibrates both DDR3 channels in 12 s, with no host involvement. The earlier images and their numbers are in docs/board.md , section 5. Mixture-of-experts models bigger than the card's 4 GiB run with their experts streamed from host storage ( docs/offload.md , section 10). The card routes each token and computes every expert, and it keeps the experts in per-layer slots in its DRAM. The host only copies missing experts from a pool file into those slots, at the link's rate (section 10.1). Measured on 2026-10-01 with build B (79c5707a), the card's own decode loop picking every token, 4-bit experts, int8 head: LFM2.5-8B-A1B (8.5B parameters, 1.7B active): 10.6 tok/s over 160 tokens. 98.5% of expert uses hit the slots, and 5.2 MB streamed per token. Qwen3.5-35B-A3B (34.7B parameters, 3.0B active): 3.95 tok/s, with Hugging Face's 16 greedy tokens. 62% of expert uses hit, and 153 MB streamed per token at 1.41 GB/s over PCIe (section 10.3). Both match the simulator bit for bit. 4-bit weights ( docs/quant.md ) use FP4 values with two-level block scales, 4.25 bits per weight, and keep the LM head in int8 for accuracy. They cut the bytes per token by about a third and raise decode speed by 40% (Qwen3.5) to 45% (Qwen3, LFM2), at a measurable cost in perplexity that docs/quant.md reports per model. The host is nearly out of the way. For LFM2 and Qwen3 the card runs one decode program compiled once, which reads the position from a register and looks up its own embedding and RoPE rows, and the logits stream back while the card is still running: the host adds 0.17 to 0.30 ms per token on omarchy (0.45 to 1.3 ms on opentpu). Qwen3.5 runs the same way for decode; its prefill still compiles each chunk's program on the host, ahead of the card. How it works Kernels in ol mlp, attention, full model layers | @ol.jit Language + compiler layouts, affine loop addressing, fusion | ISA 8 x 32-bit words per instruction | ISA simulator <======> RTL same bits, checked by the tests (Python) (SystemVerilog) | Vivado bitstream FPGA card Kintex-7 xc7k480t | PCIe Host otpu-chat, otpu-smi, otpu-lens The machine is deliberately simple. A sequencer issues one instruction per cycle to a few units: DMA moves data, the matrix unit multiplies int8 weights streamed from DRAM, the vector unit does fp32 math, and a quantizer turns results back into int8. There is no cache and no hidden scheduling: every data movement is an instruction, so a trace shows exactly where the cycles go. docs/isa.md describes the whole instruction set. A kernel looks like this: from opentpu import language as ol @ ol . jit def mlp ( h , gamma , w_gate , w_up , w_down , out , eps ): # simplified; see kernels/mlp.py x = ol . load ( h ) xs = ol . quantize ( rmsnorm ( x , ol . load ( gamma ), eps )) g = ol . dot ( xs , w_gate ) u = ol . dot ( xs , w_up ) a = ol . all_gather ( silu ( g ) * u ) y = ol . all_gather ( ol . dot ( a , w_down )) if ol . program_id () == 0 : ol . (334 points, 388 comments on Hacker News)
由 claude-code/sonnet 自動生成