Janas: Run LLMs larger than RAM by streaming experts from SSD

來源原標題

Janas: Run LLMs larger than RAM by streaming experts from SSD

作者

khalidkhair

原始標籤

hackernews

來源訊號

HN points 211　HN 留言 84

原始連結

https://github.com/prabanta-dev/janas

摘要原文

Janas 37 tok/s from Qwen3-Next-80B (a 48 GB model) on a laptop with 32 GB of memory, 24 without drafts, where llama.cpp manages 2.8; reading a prompt, 62 tok/s against 16. Janas-LLM was faster than llama.cpp on every model compared on the same laptop (Debian 13 Linux, Intel Core Ultra 9 185H, its integrated Arc GPU, 32 GB, an NVMe SSD; 9-12 October 2026; the whole machine below ): Model reading 1,024 tokens: Janas-LLM llama.cpp writing: Janas-LLM llama.cpp Qwen3-Next-80B-A3B (48.4 GB, more than the memory) 62 tok/s 16 tok/s 24 tok/s 2.8 tok/s Qwen3-Coder-Next (48.4 GB, more than the memory) 66 tok/s 14 tok/s 22 tok/s 2.2 tok/s Qwen3.6-35B-A3B 104 tok/s 79 tok/s 19 tok/s 12 tok/s Qwen3-30B-A3B 135 tok/s 122 tok/s 32 tok/s 15 tok/s Gemma-4-26B-A4B-it 84 tok/s 76 tok/s 18 tok/s 12 tok/s Qwen3.8-27B (dense, 17.6 GB) 30.6 tok/s 25.6 tok/s 4.4 tok/s 2.1 tok/s Gemma-4-12B-it 65 tok/s 46 tok/s 9.9 tok/s 7.1 tok/s Qwen3.5-9B 111 tok/s 78 tok/s 14 tok/s 8.7 tok/s Gemma-4-E4B-it 122 tok/s 109 tok/s 20 tok/s 15 tok/s Qwen3-4B 193 tok/s 164 tok/s 27 tok/s 18 tok/s Gemma-4-E2B-it 191 tok/s 150 tok/s 38 tok/s 19 tok/s Qwen3.5-2B 441 tok/s 323 tok/s 52 tok/s 28 tok/s Qwen3.5-0.8B 799 tok/s 519 tok/s 109 tok/s 32 tok/s Qwen3-0.6B 939 tok/s 670 tok/s 130 tok/s 45 tok/s The same models from other hands, in the catalog beside them: Model reading 1,024 tokens: Janas-LLM llama.cpp writing: Janas-LLM llama.cpp Qwen3-30B-A3B, bartowski's Q4_K_M (importance matrix) 131 tok/s 122 tok/s 32 tok/s 16 tok/s Qwen3-30B-A3B, unsloth's UD-Q3_K_XL (3 bits) 100 tok/s 112 tok/s 36 tok/s 11 tok/s Qwen3-4B, bartowski's Q4_K_M (importance matrix) 191 tok/s 160 tok/s 26 tok/s 18 tok/s Qwen3.8-27B: llama.cpp measured on 12 October 2026 ( llama-bench -r 2 ), Janas-LLM on 13 October ( janas-bench --prompt 1024 --gen 16 , two rounds), once its prompt went whole to the GPU in one submission: 30.6 tok/s, where it read 26.4 the day before and 18.7 before the GPU's tiles for the IQ4_XS, IQ3_S and IQ4_NL weights of unsloth's UD mix. One exception: the 3-bit Qwen3-30B reads a prompt slower than llama.cpp, since its Q3_K weights do not go to the GPU's fast prompt yet; it writes three times as fast. How they were measured, and the speeds with the drafts: What works now . On another machine the figures will be other ones: janas-bench measures yours. Try it on Linux x86-64 (Ubuntu 22.04, Debian 12 and later): curl -LO https://github.com/prabanta-dev/janas/releases/latest/download/janas-linux-x86_64.tar.gz tar xf janas-linux-x86_64.tar.gz && cd janas ./janas-chat # the models it knows: pick one, it is downloaded and checked Or on Windows 10 (1903 or later) and 11, x86-64: download janas-windows-x86_64.zip , unpack it, and in a terminal in its janas folder run .\janas-chat . On the same laptop under Windows 11 (12 October 2026) Janas-LLM wrote as fast as on Linux (Qwen3-4B 25.6 tok/s, Qwen3-30B-A3B 34.2) and read a prompt as fast with the mixture (Qwen3-30B-A3B 132 tok/s) and 1.8 times as fast with the dense model (Qwen3-4B 346 tok/s against 193), whose whole blocks go to the GPU, through Intel's own Windows driver there. The models come already converted from Hugging Face ; building from the sources: Getting started . ▄███████████▄ ▄███████████████▄ ███ ▄ █████ ▄ ███ ▄ J A N A S ███▄▄▄█████▄▄▄████ ▀█████▄▀▀▀▄█████▀ ▀ a family of tools for artificial intelligence, in C █████████████ █ █ █ █ █ █ █ █ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ Janas is a family of tools for artificial intelligence, written in C with no dependencies beyond the C library. Its first component, Janas-LLM , runs large mixture-of-experts language models on ordinary computers, with or without a GPU: the weights stay on disk, the experts each token needs are streamed from an NVMe SSD, and the ones that keep coming back stay in memory. An 80-billion parameter model runs, and answers, on a laptop with 32 GB of RAM. Designed and written by Maurizio "camauri" Cammalleri . ⚠️ Read this first This is an experiment, not a product. Stage Early development. The public API and the command line change without notice , and have changed several times a week. The models' files are the exception: the .jns files published on Hugging Face (format version 2) will be read by every later version of Janas-LLM. Stability Not verified. The engine is deliberately hard on the machine: it fills the free memory with its expert cache and reads the disk at full speed with O_DIRECT . On a machine with less headroom than it reckons, the desktop can be pushed into swap and become slow or unresponsive. Using a GPU has, once, caused a driver reset that took the desktop with it. Data It writes only in ~/.cache/janas (what it learns about your machine, and the system messages it has read, so as not to read them again) and where you tell it to put a converted model. It does not touch the model files it reads. Hardware Developed and run on one machine (below). On anything else it is untested: it may be slower, it may refuse the model, it may misbehave. If something goes wrong, Ctrl-C stops a reply and Ctrl-D leaves the chat; --cache <GiB> puts a hard limit on the memory it takes, and JANAS_GPU=0 keeps it off the GPU entirely. The machine it grew on Everything in this repository was written and measured on: Linux Debian 13 (trixie), x86-64, kernel 6.12.107; GPU driver Mesa 26.1.6 (Vulkan); built with GCC 14.2 Intel Core Ultra 9 185H (6 performance cores, 8 efficiency, 2 low-power; AVX2, FMA, F16C, AVX-VNNI; integrated Arc GPU) 32 GB of DDR5 RAM 1 TB NVMe SSD (about 5.7 GB/s reading) The Windows build was run on the same laptop under Windows 11 (build 26300), with Intel's own graphics driver: its tests pass there, and how fast it was is under Or on Windows , above. Everything the engine does is chosen from what it measures, so another machine will make other choices — that part is meant to travel. What has never been tried is the rest: other CPUs (especially without AVX-VNNI, or AMD), discrete GPUs, slower disks, more or less memory. If you run it somewhere else, the report from janas-bench is the single most useful thing you can send back (see Helping it run on more hardware ). What works now What Janas-LLM runs today, model by model and feature by feature: docs/features.md . On the machine above, with nothing else running, janas-bench reports this (measured on 10 October 2026, with the fast prompt the engine uses by default since that day): Model reading a prompt writing a reply with drafts, in a chat Qwen3.5-2B (dense, 1.3 GB) 395 tok/s 50 tok/s 67 tok/s Qwen3-4B (dense, 2.5 GB) 220 tok/s 28 tok/s see --draft below Qwen3.5-9B (dense, 5.7 GB) 103 tok/s 14 tok/s 24 tok/s Qwen3-30B-A3B (18.6 GB) 133 tok/s 34 tok/s — Qwen3.6-35B-A3B (22.3 GB) 86 tok/s 25 tok/s 34 tok/s Qwen3-Next-80B-A3B (48.4 GB) 64 tok/s 22 tok/s 37 tok/s Gemma-4-E2B-it (3.1 GB) 211 tok/s 38 tok/s 52 tok/s Gemma-4-E4B-it (5.0 GB) 133 tok/s 20 tok/s 32 tok/s Gemma-4-12B-it (dense, 7.1 GB) 71 tok/s 10 tok/s 14 tok/s Gemma-4-26B-A4B-it (18.3 GB) 90 tok/s 18 tok/s 26 tok/s Every figure is the mean of three rounds , not a best round, of the configuration the engine itself chose for each kind of pass - the one a chat runs: a prompt of 256 tokens, then 64 written. Here the integrated GPU reads every prompt (the fast prompt, below), and the threads the engine settled on were the performance cores' 12 for a prompt and for a single token on nearly every model, all the usable cores (performance and efficiency, not the low-power ones) for a few tokens at once. On another machine the engine may settle on other threads: the first passes of each kind try every core, one thread per performance core and the performance cores' threads, keep the fastest, and later try the others now and then, so that a machine that warms up can change the verdict (on an i9-14900HX every thread replied at less than half the speed of the performance cores' 16). (211 points, 84 comments on Hacker News)