ashhart/TensorFold

TensorFold TensorFold serves text models on Apple Silicon and NVIDIA GPUs through an OpenAI-compatible API. Each model family supplies its own kernels and draft verification. python -m pip install git+https://github.com/ashhart/TensorFold.git tensorfold serve Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit Use http://127.0.0.1:8080/v1 as the client base URL and the model ID from /v1/models . Python 3.11 or newer is required, and MLX 0.32.2 or newer on a Mac (pip installs it). See the runbook for installation and a first request. Models Model Checkpoint Backend Drafting Nemotron 3.5 Lightning Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit MLX, CUDA Included MTP head; context copies on MLX Qwen3.8-27B Vontra/Qwen3.8-27B-MLX-4bit MLX, CUDA z-lab/Qwen3.8-27B-DFlash2 and context copies; DFlash2 is optional on MLX Qwen3.8 Flash Next Vontra/Qwen3.8-Flash-Next-MLX-4bit-MTP MLX, CUDA Included MTP head and context copies GLM-5.3-Flash Vontra/GLM-5.3-Flash-MLX-4bit-MTP MLX on a 256 GB Mac, CUDA with two ranks MTP; optional DFlash2 on CUDA Gemma 4 26B-A4B mlx-community/gemma-4-26b-a4b-it-4bit MLX Context copies; z-lab/gemma-4-26B-A4B-it-DFlash is optional Qwen3.8-27B (EXL3, experimental) turboderp/Qwen3.8-27B-exl3 (branches 3.00bpw , 4.00bpw ; any codebook, 1 to 8 bits per weight) CUDA z-lab/Qwen3.8-27B-DFlash2 and context copies Qwen3.8 Flash Next (EXL3, experimental) turboderp/Qwen3.8-Flash-Next-exl3 (branch 3.05bpw_h5_ng5 ; any codebook, a width per tensor) CUDA Included MTP head and context copies tensorfold models lists families and checkpoints. tensorfold info MODEL checks configuration without fetching weights. serve downloads a missing checkpoint; pull downloads it ahead of time. tensorfold pull Vontra/Qwen3.8-27B-MLX-4bit z-lab/Qwen3.8-27B-DFlash2 tensorfold serve Vontra/Qwen3.8-27B-MLX-4bit Qwen3.8-27B reads MLX affine 2-, 3-, 4-, 5-, 6- and 8-bit checkpoints, including mixed layer formats. This branch adds packed row readers for groups of 32, 64 and 128 on Apple Silicon and CUDA, with hardware qualification still pending for the new paths. M5 keeps its native tensor-unit kernels for compatible formats, and other formats use the row decoder. See quantized checkpoints for the exact scope. On CUDA, pull DFlash2 before serving; without it, explicitly choose --no-drafts for the serial reference. Nemotron uses TensorFold projections and routed-expert kernels. Its load-time row check controls drafting; keep the installed MLX version within the package requirements. The named checkpoint includes mtp-4bit.safetensors , which pull and serve check for. Flash Next requires 4-bit/group-32 weights. Without an MTP head it can run without MTP drafting on MLX; on CUDA, explicitly pass --no-drafts . Nemotron CUDA requires 4-bit/group-64 weights and an MTP head unless --no-drafts is set. GLM on MLX reads 4-bit/group-64 weights and mlx-lm's mixed-bit conversions, whose 5-, 6- and 8-bit tensors take their own row kernels; it needs MLX 0.32.2 or later. GLM CUDA reads MLX 4-bit/group-64 weights and the experimental Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw conversion. GLM's optional incoai/GLM-5.3-Flash-DFlash2 checkpoint has non-commercial license terms, described in third-party notices . Gemma 4 has no draft head. It drafts copies of its context, and chains from z-lab's DFlash model when served with --drafter z-lab/gemma-4-26B-A4B-it-DFlash (pulled once). Its kernels read 4-bit weights in groups of 32 or 64 with an 8-bit router, as the mlx-community conversion stores them; serve refuses other Gemma 4 layouts before downloading. See the recipes for supported formats and backend limits. Exact decoding A draft is accepted only when it equals the token the same engine would produce serially. Sampling depends on the prompt or explicit seed, absolute position and token ID. Verify kernels keep each row's arithmetic independent of the other rows in the call. Compare a request with the same request using "draft": false to check drafted versus serial output. The MLX engine can share a round across requests. Each stream keeps its own state and sampling key, with concurrent output required to match its solo output. Load-time checks restrict window width and shared forwards where a family cannot reproduce its serial arithmetic. On CUDA, --parallel N with N greater than one enables shared rounds for Qwen3.8-27B on one or two ranks and Flash Next on one rank. Flash Next rejects concurrent two-rank execution. GLM and Nemotron CUDA serve one request at a time; CUDA --parallel auto also means one request at a time. Exactness is against the same engine, weights, runtime and settings. It does not imply identical output between MLX and CUDA, different quantizations, or different tensor-parallel rank counts. Serve options Option Meaning Backend --host , --port Listen address, default 127.0.0.1:8080 Both --name Model ID advertised to clients Both --alias Additional model IDs MLX --context N Prompt plus reply capacity Both --max-tokens N Default reply limit, 4096 Both --temperature , --top-p , --top-k Sampling defaults; temperature zero is greedy Both --thinking , --no-thinking Template thinking toggle Both --reasoning-effort Template effort, default medium MLX --thinking-budget N Token-count limit inside reasoning MLX --backend auto , mlx , cuda Select backend; auto uses MLX on macOS Both --parallel N MLX auto admits up to 8 within budget; CUDA auto is 1, explicit N enables supported shared rounds Both --no-drafts Decode serially Both --drafter auto , none , or model ID Select an optional draft model where the family supports it Both --mtp-drafts N Family-specific cap on MTP drafts Both --kv-dtype bf16 , int8 , int4 Flash Next: int8 or int4 stores keys and values with one fp16 scale per 32 values. Other families and the MLX path refuse it CUDA --mtp-confidence P Flash Next: stop a draft chain before a later draft under this probability, 0 to 1 (default 0.30) CUDA --tp 2 --rank R --master HOST Two-rank CUDA execution CUDA --prompt-cache-gib N Retained conversation-prefix budget; zero disables retention MLX --checkpoint-slots N Retained conversation prefixes (default 3 per lane, at least 8); long conversations hit this before the byte budget MLX --spill-gib N Write evicted conversation prefixes to disk (up to N GiB) and read them back instead of prefilling again; zero disables MLX --mlx-cache-gib N Reusable freed-buffer cache, default 8 GiB MLX --snapshot-dir DIR Persistent prefix snapshots; none disables them MLX --no-update-check Disable the startup release check Both The default sampling settings come from generation_config.json . Requests can override sampling and reply length. CUDA does not implement the MLX-only options above. See API fields for request scope. Context and memory On MLX, omitted --context targets the model's metadata window and reduces it to the startup memory estimate when needed, allowing room to retain a prompt for the next turn. An explicit positive value that cannot fit one request is refused at startup. --context 0 removes the metadata cap; finite engine capacity and memory admission still apply. Use the reported context when configuring client compaction. On CUDA, Qwen defaults to the affordable native capacity. GLM targets a dense 2,051-token window, and Nemotron targets 16,384 tokens; the capacity estimate can lower these defaults. Flash Next's --kv-dtype int8 or int4 counts its smaller cache, so the same memory admits a longer window. Explicit --context 0 targets the affordable native capacity for every CUDA family. A positive CUDA value must fit both the native window and the capacity estimate on every rank; otherwise startup refuses it with fitting guidance. Increasing GLM beyond its dense window enables its sparse-attention path. The startup report distinguishes native and allocated capacity. MLX defaults to a process budget of 70% of RAM. A family can state a larger share: GLM-5.