raullenchai/Rapid-MLX

Rapid-MLX Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm), 1.5× on a typical task — same model, same weights. Decode speed, Qwen3.5-9B 4-bit on a Mac mini M4 Pro, greedy decoding, both servers at their defaults — per-task results, method, and raw data . rapidmlx.com · Docs · Model mirror · Desktop app · Discord How it compares Feature facts only, taken mostly from each project's own documentation as of 2026-09-26 (sources below). For speed, see the measured results below the table. Rapid-MLX oMLX Ollama LM Studio mlx-lm ( mlx_lm.server ) License Apache-2.0 Apache-2.0 MIT Closed-source app; its MLX engine and lms CLI are MIT MIT Inference engine on a Mac MLX MLX GGML/GGUF engine, plus an MLX engine for safetensors models llama.cpp (GGUF) and MLX engines MLX Local API OpenAI ( /v1/chat/completions , /v1/responses ) and Anthropic Messages ( /v1/messages ) OpenAI and Anthropic Messages Ollama API, OpenAI-compatible, Anthropic-compatible (subset) OpenAI-compatible, Anthropic-compatible, LM Studio REST API OpenAI-style chat API GUI Rapid-MLX Desktop (macOS app) macOS menu-bar app and web admin panel Desktop app (macOS, Windows) Desktop app None (CLI / Python) Concurrent requests Continuous batching Continuous batching (mlx-lm BatchGenerator ) Parallel requests per model ( OLLAMA_NUM_PARALLEL , default 1) Continuous batching (llama.cpp engine; MLX engine since 0.4.2) Continuous batching ( BatchGenerator ); one request at a time with a quantized KV cache Prompt / KV cache Radix prefix cache in memory (with state snapshots for hybrid models), saved to disk on shutdown and restored at startup; quantized KV cache Tiered KV cache: in-memory hot tier plus SSD cold tier, reused across restarts Reuses the previous request's prompt KV cache; KV cache quantization ( OLLAMA_KV_CACHE_TYPE ) — In-memory prompt cache; quantized KV cache ( --kv-bits ) Tool-call parsing 27 parser modules, including an auto-detect fallback ( rapid_mlx/tool_parsers ) mlx-lm formats plus the auto-detected families listed in its README Tool calling Tool use via the OpenAI-compatible API Tool calls for models whose tokenizer declares tool-calling support Sources: oMLX README · Ollama OpenAI and Anthropic compatibility, FAQ , MLX engine · LM Studio Anthropic compatibility , parallel requests , 0.4.2 changelog , app terms , mlx-engine · mlx-lm server docs and server.py . Ollama prompt-cache reuse is from our benchmark notes . — = not verified for this table. Spot an error? Please open an issue . Measured speed vs mlx-lm: Rapid-MLX 0.15.6 against mlx-lm 0.32.0 ( mlx_lm.server ) on a 48 GB M4 Pro Mac mini, Qwen3.5-9B 4-bit, same weights, greedy decoding, both at defaults (measured 2026-10-04). Decode was 1.50× at the median of 18 streaming tasks (lowest 1.28×) and up to 4.27× on a whole-file code edit (46.4 → 198.4 tok/s; 4.00× against mlx_lm.generate , hence "4×"). The gain is speculative decoding (MTP plus prompt lookup), on by default for this model; a long-prompt agent turn was close to a tie (1.05× end to end), and Qwen3.5-4B, where it is off by default, was 1.12× at the median ( per-task results and raw data ). Measured speed vs Ollama (published 2026-08-11): at 8 concurrent streams on Qwen3.6-35B-A3B, aggregate decode was 3.0× Ollama's, 82.9 vs 27.2 tok/s (32 GB M2 Pro Mac mini, Rapid-MLX 0.12.11 vs Ollama 0.32.7). Including prefill, whole-batch throughput was 1.6×; single-stream decode was about 1.5×; a dense 12B model was no faster; and llama.cpp-family engines prefilled cold prompts faster ( method and raw data ). Head-to-head with oMLX and Ollama on the same Macs (18 GB and 48 GB) — decode, agent sessions, restart, concurrency, and where Rapid-MLX is slower, with raw data: rapidmlx.com/compare . Works with your AI stack Use Rapid-MLX as a local backend for agents, apps, or your own code. If a client accepts an OpenAI- or Anthropic-compatible endpoint, it can usually use Rapid-MLX without an adapter. Five Tier-1 agents are exercised end-to-end on real weights before release. See the tested compatibility matrix for exact API coverage and setup status. Install Desktop — macOS (Apple Silicon) The easiest way to chat locally, manage models, and use vision, files, voice, and image generation from one app. Download Rapid-MLX Desktop Browse signed Desktop releases Requires an M-series Mac; Windows and Linux desktop builds are not available yet Share Compute setup and credit ledger CLI and server — macOS (Apple Silicon) # Homebrew — prebuilt bottle from homebrew-core brew install rapid-mlx # Or the guided installer — detects RAM and recommends a starter model curl -fsSL https://rapidmlx.com/install.sh | bash Both install the same rapid-mlx CLI. Prefer uv or pip , or want to verify the installer before running it? See alternative install methods and install security . The guided installer prefers a runnable model already cached on this Mac when it fits the RAM tier. Otherwise its quick first-chat download is lfm2.5-1b-4bit below 16 GB and qwen3.5-4b-4bit at 16 GB or above; larger quality picks remain available through rapid-mlx recipe and the model picker. Submit a Community Benchmark Help other Mac users choose a model by sharing a standardized result from your own Apple Silicon Mac. Runs stay local until you explicitly choose to publish; you do not need a GitHub account or a benchmark JSON pull request. Desktop: Enable Community Benchmark in Settings → Experimental , then choose a model in the Community Benchmark tab, run it, review the result in My Results , and select Publish if you want to share it. CLI: Run rapid-mlx benchmark catalog , then rapid-mlx benchmark run <model> . Review the saved result and publish it with rapid-mlx benchmark share <run-id> only if you choose to. Explore the Community Benchmark leaderboard or read the protocol and privacy details . Quick Start (60 seconds) 1. Chat with a model right now: rapid-mlx chat Defaults to qwen3.5-4b-4bit . First run downloads the weights (~3 GB) with a progress bar and drops you into a REPL. Type /help for slash commands, /exit to quit. 2. Or serve it for use from other apps: rapid-mlx serve qwen3.5-4b-4bit Starts an OpenAI-compatible HTTP server on the first free port in 8000 – 8009 when --port is omitted. An explicit --port never falls back. The examples below assume port 8000 was selected. Point any client that supports a local custom endpoint (Aider, LangChain, OpenCode, PydanticAI, your own scripts) at http://localhost:8000/v1 ; Claude Code / Anthropic SDK uses http://localhost:8000 (the Anthropic messages route lives at /v1/messages under the same host). curl http://localhost:8000/v1/chat/completions \ -H " Content-Type: application/json " \ -d ' {"model":"default","messages":[{"role":"user","content":"Say hello"}]} ' from openai import OpenAI client = OpenAI ( base_url = "http://localhost:8000/v1" , api_key = "not-needed" ) print ( client . chat . completions . create ( model = "default" , messages = [{ "role" : "user" , "content" : "Say hello" }], ). choices [ 0 ]. message . content ) 3. Or wire up your coding agent — one command: rapid-mlx launch claude-code With a server running (step 2), this patches Claude Code's local config ( ~/.claude/settings.json ) to route at http://localhost:8000 — no manual env vars, no editing JSON by hand. You get a fully local Claude Code: $0 per token, nothing leaves your Mac. Swap in cline or continue-dev for the other IDE clients, or run rapid-mlx launch list to see what's detected on this machine. Cursor: Cursor currently routes BYOK requests through its own servers, so its servers cannot reach a Rapid-MLX endpoint on localhost . Rapid-MLX therefore does not generate a Cursor localhost config.