MakazhanAlpamys/Soup

🌍 English | Türkçe | العربية | 日本語 | 中文 Soup Fine-tune and post-train LLMs in one command. No SSH, no config hell. Website · Quick Start · Web UI · Config · Docs · Commands · Models · Discord · Telegram · Product Hunt Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done. pip install " soup-cli[train] " # add [train] to fine-tune; bare `soup-cli` is the light CLI soup init --template chat soup train Fine-tune an 8B model on a 4 GB laptop GPU. Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one decoder layer at a time. Measured on an RTX 3050 Laptop 4 GB: Llama-3.1-8B-Instruct + NF4 at 119.6 tok/s at 952 MHz, 3.32 GB peak (pre-repair) — bit-exact against a normal resident run, and reproduced independently on an H100 at 113.00 tok/s in the same 3.32 GB. (Both figures were measured on v0.72.2, before the v0.73.0 correctness repair that cost −4.8% at 32B. The post-repair run, on another RTX 3050 Laptop 4 GB (@umran666, #1103), reads 208.6 tok/s at 1935–1957 MHz in a 2.40 GB peak . That is 59% of its same-session GEMM ceiling against 68%: about 14% slower per unit clock, so the raw 1.74x gain is the clock, not a speedup. Large-layer streaming also changed the memory pattern between the two rows, so even the fraction is not like-for-like — the record .) Opt-in ( stream_layers: true ) and still BETA — how it works · all measurements · paper · check it yourself on a free Colab T4 (caps the process to 4 GB, then asserts a streamed model is bit-identical to a normal one) Llama-3.1-8B-Instruct + NF4, LoRA, batch 1, seq 512 on an RTX 3050 Laptop 4 GB — 3.32 GB peak, 119.6 tok/s at 952 MHz (pre-repair, v0.72.2; post-#331: 208.6 tok/s at 1935 MHz in 2.40 GB, not directly comparable, @umran666, #1103). Full video (90s) Why Soup? Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that. Zero SSH. Never SSH into a broken GPU box again. One config. A simple YAML file is all you need. Auto everything. Batch size, GPU detection, quantization — handled. Works locally. Train on your own GPU with QLoRA. No cloud required. What's New v0.76.0 — settings Soup accepted and never applied now stop the load. Options that were validated and documented, and read by nothing on the path they claimed to configure, are now refused at load, naming the setting and what to change; the rest warn and name the release that will refuse them. A config that loaded before can stop loading: run soup doctor --config soup.yaml first. Breaking: a setting that never took effect is refused at load. Among them training.quantization_aware: true (it never ran quantization-aware training), training.use_longlora , training.ebft_variant , training.gdpo_variant , training.rope_scaling_type where no setup reads it, freeze_layers / freeze_ratio outside the text sft / tts path on the transformers backend, auto_mixed_precision outside sft / tts , and backend: unsloth on seven further tasks that have no unsloth setup. Each message names the field and what to change. lr_groups , early_stop_patience , forgetting_eval_steps and other unread knobs warn at load now and are refused from v0.77. Validation loss is now evaluated. data.val_split held rows out of training on every run, and no trainer on the transformers backend evaluated them. Every task that gets a split now evaluates it at the end of each epoch, or every N steps with training.eval_steps , and records the val loss; runs take a few percent longer. Breaking for grpo , ppo and online_dpo : by default they no longer hold rows out ( data.val_split is ignored with a note), so every row trains. Breaking: every non-standard grpo_variant applies the grpo_beta KL penalty. gspo , dapo , dr_grpo , bnpo , two_sided and rft validated grpo_beta and dropped it, so existing variant configs will not reproduce prior runs. grpo_beta: 0 now runs KL-free and skips the reference forward pass. Layer streaming reads its disk tier on a background thread, through direct I/O, and can stripe it across NVMe drives. On a 36 GB 70B-shaped NF4 store larger than RAM (one RTX 5070 Laptop 8 GB box), a cold step went from 110.5 s on the synchronous read to 16.2-17.3 s, and from 17.4 s to 9.97 s across two drives with SOUP_LAYER_STREAM_STRIPE_DIRS (a sustained-load probe: two rounds, 1.28x and 1.41x, no formal verdict; record ). A store that fits RAM is still faster on the RAM tier, which stream_source: auto picks. Opt-in ( stream_layers: true ) and still BETA; records: direct I/O , striping . New commands and options. soup adapters check scans a LoRA adapter for silent failure; soup bench train trains a fixed number of steps and exits 1 when the model was not training (it covers task: sft on the transformers backend; soup bench is now a group, and soup bench infer is the old inference benchmark); soup data clean sanitizes and repairs a dataset without touching the input file; soup recipes verify checks on the meta device, with no weights downloaded, that every shipped recipe can attach a LoRA adapter; soup rewind names the dataset rows behind a loss spike. soup infer gains opt-in --batch-size and an experimental --cuda-graphs (PyTorch 2.14 or newer; on one RTX 5070 Laptop, unpinned, Qwen2.5-1.5B in FP16, the slowest run with it decoded 2.9x faster than the fastest without). soup ab runs a two-sided, variance-robust test, and soup adapters sign --backend sigstore and soup attest emit --sign sigstore sign with Sigstore keyless OIDC ( pip install "soup-cli[sigstore]" ). Deprecated: soup export --format awq and --format gptq will be removed in v0.77; both upstream projects are archived and neither extra installs next to [train] . Python 3.10–3.12 only. On 3.13+, pip used to resolve untested PyTorch wheels that crash in the native extension before Soup runs at all. Older highlights live on the GitHub Releases page. Quick Start 1. Install Soup is a command-line application, so the cleanest install gives it its own environment and puts soup on your PATH : # Light core: CLI + config + data tools, no PyTorch pipx install soup-cli uv tool install soup-cli # same idea, if you already use uv # Add the training stack (torch, transformers, peft, trl, datasets, …) pipx install " soup-cli[train] " # Everything (train + serve + ui + data) in one shot pipx install " soup-cli[all] " # Or from GitHub (latest dev) pipx install " git+https://github.com/MakazhanAlpamys/Soup.git " Already inside a virtualenv, a Colab notebook, or a Docker image? Use pip directly, with the same names and extras: pip install soup-cli pip install " soup-cli[train] " pip install " soup-cli[all] " pip install git+https://github.com/MakazhanAlpamys/Soup.git Use pip rather than pipx if you also want to import soup_cli from your own code, since pipx deliberately isolates the application from everything else. The full extras table ( fast , mlx , serve , eval , ui , vision , audio , …) lives in docs/models.md . error: externally-managed-environment ? That is PEP 668 , not a Soup problem. Debian 12, Ubuntu 23.04 and later stop pip from writing into the system Python, because apt manages those files too. pipx and uv tool sidestep it by giving Soup its own environment, which is why they are listed first above. python3 -m venv .venv && source .venv/bin/activate then plain pip works just as well. Double quotes, not single. "soup-cli[train]" is the only spelling that works in every shell — cmd.exe , PowerShell, bash and zsh. If you copied 'soup-cli[train]' from an older tutorial and pip rejected it, that is the reason: why, and the exact error . soup init , soup data … , and the other data/inspection commands work on the light install. Fine-tuning ( soup train ) needs the [train] extra. 2.