Kev: Tiny Jev-like family of decision models built on top of Qwen3.5

來源原標題

Kev: Tiny Jev-like family of decision models built on top of Qwen3.5

作者

tosh

原始標籤

hackernews

來源訊號

HN points 361　HN 留言 163

原始連結

https://github.com/jaredpalmer/kev/tree/main

摘要原文

Kev Small Jev-like decision models you can train and run yourself. Kev is a family of small decision models built on Qwen3.5 and based on the architecture described in Jev's Architecture Unmasked . You can use the pretrained weights or train your own. The API matches TypeSafe's System One , so you can point their Python SDK at your local server. Highlights 0.8B, 4B, and 9B models, with training code and evaluation data. Yes/no ( noul ), multiple-choice ( choice ), and rating ( score ) questions in the same request. Questions share the input text but can't read each other. Runs on CUDA, ROCm, and Apple Silicon. The 4B and 9B models fit a 32 GB Mac using bf16; see Serving Performance for what to expect on a Mac. A web playground for trying your own inputs and checking how option order affects the answers. Or try Kev-4B and Kev-0.8B in the browser at huggingface.co/spaces/jaredpalmer/kev , no install needed. Quick Start You'll need Python 3.12+ and uv . git clone https://github.com/jaredpalmer/kev.git && cd kev uv sync --extra serve KEV_DTYPE=bf16 uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b --port 8009 This starts Kev-4B locally. The first run downloads the adapter and base model. --run also accepts a local checkpoint directory or a Hub revision, such as jaredpalmer/kev-4b@qwen3 for the previous generation. In another terminal, send it a ticket: curl -s localhost:8009/v1/systemone -H ' content-type: application/json ' -d ' { "state": "Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.", "model": "kev-latest", "questions": { "department": {"type": "choice", "instructions": "Which team should handle this?", "criteria": {"returns": "Exchanges, refunds, wrong or damaged items", "shipping": "Delivery status, delays, lost packages", "billing": "Charges, invoices, payment problems"}}, "escalate": {"type": "noul", "instructions": "Does this need urgent human attention?"}, "frustration": {"type": "score", "instructions": "How frustrated is the customer?", "criteria": ["Calm", "Frustrated", "Very angry"]} }} ' Example response from Kev-4B, running in bf16 on an Apple M5: { "model" : " kev-latest " , "answers" : { "department" : { "type" : " choice " , "choice" : " returns " , "confidence" : 0.21 , "probabilities" : { "returns" : 0.47 , "shipping" : 0.28 , "billing" : 0.25 } }, "escalate" : { "type" : " noul " , "noul" : 0.93 }, "frustration" : { "type" : " score " , "score" : 1.44 , "confidence" : 0.78 , "legend" : { "0" : " Calm " , "1" : " Frustrated " , "2" : " Very angry " }, "probabilities" : { "0" : 0.00 , "1" : 0.56 , "2" : 0.44 } } }, "usage" : { "input_tokens" : 101 , "output_tokens" : 161 }, "latency_ms" : 495 } The ticket mentions a return, a late delivery, and a billing problem, and the department probabilities say so. That is the point of getting probabilities back instead of a single label. Python The TypeSafe SDK is included in uv sync --extra serve : from typesafe_sdk import Choice , Noul , Score , TypeSafeClient client = TypeSafeClient ( api_key = "local" , base_url = "http://127.0.0.1:8009" , model = "kev-latest" , ) response = client . system_one ( state = "I was charged twice. Please fix this ASAP." , questions = { "billing" : Noul ( instructions = "Is this ticket about billing?" ), "tone" : Choice ( instructions = "What is the customer's tone?" , criteria = { "calm" : None , "frustrated" : None , "angry" : None }, ), "urgency" : Score ( instructions = "How urgent is this ticket?" , criteria = [ "can wait" , "this week" , "today" ], ), }, ) print ( response . nouls [ "billing" ]. noul ) print ( response . choices [ "tone" ]. choice ) print ( response . scores [ "urgency" ]. score ) Playground With the server still running, open another terminal. You'll need Node 20.9+: cd playground npm install npm run dev -- -p 3001 Open localhost:3001 , load a preset, and edit the text and questions. Press ⌘↵ to run it. "Packed vs separate" compares asking all questions at once with asking them one at a time. "Permute" runs a Choice question with six option orders. There are also presets for testing question isolation and fake delimiter tokens. There's a chess demo , too. The board is the input, legal moves are Choice options, and a Score question rates the position. You can play against Kev or let it play itself. Games are saved in localStorage . Models Start with Kev-4B. Use Kev-9B when accuracy and calibration matter more than memory. Use Kev-0.8B if you need the smallest model. All three are built on Qwen3.5 bases with the same training data and settings. Model Base Accuracy: Trained Sources Accuracy: New Sources Brier: New Sources Model Card Kev-0.8B Qwen3.5-0.8B-Base 0.825 / 0.834 0.652 / 0.684 0.499 / 0.460 Details Kev-4B Qwen3.5-4B-Base 0.872 / 0.871 0.797 / 0.837 0.299 / 0.255 Details Kev-9B Qwen3.5-9B-Base 0.872 / 0.874 0.822 / 0.852 0.286 / 0.237 Details Jev Hosted 0.845 / – 0.857 / – 0.211 / – – Each cell is development / test . "Trained sources" means held-out examples from the datasets used to train Kev. "New sources" means datasets and policy rule types Kev wasn't trained on. Every model was evaluated on the same development sets ( decision-v7 , transfer-v4 ) and the same test sets, which were read once per released checkpoint, after model selection. Lower Brier is better. Kev-9B trails Jev by 3.5 points on the new-source development set (0.822 vs 0.857) and scores 0.852 on the test set, which Jev hasn't been run on. We don't know which datasets Jev was trained on, so this isn't a controlled comparison of the two architectures. All three models were updated on 2026-09-21 with a short second training pass on generated examples: policy cases with explicit day counts, and cases whose deciding evidence was removed, trained toward a uniform answer. On the test set this moved Kev-9B from 0.837 to 0.852 (95% CI +0.8 to +2.9 points), Kev-4B from 0.832 to 0.837, and Kev-0.8B from 0.668 to 0.684. The previous weights are at revision v7-base . Details and costs are in the model cards and PLAN.md . Probabilities are calibrated by default. Each checkpoint stores a temperature (about 2.1–2.4) fitted on its in-distribution development set, and the pointer head applies it when the model is loaded. It never changes an answer: on new sources Kev-9B's calibration error goes from 0.106 to 0.042 and its confident errors (wrong answers with probability ≥ 0.9) from 8.7% to 4.0%, about Jev's 3.7%, with accuracy identical. Set KEV_TEMPERATURE=1.0 for the raw logits. The accuracy numbers in the table are the same either way; the Brier numbers are for the raw logits. One optional setting: KEV_DATE_FACTS=1 appends the number of days between any two absolute dates found in the state ("June 26, 2026 is 8 days before July 4, 2026"). Kev can't subtract dates reliably but it can use a stated day count: on the deadline policy questions Kev-9B goes from 0.80 to 0.90 (Jev 0.93). The table above doesn't use it. All weights are in the Kev collection and the GitHub release , which includes tarballs and SHA-256 checksums. Previous generation (Qwen3) and the prototype The first Kev family used Qwen3 bases with the same data and settings. Those weights stay published and are the faster choice on a Mac (see Serving Performance ), but they are no longer developed. Model Base Accuracy: Trained Sources Accuracy: New Sources Brier: New Sources Model Card Kev-0.6B (Qwen3) — jaredpalmer/kev-0.6b Qwen3-0.6B-Base 0.801 / 0.808 0.620 / 0.642 0.536 / 0.483 Details Kev-4B (Qwen3) — jaredpalmer/kev-4b@qwen3 Qwen3-4B-Base 0.854 / 0.856 0.790 / 0.806 0.328 / 0.294 Details Kev-8B (Qwen3) — jaredpalmer/kev-8b Qwen3-8B-Base 0.863 / 0.870 0.796 / 0.780 0.337 / 0.327 Details Because only the base changed, the two generations are a controlled comparison. On the development set the accuracy gain is within noise; on the test set Kev-9B is 7.3 points ahead of Kev-8B (95% CI +2.8 to +11.7) with a Brier score 0.08 lower, Kev-4B is 2. (361 points, 163 comments on Hacker News)