Canary rollouts: upgrade models in production without downtime

來源原標題

Canary rollouts: upgrade models in production without downtime

機構

Together AI

原始連結

https://www.together.ai/blog/canary-rollouts-upgrade-models-in-production-without-downtime

摘要原文

Summary Rollouts move live traffic from your current model to a new checkpoint in gated steps. Health checks always run before traffic moves; on a canary you can also add metric gates (say p95 latency or error rate) that run after each step. If a gate trips, the rollout pauses at the canary share and you cancel it and run it in reverse. Below we run one for real: a Qwen2.5-7B → Qwen3.5-9B canary whose gate caught a 137% p95 regression at 10% of traffic; we canceled and reversed it with live requests served and none failed. Swapping models is common practice If you run a model in production, you already know the need to swap in a new checkpoint or a new model family: the open model ecosystem moves fast, and the candidate usually looks great in evals or promises better throughput. You want it in front of every user without hiccups, and a way to rollback if it disappoints. The usual options force a tradeoff: Hard swap: point the endpoint at the new model and every user is exposed at once. If p95 doubles, you find out from your dashboard or, worse, from your customers, and you roll back under pressure onto a cold-started old model. DIY staged cutover: a second deployment plus a script that nudges traffic percentages while you watch Grafana, remembering to scale the old deployment back up before you shift traffic back. Both of these options put a human in the loop as the safety mechanism. Rollouts move that mechanism into the platform: you describe the source, the target, the steps, and what "healthy" means, and the platform works against this plan at every stage. How rollouts work A rollout migrates traffic between two deployments on the same endpoint: a source (what's serving today) and a target (what you want to serve tomorrow). You pick one of three strategies: Canary: traffic moves in staged percentages you define (say 10% → 50% → 100%; the default ladder is 5% → 25% → 50% → 100%), with a wait period and optional metric checks between steps. Blue-green: one gated 0% → 100% cutover. You can think of this as a single-step canary. Rolling: an in-place, replica-by-replica swap that preserves total capacity. Best when capacity is constrained, especially for same-model config changes that don't need a traffic ramp. Here's what happens inside every canary step: We chose this ordering deliberately; each item prevents a class of incidents: The target scales up before any traffic moves. No capacity for the new deployment means no redirected requests: the rollout parks first. The health gate runs before the traffic shifts. Traffic only reaches replicas whose engine is loaded and answering, not merely started. A propagation wait sits between the shift and the drain. Routing caches converge before any source capacity is removed. The source drains after traffic has moved. Capacity leads traffic on the way up; traffic leads capacity on the way down. The wait period and the metric gate come before the step is recorded as complete. A step that regressed is never marked passed. Through the API or the console, a rollout is created in a PENDING state and does nothing until you explicitly start it (the CLI's rollout command creates and starts in one step). This two-step create/start is intentional because you can create the rollout, review it (or have a teammate review it), and start it when you're actually watching. Two states in the diagram above deserve a note: PAUSED means you pressed pause. The rollout holds exactly where it is and resumes from the same step. SYSTEM_PAUSED means the platform found something that went wrong, such as a failed metric gate, a capacity shortfall or missing metrics, and stopped to wait for human approval. It pauses, notifies you and waits; canceling is always your call. There is no FAILED end state that leaves traffic in limbo: a rollout ends COMPLETED (the target serves) or CANCELED (the split is frozen where it was, and you run the rollout in reverse to go back). Anatomy of a step The following is a breakdown of what happens in a single canary step, measured on the run at the end of this post (Qwen2.5-7B → Qwen3.5-9B on one H100 each). The propagation wait is what keeps stale global routing caches from sending requests to a shrinking source. The wait period is grown to the metric window plus ingestion lag. The cold start dominates the first step; later steps add replicas to a target that is already serving and warm. Choosing a strategy at a glance All three strategies run through the same engine and the same health gates; they differ in how traffic moves, how much extra capacity the overlap costs, and whether there is a wait window for a metric gate. Canary Blue-green Rolling Traffic pattern Steps through shares you choose (default 5% → 25% → 50% → 100%), each held for a wait window One cutover, 0% → 100%, once the target is healthy Replica by replica, traffic following the replica ratio Extra capacity Near source size; the target grows one step before the source drains that share Both deployments at full size until the source drains Source's replica count at each step; one extra replica mid-step Typical duration One cold start plus a wait per step (at least 390 s each with a metric gate) One cold start plus 30 s propagation; a few minutes One cold start per replica; slowest on large deployments Metric gates Yes, after every step No (no wait window) No How to go back Cancel freezes the current share, then run the rollout in reverse Run the rollout in reverse; --final-source-replicas 1 keeps the old model warm for an instant return Run the rollout in reverse Best for Measuring on live traffic before taking 100% The fastest switch, when you can briefly afford double capacity Same-model engine or config changes at a constant GPU footprint ‍ Creating a rollout Here's a three-step canary from a deployment serving your current model to one serving the candidate, with a latency regression gate. The CLI ships as tg in the together Python package (2.34.0 or newer). You pass the target deployment; the source is inferred when exactly one deployment is receiving traffic, otherwise pass --source : # 1. Create AND start the rollout in one command. # Intervals and windows are seconds with an "s" suffix ("600s", not "10m"). tg beta endpoints rollout $TARGET_DEPLOYMENT_ID \ --source $SOURCE_DEPLOYMENT_ID \ --canary \ --steps 10,50,100 \ --interval 600s \ --metric router_latency --metric-stat p95 \ --metric-max-regression 10 --metric-direction higher-is-worse \ --metric-window 300s # 2. Watch it move: pass the rollout ID printed under "Active Rollout", # or the endpoint ID for the endpoint summary tg beta endpoints get $ROLLOUT_ID tg beta endpoints get $ENDPOINT_ID # 3. Control it: pass the endpoint ID plus exactly one control flag tg beta endpoints rollout $ENDPOINT_ID --pause --reason "holding for review" tg beta endpoints rollout $ENDPOINT_ID --resume tg beta endpoints rollout $ENDPOINT_ID --promote tg beta endpoints rollout $ENDPOINT_ID --cancel --reason "latency regression on target" The source drains to zero replicas and stops when the rollout completes ( --final-source-replicas defaults to 0), and the target lands with the source's replica count as its floor ( --final-target-replicas ). The CLI attaches one metric gate per rollout; for several rules use the console or the API. The same via the REST API, where create and start are separate calls and a rollout can carry several metric rules: # 1. Create the rollout. It comes back in state PENDING; save its "id" (rol_...) as $ROLLOUT_ID. curl -s -X POST \ "https://api.together.