RADAR: Catch gray failures with anomaly detection

來源原標題

RADAR: Catch gray failures with anomaly detection

機構

Databricks

原始標籤

Industries

原始連結

https://www.databricks.com/blog/radar-catch-gray-failures-anomaly-detection

摘要原文

Some of the most damaging outages are the ones your monitoring never flags: a slice of your customers quietly fails while every health check still reads normal. These "gray failures" leak users and revenue for hours before anyone connects the dots. This post is about catching them early with anomaly detection — how we do it at Databricks with a system called RADAR, and how you can build the same thing for whatever metric matters most to your business. It's written for the people who own service reliability: SREs, platform and data engineers, on-call responders, and the engineering leaders they answer to. Picture a normal Wednesday. Keeping a customer-facing service reliable is your job, and every dashboard on your wall is green — CPU healthy, latency fine, servers up, database connected. By every signal your team watches, the system looks perfect. For nearly seven hours, your monitoring insisted everything was fine while customers walked and revenue leaked. That Wednesday is a textbook gray failure. On the surface everything looks healthy; underneath, one specific piece has quietly stopped working — and it hurts customers without ever tripping an alert. Two things make gray failures so sneaky: Think of it as smoke behind the wall. From the outside the house looks fine, but inside the damage is spreading — and the longer you wait, the bigger the blast radius. Researchers have a name for the underlying problem, too: Microsoft’s Gray Failure: The Achilles’ Heel of Cloud-Scale Systems calls it differential observability — your failure detectors don’t notice a problem even while your users clearly do. Most teams handle gray failures exactly the way that Wednesday played out: they wait for customers to tell them. Customer reports matter — they’re real human pain — but your customers shouldn’t be your monitoring system. Leaning on reports alone has three problems: The fix isn’t to stop reading tickets — keep doing that. It’s to add automatic detection that runs all the time and catches what people miss. Concretely, you want something that fires the moment a lot more customers than usual start hitting the same issue at the same time. That’s the idea behind RADAR — Reliability Anomaly Detection, Alerting, and Root-cause analysis. We built it at Databricks to catch gray failures in minutes instead of hours. The name fits: when visibility is low, you don’t wait until something hits you — you scan for weak signals early. Here’s how we point it at one especially useful signal: user errors. A gray failure often shows up as a sudden spike in errors that look like the user’s fault. Picture a bunch of users in one region who suddenly can’t spin up a certain type of cluster. Each request fails with INVALID_ARGUMENT - an error that is politely saying, “this one’s on you.” But when many users hit the same “your fault” error at the same moment, it stops being their fault. It’s ours. That spike is exactly the pattern RADAR is built to catch. RADAR turns that instinct into a pipeline with four stages: Running RADAR on ourselves changed the shape of these incidents. Before, we waited on customer tickets to discover such incidents, leading to days of delay. With RADAR, we achieved a 95% reduction in incident-discovery time , at over 90% precision , with no human needed to spot the pattern. As a result, we are able to keep the blast radius of gray failures contained. Here’s the part that matters most for you: RADAR doesn’t care what the metric is. We happen to point it at user errors, but the same pattern works anywhere a number can quietly go wrong: It’s the same pattern under different settings. Anywhere you have something that could quietly go wrong, RADAR applies. The best news: every piece you need is already on Databricks. Map the four stages to the platform and it looks like this: And the whole thing deploys as a single unit through a Declarative Asset Bundle (DAB) . Wiring all those parts together by hand is the annoying bit — so we removed it. We distilled the entire internal RADAR system into a single scaffold: one markdown file that works like a recipe, mapping each part of RADAR to a specific Databricks component (collect and store → a Delta table; detect the anomaly → a job; alert and dedupe → a ticket; visualize → a dashboard). Then comes the payoff. You bring your own metric — wherever your signal lives — and hand the metric, the scaffold, and a short prompt to an AI agent. It builds the whole RADAR system for you, live on Databricks. You can follow the Github instructions on how to build one from a single prompt. Get the RADAR scaffold on GitHub Because the best outcome isn’t a faster response to angry customers — it’s that your customers never have to discover your incidents for you. Subscribe to our blog and get the latest posts delivered to your inbox.