Once Claude can measure something, it can make it faster

來源原標題

Once Claude can measure something, it can make it faster

作者

matthieu_bl

原始標籤

hackernews

來源訊號

HN points 220　HN 留言 148

原始連結

https://claude.dev/blog/how-we-made-claude-ai-faster/

摘要原文

This August, we made the core user experience of claude.ai and the Claude desktop app about 3x faster in a two-week sprint. Users had been telling us it was slow, and they were right. We ran everything from a single Slack channel, with Claude in every thread. We focused on four journeys that make up 95% of user activity. At the 75th percentile, time to a typeable page on a fresh load of claude.ai went from 3.1 seconds to 0.55, starting a new Claude Code session went from 0.8 seconds to 0.3, and loading a Claude Cowork cloud session went from 2.6 seconds to 0.73. In aggregate, we estimate that saves tens of thousands of user-hours of waiting every day. Core user journeys, p75 Real user monitoring, per platform and product, August 13 vs. August 27 Launching the app claude.ai web · fresh load 5.6x faster −82% 3,085 → 550 ms Desktop app cold start 1.9x faster −47% 6,310 → 3,328 ms Starting a conversation Chat web 1.5x faster −34% 416 → 273 ms Chat desktop 2.1x faster −51% 460 → 224 ms Claude Code desktop 2.4x faster −59% 837 → 347 ms Loading a conversation Chat web 2.4x faster −59% 1,557 → 646 ms Chat desktop 2.8x faster −64% 1,353 → 488 ms Claude Cowork desktop · cloud 3.5x faster −72% 2,566 → 728 ms Claude Code desktop 2.1x faster −52% 545 → 262 ms Sending a message client-side share Chat web 3.1x faster −67% 180 → 59 ms Chat desktop 2.2x faster −54% 140 → 64 ms Claude Cowork desktop · cloud 19x faster −95% 928 → 48 ms Claude Code desktop 4.8x faster −79% 250 → 52 ms Thirteen measurements across four journeys, before and after: 3.1x faster on average (geometric mean). Core user journeys, p75 Real user monitoring, per platform and product, August 13 vs. August 27 Launching the app claude.ai web · fresh load 5.6x faster −82% 3,085 → 550 ms Desktop app cold start 1.9x faster −47% 6,310 → 3,328 ms Starting a conversation Chat web 1.5x faster −34% 416 → 273 ms Chat desktop 2.1x faster −51% 460 → 224 ms Claude Code desktop 2.4x faster −59% 837 → 347 ms Loading a conversation Chat web 2.4x faster −59% 1,557 → 646 ms Chat desktop 2.8x faster −64% 1,353 → 488 ms Claude Cowork desktop · cloud 3.5x faster −72% 2,566 → 728 ms Claude Code desktop 2.1x faster −52% 545 → 262 ms Sending a message client-side share Chat web 3.1x faster −67% 180 → 59 ms Chat desktop 2.2x faster −54% 140 → 64 ms Claude Cowork desktop · cloud 19x faster −95% 928 → 48 ms Claude Code desktop 4.8x faster −79% 250 → 52 ms Thirteen measurements across four journeys, before and after: 3.1x faster on average (geometric mean). We used Claude Tag (beta), running an internal research model roughly comparable to Opus 5.5. Claude found bottlenecks, built benchmarks, shipped improvements, and watched every deploy. We steered by setting goals, making tradeoffs, and approving every change. With that approach, we merged more than three thousand changes without a single customer-facing incident or rollback. This post covers what we shipped, how we measured it, and the loop we built with Claude to do it safely. THE BRIEF Before the sprint, we created a Slack channel with the following standing instructions : @Claude Your job is to facilitate all things related to the performance of the claude.ai website and desktop app. Your responsibilities include monitoring deploys for performance regressions, assessing the accuracy and comprehensiveness of existing telemetry, maintaining well-curated observability dashboards, proactively implementing solutions for observed issues and low-hanging fruit, proposing performance project opportunities, and communicating with your human teammates. […] The ultimate goal for this channel is for you to become as autonomous as possible, but today we know that isn’t yet possible. We asked Claude to analyze usage data through the Datadog MCP server. It identified the four highest-impact user journeys: launching the app, starting a conversation, loading an existing conversation, and sending a message. Between web and desktop, and across our products, those journeys came to thirteen distinct measurements. To establish baselines, we added instrumentation until they were directly comparable: each started with a user interaction, ended once the result was rendered, and disambiguated client and server work. We kicked off the sprint with a list of about twenty hand-picked projects, each targeting a specific journey. Claude estimated the impact of each project in milliseconds, and we aggregated those estimates to set our targets for the sprint. Some of the projects were fairly large, but we thought we could probably achieve most of them within two weeks. We hit twelve of the thirteen targets by day three. The planned projects landed early. For faster launches, we baked a static composer into the HTML so users can type during React initialization, and precompiled a V8 code cache so the desktop shell’s main process doesn’t recompile from scratch. For faster navigations, we kept the composer mounted between conversations, prefetched sessions when the user hovered over them, and cut sidebar re-renders by 90%. We had also left room for Claude to identify opportunities and propose new workstreams. Those workstreams quickly ramped into full projects of their own, which far exceeded our initial targets. So we set new targets, then looked for more things to measure: @Claude we’ve ended up funding nearly every project in the original projects list and more. let’s do a refresh […] what have we not explored, what can we hill climb on, where is the most opportunity at this point? […] i am open to WACKY ideas ANYTHING CAN BE HILL CLIMBED From the start, we knew we wanted to iterate faster than our deploy cadence. Claude could work asynchronously for many hours, even overnight, and we wanted to let it validate its prototypes without waiting for field reads. To achieve that, we looked for other ways to measure performance in the lab. Sam found the first lead: # claude-apps-perf · thread S Sam 12:34 PM What can we do instead of wall-clock timing? Can we measure JS instruction counts for instance? Claude App 12:42 PM Yes. For pure-JS hot paths, literal instruction counts: run the benchmark under Valgrind with node --predictable and compare to a checked-in baseline — one run, no statistics needed. For browser paths there’s no instruction counting under Chromium, but there’s a ladder of other deterministic counts: React commits per interaction, function call counts from V8’s precise coverage, layout and style-recalc counts, DOM mutations. Which do you want first? S Sam 12:49 PM Let’s explore valgrind + Ir + --predictable in one thread and each of the browser/react benches in new threads. Ping me in all of them. You know what we want. Let’s go Recreated from a real conversation. Eleven minutes later, five threads were running, each focused on a different measurement: instruction counts, V8 call counts, React commits, style recalculations, and DOM mutations. We treated every new benchmark with some skepticism. Each one had two jobs: first, a metric Claude could move in the lab; second, a guardrail in CI with a number that could only ratchet down. If a benchmark was flaky, or if it didn’t actually correlate with user latency, we threw it out rather than let Claude climb the wrong hill. @Claude please prove that hill climbing against each of these can result in measurable wall clock perf wins. we’ll unship the benches for any candidates that cannot prove that Wall-clock time is what users feel, but it’s noisy, and milliseconds are too flaky to use as a CI gate. Instruction counts were appealing because they were deterministic, but we still needed Claude to prove they tracked wall-clock time. So we asked Claude to drive the count down on two hot paths: the routine that assembles a conversation’s message tree, and a scanner for status lines in Claude Code output. (220 points, 148 comments on Hacker News)