AI Evals: Everything You Need to Know

來源原標題

AI Evals: Everything You Need to Know

作者

Hamel Husain

原始標籤

blog

原始連結

https://hamel.dev/blog/posts/evals-faq/

摘要原文

This document curates the most common questions Shreya and I received while teaching 700+ engineers & PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment. Browse the questions that interest you, or choose a guide below for a curated reading path through the FAQs and related articles. Browse all questions by section. AI evals are tests that tell you whether an AI system is doing what you want. They give your team feedback when the product drifts from user needs or business goals. The failures they catch also become data you can use to improve the system. More formally, evaluation is the systematic measurement of quality. Each eval checks one behavior on relevant examples and returns a score or structured review. Most AI products need several evals because they can fail in different ways. When you hear the word “evals,” it usually refers to one of two things: model benchmarks or product evals. Model benchmarks compare general-purpose models on shared tasks. Model providers publish these benchmark results when they release new models. Common examples include GPQA Diamond for graduate-level science reasoning, Terminal-Bench for agents doing complex work in command-line environments, and MMLU for knowledge and reasoning across a wide range of subjects. These scores can help you choose a promising model as a starting point. To assess quality on your own tasks you need product evals, which we discuss next. Product evals measure whether your specific AI product does what you want it to do. They turn your judgment about what a good product experience looks like into metrics you can track. Product evals encompass all components of your product, including the model, prompts, retrieval, tools, and application code. This flavor of evals are focused on capturing failures that matter to users and the business. Consider an order-cancellation agent. Its product evals might check whether it selected the correct order and waited for the cancellation tool to succeed before telling the user the order was canceled. A high score on GPQA Diamond or Terminal-Bench gives you little information on this, because those benchmarks don’t have access to your systems. There are several mechanisms you can use to implement product evals, including code assertions, human review, LLM judges, and online experiments. The right method depends on the failure being measured, and is discussed in greater detail in this series . In the rest of the AI Evals FAQ , we focus on product evals. It starts with analyzing traces to discover real failure modes. We then turn important failures into targeted evals and use the results to guide changes. Finally, rerunning the evals tells us whether the system improved. If you are completely new to product-specific evals, see these posts: A trace is the complete record of all actions, messages, tool calls, and data retrievals from a single initial user query through to the final response. It includes every step across all agents, tools, and system components in a session: multiple user messages, assistant responses, retrieved documents, and intermediate tool interactions. Note on terminology: Different observability vendors use varying definitions of traces and spans. Alex Strick van Linschoten’s analysis highlights these differences (screenshot below): Start with error analysis, not infrastructure. Spend 30 minutes manually reviewing 20-50 LLM outputs whenever you make significant changes. Use one domain expert who understands your users as your quality decision maker (a “benevolent dictator”). Use a notebook to review traces and analyze data, or build your own custom annotation interface with an AI coding assistant like Claude or Codex. Either way, you can write arbitrary code, visualize data, and iterate quickly. The video below shows a simple annotation interface built inside a notebook. Watch “Build Your Own Eval Tools With Notebooks!” on YouTube It’s important to recognize that evaluation is part of the development process rather than a distinct line item, similar to how debugging is part of software development. You should always be doing error analysis . When you discover issues through error analysis, many will be straightforward bugs you’ll fix immediately. These fixes don’t require separate evaluation infrastructure as they’re just part of development. The decision to build automated evaluators comes down to cost-benefit analysis. If you can catch an error with a simple assertion or regex check, the cost is minimal and probably worth it. But if you need to align an LLM-as-judge evaluator, consider whether the failure mode warrants that investment. In the projects we’ve worked on, we’ve spent 60-80% of our development time on error analysis and evaluation . Expect most of your effort to go toward understanding failures (i.e. looking at data) rather than building automated checks. Be wary of optimizing for high eval pass rates . If you’re passing 100% of your evals, you’re likely not challenging your system enough. A 70% pass rate might indicate a more meaningful evaluation that’s actually stress-testing your application. Focus on evals that help you catch real issues, not ones that make your metrics look good. Yes. Even with perfect models, you still need to verify they’re solving the right problem. The need for systematic error analysis, domain-specific testing, and monitoring will still be important. Today’s prompt engineering tricks might become obsolete, but you’ll still need to understand failure modes. Additionally, a LLM cannot read your mind, and research shows that people need to observe the LLM’s behavior in order to properly externalize their requirements. For deeper perspective on this debate, see these two viewpoints: “The model is the product” versus “The model is NOT the product” . Don’t try to sell your team on “evals”. Instead, show them what you find when you look at the data. Start by doing the error analysis yourself. Look at 50 to 100 real user conversations and find the most common ways the product is failing. Use these findings to tell a story with data. Frame evaluation as part of development, not optional testing. Keep a running log of the errors you catch, what you learned, the fix, and the likely impact you avoided. Share it weekly or monthly. A concrete report such as “we caught 47 issues before users saw them” makes the value easier to see than an abstract pitch about evals. This approach builds trust. Don’t just show dashboards and metrics; tell the story of what you’re finding in the data. By narrating your findings, you teach the team what you’re learning, providing immediate value. When you fix an issue, show how the error rate for that specific problem went down. Soon, your team will see the progress and ask how you’re doing it. Let results instead of methods lead the conversation. This is similar to classic machine learning projects, where outcomes are speculative and progress is bounded by iterating on experiments . In this situation, it’s important that you share the learnings from each experiment to show progress and encourage investment. Error analysis is the most important activity in evals . Error analysis helps you decide what evals to write in the first place. It allows you to identify failure modes unique to your application and data. The process involves: Gathering representative traces of user interactions with the LLM. If you do not have any data, you can generate synthetic data to get started. Human annotator(s) (ideally a benevolent dictator) review and write open-ended notes about traces, noting any issues. This process is akin to “journaling” and is adapted from qualitative research methodologies. Start by annotating at least 30 traces yourself before reviewing suggestions from an agent.