How TReNDS automates root-cause analysis with Amazon Bedrock

來源原標題

How TReNDS automates root-cause analysis with Amazon Bedrock

作者

Vitaly Omelchenko

原始標籤

Advanced (300) · Amazon Bedrock

原始連結

https://aws.amazon.com/blogs/machine-learning/how-trends-automates-root-cause-analysis-with-amazon-bedrock/

摘要原文

This is a guest post co-written with Vitaly Omelchenko from the TReNDS Center at Georgia State University. At the Center for Translational Research in Neuroimaging and Data Science (TReNDS) , a joint center of Georgia State University, Georgia Institute of Technology, and Emory University, we develop and apply advanced analytical methods and neuroinformatics tools for brain health research. We’ve been running our infrastructure on Amazon Web Services (AWS) since 2019, and over the years we’ve built a diverse set of applications, including research tools and APIs, all running on Amazon Elastic Kubernetes Service (Amazon EKS) with logs shipped to Amazon CloudWatch using FluentBit . As our application grew, so did the volume of errors we needed to investigate. When we started exploring Amazon Bedrock , we saw an opportunity we had wanted for a long time. We could automate the most time-consuming part of incident response, the root-cause investigation itself. In this post, we share the architecture we built and use in production at TReNDS. It combines Amazon CloudWatch subscription filters , AWS Lambda , the Strands Agents SDK , and Amazon Bedrock to detect errors in real time, enrich them with log context and source code from GitHub, and deliver AI-powered root-cause analysis to our team. The architecture and recommendations in this post reflect our team’s experience at the TReNDS Center and do not represent official guidance from Georgia State University, Georgia Institute of Technology, or Emory University. Like many teams, we had alerting and monitoring in place. We knew when things broke. However, knowing that something failed and understanding why it failed are different things. Our engineers still had to open Amazon CloudWatch Logs, read through stack traces, find the relevant source files, and mentally trace the execution path. For straightforward errors, this took 15–30 minutes. For complex issues spanning multiple services, much longer. We realized that this investigation process is exactly the kind of work a foundation model with the right tools can do. The model does more than summarize the error message. It investigates the error by pulling the surrounding log context, reading the source code, and producing a structured analysis. That is what we set out to build. Here’s the architecture we arrived at: Figure 1 — Architecture for automated root-cause analysis Our applications on EKS send logs to CloudWatch using FluentBit. A CloudWatch subscription filter watches for error-level patterns ( ERROR , Exception , FATAL , CRITICAL ) and invokes a Lambda function when a match occurs. The Lambda runs a Strands Agent powered by Amazon Bedrock that investigates the error, then publishes the analysis to an Amazon Simple Notification Service (Amazon SNS) topic for delivery to our team. The core of the system is Amazon Bedrock. The foundation model (FM) does the actual reasoning about errors, code, and root causes. We use the Strands Agents SDK on top of Amazon Bedrock to handle tool-use orchestration. We define what tools are available, and the model decides when and how to call them. Given a stack trace, the agent might fetch the relevant source file, realize it needs more context, search for related error handling, and produce a structured analysis, without us hardcoding that investigation path. Because TReNDS works with health-related research data, data residency and compliance are important considerations. Amazon Bedrock processes requests within our AWS account, so log data and source code stay within the same environment as the rest of our application. The AI analysis doesn’t require sending data to external endpoints. This keeps data flows within boundaries we already manage. This is particularly important for our work, because TReNDS handles health-related research data that might fall under HIPAA requirements. For more on Health Insurance Portability and Accountability Act (HIPAA)-eligible AWS services, see the AWS HIPAA Eligible Services Reference . While our setup uses EKS and FluentBit, this pattern works with other applications that send logs to CloudWatch, including ECS, Lambda, EC2, or on-premises workloads using the CloudWatch Agent. To implement this solution, you need the following: The agent’s capabilities come from the tools we give it. Of all the tools we built, source code retrieval is the most critical. Stack traces reference file paths and line numbers, but without access to the actual implementation, the agent would be limited to log pattern matching. By giving the agent the ability to read source files, it can trace execution paths and identify the specific code that caused the failure. With the Strands Agents SDK, you define a custom tool by decorating a Python function with @tool . Here’s the tool we built to fetch source code from our GitHub repositories: The docstring and type hints matter. Strands uses them to tell the model what the tool does and what parameters it expects. The model then decides when to call this tool based on what it finds in the error. See the custom tools documentation for more patterns. For deployment, we use the Strands Agents official Lambda layer . There’s no need to bundle the SDK manually. When an error occurs in one of our applications, the pipeline moves through four stages automatically. First, CloudWatch detects the error pattern and invokes our Lambda function with the compressed log data. The Lambda decodes the event, and the Strands Agent takes over from there. The agent fetches additional log context from the same container, retrieves relevant source code from GitHub, and reasons through the root cause. Finally, it publishes a structured analysis to SNS for delivery to our team. The following sections walk through each stage in detail. CloudWatch subscription filters send base64-encoded, gzip-compressed log events to Lambda. Each invocation contains one or more log events that matched the filter pattern within a short time window. The Lambda handler decodes the information, extracts the log group name and matching events, and passes them to the agent for analysis. See the CloudWatch Logs subscription filter documentation for the standard decoding pattern. The subscription filter delivers the matching log line, but a single line is rarely enough. The CloudWatch event information includes the logStream, which identifies the specific container that produced the error. We built a second @tool that fetches surrounding logs from the same stream. This gives the agent the full stacktrace and the request context that led to the failure, without noise from other concurrent requests: By scoping to the log stream, we get a clean, chronological sequence of events from the same container. This includes the request that triggered the error, preceding warnings, and the full exception trace. The agent receives the error plus context, then autonomously decides what to investigate. Unlike a rule-based system that follows predefined decision trees, the agent interprets the error message, identifies file paths and class names in the stack trace, and determines which source files to retrieve. If the initial code review reveals that the error originates in a dependency or a shared utility, the agent follows that chain without additional prompting from us. We shaped the output format through the system prompt: The system prompt defines a structured output format but leaves the investigation strategy to the model. The agent decides which tools to call based on what it finds in the error. A stack trace with clear file paths triggers fetch_source_code calls. An error without a stack trace might lead the agent to search the code base for the error message string. This flexibility is the core value of the agentic approach. We did not need to anticipate every type of error our applications can produce.