Automated web insight extraction with Amazon Bedrock AgentCore

來源原標題

Automated web insight extraction with Amazon Bedrock AgentCore

作者

Louisa Liu

原始標籤

Advanced (300) · Amazon Bedrock · Amazon Bedrock AgentCore · Technical How-to

原始連結

https://aws.amazon.com/blogs/machine-learning/automated-web-insight-extraction-with-amazon-bedrock-agentcore/

摘要原文

Extracting insights from dozens of websites often means manually checking each one, a process that quickly becomes overwhelming. Design teams need to track competitor products, marketing teams want to monitor content trends, and product managers need to stay on top of market intelligence. But doing this manually means someone has to visit sites, copy content, and organize information before real analysis can begin. Rule-based scrapers offer some automation, but they are tightly coupled to page structure. A site redesign or a migration to a JavaScript-rendered frontend can silently break the pipeline for days before the team notices. Amazon Bedrock AgentCore is a platform to build, connect, and optimize agents at scale, with any framework or model. The solution in this post uses the AgentCore Browser, a capability of Amazon Bedrock AgentCore. This fully managed browser service renders JavaScript-heavy pages reliably, so your pipeline is more resilient when websites change. This post demonstrates how to deploy an automated insight extraction solution using Amazon Bedrock AgentCore Browser, Amazon Bedrock for AI-powered analysis, Amazon OpenSearch Serverless for semantic search, and AWS Lambda for orchestration. You build a system that monitors RSS feeds, retrieves web content using the AgentCore managed browser, extracts insights with AI, and makes everything searchable through a web interface. This solution was built for design and product teams who need to track industry trends, but the architecture applies to broader scenarios: At the end of this post, you will understand how the event-driven architecture separates content collection from AI processing, how the browser automation in Amazon Bedrock AgentCore handles JavaScript-heavy pages, and how vector embeddings in Amazon OpenSearch Serverless power semantic search across your collected insights. The complete implementation is available in this GitHub repository . If you are new to the services used in this solution, the following resources provide foundational context: The architecture follows an event-driven pattern that separates content collection from processing. The following diagram illustrates the end-to-end system, organized into three functional layers. Figure 1 — Architecture diagram for the automated web insight extraction system RSS feed collection: An Amazon EventBridge schedule triggers an AWS Lambda function every 15 minutes to check configured RSS feeds for new articles and deduplicate against Amazon Simple Storage Service (Amazon S3). Browser-based content retrieval: For each new article, the Lambda function opens a browser session through Amazon Bedrock AgentCore and connects to it using Playwright over the Chrome DevTools Protocol (CDP). Unlike standard HTTP requests, the remote browser renders the full page including JavaScript-heavy content, waits for dynamic elements to load, takes a screenshot, and downloads images. This is the step that makes the rest of the pipeline possible: without reliable page rendering, the AI extraction downstream would receive incomplete or broken content. The artifacts are uploaded to Amazon S3 in a structured format. Event-driven processing: Amazon S3 upload events publish messages to an Amazon Simple Queue Service (Amazon SQS) queue, and a second Lambda function extracts clean text from the raw HTML. AI-powered insight extraction: Amazon Bedrock generates summaries, identifies themes and entities, extracts actionable insights, and creates vector embeddings from the cleaned content. Indexing and semantic search: The enriched results are indexed into Amazon OpenSearch Serverless, which supports both keyword and vector search. User and API access: End users authenticate through Amazon Cognito and access a React-based frontend on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate . For programmatic access, a Model Context Protocol (MCP) server on Amazon ECS with Fargate exposes the system through Amazon CloudFront . MCP is an open standard that lets AI assistants and tools connect to external data sources through a unified interface. The following sections describe the main components of the solution. The RSS sync Lambda function does more than fetch feeds. It parses each RSS feed, filters for articles published within a configurable time window (24 hours by default), and deduplicates using a URL hash as a unique ID in Amazon S3. For new articles, the function opens a remote browser session through Amazon Bedrock AgentCore and connects via Playwright over CDP. This isn’t a headless browser running in Lambda. It is a managed browser service: AgentCore hosts the remote browser, and Playwright controls it through a WebSocket connection. The browser renders the full page, waits for JavaScript to load, and captures the complete rendered output. The function then extracts images from the page, downloads them, and uploads everything to Amazon S3 in a structured format: The metadata.json file includes the original URL, title, timestamp, and references to downloaded assets. When the metadata file is uploaded to Amazon S3, it triggers the processing pipeline automatically. This separation means content collection and AI processing scale independently. If you’re monitoring dozens of RSS feeds, the collection Lambda function handles them concurrently while the Amazon SQS queue with a dead-letter queue provides automatic retries for processing failures. The idea extraction Lambda function preprocesses HTML to reduce token consumption, a step that significantly improves the consistency of AI-generated output. When it receives a message from Amazon SQS, it pulls the HTML from Amazon S3 and runs it through a preprocessing step. Large HTML files (over 1 MB) get simplified using html-to-text to avoid token limits. Smaller files use Mozilla’s Readability library, which does a better job of extracting the main content and identifying the primary image. The cleaned content then goes to Amazon Bedrock with a customizable prompt. You can use models supported by Amazon Bedrock. For model availability by AWS Region, refer to Supported models by AWS Region in Amazon Bedrock . The prompt asks the AI to extract: The Lambda function also generates vector embeddings for semantic search, so you can find related content even when the exact words don’t match. The function writes everything to Amazon OpenSearch Serverless: the original content, AI-extracted metadata, and the embedding vector. The index is configured for both keyword and vector search. A query like “what are emerging design trends” returns relevant results even if those exact words don’t appear in the source content. Because this pipeline sends third-party web content to a foundation model and publishes the AI-generated output to your team, production deployments should include safeguards around the extraction step. Amazon Bedrock Guardrails lets you apply these controls without changing the extraction prompt: Because the pipeline ingests content from external websites, treat the scraped text as untrusted input: guardrails act as a control point between raw web content and the insights your organization consumes. The React-based frontend with Amazon Cognito authentication provides a search interface where teams can browse and explore insights. The MCP server provides an API layer for integration with other AI tools, supporting programmatic search and retrieval through Amazon CloudFront. MCP is an open standard that defines how AI assistants discover and call external tools, so other AI agents in your organization can query the insight database directly. Processing components run within an Amazon Virtual Private Cloud (Amazon VPC), with Amazon CloudWatch providing observability and AWS Identity and Access Management (IAM) enforcing least privilege access. Building a real-world insight extraction pipeline involves more than wiring together managed services.