NarrateAI: production-ready LLM quality assurance on Amazon Bedrock

來源原標題

NarrateAI: production-ready LLM quality assurance on Amazon Bedrock

作者

Felicia Yue Xiong

原始標籤

Advanced (300) · Amazon Bedrock · Technical How-to

原始連結

https://aws.amazon.com/blogs/machine-learning/narrateai-production-ready-llm-quality-assurance-on-amazon-bedrock/

摘要原文

Executives need to make data-driven decisions during live business reviews, where accuracy and speed matter. A conversational agentic AI assistant can meet this need by answering data questions instantly. But the stakes are high: a wrong number or a slow response in front of leadership carries immediate professional consequences, and a capable large language model (LLM) alone can’t guarantee either. In production, that gap shows up as hallucinated metrics, API throttling, validation latency, and subjective language. Closing it requires production-ready quality assurance built into every step, from data retrieval to response delivery. This post details five techniques, implemented on Amazon Bedrock , that work together to deliver it: adaptive pipeline orchestration, cross-account multi-model failover, real-time streaming evaluation, composite evaluation framework, and data accuracy verification. Each addresses a distinct failure mode while operating as a coordinated system. This post is the second in our NarrateAI series. NarrateAI transforms business intelligence for over 4,000 AWS executive leaders through a two-layer architecture built on Amazon Bedrock AgentCore , a platform to build, connect, and optimize agents at scale, with any framework or model. The architecture comprises an Automated Narrative Generation Layer for batch processing and a Conversational AI Interface Layer for real-time interaction. Our previous post covered the business challenges, overall architecture, user experience, and enterprise deployment. This one goes deeper into the engineering, showing how the five techniques achieve approximately 99 percent numerical accuracy while streaming responses in real time. It’s written for engineers and architects building LLM applications who are familiar with LLM APIs and streaming responses. This post focuses exclusively on the advanced quality assurance mechanisms within the real-time layer. Consider an executive asking “Which regions aren’t meeting their targets and why?” The five techniques work together so that the response is numerically accurate and arrives in real time. It also maintains throughput under concurrent global usage and is professionally formatted for direct use in business reviews. Figure 1 shows how these five techniques form a layered dependency chain, where each layer feeds the one below it to produce a validated real-time response from raw queries. Adaptive Pipeline Orchestration routes queries by data volume so that most complete in a single fast pass while complex queries receive full parallel treatment. Cross-Account Multi-Model Failover expands available inference capacity across independent model-account quota spaces, reducing user-visible throttling. Real-Time Streaming Evaluation validates each paragraph the moment it’s produced, overlapping quality checks with generation. The Composite Evaluation Framework runs multiple independent evaluators in parallel against each paragraph. Data Accuracy Verification catches numerical hallucinations through a two-stage cascade. It starts with cheap exact matching and escalates to semantic verification only when needed. The following sections walk through each technique in the order it appears in the pipeline, from data retrieval through generation to quality assurance. Figure 1: How the five quality assurance techniques layer together to turn raw queries into validated real-time responses Enterprise knowledge documents contain extensive information and queries vary enormously in how much of it they need. A focused question like “What’s my team’s quarterly attainment?” might draw on a handful of sections (discrete chunks of enterprise knowledge documents retrieved). A comprehensive request like “Give me a full regional performance analysis” requires synthesis across hundreds. Users expect the same response time either way. This heterogeneity requires a routing strategy that handles the common case quickly without sacrificing thoroughness for complex scenarios. No single processing strategy works for all queries, because the right approach depends on how much data a query actually requires. Single-pass concatenation is fast and cheap, but it breaks down when aggregated sections exceed the model’s context window (typically 200K tokens), forcing truncation and quality loss. Fixed multi-pass batch processing avoids truncation, but it pays for multiple LLM invocations on every query, including the roughly 90 percent that don’t need them. Our approach routes each query based on its total aggregated section volume ∣D∣ (where D is the set of retrieved document sections): single pass when the data fits, batch processing only when necessary. The result is a three-phase pipeline that classifies each query once and then processes it along the cheapest path that preserves quality. The pipeline processes every query in up to three phases: Mode-Aware Consolidation packs retrieved sections and picks a route, Bifurcated Analysis executes that route as either a single fast-path call or parallel normal-path batches, and Conditional Consolidation merges the parallel results when needed. The following table summarizes how the two paths move through these phases before we walk through each phase in detail. The first phase, Mode-Aware Consolidation, operates on retrieved document sections. It applies a greedy first-fit packing strategy to fill context windows up to the token limit while preserving section priority order and section boundaries. For smaller volumes (∣D∣≤θ, where θ is an empirically calibrated character-count threshold), all sections are concatenated into a single chunk, and this is called “fast path”. For large volumes (∣D∣>θ), sections are packed into optimal batches that respect document boundaries and this is called “normal path”. Threshold θ is set relative to the model’s context window limit and calibrated to capture 70–90 percent of queries on the fast path (in practice, approximately 90 percent take this path, as shown in the Results section). Because the execution mode is fixed in this first phase, each downstream phase can optimize for its specific path rather than handling both. The second phase, Bifurcated Analysis, executes the chosen path. The fast path makes a single LLM call with the complete context and streams results directly to the evaluation pipeline. The normal path distributes batches across N parallel LLM invocations (where N is the number of batches produced by Phase 1) that analyze chunks independently to achieve near-linear speedup. The third phase, Conditional Consolidation, operates exclusively on the normal path by synthesizing N independent analyses into a unified response. The consolidation LLM receives partial analyses with the original question and applies conflict resolution heuristics while streaming the final answer. The fast path bypasses this phase entirely because the final response only uses one analysis. Figure 2 shows the complete flow, with both paths converging on the streaming response. Figure 2: End-to-end flow of the three-phase pipeline, showing the fast path bypassing consolidation In production, this threshold-based routing delivers outsized efficiency. Approximately 90 percent of queries take the fast path (single LLM call, time-to-first-token (TTFT) within a few seconds, total latency typically under 25 seconds). The remaining 10 percent (complex multi-document queries) receive the full parallel batch treatment (approximately 50–75 seconds compared to under 25 seconds on the fast path) commensurate with their complexity. With N=4 typical batches for normal-path queries, the blended cost per query is approximately 1.4 LLM invocations. That is a 72 percent reduction in LLM invocations per query versus always using the multi-pass strategy, while maintaining full quality coverage for complex queries.