Natural language security queries at Orca Security using Amazon Bedrock

來源原標題

Natural language security queries at Orca Security using Amazon Bedrock

作者

Deniz Yilmaz

原始標籤

Amazon Bedrock · Customer Solutions

原始連結

https://aws.amazon.com/blogs/machine-learning/natural-language-security-queries-at-orca-security-using-amazon-bedrock/

摘要原文

Security analysts need fast answers about their cloud infrastructure without deep familiarity with domain-specific query languages. Orca Security ’s tool covers over 900 cloud models (the resource types Sonar tracks, such as an Amazon Elastic Compute Cloud (Amazon EC2) instance or an Amazon Simple Storage Service (Amazon S3) bucket). Querying that data required precise model names, nested relationship navigation, and selection from 18 operators (comparison functions like equals or contains used to filter results). To solve this, Orca Security partnered with AWS to build Discovery AI, a natural language query agent on Amazon Bedrock . Orca provides context-aware security for cloud, code, and AI, and Sonar is its domain-specific query system where users search, filter, and explore across the entire data graph. In this post, we share the architecture and results of Discovery AI, which converts a natural language question into a Sonar query. As a result, Orca reduced query costs by 90 percent, input tokens by 93 percent, and latency by 75 percent compared to the initial baseline. Orca Security’s Sonar query system supports 900+ cloud infrastructure and application models with complex relationships between them. To query this data, users construct JSON objects that specify exact model names, select from 18 operators, and navigate parent-child hierarchies across cloud providers. Doing this well requires deep familiarity with the Sonar schema. To build this capability, the Orca team set out to build a natural language interface that can generate valid Sonar queries from plain text. Their initial proof of concept loaded the full Sonar schema into a single prompt on the largest available model. This included over 900 model definitions, every relationship type, and a generous set of example queries. The output quality was strong (design partners and internal users responded positively) but three significant drawbacks emerged at production scale. Token consumption was inefficient, at roughly 2 million input tokens per workload, which drove up cost. Latency was high, with end-to-end response times of around 60 seconds per workload. Accuracy on complex, multi-model queries was limited because nested relationships and multi-condition queries often required additional refinement. The Orca team validated that the concept worked. The next challenge was making it cost-effective, fast, and accurate enough for production. To address this, the Orca team designed and deployed a solution that handles thousands of queries per day at 15-second response times using Amazon Bedrock. Orca designed Discovery AI as a hybrid solution on Amazon Bedrock that combines Retrieval Augmented Generation (RAG) with schema-aware query construction. The system converts natural language questions into valid Discovery filters through a multi-stage pipeline, with each stage matched to the most appropriate Amazon Bedrock model. Orca chose Amazon Bedrock because it provides multi-model access through a single API, letting the team allocate different foundation models to each pipeline stage without managing separate integrations. Amazon Bedrock also inherits AWS governance controls through AWS Identity and Access Management (IAM), virtual private cloud (VPC) endpoints, and AWS CloudTrail, meeting Orca’s security requirements. The model-agnostic architecture meant the team could swap in newer model versions as they became available without rearchitecting the pipeline. The following diagram illustrates the nine-stage pipeline architecture. Figure 1: Multi-stage Discovery AI pipeline architecture showing the flow from natural language input to validated Discovery filter queries using foundation models on Amazon Bedrock The Discovery AI pipeline converts a natural language request into a validated Discovery filter through nine steps. Example retrieval emerged as the single biggest accuracy lever in the pipeline. Strong examples teach the Generator how to translate questions into Sonar queries more efficiently than schema alone, so the Orca team benchmarked several retrieval strategies on Amazon Bedrock. Each strategy traded off relevance, diversity, or speed in a different way, and comparing them directly showed which combination worked best. Pure top-K semantic search on Amazon Titan embeddings performed quickly, but it tended to return near-duplicate examples that starved the Generator of diversity. Maximal Marginal Relevance (MMR) improved diversity, but it traded off relevance when the top candidate was actually the right answer. Retrieving a larger candidate set and then reranking it pulled in a broader pool and used a reranker to select the most relevant subset, which pointed toward the winning approach. The winner was top-20 nearest-neighbor retrieval using Amazon Titan embeddings. Orca then reranked these results with Cohere Rerank 3.5 down to the top five. This approach gives the Generator a small, diverse, highly relevant set of examples without forcing a trade-off between coverage and precision at retrieval time. A core constraint shaped the architecture. It wasn’t feasible to include 900+ Sonar model schemas into every prompt. Even with long-context models, doing so degrades both cost and accuracy. The Model Selector resolves this. Running on Anthropic Claude Haiku 4.5, it works against compact, semantically rich representations of each model (name, purpose, key fields, and key relationships). Given the user’s question, it returns only the models the Generator actually needs. The Generator then receives full schema detail only for those selected models plus the relationships that connect them. To iterate confidently, the Orca team built a continuous evaluation harness early in the project. The harness combines a curated dataset, an automated scoring layer, and a slice-based measurement approach so the team can trust every change before it ships. Together these elements turn evaluation into a routine step rather than an afterthought. The curated evaluation dataset spans the question types, model families, and complexity tiers seen in production, so it reflects real usage. An LLM-as-judge scoring layer grades generated queries on correctness, completeness, and semantic equivalence to a reference query, which gives the team a consistent quality signal. The measurement methodology then breaks deltas down by slice, allowing the team to identify exactly where regressions occur and what to correct. Every change to a prompt, a model, a retrieval parameter, or an example flows through the harness before it ships to production. Sustaining accuracy as customer environments evolve and new question patterns emerge required more than a strong initial deployment. The Orca team built an internal labeling tool for product managers, support engineers, and security researchers. With the tool, they can flag queries, correct outputs, and push corrections into the example bank in one action. Furthermore, the system automatically collects production queries with low-confidence matching examples and routes them to the appropriate domain expert for daily labeling. The next pipeline run retrieves the new example through Cohere Rerank 3.5, and accuracy on that class of question improves the same moment, with no deployment. “The labeling UI is the single most important investment we made for long-term quality. It turns Discovery AI from a static deployment into a living system, with our domain experts continuously improving it inside their normal workflow.” — Or Gabay, Director of AI, Orca Security Combined with end-to-end pipeline observability and shared dashboards spanning product, engineering, and customer-facing teams, this feedback loop is what keeps Discovery AI improving after initial launch. After deploying Discovery AI to production, the new solution outperformed the initial baseline across cost, latency, and accuracy.