Automated Reasoning policy refinement in Amazon Bedrock

來源原標題

Automated Reasoning policy refinement in Amazon Bedrock

作者

Nafi Diallo

原始標籤

Amazon Bedrock · Announcements · Intermediate (200)

原始連結

https://aws.amazon.com/blogs/machine-learning/automated-reasoning-policy-refinement-in-amazon-bedrock/

摘要原文

Refining an Automated Reasoning policy in Amazon Bedrock has been a manual cycle of diagnose, hand-edit, retest, and repeat. Today, we are announcing automatic policy refinement, which automates the diagnose-and-fix work in that cycle. The refinement engine diagnoses failing tests and proposes formal-logic fixes. You approve every change before it takes effect. Automated Reasoning checks in Amazon Bedrock Guardrails use formal verification to prove answer correctness. On unambiguous translations from natural language to formal logic, they deliver up to 99% verification accuracy, as reported in the GA announcement . To get started, you build an Automated Reasoning policy from a source document and validate it with test cases. Customers told us that this iterative tuning creates the biggest friction point in policy development. In this post, we walk through two new refinement modes: Iterative Refinement for rule issues, and Ambiguous Variable Refinement for language issues. For each mode, we show a complete API workflow (start, poll, retrieve) and a repeatable console workflow for turning failing policies into passing ones. Automated Reasoning checks translate natural language into formal logic, then apply automated reasoning techniques to produce a finding: VALID , INVALID , SATISFIABLE , IMPOSSIBLE , or TRANSLATION_AMBIGUOUS . For a full introduction to how policies work, refer to our GA announcement post . For this post, the key concept is the two-step validation pipeline . First, the translate step maps natural-language input/output to variable assignments using the variable descriptions in your policy. Second, the validate step applies your formal rules to those assignments. When a test fails, the root cause lives in one of those two steps, and each refinement mode targets a different one. Figure 1 traces that pipeline end to end. Figure 1: How Automated Reasoning checks validate a response at runtime. Automated Reasoning checks translate natural language into variables using the policy’s variable descriptions, then validate those variables against the policy’s formal rules to return a finding. This two-step pipeline is why refinement has two modes. Testing your policy. You validate a policy by attaching tests: each test is input/output text plus the result you expect. Run tests individually or as a batch. Failures tell you exactly where the policy diverges from your intent. Recall the two-step pipeline: translate (natural language to variable assignments) and then validate (formal logic to finding). A failed test means one of these steps produced something you didn’t expect. Automated Reasoning checks surface two distinct failure signals that map cleanly to each step. In a rule-issue failure, the translation works correctly: the right variables have the right values, but the validation result doesn’t match your expectation. The problem lives in your rules: a rule is too permissive, too restrictive, or missing entirely. Concretely, you expected INVALID but got SATISFIABLE because a missing or too-permissive rule lets a bad answer through. Or you expected SATISFIABLE but got INVALID because an overly strict rule blocks a correct answer. Mental model: The system understood the question perfectly but applied the wrong logic. You need to fix the rules . When a test returns TRANSLATION_AMBIGUOUS , the validation engine runs and produces different outcomes depending on which interpretation it follows. In some cases, the translation models disagreed on how to map the natural-language input to your policy’s variables, and each competing interpretation led to a different validation result. The finding surfaces two or more options, each with its own translation and conclusion, plus differenceScenarios showing where the interpretations diverge in practice. Common root causes include overlapping variable definitions (“tenure” compared to “years of service”), vague descriptions, and inconsistent value formats (5 compared to 0.05 for “5%”). This table summarizes which refinement mode addresses which failure type: Use Ambiguous Variable Refinement when the system cannot determine a single translation. The next two sections walk through each mode in turn: what it does, when to use it, how the review gate works, and how to launch it programmatically. We start with Iterative Refinement because rule-issue failures are the more common case. When tests fail because the logic is wrong (the translation is clean but the validation result doesn’t match your expectation), the problem lives in your rules. Iterative Refinement ( ITERATIVELY_REFINE_POLICY ) automates the diagnose-and-fix cycle so you don’t need to manually trace each rule, hypothesize a correction, and hand-edit formal logic. Consider a policy with 10–30 rules. Previously, a fix would take a subject matter expert multiple rounds of manual diagnosis and hand-editing of SMT-LIB formal logic. That work now compresses to a single review-and-approve step, with no formal logic written by hand. Iterative Refinement takes three inputs. The first is your existing policy definition (the current rules, variables, and types). The second is a source document containing the authoritative natural-language text that describes how things should work. The third input is optional: natural language feedback with explicit instructions describing the change you want. For example, the feedback field might contain: “Update the tenure requirement for parental leave from 12 months to 6 months, as specified in section 3 of the revised document.” Given these inputs, the refinement engine analyzes how the current rules diverge from the source document and your feedback. It proposes a set of candidate changes (new rules, edited rules, added variables) that bring the policy in line. Iterative Refinement, as the name suggests, iterates. Behind the scenes, the engine generates a candidate change, simulates its effect on your saved tests, checks whether the previously failing tests now pass, and adjusts if they don’t. This can involve several internal cycles for a single request, especially when a fix in one rule ripples into others. The iteration happens internally , though: you don’t observe each intermediate attempt, and you don’t need to shepherd it. What you receive is the converged result: a proposed diff that shows exactly which rules changed, which variables changed, and how the change affects every test in your suite. After convergence, the Review policy changes screen appears. You then select Accept changes or Discard changes . Accepting writes the changes to your DRAFT policy. Discarding leaves everything exactly as it was. Iterative Refinement requires at least one test attached to your policy. Without a failing test signal, there’s nothing to drive the refinement. Use this mode when the translation is correct (right variables, right values) but the validation result is unexpected. Do not use it when the finding is TRANSLATION_AMBIGUOUS . That’s a language problem better addressed by Ambiguous Variable Refinement . Refinement runs as an asynchronous build workflow. Using the AWS SDK for Python (Boto3), the flow has four steps: export the current policy definition, start the workflow, poll for completion, and retrieve the proposed changes. Set buildWorkflowType to ITERATIVELY_REFINE_POLICY . The iterativeRefinementContent block accepts one to five source documents (required) and up to 4,000 characters of optional feedback: The call returns immediately with a buildWorkflowId , not the proposed changes. The workflow moves from SCHEDULED to BUILDING until it reaches COMPLETED , FAILED , or CANCELLED . Convergence typically takes one to a few minutes, depending on policy size. Poll get_automated_reasoning_policy_build_workflow until the status reaches a terminal state.