Build visibility for Codex on Amazon Bedrock with OpenTelemetry and Amazon CloudWatch

來源原標題

Build visibility for Codex on Amazon Bedrock with OpenTelemetry and Amazon CloudWatch

作者

Claudio Mazzoni

原始標籤

Advanced (300) · Amazon Bedrock · Amazon CloudWatch · Technical How-to

原始連結

https://aws.amazon.com/blogs/machine-learning/build-visibility-for-codex-on-amazon-bedrock-with-opentelemetry-and-amazon-cloudwatch/

摘要原文

As organizations move from experimenting with coding agents to adopting them across engineering teams, the leadership question changes. It is no longer only, “Can this tool help a developer?” It becomes, “How do we understand adoption, manage consumption, maintain reliability, and scale access responsibly?”. Codex can emit OpenTelemetry (OTel) metrics about its activity. When local Codex clients use OpenAI models through Amazon Bedrock and authenticate with AWS IAM Identity Center, you can route those metrics through a local OTel collector to Amazon CloudWatch. The result is an AWS native view of Codex usage that can be organized by user, team, department, organization, or cost center. This approach does not add a centralized proxy to the model-request path. Developers continue to use Codex locally. A collector running on each developer workstation receives metrics on the local host and enriches them with organizational context. It then sends them to the regional CloudWatch OpenTelemetry Protocol (OTLP) endpoint using AWS Signature Version 4 (SigV4). The reference deployment creates a CloudWatch dashboard, not an Amazon Elastic Container Service (Amazon ECS) service, load balancer, virtual private cloud (VPC), or public ingestion endpoint. In this post, we explain how this pattern supports governed adoption, review its architecture, and summarize the implementation in the Codex on AWS guidance repository . Telemetry is most useful when it answers a decision, not simply when it produces another dashboard. The bundled CodexOnBedrock dashboard includes rolling 24-hour totals for active users, conversation turns, API requests, and token usage. It also provides views by model, token type, user, department, team, cost center, organization, and session source. These signals can help technology leaders distinguish broad adoption from isolated experimentation. For example, an increase in active users across several teams suggests a different enablement need than high consumption concentrated among a small group. Tool-call activity can help teams see where agentic workflows are taking hold. Request and duration metrics can support investigation of degraded experiences. The distinction in the last row is important. CloudWatch OTel metrics show usage volume and operational behavior. Token counts can support trend analysis, but they are not a billing ledger. List-price estimates can diverge from actual charges because of pricing changes, discounts, credits, and billing adjustments. For realized spend, use IAM principal cost allocation from AWS Cost and Usage Reports (CUR) 2.0 or the applicable Amazon Bedrock cost-management reports. The architecture uses the same AWS identity for model access and telemetry publishing. A developer signs in through IAM Identity Center and runs Codex with Amazon Bedrock as the model provider. Codex sends metrics to a collector listening only on 127.0.0.1 . The collector adds identity and organizational attributes, batches the metrics, and signs requests to CloudWatch with temporary AWS credentials. Local Codex telemetry flow to Amazon CloudWatch through a local collector that publishes metrics with SigV4 authentication and stays out of the Amazon Bedrock inference path Codex emits metrics such as codex.api_request , codex.api_request.duration_ms , codex.turn.e2e_duration_ms , codex.turn.token_usage , codex.turn.tool.call , codex.thread.started , and codex.conversation.turn.count . The current Codex configuration documentation describes OTel as opt-in and documents separate exporters for logs, metrics, and traces. The local collector adds user.id and user.email as required resource attributes. Optional attributes include department , team.id , cost_center , organization , location , role , and manager . The reference collector copies these attributes onto each metric datapoint as well. This approach gives the dashboard consistent Prometheus Query Language (PromQL) groupings across the local sidecar and other supported ingestion patterns. The complete commands and templates are in the native AWS access quickstart . The following five stages summarize the implementation. The reference runbook begins by enabling OTel enrichment and resource tags for telemetry in the target Region: The current CloudWatch OTel documentation describes native OTLP ingestion and PromQL querying. The start-otel-enrichment operation enables enrichment and PromQL access for supported AWS vended metrics. The start-telemetry-enrichment operation enables resource-tag enrichment. Confirm which account-level settings are already enabled before changing them. Clone the repository, then deploy the dashboard and obtain the collector binary: The AWS CloudFormation stack deploys the CodexOnBedrock dashboard. The collector runs on developer workstations, so this step does not create centralized collector compute or networking infrastructure. Generate the collector configuration from the developer’s authenticated AWS profile: With --auto-lookup , the script can read organizational attributes from the IAM Identity Center identity store. Explicit command-line values can override discovered values. If an optional attribute is unavailable, omit its entire configuration block. Do not send placeholder strings as dimensions. They create low-value series and weaken reporting quality. For a fleet rollout, generate and distribute this configuration through your existing endpoint-management process. Treat organizational metadata as governed data: establish approved values, ownership, and update procedures before using it for executive reporting. Point the Codex metrics exporter to the local collector. Include the full /v1/metrics path because Codex does not append it: The collector forwards metrics to the regional CloudWatch OTLP metrics endpoint , such as https://monitoring.us-west-2.amazonaws.com/v1/metrics . SigV4 is the recommended authentication method for short-term AWS credentials. The publishing identity requires cloudwatch:PutMetricData . No log-group or ECS permissions are required for this metrics path. Retain log_user_prompt = false . This design is intended to measure operational and adoption signals, not collect source code or prompt content. Start the service-specific collector with the generated configuration, run a Codex task, and then open the CodexOnBedrock dashboard in the CloudWatch console. You can also use CloudWatch Query Studio or the repository’s check-otel-pipeline.sh script to confirm that codex.turn.token_usage is arriving. Codex metrics flush periodically and on a clean process exit. The reference runbook documents a 60-second interval and suggests OTEL_METRIC_EXPORT_INTERVAL=1000 as an optional safeguard for error paths that might skip the exit flush. If no metrics appear, also confirm that managed configuration has not set [analytics] enabled = false , because that setting disables the Codex metrics pipeline. The same dimensions that make the dashboard useful can create privacy and governance concerns if they are exposed too broadly. Use aggregated team, department, and cost-center views for executive reporting. Restrict per-user dashboards to approved system, operations, security, or finance roles, and align access with employee-monitoring and data-retention policies. Control metric cardinality as you scale. Standardize attribute names and values, omit fields that do not support a defined decision, and avoid adding project names or ephemeral identifiers without a retention and query plan. CloudWatch OTel metrics use per-gigabyte ingestion pricing, and PromQL queries are priced according to samples scanned. Review the current CloudWatch OTel pricing documentation and Amazon CloudWatch Pricing before a broad rollout. This pattern provides visibility and soft controls. You can use CloudWatch alarms and Amazon Simple Notification Service (Amazon SNS) notifications to alert when a user or team crosses a defined usage threshold.