For decades, the observability sector has been defined by a binary question: Is the system running? Whether it was a server, a database, or a microservice, if it responded to pings and met latency targets, the system was considered healthy. However, the rise of agentic artificial intelligence has rendered this traditional performance model obsolete. An AI agent can operate with perfect uptime, meet all latency goals, and throw zero technical errors, yet still provide an incorrect response, execute an unauthorized tool, or hallucinate using stale data.
Amazon Web Services Inc. is looking to solve this “trust gap” with the general availability of Amazon CloudWatch Omni. Built on OpenTelemetry and designed for an era where IDC forecasts suggest over one billion agents will be deployed by 2029, Omni shifts the industry’s focus from infrastructure health to behavioral accountability. As enterprise IT teams transition from managing static code to governing nondeterministic AI behavior, the ability to answer the question, “Why did my agent do that?” has become a business imperative.
At its core, Omni is an evaluation engine rather than a traditional monitoring dashboard. It ships with 17 built-in evaluators that score agents on metrics such as faithfulness, coherence, and routing correctness. By moving evaluation to the forefront, AWS enables teams to run continuous quality checks against production traffic. For early adopters like Sony, this capability is essential for managing the sheer scale of production-ready AI. “At this scale, observability and evaluation are essential,” said Masahiro Oba, senior general manager of the AI Acceleration Division at Sony. The ability to move from a single trace to an AI-driven analysis or dataset creation in one click, he noted, removes the bottleneck of manual data preparation.
Perhaps the most significant shift for DevOps teams is Omni’s departure from the standard AWS Management Console. Recognizing that AI engineers and site reliability engineers require tools that align with their existing workflows, AWS has introduced native extensions for Visual Studio Code, Cursor, and Kiro. Developers can now instrument and debug agents locally without requiring an AWS account, while operators utilize a standalone web experience that integrates with existing identity providers such as Okta and Microsoft Entra ID. Because both interfaces share a single, unified data layer, the trace a developer investigates during the build phase is the same data an operator analyzes in production.
This unified approach allows organizations to correlate disparate signals—from an API failure in an application to an exhausted database connection pool—within a single context. Capital One, a design partner for the project, has emphasized the importance of this topology-aware intelligence. For regulated industries, the ability to maintain a complete investigation history serves as critical audit evidence for how an AI incident was handled and rectified.
Despite its integration with the AWS ecosystem, Omni leans heavily into openness. It supports industry standards including OpenInference and the AWS Distro for OpenTelemetry, ensuring compatibility with frameworks like LangChain, CrewAI, and the OpenAI Agents SDK. While this reduces vendor lock-in, AWS remains the primary beneficiary, as the “intelligence layer”—the topology mapping and evaluators—is tethered to the AWS backend.
For enterprise leaders, the adoption of Omni requires a fundamental shift in operational discipline. The platform is not merely a monitoring tool; it is a governance framework. As teams prepare for the complexities of agentic AI, they must define specific benchmarks for what constitutes a “good” outcome, standardize instrumentation through OpenTelemetry early in the development lifecycle, and carefully model the telemetry costs associated with agentic “chattiness.” As the industry enters a new phase of AI maturity, the companies that succeed will be those that treat evaluation not as an afterthought, but as a core component of their operational strategy.
Source: CIO.com
