Leaning into AI, ML, and observability to manage your ever-growing infrastructure

As systems become more complex, intuitively they generate a lot more telemetry and data. Without help, observability costs will rise in lockstep with infrastructure scale, posing a significant financial burden to organizations.

There are a variety of existing techniques that can be employed to manage this deluge of observability data. While techniques such as metric downsampling, trace sampling, and log deduplication can be helpful in specific scenarios, they come with significant caveats if not employed carefully. Strategies that ignore data sources or surgically remove contextual metadata are generally not recommended. As we will discuss later, the future of observability is highly dependent on assistance from machine learning (ML) and AI. These technologies thrive on and require high fidelity data. Finally, speculating on the value of new log files is a full-time job; your time is better spent resolving issues.

Fortunately, there is a better solution: By making storage (a lot) cheaper, you can simply store everything without compromising searchability. How? Start by ingesting data in such a way as to separate index metadata from the data itself. This allows the backend to still support fast searches without requiring all of the backing data to always be loaded in a cache. Then apply state-of-the-art data organization and compression. Finally, store the data in super cheap object storage with a caching layer. This strategy allows an organization to maintain the full data fidelity required by ML and AI without becoming cost prohibitive.

Perhaps the single largest change in modern observability is the advent of OpenTelemetry (OTel). OTel is an open source project that provides a standardized way to collect application and infrastructure logs, metrics, and traces. It offers a common set of tools, APIs, and SDKs for instrumenting applications and infrastructure.

Why is it such a big deal? OTel removes vendor lock-in and the need for proprietary agents. This in turn encourages more use of tracing as an observability signal. Further, because APIs are not vendor-specific, developers are encouraged to add application-specific attributes to their logs and spans. OTel makes the resulting data significantly more valuable both for SREs as well as ML and AI-based tools — imagine being able to analyze all of your telemetry data by customer ID, for example. Standardization of tracing APIs also encourages third parties to add instrumentation to their services, allowing visibility into infrastructure like reverse proxies and ingress controllers. Finally, a common format and tooling makes it easy for operators to process, route, and duplicate telemetry data to any OTel-compatible observability solution.

Adding OTel instrumentation to your systems is easier than ever: With the OTel Kubernetes Operator, instrumentation can be automatically injected into pods at runtime, without requiring changes to your devops workflow.

Collecting logs, metrics, and traces is a great first step, but the real power of these signals emerges when they are tied together with common correlating metadata. Correlation is a key tenet of the future of modern observability. Without it, debugging issues becomes a high-friction process, requiring analysts to manually correlate data across separate systems with inconsistent service names, timestamps, and contextual metadata. While humanly possible, this becomes a nearly impossible task for ML and AI-based tools.

OTel solves half of this problem by facilitating the consistent application of common metadata across observability signals. The other half of the problem is solved by bringing together logs, traces, and metrics into a single backend equipped for correlation.

Doing so gives SREs a superpower: They can seamlessly pivot between signals without friction, jumping from an alert to the associated traces and then drilling down into the related logs within seconds. This same level of correlation also makes it possible for ML and AI tools to look at the same problem from multiple angles, giving additional confidence and insight into a root cause.

As infrastructure and applications increasingly adopt ML and AI, observability solutions must also useML and AI to be successful. What is “success” for a modern observability solution? Preventing the mean time to resolution (MTTR) of an issue from scaling in lockstep with infrastructure complexity. Ideally, MTTR remains relatively constant regardless of infrastructure scale.

Modern distributed applications and cloud infrastructure have grown increasingly complex. The solution isn't to eliminate that complexity, but rather to monitor it with confidence. 

This requires four foundational capabilities: cost-effective storage, standardized data collection via OpenTelemetry, correlation between signals through unified metadata, and ML and AI-driven tools that democratize knowledge and make alerts actionable. Together, these capabilities will help constrain the mean time to resolution of an issue constant even as infrastructure continues to scale without bound.

With the advent of OpenTelemetry and AI, vendors will no longer differentiate themselves on ingestion technologies, basic analytics, and static dashboards; those things are becoming commodities. Future differentiation will come through innovations in storage (speed and cost) and AI-based workflow automation (alerts and dynamic dashboards).

This isn't just about monitoring; it's about enabling our systems, and the people who manage them, to operate effectively at a scale previously thought unimaginable. 

Want to learn more? Watch the Future of Modern Observability webinar to hear more about Elastic’s vision for how observability will evolve moving forward.