How The Great Cat Escape game works

In this detective game, the rules that you eventually create in your Elastic cluster govern whether you win or lose against these cats. Each cat creeps toward the fence at its own accelerating pace. If any cat hits 10 escape attempts, that cat is free and you lose your detective license. Your job, gumshoe, should you choose to accept it, is to configure an Elastic Observability alert that fires before any cat hits 10. The earlier it fires, the higher your score.

 Requirements: Elastic Observability, Python, and a tunnel

• Your own fork/branch of The Great Cat Escape

• Elastic Observability v. 9.3.0 or higher (get a seven-day free trial of Elastic Cloud Serverless), 

• Python v. 3.11.0 or higher (latest version)

• ngrok or a similar tunnel, since Kibana's webhook needs a way to reach your laptop

The Great Cat Escape game is governed through python-dotenv with credentials from an environment (.env) file. Change these to match your Elastic cluster, and make sure that the connection is through Elastic APM. You can input your secret token through your agent by going to Fleet → Agent Policies → click on the policy you want to change → Elastic APM → Settings → and then scroll down until you see "Agent Authorization."

To populate your environment, take a copy of the .env.example and rename it .env. Your .env file needs two values:

NUM_CATS is optional (default 1, max 6). More cats means more chaos, but more chaos means a potentially higher score for you, detective.

Instrumenting a Python app with OpenTelemetry for Elastic APM

Catching rogue cats is hard. What should we trigger on? What should be measured? Why? To answer these questions, we need to know what our cats are doing at any given time. That’s where instrumentation comes in. In observability, instrumentation is the process of adding code or configurations to your application that can generate and collect telemetry data, as well as export it. This telemetry data can include metrics and logs, along with traces. So let’s get into the code that will help your investigation.

To instrument our application, we’ll use OTel, which is a standardized framework for generating, collecting, and exporting telemetry data. It’s the industry standard for observability. Before we get started, we need to define a resource in our project. In OTel instrumentation, a resource represents the entity that’s producing telemetry data, which, in this case, is our “escape-room-cat” service.

The “service.name” key is a standard semantic convention from OTel. Semantic conventions are standardized and vendor-neutral naming rules used throughout OTel to structure your data. You’ll see more of these conventions as we set up our signals in the sections that follow.

Setting up traces, metrics, and logs with OTLP exporters

We set up our three pillars and our logs, metrics, and traces through OTel in the following code. Let’s go through the code for each signal.

With this code, we set up the TracerProvider to start collecting traces, which are then exported in batches through the OTLPSpanExporter via the OpenTelemetry Protocol (OTLP). A trace is a collection of spans, and spans can be enriched with span attributes. We go over the span "cat.escape_attempt" later in this article.

Next, we set up our metrics through a MeterProvider and a PeriodicExportingMetricReader. Metrics are polled on a timer according to the reader’s interval and then exported in OTLP format through OTLPMetricExporter, rather than being pushed when events arrive, like traces and logs. The MeterProvider is still necessary despite the PeriodicExportingMetricReader because it’s the configuration layer from the OTel SDK that actually processes and collects your telemetry and exports it to your Elastic Observability project. This is true for the TracerProvider and for the LoggerProvider (see below):

LoggerProvider works slightly differently than the rest of the signals. The code here bridges Python’s built-in logging module, LoggingHandler, with OTel, instead of creating a new logging system. This handler routes any logging.info call in the Python application into the OTel pipelines, which means that your structured log records get exported alongside the rest of your signals.

Counters, gauges, and spans: The signals this game emits

Now that we know how OTel works to instrument our game, let’s get into the details of the specific signal configurations that we’ll use in this application.

First, let’s go over the most important counter (a data point that only goes up):

This metric is important to the game because it determines whether you win or lose. If the ”cat.escape.attempts” counter for any cat hits 10, you lose and the cat gets free. 

The next metric is an observable gauge governed by a function from our game:

This observable gauge meter indicates that, at each collection interval, a cat’s current suspicion level is recorded. “cat.suspicion.level” is governed by a function that picks a random floating number between 8 and 15 and appends that to the current suspicion level. An observable gauge represents a value that can go up and down, unlike a counter that only goes up. We’re using an observable gauge instead of a counter because we’re trying to determine How suspicious is this cat right now? as opposed to How much suspicion has accumulated? In this game, cat suspicion is another meter that you might be able to create a rule on.

Below is the central event, a span that fires every time a cat attempts to break out. This span identifies which cat it was, the escape attempt number, the cat’s position, whether the attempt was successful, and the cat’s suspicion level. The next span is also collected whenever the cat moves:

When you view the traces data in Kibana, the trace and its span attributes appear as follows:

The fields translated from our game are prepended with “numeric_labels” or “labels.” This is the result of our data being translated from OTel into the Elastic Common Schema (ECS). However, you can rename these in Kibana, if you choose, to shorten the name or make searching easier for you. 

Emitting structured logs with the Python logging module

Logs are the most human-readable of the three signals, and this game emits them through semi-structured logging:

The game sends out several logs and communicates them via multiple methods. When it comes to logging, it’s generally good practice to keep your logs clear and concise so that they’re easily recognizable by anyone debugging, especially for errors. In this Flask application, we used the logging module by Python. The logs sent to Kibana are “logger.info”, “logger.warning”, and “logger.error”, and they indicate the log severity level. logger.info is used for generic information from your application, and logger.warning indicates unexpected events that the system has recovered from. logger.error notes a failure or critical error. In The Great Cat Escape, the only error is when you lose the game. These logs are emitted through the standard Python logger and exported via the OTLP log exporter, so they show structured severity_text and resource attributes, along with trace correlation in Kibana:

The next logging system is _log, a custom function written for in-game use:

This log function shows the player the same information as Kibana receives by grabbing the current time and by using game_lock through game_lock = threading.Lock(). In this game, since there are requests from multiple threads (cats), the game_lock helps by preventing data loss if two threads reach out to the same resource at the same time, which can lead to data corruption. 

Now that you understand the instrumentation, the final step before you write your rules and play the game is to understand the alert_fired API and wire the needed webhook so that your alert can be received by your URL. 

Wiring a webhook connector for your alert

To send out a webhook alert, you use ngrok or a similar tunnel to expose your port to Kibana, enabling it to POST to that port over the internet. You front your own port with a tunnel, and that public URL is the <your-tunnel> you point Kibana at. Add a connection of type “Webhook” in Kibana with these settings:

In Kibana 9.4, the connector lives under Stack Management → Connectors. Save it once, and you can reuse it across all of your rules.

The {{context.group}} field shows which cat triggered the rule, and the fired_at {{context date}} field tells you when the alert happened. These are purely informational parameters, but the winning endpoint, /api/alert_fired takes more precaution with “current_count” and “rule_name.” The game's /api/alert_fired endpoint, shown in the following code, has several checkpoints based on the parameters above that help with specificity in this game. These checkpoints are also important to real alerting in systems to ensure that the correct source, system, or alert was sent via your alerting webhook.

The endpoint runs three checkpoints before it accepts a win, matching the numbered comments in the code above:

1. Parse the request body. request.get_json takes the body sent by the webhook and parses it into a Python dict. Passing silent=True means an empty or malformed body won't raise an error; the endpoint keeps working and fills in unknown values with {}.

2. Check the rule name. "Cat Escape Alert" is the enforced rule_name, so a stray request from another source can't trigger a win. Use this name exactly, or change it in the code. (See EXPECTED_RULE_NAME = "Cat Escape Alert".)

3. Check the attempt count. This gate stops an alert from counting as a win when no cat has actually moved. It's set to one, so at least one escape attempt must have registered. If you want a higher bar, change it in the code. (See MIN_COUNT = 1.)

How to create a Kibana alert rule with ES|QL

This game ends one of two ways. You win if your created rule alerts properly; and you lose if any cat reaches 10 escape attempts. 

To create your rule and play the game, follow these instructions. Lettered subsections give you a full breakdown of how to create a winning rule.

1. Go to Kibana → Stack Management → Rules.

2. Create new rule → Elasticsearch query.

3. Create a rule to alert on logs or metrics. You can use Kibana Query Language (KQL) or Lucene, Query DSL, or ES|QL.

• To create a winning rule, choose ES|QL. Enter the code block below into the box for creating a query. The service for this game is “escape-room-cat,” and “cat.escape.attempts” IS NOT NULL discards every row that isn’t an escape.attempts measurement. Since the cat.escape.attempts is an OTel counter, it only climbs and doesn’t reset when you restart your game. To circumvent this counter, we take the change in the counter over the window (EVAL attempts = hi - lo) instead of its absolute cumulative value.

• ES|QL query: 

• Change the rule schedule for every 30–45 seconds.

4. Add an action with your configured webhook, and use the configuration from the ″A webhook connector for your alert″” section of this post.

5. Click Start Game, and wait. 

6. If your alert signals in time, you’ve won, detective. You sleuthed out the right cat and caught it in time. If your rule doesn’t alert by the tenth escape attempt, you lose. Try configuring a new rule and checking that it matches the correct rule_name for the checkpoint (including whitespace and casing) or changing your rule’s threshold or timing.

From cat escapes to credential stuffing detection

Beneath this fun, interactive cat-catching game is an application that demonstrates simple and correctly wired observability architecture. The pieces assembled from this application are the building blocks of a production observability stack. They include OTLP/HTTP exporters shipping to an Elastic Observability endpoint and signals stamped with a service.name by a resource, along with the separate MeterProvider, LoggerProvider, and TracerProvider pipelines. They also include a counter that’s emitted alongside the span which describes the same event, an observable gauge sample through a callback, ES|QL queries transformed for alerting on delta values, and more. And, since the Elastic 9.4.0 also allows you to natively use Prometheus Query Language (PromQL) to query time series data, you have even more query possibilities.

If you replace “escape attempt” with “failed authentication” and cat.name with user.id, the same rule that you created to catch a fence-jumping tabby can become a credential-stuffing detector. Try out different rules and types of alerting, along with other query rates, and see how creative you can be with Elastic Observability in this game. 

If you need any help with The Great Cat Escape, feel free to contact the author.

Daylight streamed through the windows. With a robust rule in hand and an alert in the other, the detective sighed with relief. Nights of grepping had turned into quick decisions and high scores for the seasoned private eye. This case had proved difficult, but no cats had escaped the watchful observability back end. Although this investigation was successful, there would be more cases. And more cats. But now the detective was ready for them.

Observability for developers has lately been distilled into implementing auto-instrumentation, allowing you to instantly connect your code with the larger observability world. This way of utilizing an upstream SDK is certainly the simplest and most production-ready, and works efficiently with the Elastic Cloud Managed OTLP Endpoint.

But what if you could not only add powerful tracing to your Go service but also truly understand how the magic works, rather than just copy-pasting configuration files or a line of code? In the same way that you build your knowledge of software development systems, observability, modernized by OpenTelemetry (OTel) standardization, is a rich, broad system that is valuable to understand. Here is an in-depth technical breakdown of every piece of simple OTel instrumentation using the Elastic Distributions of OpenTelemetry (EDOT) and Golang, from the ground up.

Telemetry is the automated collection, transmission and analysis of data from your application, which can apply to any observable distributed system. This data can range from regular health check calls with your application to real-time information about user interactions, requests, and transactions. Using the example application repository here, we’ll build a strong observability foundation to start observing our applications with confidence.

Understanding the OpenTelemetry Flow

Below, you will see the basic flow of your data when implementing observability with OTel in your system. Before we dive in, let’s explain some of the key terms within OTel. Let’s go over these base key players that we need to implement observability solutions with OTel:

• Span: This is a single, timed unit of a distributed trace that can represent a specific operation, such as a database query or an HTTP handler.

• Trace: This is a detailed record of a single request’s journey through your system, AKA a hierarchy of your spans.

• Tracer: This is the handle for generating spans. You will typically have one per instrumentation library, for example myapp/http.

• Tracer Provider: This is the cornerstone of the SDK. This creates Tracer instances, and you can configure it on application start up.

• Context Propagation: The mechanism for passing trace context between operations and services, maintaining the relationship between parent and child spans.

• Exporter: This is the part that is responsible for sending your telemetry data to a vendor backend, and you can decide if you are sending it to the OTel Collector, EDOT Collector or an OTLP Endpoint.

Installing the Magic, Instrumentation Style

OpenTelemetry provides instrumentation libraries that handle much of the tracing complexity for you. These libraries wrap common frameworks and libraries (like net/http/otelhttp) and automatically capture telemetry without requiring you to manually create spans for every operation.

However, before you're able to send any telemetry, OTel needs to know who (which service) is sending that data.

A resource represents the specific entity, in this case "simple-go-service", that is producing your telemetry data. Its identity is recorded as resource attributes, and resource attributes can include pod names, service names or instances, deployment environments; Basically anything important to identifying your resource. This resource is your service's identity card that gets attached to every span and metric that it emits with its attributes. Once your trace arrives, these attributes can answer "what version was running?" or "which service is this from?"

func initOTel(ctx context.Context, endpoint string) (func(context.Context) error, error) {
res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName("simple-go-service"),
            semconv.ServiceVersion("1.0.0"),
        ),
    )
    if err != nil {
        return nil, err
    }

In the code above, resource.New() constructs the "identity card" of our Go service. The attributes that will be attached to it will use semantic conventions(semconv), standardized names for common metadata fields. These semantic conventions make sure that every single OTel-compatible observability backend knows their meaning.

Now that we've bootstrapped our application with the initOtel function, we can continue to configure everything else!

Let’s begin instrumenting this application by building all the app components that we will need to implement modern observability tools. Below is our instrumentation using otelhttp, which will handle span creation after calling the specified API routes. 

http.Handle("/hello", otelhttp.NewHandler(http.HandlerFunc(handleHello), "hello"))
http.Handle("/api/data", otelhttp.NewHandler(http.HandlerFunc(handleData), "data"))
http.HandleFunc("/health", handleHealth)

// Example of a tracer within our handleHello() function
tracer = tp.Tracer("simple-go-service")

ctx, span := tracer.Start(ctx, "process-hello")
defer span.End()

The key insight here is that otelhttp.NewHandler handles all the span lifecycle management for HTTP requests. You don't need to manually call tracer.Start()or span.End() for basic HTTP tracing since the library does this for you.

On application start up, the SDK will use the tracer provider set up below in order to create Tracer instances. These instances help create and manage the spans contained within traces.

traceExporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint(endpoint),
        otlptracegrpc.WithInsecure(),
    )

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(traceExporter),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    tracer = tp.Tracer("simple-go-service")

Within our initOTel function, we will set up one of our most important signals: logs. First, we initialize the logExporter that will send logs to our OTel Collector using gRPC protocol. Then the LoggerProvider will create the base of the logExporter that batches log entries together before sending those batches to your exporter, attaching metadata about the services along the way. Lastly, the LoggerProvider also creates a standard Go structured logger (slog) that automatically includes trace context (such as span IDs) and batches your log with other logs. These are sent to your observability backend through the exporter along with your metrics and traces. 

logExporter, err := otlploggrpc.New(ctx,
        otlploggrpc.WithEndpoint(endpoint),
        otlploggrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    lp := sdklog.NewLoggerProvider(
        sdklog.WithProcessor(sdklog.NewBatchProcessor(logExporter)),
        sdklog.WithResource(res),
    )
    logger = slog.New(otelslog.NewHandler("simple-go-service", otelslog.WithLoggerProvider(lp)))

Below you can see how you can view your logs through Kibana in the APM UI. These logs are also color - coordinated; Coded warnings are in yellow, errors are in red, and regular logs are in green.

Metrics are set up in the next part of our code. Metrics are telemetry signals that track the quantitative data from your application, such as response times and request counts. The metric exporter is initialized to send metric data to our EDOT Collector then to our observability backend, Elastic Observability in this case, using gRPC. The meter provider in the next portion periodically collects and exports our metrics data and measurements, the same as the tracer provider creates tracers. The only difference between the two providers is that the meter provider works on a timer while the trace provider exports spans as they complete.

metricExporter, err := otlpmetricgrpc.New(ctx,
        otlpmetricgrpc.WithEndpoint(endpoint),
        otlpmetricgrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    mp := metric.NewMeterProvider(
        metric.WithReader(metric.NewPeriodicReader(metricExporter)),
        metric.WithResource(res),
    )
    otel.SetMeterProvider(mp)

    meter := mp.Meter("simple-go-service")
    requestCounter,  = meter.Int64Counter("http.requests")
    requestDuration,  = meter.Float64Histogram("http.duration")

In order to finish initializing OpenTelemetry, we set up our propagators for context propagation. The set text map propagator automatically injects the trace ID and the span ID of your service making an outbound HTTP request to another service, following the W3C Trace Context standard. In short, this maintains the parent-child relationship between spans.

otel.SetTextMapPropagator(propagation.TraceContext{})

    return func(ctx context.Context) error {
        tp.Shutdown(ctx)
        mp.Shutdown(ctx)
        lp.Shutdown(ctx)
        return nil
    }, nil

Now that you know how these pieces work together, try to run the repository linked here, using the readme as your guide.

Sidenote: Adding Custom Spans

For getting an application emitting traces, this instrumentation works great! If you visit localhost:8080/hello after starting the docker containers, the otelhttp middleware automatically creates spans for each HTTP request. However, basic instrumentation only shows essential application telemetry, such as response duration, URL paths, and status codes. You won’t know what happens between the request coming in and request completion. The moment OpenTelemetry truly gains power is when you add custom spans. Unlike auto-instrumentation where spans are created as well as closed automatically, custom spans require you to explicitly start and stop them.

Custom spans can track your application’s logic, such as specific business events or marking expensive operations, using a detailed hierarchy within each trace. In the application for this article, there are several custom spans that were created to track important operations:

• background-work: This traces asynchronous processing that happens with the main request.

• computation: This measures computations and then captures those results, and the computation type.

Custom spans add granular visibility into your application's behavior. For example, in performComputation:

ctx, span := tracer.Start(ctx, "computation")
defer span.End()

result := rand.Float64()
span.SetAttributes(
    attribute.String("comp.type", compType),
    attribute.Float64("comp.result", result),
    )

    logger.InfoContext(ctx, "Computation completed", "type", compType, "result", result)

if result < 0.3 {
span.AddEvent("Low confidence result")
    logger.WarnContext(ctx, "Low confidence computation", "result", result)
}
}

The attributes set above become searchable and filterable in our Elastic Observability backend, allowing for attribute filtering by attribute.result and attribute.compType. If you query your data with “show me all computations where results are less than 0.3,” then you will notice the span event span.AddEvent(“Low confidence result”) tacked on with a timestamped marker. This appears on your trace timeline as well, adding even more visibility to any unusual events. Below is a small example of the filtering that Kibana can accomplish from custom spans.

The Data Pipeline: From Code to IRL

Now that you can export your custom spans and data to OTLP which sends it to the EDOT Collector and then to an observability backend, the best hub for your telemetry data will be the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/. It is a simple, standalone process that is able to receive, process and export all of your telemetry data. Within this project, we use the Elastic Distributions of OpenTelemetry (EDOT) Collector, an optimized Collector for usage within your Elastic Stack. Since this is a self-managed Elastic instance, this article and connected repository utilize the EDOT Collector through elasticapm, but for Elastic Cloud or Serverless projects, you can use the Elastic Managed OpenTelemetry Protocol (OTLP) Endpoint. As noted in the quickstart documentation here, the Elastic Cloud Managed OTLP Endpoint endpoint helps get your data quickly and efficiently into your Elastic Stack through OTLP, without schema translation! This means that your telemetry hits Elastic instantly and your telemetry data remains vendor-neutral.

For most developers and SREs, this Collector is an amazing tool. It allows you to decouple your code from the observability backend. Your application does not need to know its final destination, it can just send the data to the Collector. Your observability backend can change constantly without it even touching your code. The OpenTelemetry Collector also acts as a gateway for multiple streams of data, and is able to accept various formats in order to unify them for exportation. Lastly, the OpenTelemetry Collector is able to offload processing power from your application - tasks such as retries, batching and filtering can happen in the Collector, not your application.

After trying out this article’s repository, try auto-instrumenting your application with Elastic Distributions of OpenTelemetry (EDOT) so that you can utilize the APM UI to its full potential! With the latest version of Elasticsearch and Kibana start-local, you can use Docker to install and run the services and instantly start monitoring your application. 

Understanding the Collector Configuration

The Collector's behavior is defined in a configuration file (otel-collector-config.yaml). Let's break down each component.

Receivers define how the Collector accepts telemetry data. Here, we're listening for both gRPC and HTTP traffic.

receivers:
  # Receives data from other Collectors in Agent mode
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

Connectors are specialized components that sit in between pipelines, and in this case, we are using the elasticapm Connector. This APM Connector exports our metrics, logs, and traces, while simultaneously acting as a receiver for the metrics/aggregated-otel-metrics pipeline (see below). Without it, your raw OTLP data lands in Elasticsearch, but the APM UI has nothing to build its views from.

connectors:
  elasticapm: {} # Elastic APM Connector

Processors transform, filter, or enrich data as it passes through the EDOT Collector. The batch processor aggregates spans before export, reducing network overhead and improving efficiency, as well as limiting batch sizes. The batch/metrics processor does this as well, but for APM metrics. Lastly, there is the Elastic APM processor. This processor ensures that your spans fields are aligned, your traces views are complete, and it overall bridges the gap between Elastic's expectations and OpenTelemetry's formatting of your traces.

processors:
  batch:
    send_batch_size: 1000
    timeout: 1s
    send_batch_max_size: 1500
  batch/metrics:
    send_batch_max_size: 0 # Explicitly set to 0 to avoid splitting metrics requests
    timeout: 1s
  elasticapm: {} # Elastic APM Processor

As mentioned previously in the article, exporters send data to your observability backend. The debug exporter logs telemetry to the console (useful for development), while the Elasticsearch exporter sends traces to your Elastic stack.

exporters:
  debug: {}
  elasticsearch/otel:
    endpoints:
      - ${ELASTIC_ENDPOINT} # Will be populated from environment variable
    user: elastic
    password: ${ELASTIC_PASSWORD}
    tls:
      ca_file: /config/certs/ca/ca.crt
    mapping:
      mode: otel

Pipelines connect receivers, processors, and exporters into a data flow. These EDOT Collector pipelines receive OTLP traces, batches them, and exports to the debug, elasticapm and elasticsearch/otel exporters. It also exports metrics to the debug and elasticsearch/otel exporters.

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch/metrics]
      exporters: [debug, elasticsearch/otel]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, elasticapm, elasticsearch/otel]
    traces:
      receivers: [otlp]
      processors: [batch, elasticapm]
      exporters: [debug, elasticapm, elasticsearch/otel]
    metrics/aggregated-otel-metrics:
      receivers:
        - elasticapm
      processors: [] # No processors defined in the original for this pipeline
      exporters:
        - debug
        - elasticsearch/otel

Debugging Your Code with Confidence in Kibana

Elastic Observability, utilizing Kibana and Streams, has native support for the OTLP Endpoint through the EDOT Collector, which was used in this project. Below, you can see that your data is automatically connected to Streams from the beginning, requiring no extra leg work! You can add conditions or any Grok processors as your data is streaming in, and you'll be able to instantly see your data's schema and data quality.

Elastic also provides the Elastic Cloud Managed Endpoint for even easier storage, data-processing, and scaling. If you use this Managed Endpoint, it means that you can configure OpenTelemetry to send data directly to Elasticsearch, without ANY specialized Collectors. Any way you choose, once your traces are flowing, Kibana’s APM UI provides powerful visualization and analysis capabilities will be everything you need to debug your code. You are able to drill down into individual requests, identify bottlenecks, find anomalies and troubleshoot any issues that arise with confidence.

Here is one span of interest from this repository. Within Kibana, you can immediately filter by the Trace ID, finding other spans with the same Trace ID to visually see the entire trace.

Kibana Discover also allows you to switch indices instantly without losing your filters, ensuring that you can also see the logs that correspond with the same Trace ID.

In addition to the manually checking your traces, you can automatically check them within the APM UI (shown below). This is easy trace visualization using the Kibana APM UI is readily available while using the elasticapm connector. Below is a visualization of a trace comprised of spans within our project. Knowing both methods of correlating spans is beneficial to build the foundation of utilizing Kibana and the APM UI for observability.

Here is a fully built out dashboard built from the repository featured in this article. The possibilities with Elastic Observability are endless!

Congrats, You’re Not “Just” a Developer Anymore!

We’ve broken down the why and how behind OpenTelemetry’s basic components, including the TraceProvider, the span, the exporter and the Collector. Here, you’ve done more than just implement your tracing tool. You now understand the complete data flow from your code to the graphs on your dashboard.

You can now speak the language of observability with confidence, not because you memorized a configuration file, but because you now understand the data flow from your code to the graph on your dashboard. You understand how telemetry moves through your system. You aren’t “just” a developer anymore; you’re now a developer who can truly see.

Try out the code repo above! Included in the [repository]() is a generate-traffic.sh script file. You can run this repeatedly in order to generate logs, traces, and metrics for you to play with within the APM UI. Also, check out our latest releases in our release docs page for exciting Elastic updates.

Elastic has been leading the charge with AIOps, especially in the recent 9.2 update of Elastic Observability with Streams. The conversation around AIOps has shifted dramatically as we move through the year. DevOps and SRE teams aren't asking whether they need AIOps, they're asking how to leverage it more effectively to stay ahead of exponentially growing complexity.

The current challenge of AIOps is that modern cloud-native environments generate massive volumes of telemetry data that are magnitudes larger than past environments. But here's what many teams overlook: logs are the richest source of operational intelligence you have. Logs are able to tell you exactly what happened and why, while metrics only tell you something is wrong, and traces only tell you where. The problem is that most organizations are drowning in logs. Microservices, such as user authentications or inventories, serverless functions, and Kubernetes generate millions of log entries daily. Without AI and machine learning, finding meaningful patterns in this data takes too much time and energy.

Log Intelligence Improvement: What's New in 2025

Historically in observability, unlocking your log intelligence included long manual effort that required not only parsing through logs, but also structuring those logs. Elastic Observability has drastically changed how teams extract value from logs. Observability is not just simple signal analysis - modern tools need to have proactive, log-driven investigations. At Elastic, this modernity is Streams.

Streams, a new release from Elastic, is a collection of AI-driven tools that identify significant events in parsed raw logs by enriching logs with meaningful fields. With Streams, SREs can maximize the value of their data, their logs, and their systems. With system reliability as the goal, Streams helps to reduce pipeline management overhead and accelerates observability analysis. And it takes nearly no time to set up!

Here is how Streams powers the Elastic Observability capabilities available now.

Advanced Log Rate Analysis

Log rate analysis can go far beyond only detecting spikes. Elastic's machine learning automatically identifies when log volumes deviate from expected baselines, then contextualizes these changes within your broader system performance. When your application suddenly generates more error logs, Elastic’s AIOps doesn't just alert you, it also determines whether it's a critical issue requiring immediate attention or just a temporary anomaly.

This matters to your analysis because not all log spikes are equal. A 10x increase in DEBUG logs might indicate verbose logging accidentally enabled in production. A 2x increase in ERROR logs could signal a cascading failure. Log rate analysis distinguishes between these scenarios automatically, giving your team the context needed to respond appropriately.

Intelligent Log Categorization with Streams

This is where AIOps shines with log data. Streams uses machine learning algorithms in order to automatically classify and group similar log patterns, dramatically reducing noise. Instead of manually parsing millions of entries, the system identifies common structures, groups related events, and surfaces the categories that matter most.

Logs are unstructured by nature, making them difficult to analyze at scale. Streams corrals chaotic log streams into organized, queryable patterns. Instantly, you can see that 80% of your errors fall into three categories, helping you prioritize where to focus remediation efforts. This approach helps you reduce noise and accelerate analysis, allowing teams to act on insights faster.

Multi-Dimensional Anomaly Detection

Anomaly detection now simultaneously examines relationships between logs, metrics, and traces. A slight increase in response time might not trigger an alert by itself, but when correlated with unusual log patterns and memory consumption changes, the system recognizes it as an early warning sign.

Logs contain a myriad of contextual information that metrics and traces can't capture: stack traces, user IDs, transaction details, error messages, etc. By correlating log anomalies with other signals, you get the full picture of what's happening in your system. This whole holistic view enables teams to catch issues earlier, as well as understand their full impact across the stack.

Enhanced Root Cause Analysis Powered by Significant Events

When an issue occurs, Elastic's Streams accelerates root cause analysis through AI-assisted parsing of logs and bringing about “Significant events.” Significant event queries can be defined by AI or manually, depending on if you know what logs you are looking for or not. Then, Elastic’s AIOps traces the problem through your entire stack using these events, as well as enriched log data combined with distributed tracing. This system is able to correlate failed transactions with specific log entries, deployment events, and infrastructure changes. This helps you understand not just what broke, but why and when.

Streams makes the analysis of your logs quick and automatic by going across your entire distributed system within seconds, grabbing relevant log entries such as stack traces, state information, error messages, and more. What used to require hours of manual investigation and deduction now happens automatically, freeing you and your team from tedious detective work and enabling faster resolution. 

Logs in Action: Real-World Impact

Let's look at how these capabilities work together in practice. Imagine your payment processing service is experiencing intermittent failures - only 0.5% of transactions, but enough to concern your team. Traditional monitoring shows everything is mostly okay, but customers are still complaining.

Without Streams, an SRE might initially run some broad queries, manually sift through thousands of logs, struggle to connect all the dots, and ultimately not understand the correlation between the errors and recent system changes. 

With Elastic Streams and AIOps, many of these potential problems are instantly mitigated:

• Streams automatically parse the payment service, adding connection timeouts to a new category of significant events

• Log rate analysis with Streams reveal that this significant event category has been slowly growing over the past month, showing growth of the timeouts from a small number of occurrences into a larger amount

• Elastic’s built-in anomaly detection correlates these significant events with deployment data, and identifies that they started appearing after a recent load balancer configuration

• Root analysis pinpoints the exact database connection pool setting that is too restrictive for peak load by tracing affected transactions through previously enriched logs

What usually takes 4-8 hours of manual log analysis is resolved in minutes, with Elastic automatically highlighting the relevant log entries that tell the complete story. This is the power of AIOps and Streams as applied to log intelligence.

The Power of Unified Log Intelligence

What sets Elastic apart is treating logs as a priority in your observability strategy. Elastic provides comprehensive log ingestion that centralizes petabytes of logs from across your infrastructure with flexible parsing and enrichment. The platform uses purpose-built machine learning models that understand log patterns, not generic algorithms retrofitted for log analysis.

Logs don't exist in isolation, which is why Elastic correlates log data with metrics, traces, and business events to provide complete context. And because log volumes can be massive, Elastic's tiered storage approach means you can retain years of logs for compliance and historical analysis without breaking the budget.

Why Logs Matter More Than Ever

Logs have become the cornerstone of effective AIOps for three critical reasons.

First off, logs capture what metrics can't. A metric tells you the CPU is at 80%, but a log tells you which process is consuming resources and why. This level of detail is essential for understanding not just that something is wrong, but what specifically is causing the problem.

Second, logs provide business context. Error messages contain user IDs, transaction ldetails, and business logic failures that help you understand customer impact. When you're troubleshooting an issue, knowing which customers are affected and what they were trying to do is invaluable for prioritizing your response.

Third, logs enable true root cause analysis. Stack traces, error messages, and application state captured in logs are essential for understanding the why behind every incident. Without this information, teams are left guessing at root causes rather than definitively identifying and fixing them.

The teams winning with AIOps in 2025 aren't just monitoring metrics, they're extracting intelligence from their logs at scale, turning operational data into actionable insights.

Transform Your Log Strategy Today

Every hour your team spends manually searching through logs is an hour they're not spending on innovation. Every incident that could have been prevented through intelligent log analysis represents both technical debt and business risk.

Elastic Observability provides the foundation you need to unlock the intelligence hidden in your logs. With automatic categorization, anomaly detection, and ML-powered analysis, you can start seeing value immediately. Check out this recent article to get started with Elastic Streams and Observability today!