How The Great Cat Escape game works

In this detective game, the rules that you eventually create in your Elastic cluster govern whether you win or lose against these cats. Each cat creeps toward the fence at its own accelerating pace. If any cat hits 10 escape attempts, that cat is free and you lose your detective license. Your job, gumshoe, should you choose to accept it, is to configure an Elastic Observability alert that fires before any cat hits 10. The earlier it fires, the higher your score.

 Requirements: Elastic Observability, Python, and a tunnel

• Your own fork/branch of The Great Cat Escape

• Elastic Observability v. 9.3.0 or higher (get a seven-day free trial of Elastic Cloud Serverless), 

• Python v. 3.11.0 or higher (latest version)

• ngrok or a similar tunnel, since Kibana's webhook needs a way to reach your laptop

The Great Cat Escape game is governed through python-dotenv with credentials from an environment (.env) file. Change these to match your Elastic cluster, and make sure that the connection is through Elastic APM. You can input your secret token through your agent by going to Fleet → Agent Policies → click on the policy you want to change → Elastic APM → Settings → and then scroll down until you see "Agent Authorization."

To populate your environment, take a copy of the .env.example and rename it .env. Your .env file needs two values:

NUM_CATS is optional (default 1, max 6). More cats means more chaos, but more chaos means a potentially higher score for you, detective.

Instrumenting a Python app with OpenTelemetry for Elastic APM

Catching rogue cats is hard. What should we trigger on? What should be measured? Why? To answer these questions, we need to know what our cats are doing at any given time. That’s where instrumentation comes in. In observability, instrumentation is the process of adding code or configurations to your application that can generate and collect telemetry data, as well as export it. This telemetry data can include metrics and logs, along with traces. So let’s get into the code that will help your investigation.

To instrument our application, we’ll use OTel, which is a standardized framework for generating, collecting, and exporting telemetry data. It’s the industry standard for observability. Before we get started, we need to define a resource in our project. In OTel instrumentation, a resource represents the entity that’s producing telemetry data, which, in this case, is our “escape-room-cat” service.

The “service.name” key is a standard semantic convention from OTel. Semantic conventions are standardized and vendor-neutral naming rules used throughout OTel to structure your data. You’ll see more of these conventions as we set up our signals in the sections that follow.

Setting up traces, metrics, and logs with OTLP exporters

We set up our three pillars and our logs, metrics, and traces through OTel in the following code. Let’s go through the code for each signal.

With this code, we set up the TracerProvider to start collecting traces, which are then exported in batches through the OTLPSpanExporter via the OpenTelemetry Protocol (OTLP). A trace is a collection of spans, and spans can be enriched with span attributes. We go over the span "cat.escape_attempt" later in this article.

Next, we set up our metrics through a MeterProvider and a PeriodicExportingMetricReader. Metrics are polled on a timer according to the reader’s interval and then exported in OTLP format through OTLPMetricExporter, rather than being pushed when events arrive, like traces and logs. The MeterProvider is still necessary despite the PeriodicExportingMetricReader because it’s the configuration layer from the OTel SDK that actually processes and collects your telemetry and exports it to your Elastic Observability project. This is true for the TracerProvider and for the LoggerProvider (see below):

LoggerProvider works slightly differently than the rest of the signals. The code here bridges Python’s built-in logging module, LoggingHandler, with OTel, instead of creating a new logging system. This handler routes any logging.info call in the Python application into the OTel pipelines, which means that your structured log records get exported alongside the rest of your signals.

Counters, gauges, and spans: The signals this game emits

Now that we know how OTel works to instrument our game, let’s get into the details of the specific signal configurations that we’ll use in this application.

First, let’s go over the most important counter (a data point that only goes up):

This metric is important to the game because it determines whether you win or lose. If the ”cat.escape.attempts” counter for any cat hits 10, you lose and the cat gets free. 

The next metric is an observable gauge governed by a function from our game:

This observable gauge meter indicates that, at each collection interval, a cat’s current suspicion level is recorded. “cat.suspicion.level” is governed by a function that picks a random floating number between 8 and 15 and appends that to the current suspicion level. An observable gauge represents a value that can go up and down, unlike a counter that only goes up. We’re using an observable gauge instead of a counter because we’re trying to determine How suspicious is this cat right now? as opposed to How much suspicion has accumulated? In this game, cat suspicion is another meter that you might be able to create a rule on.

Below is the central event, a span that fires every time a cat attempts to break out. This span identifies which cat it was, the escape attempt number, the cat’s position, whether the attempt was successful, and the cat’s suspicion level. The next span is also collected whenever the cat moves:

When you view the traces data in Kibana, the trace and its span attributes appear as follows:

The fields translated from our game are prepended with “numeric_labels” or “labels.” This is the result of our data being translated from OTel into the Elastic Common Schema (ECS). However, you can rename these in Kibana, if you choose, to shorten the name or make searching easier for you. 

Emitting structured logs with the Python logging module

Logs are the most human-readable of the three signals, and this game emits them through semi-structured logging:

The game sends out several logs and communicates them via multiple methods. When it comes to logging, it’s generally good practice to keep your logs clear and concise so that they’re easily recognizable by anyone debugging, especially for errors. In this Flask application, we used the logging module by Python. The logs sent to Kibana are “logger.info”, “logger.warning”, and “logger.error”, and they indicate the log severity level. logger.info is used for generic information from your application, and logger.warning indicates unexpected events that the system has recovered from. logger.error notes a failure or critical error. In The Great Cat Escape, the only error is when you lose the game. These logs are emitted through the standard Python logger and exported via the OTLP log exporter, so they show structured severity_text and resource attributes, along with trace correlation in Kibana:

The next logging system is _log, a custom function written for in-game use:

This log function shows the player the same information as Kibana receives by grabbing the current time and by using game_lock through game_lock = threading.Lock(). In this game, since there are requests from multiple threads (cats), the game_lock helps by preventing data loss if two threads reach out to the same resource at the same time, which can lead to data corruption. 

Now that you understand the instrumentation, the final step before you write your rules and play the game is to understand the alert_fired API and wire the needed webhook so that your alert can be received by your URL. 

Wiring a webhook connector for your alert

To send out a webhook alert, you use ngrok or a similar tunnel to expose your port to Kibana, enabling it to POST to that port over the internet. You front your own port with a tunnel, and that public URL is the <your-tunnel> you point Kibana at. Add a connection of type “Webhook” in Kibana with these settings:

In Kibana 9.4, the connector lives under Stack Management → Connectors. Save it once, and you can reuse it across all of your rules.

The {{context.group}} field shows which cat triggered the rule, and the fired_at {{context date}} field tells you when the alert happened. These are purely informational parameters, but the winning endpoint, /api/alert_fired takes more precaution with “current_count” and “rule_name.” The game's /api/alert_fired endpoint, shown in the following code, has several checkpoints based on the parameters above that help with specificity in this game. These checkpoints are also important to real alerting in systems to ensure that the correct source, system, or alert was sent via your alerting webhook.

The endpoint runs three checkpoints before it accepts a win, matching the numbered comments in the code above:

1. Parse the request body. request.get_json takes the body sent by the webhook and parses it into a Python dict. Passing silent=True means an empty or malformed body won't raise an error; the endpoint keeps working and fills in unknown values with {}.

2. Check the rule name. "Cat Escape Alert" is the enforced rule_name, so a stray request from another source can't trigger a win. Use this name exactly, or change it in the code. (See EXPECTED_RULE_NAME = "Cat Escape Alert".)

3. Check the attempt count. This gate stops an alert from counting as a win when no cat has actually moved. It's set to one, so at least one escape attempt must have registered. If you want a higher bar, change it in the code. (See MIN_COUNT = 1.)

How to create a Kibana alert rule with ES|QL

This game ends one of two ways. You win if your created rule alerts properly; and you lose if any cat reaches 10 escape attempts. 

To create your rule and play the game, follow these instructions. Lettered subsections give you a full breakdown of how to create a winning rule.

1. Go to Kibana → Stack Management → Rules.

2. Create new rule → Elasticsearch query.

3. Create a rule to alert on logs or metrics. You can use Kibana Query Language (KQL) or Lucene, Query DSL, or ES|QL.

• To create a winning rule, choose ES|QL. Enter the code block below into the box for creating a query. The service for this game is “escape-room-cat,” and “cat.escape.attempts” IS NOT NULL discards every row that isn’t an escape.attempts measurement. Since the cat.escape.attempts is an OTel counter, it only climbs and doesn’t reset when you restart your game. To circumvent this counter, we take the change in the counter over the window (EVAL attempts = hi - lo) instead of its absolute cumulative value.

• ES|QL query: 

• Change the rule schedule for every 30–45 seconds.

4. Add an action with your configured webhook, and use the configuration from the ″A webhook connector for your alert″” section of this post.

5. Click Start Game, and wait. 

6. If your alert signals in time, you’ve won, detective. You sleuthed out the right cat and caught it in time. If your rule doesn’t alert by the tenth escape attempt, you lose. Try configuring a new rule and checking that it matches the correct rule_name for the checkpoint (including whitespace and casing) or changing your rule’s threshold or timing.

From cat escapes to credential stuffing detection

Beneath this fun, interactive cat-catching game is an application that demonstrates simple and correctly wired observability architecture. The pieces assembled from this application are the building blocks of a production observability stack. They include OTLP/HTTP exporters shipping to an Elastic Observability endpoint and signals stamped with a service.name by a resource, along with the separate MeterProvider, LoggerProvider, and TracerProvider pipelines. They also include a counter that’s emitted alongside the span which describes the same event, an observable gauge sample through a callback, ES|QL queries transformed for alerting on delta values, and more. And, since the Elastic 9.4.0 also allows you to natively use Prometheus Query Language (PromQL) to query time series data, you have even more query possibilities.

If you replace “escape attempt” with “failed authentication” and cat.name with user.id, the same rule that you created to catch a fence-jumping tabby can become a credential-stuffing detector. Try out different rules and types of alerting, along with other query rates, and see how creative you can be with Elastic Observability in this game. 

If you need any help with The Great Cat Escape, feel free to contact the author.

Daylight streamed through the windows. With a robust rule in hand and an alert in the other, the detective sighed with relief. Nights of grepping had turned into quick decisions and high scores for the seasoned private eye. This case had proved difficult, but no cats had escaped the watchful observability back end. Although this investigation was successful, there would be more cases. And more cats. But now the detective was ready for them.

In this article, we follow six audit events from a company’s admin console through the Elastic Distribution of OpenTelemetry (EDOT) Collector into a dedicated data stream. One of them is a user named alex, who fails two login tries but gets in on the third and then attempts a role change. We reconstruct that timeline with one ES|QL query and read back the retention policy to confirm that it applied. Then we rerun the same query on a snapshot-backed index. The evidence comes back in a form that you can hand to an auditor and explain.

Prerequisites

• Elasticsearch and Kibana 9.5+

• Docker, to run the elastic-otel-collector:9.5 container

• An API key created by a superuser

You can find the companion notebook here. It reproduces the whole lab from this article, including the collector configuration and the searchable snapshot, and the steps in between, and it cleans up at the end.

What audit logs tell you that application logs don’t

Application logs usually explain how software behaved, such as a timeout or an exception, or even a failed connection. Audit logs establish accountability by recording an actor, action, target, outcome, and time. They can help debugging, too, but their value is in reconstructing activity and demonstrating how controls operated.

A Kubernetes RoleBinding change captured in full

In Kubernetes, a Role defines permissions and a RoleBinding grants them to an identity within a namespace. Changes to both are auditable when the audit policy includes those resources. Here’s a real RoleBinding creation that was captured in an isolated Kubernetes 1.35.0 lab. It was collected through OpenTelemetry (OTel) and queried in Elasticsearch

The successful request grants the deploy-bot service account the existing pod-reader role in payments. The RoleBinding's subjects and role reference come from its request body, captured by this lab's targeted Request audit level. Metadata logging alone would omit that body.

Where audit logs come from

Common sources cover different parts of the story:

Source

Questions it can answer

Kubernetes API audit logs

Who requested a resource change, and how did the API server respond?

Cloud provider trails

Which principal performed an API operation on which resource?

Authentication systems

Who signed in or failed authentication? Who changed privileges?

Database audit logs

Who accessed a sensitive object or changed database permissions?

Collection starts with enabling the right events at the source. Cloud data-access events may need separate configuration; database slow-query logs aren’t a complete access audit. For Kubernetes setup, use the existing managed Kubernetes audit-log tutorial.

Four requirements for an audit logging pipeline

Four requirements shape an audit pipeline:

Requirement

Design decision

Long retention

Keep audit data in streams with an explicit retention policy, separate from short-lived operational logs.

Immutability

Define who shouldn’t be able to alter or delete records, and for how long. A read-only index doesn’t prevent privileged administration.

Tamper-evidence

Preserve originals and independently protected verification material so changes or missing records can be detected.

Access control

Separate collection, investigation, and retention administration; record changes to those controls.

For System and Organization Controls 2 (SOC 2) or ISO/IEC 27001, logs support the evidence behind your controls; storing them doesn’t establish compliance. Agree which activities to record, how long to retain each copy, who can retrieve it, and how exceptions or legal holds work. Neither framework gives this lab a universal 365-day retention requirement.

Search and preservation need complementary controls. For example, CloudTrail integrity validation uses signed digest files that must actually be validated. Where write-once retention is required, design a source archive using appropriate controls, such as S3 Object Lock. An indexed JSON string isn’t a substitute for that archive or its validation material.

How to collect audit logs with the EDOT Collector

The retention walk-through uses six fictitious application-audit JSON events: four for the user alex and two for a second user, priya, so the actor filter has something to exclude. The EDOT Collector runs as a container and ships to Elasticsearch and Kibana 9.5.3 on Elastic Cloud Hosted (ECH). The configuration is based on Elastic's documented file log collection and JSON parsing procedure: a filelog receiver with a json_parser operator and a resource processor that sets data_stream.dataset and data_stream.namespace, along with the elasticsearch exporter, in otel mapping mode. This article focuses on the decisions and results; the complete configuration and commands are in the companion notebook. 

Why the json_parser uses parse_to: attributes

The collector parses the source timestamp and fields and then sends records to the dedicated logs-audit_demo.otel-lab data stream. The exporter builds that name from the dataset and namespace and appends .otel to the dataset.

Unlike the documented example, which uses parse_to: body, the lab's json_parser uses parse_to: attributes, so the parsed values go under attributes and the original JSON string stays in body.text. With parse_to: body, the fields would end up in body.structured and the original string would be lost. All six stored strings matched the fixture lines in this test. Elastic Common Schema–based (ECS-based) integrations can use root fields, such as user.name, so check the schema before reusing a query.

Reconstruct an actor’s timeline with one ES|QL query

In Kibana Discover, select ES|QL, Elastic's piped query language. The time picker uses the time zone of the dateFormat:tz advanced setting, which defaults to the browser's, so set it to UTC in Stack Management > Advanced Settings before entering the absolute window 2026-09-07T04:59:00Z through 2026-09-07T05:06:00Z. Then run the following:

Notice that the last event is a failed role change. So far, we only know that alex tried to change a role and was denied, which isn’t the same as a permission change or a compromised account. To find out what happened, the next step is to look at identity-provider and resource activity within the same tenant and time window.

When comparing the screenshot with the query, keep in mind that the fixture uses UTC and Kibana displays timestamps in the browser's time zone (in this case, UTC−05:00). That’s why the rows show 00:00, while the time window is set in UTC.

Discover's time picker constrains this query; API clients need an explicit timestamp filter, and the notebook adds one. When handing evidence to a reviewer, record:

• The query.

• The UTC window.

• The source.

• The retrieval time.

• The result limit.

LIMIT 100 is a bounded investigation view, not an exhaustive export.

Confirm that every audit event arrived

The data stream held six records with six distinct event IDs before and after a normal collector restart, because the filelog receiver keeps its file offsets and the exporter keeps its sending queue in the file_storage extension. At scale, also monitor source coverage, parse failures, queue saturation, and delivery lag. This six-event check is a functional test, not an ingestion benchmark or an exactly once guarantee.

Set a log retention policy that you can verify

A retention setting should be something that you can verify. On ECH, the built-in logs-*-* templates hand new data streams to the logs index lifecycle management (ILM) policy, so the lab gave its stream a dedicated index template, configured as follows: 

• Priority 500, higher than the built-in logs templates that also match the stream name, so Elasticsearch applies this template instead of them.

• Copies the built-in logs-otel@template component list.

• Restricted to logs-audit_demo.otel-*.

• Sets index.lifecycle.prefer_ilm to false.

• Defines a 365-day data stream lifecycle.

The data stream API then reported the backing index as managed by data stream lifecycle, and the lifecycle read-back reported effective_retention: 365d:

Why your audit data stream ignores the retention policy you set

Data stream lifecycle retention is a minimum retention mechanism based on backing-index lifecycle timing, not an exact per-event deletion deadline. A successful read-back proves configuration, not a year of availability. Rehearse retrieving the oldest required records, and review source coverage regularly. For the retention interface and ILM comparison, see retention management with Streams.

Capacity planning for long log retention

Long retention also changes capacity planning. Daily ingest volume multiplied by retention days gives a starting raw volume; mappings, compression, replicas, query concurrency, and tier placement affect the actual resources needed. Size and test against representative investigations across realistic data ages.

Query older audit logs with searchable snapshots

Searchable snapshots keep snapshot-backed indices queryable through Elasticsearch. Fully mounted indices can serve older data from the cold tier; partially mounted indices use a shared local cache in the frozen tier. This gives recent and older evidence a common query surface, with storage and retrieval tradeoffs to test for your workload:

The audit lab took a snapshot of its six-event backing index and mounted it. It then reran the same actor query:

The same four events were returned. This used full_copy on available hot nodes with an Enterprise license; it didn’t benchmark cold/frozen costs or latency. The backing snapshot must remain available while its mounted index uses it, and local cache isn’t a backup. The separate lab mount also needs its own retention and cleanup plan.

Who can read and change your audit logs

Use role-based access to give investigators read and view_index_metadata on the required audit data, plus Discover read access in the intended Kibana space. Keep retention and snapshot administration in separate roles. Roles are additive, so a reader role doesn’t remove permissions granted elsewhere.

The collector's API key only carries create_doc and auto_configure on its stream, and the has privileges API confirmed that it has no index, write, delete, or manage privilege there. That limits the collector's authority; it doesn’t make the records immutable against administrators. Also plan Elasticsearch and Kibana audit logging for the evidence platform itself, subject to deployment and subscription requirements.

Connecting Kubernetes audit logs to container runtime data

For the Security Labs crossover, correlating Kubernetes audit logs with container runtime data shows how API activity connects to what runs inside a workload. Stable identities and resource context make that connection possible. Our small application fixture demonstrates timeline reconstruction; it doesn’t validate those Kubernetes detection rules.

Where to start with audit log management

Start with one audit source and one question that you must answer months later. Verify delivery, preserve the required originals, rehearse searches across recent and retained data, and review who can change the evidence path. Elastic brings collection, structured investigation, searchable retention, and scoped access together; your policy defines the evidence that those capabilities must preserve.

In this article we instrument four Python services with the Elastic Distribution of OpenTelemetry (EDOT), saturate one of them, and use ES|QL to recover which service failed first. Metrics tell you that something is wrong; logs tell you why, once you group them by request instead of by service.

Prerequisites

• Elasticsearch and Kibana 9.1 or later.
• Python 3.10 or later.
• The Elastic Cloud Managed OTLP endpoint and an API key for it. Kibana shows both under Add data > OpenTelemetry.

A companion notebook replays this incident and runs every query below, so you can follow along against your own cluster.

Root cause analysis from logs: why error counts cannot build a causal graph

A causal graph is a directed graph whose nodes are the failing components and whose edges record which failure produced which. Reading it is mechanical: walk the edges backwards from the service the user complained about, and the node you cannot walk back from is the root cause.

Causal edges can be declared in advance from a service dependency model, which covers every failure the model anticipates and has to be kept in step with the architecture as it changes. This article derives them from log fields after the fact instead. That covers one incident rather than every possible failure and needs no model to maintain.

Error counts alone give you nodes and no edges. Every service in a cascade logs one error per failed request, so their counts converge. Meanwhile, a healthy service that logs an error on a routine event outranks all of them as soon as that event is common enough. In this lab catalog-api serves less traffic than checkout-api and still logs almost twice as many errors, because it errors on more than half of its requests while the cascade services error on a fifth of theirs.

The trace.id field supplies the structure that counts cannot:

• Two errors with the same trace.id belong to the same request, so one may have caused the other.
• Two errors with different trace.id values are unrelated, however close their timestamps.

Four Python services instrumented with EDOT

checkout-api receives the order and calls payment-gateway, which calls ledger-service, which holds a fixed pool of 8 database connections. catalog-api sits outside that path and logs an ERROR on every image cache miss while still returning 200.

Install EDOT Python and the instrumentation for the libraries in use, following the documented setup:

pip install elastic-opentelemetry flask requests
edot-bootstrap --action=install

edot-bootstrap inspects the installed packages and adds the matching instrumentation, including opentelemetry-instrumentation-logging, which attaches trace context to log records.

Each service is ordinary Flask code using the standard library logger. This is ledger-service, the one that fails:

POOL_SIZE = 8
ACQUIRE_TIMEOUT_S = 0.25

pool = threading.BoundedSemaphore(POOL_SIZE)
state = {"hold_ms": 15}

@app.post("/reserve")
def reserve():
    order_id = request.json.get("order_id")
    if not pool.acquire(timeout=ACQUIRE_TIMEOUT_S):
        logger.error(
            "connection pool exhausted, no connection available after %.0fms",
            ACQUIRE_TIMEOUT_S * 1000,
            extra={
                "db.connection_pool.size": POOL_SIZE,
                "db.connection_pool.available": 0,
                "error.kind": "pool_timeout",
                "order.id": order_id,
            },
        )
        return jsonify({"error": "pool_timeout"}), 503
    try:
        time.sleep(state["hold_ms"] / 1000.0)
        return jsonify({"reservation_id": f"res-{order_id}"}), 200
    finally:
        pool.release()

The ledger-service file contains no OpenTelemetry code. Every key in extra becomes a searchable field in Elasticsearch.

The two callers add one field; the others do not. When payment-gateway fails because its dependency failed, it records which dependency:

logger.error(
    "ledger rejected reservation with status %s, cannot authorize payment",
    resp.status_code,
    extra={
        "error.kind": "authorization_failed",
        "upstream.service": "ledger-service",
        "upstream.status_code": resp.status_code,
        "order.id": order_id,
    },
)

Point the services at your OTLP endpoint and start them under opentelemetry-instrument:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://<your-managed-otlp-endpoint>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey%20<your-api-key>"
export OTEL_RESOURCE_ATTRIBUTES="service.name=ledger-service,deployment.environment=obs-labs-causal-graph"

opentelemetry-instrument python ledger_service.py

OTEL_EXPORTER_OTLP_HEADERS is URL-encoded, so the space after ApiKey has to be written as %20.

The traffic generator drives roughly 12 checkouts and 8 product lookups per second, then raises the ledger connection hold time from 15ms to 900ms. At 8 connections and a 900ms hold, the service can serve about 8.9 requests per second, so the pool saturates and requests time out on acquire. No service returns a hardcoded error. The recorded run is 2 minutes of baseline, 4 minutes of saturation, and 1 minute of recovery, producing 4,053 successful checkouts and 956 failures.

What does a trace-correlated log record contain?

Sending OTLP to the Managed OTLP endpoint stores the logs in logs-generic.otel-default in their native OpenTelemetry shape, with no schema translation. This is what an ERROR record from the run looks like, with scope and host metadata trimmed:

{
  "@timestamp": "2026-07-26T09:03:45.139Z",
  "resource": {
    "attributes": {
      "service.name": "ledger-service",
      "deployment.environment": "obs-labs-causal-graph"
    }
  },
  "trace_id": "f9b32dca62c1e142537413a047fb9b69",
  "span_id": "c4548cab42cbdd15",
  "severity_text": "ERROR",
  "body": { "text": "connection pool exhausted, no connection available after 250ms" },
  "attributes": {
    "error.kind": "pool_timeout",
    "order.id": "10c9e056ae8c",
    "db.connection_pool.size": 8,
    "db.connection_pool.available": 0
  }
}

Three details decide what the queries can do:

• trace_id and span_id arrive without any application code setting them, because the logging instrumentation attaches the active span context to records emitted inside a request.
• Every key in extra lands under attributes with its dots intact, and Elasticsearch exposes each one by its short name. The field the code wrote, upstream.service, is the field the queries use. Resource attributes work the same way, so service.name and deployment.environment are queryable as written.
• The ECS names still resolve. trace.id, span.id, log.level, and message are built-in aliases for trace_id, span_id, severity_text, and body.text, so the queries below use whichever reads better.

The four steps that follow add one piece of the graph each, starting from nodes with no edges:

Step 1: rank services by failure rate and by error volume

Start with the failure rate per service:

FROM traces-*.otel-*
| WHERE @timestamp >= "2026-07-26T08:59:00.000Z" AND @timestamp < "2026-07-26T09:07:00.000Z"
    AND deployment.environment == "obs-labs-causal-graph" AND kind == "Server"
| STATS failed = COUNT(*) WHERE status.code == "Error", total = COUNT(*) BY service.name
| EVAL failed_pct = ROUND(100.0 * failed / total, 1)
| SORT failed_pct DESC

checkout-api, ledger-service, and payment-gateway all sit at 19.1%, with 956 failures each. The rates match because in a synchronous call chain, every hop fails when the deepest hop fails. The metric names the affected services and gives no way to order them.

Now count ERROR records per service over the same window:

FROM logs-*.otel-*
| WHERE @timestamp >= "2026-07-26T08:59:00.000Z" AND @timestamp < "2026-07-26T09:07:00.000Z"
    AND deployment.environment == "obs-labs-causal-graph" AND log.level == "ERROR"
| STATS errors = COUNT(*), traces = COUNT_DISTINCT(trace.id) BY service.name
| SORT errors DESC

catalog-api leads with 1,820 errors, almost double any other service, and it returned 200 to all 3,345 of its requests. Sorting by error count ranks services by log verbosity, which is independent of where the failure started.

Step 2: group error logs by trace ID

Group by trace.id and count how many distinct services logged an error inside each request:

FROM logs-*.otel-*
| WHERE @timestamp >= "2026-07-26T08:59:00.000Z" AND @timestamp < "2026-07-26T09:07:00.000Z"
    AND deployment.environment == "obs-labs-causal-graph" AND log.level == "ERROR"
| STATS services = COUNT_DISTINCT(service.name) BY trace.id
| STATS traces = COUNT(*) BY services
| SORT services ASC

The population splits into two. Traces with one erroring service are local problems that never propagated; traces with three are cascades.

| Trace type | Traces | Services that logged an error | | :---- | :---- | :---- | | Single-service, never propagated | 1,820 | catalog-api | | Three-service cascade | 956 | checkout-api, ledger-service, payment-gateway |

catalog-api appears in no cascade, so the query removes the highest-volume service from the investigation without anyone reading a message.

Step 3: find which service failed first in each trace

Within a cascade, the service that logged the first error is the candidate cause. ES|QL has no window functions, so build a sortable string of timestamp plus service name, take the minimum per trace, and slice the service name back out:

FROM logs-*.otel-*
| WHERE @timestamp >= "2026-07-26T08:59:00.000Z" AND @timestamp < "2026-07-26T09:07:00.000Z"
    AND deployment.environment == "obs-labs-causal-graph" AND log.level == "ERROR"
| EVAL marker = CONCAT(DATE_FORMAT("yyyy-MM-dd HH:mm:ss.SSS", @timestamp), "|", service.name)
| STATS origin_marker = MIN(marker), services = COUNT_DISTINCT(service.name) BY trace.id
| WHERE services > 1
| EVAL origin_service = SUBSTRING(origin_marker, 25)
| STATS cascade_traces = COUNT(*) BY origin_service
| SORT cascade_traces DESC

MIN() accepts keyword fields from 8.16.0 onward, and the formatted timestamp is fixed at 23 characters, so the earliest marker is also the lexicographically smallest one. SUBSTRING(origin_marker, 25) skips the timestamp and the separator.

ledger-service is the origin in 955 of 956 cascades. checkout-api is the origin in one.

How reliable is timestamp ordering across services?

Measure it rather than assuming it:

FROM logs-*.otel-*
| WHERE @timestamp >= "2026-07-26T08:59:00.000Z" AND @timestamp < "2026-07-26T09:07:00.000Z"
    AND deployment.environment == "obs-labs-causal-graph" AND log.level == "ERROR"
| EVAL ms = DATE_FORMAT("yyyy-MM-dd HH:mm:ss.SSS", @timestamp)
| STATS spread_ms = DATE_DIFF("milliseconds", MIN(@timestamp), MAX(@timestamp)),
        distinct_ms = COUNT_DISTINCT(ms),
        services = COUNT_DISTINCT(service.name) BY trace.id
| WHERE services > 1
| STATS cascade_traces = COUNT(*),
        median_spread_ms = MEDIAN(spread_ms),
        max_spread_ms = MAX(spread_ms),
        traces_with_a_collision = COUNT(*) WHERE distinct_ms < services

Across 956 cascades, the median spread between the first and last error is 2ms, the maximum is 16ms, and 259 traces (27% of the total) contain at least one millisecond collision. Collisions are this common because the hops are local HTTP calls, while @timestamp only has millisecond resolution.

A collision is not automatically a wrong answer. Alphabetically, checkout-api sorts before ledger-service, which sorts before payment-gateway. When ledger-service and payment-gateway land in the same millisecond, MIN() still returns ledger-service, which is the correct origin. Only a tie between checkout-api and ledger-service flips the result. That is why 259 traces contain a collision and just one of them reports the wrong origin.

Timestamp ordering degrades on fast hops and on hosts whose clocks disagree by more than the gap being measured. Treat it as a strong hint rather than proof.

Step 4: build the causal edge list from upstream.service

The upstream.service field records which dependency the caller was waiting on, so every error carrying it is a directed edge. One CONCAT turns the set of them into an edge list:

FROM logs-*.otel-*
| WHERE @timestamp >= "2026-07-26T08:59:00.000Z" AND @timestamp < "2026-07-26T09:07:00.000Z"
    AND deployment.environment == "obs-labs-causal-graph"
    AND log.level == "ERROR" AND upstream.service IS NOT NULL
| EVAL edge = CONCAT(upstream.service, " -> ", service.name)
| STATS traces = COUNT_DISTINCT(trace.id), first_seen = MIN(@timestamp), last_seen = MAX(@timestamp) BY edge
| SORT traces DESC

Two edges, each observed in all 956 cascades:

ledger-service  -> payment-gateway   956 traces
payment-gateway -> checkout-api      956 traces

This edge list uses no timestamp comparison, so it needs no clock synchronization and no tie-breaking. It agrees with the timestamp method on all 956 traces, including the one the timestamp method got wrong.

| | Timestamp ordering | Declared upstream | | :---- | :---- | :---- | | Code changes needed | none | one field per error log | | Correct in this run | 955 of 956 | 956 of 956 | | Fails when | hops are sub-millisecond, clocks drift | the field is missing |

Which services fail without naming an upstream?

The root of the graph is the node that appears as a source but never as a target. Query for candidates by asking which services fail without naming an upstream:

FROM logs-*.otel-*
| WHERE @timestamp >= "2026-07-26T08:59:00.000Z" AND @timestamp < "2026-07-26T09:07:00.000Z"
    AND deployment.environment == "obs-labs-causal-graph" AND log.level == "ERROR"
| STATS errors = COUNT(*), traces = COUNT_DISTINCT(trace.id),
        names_an_upstream = COUNT_DISTINCT(upstream.service) BY service.name
| WHERE names_an_upstream == 0
| SORT errors DESC

This returns catalog-api and ledger-service. Intersect it with the cascade participants from step 2 and one service remains: ledger-service.

What these two queries produce is the propagation structure of one incident, not a reusable causal model. The edges carry no probabilities and say nothing about failures that did not occur during the window. They are observations, and they hold for the traces you queried.

What the root cause service reported

Now read what that service reported, keeping the pool numbers so the evidence travels with the failure mode:

FROM logs-*.otel-*
| WHERE @timestamp >= "2026-07-26T08:59:00.000Z" AND @timestamp < "2026-07-26T09:07:00.000Z"
    AND deployment.environment == "obs-labs-causal-graph"
    AND log.level == "ERROR" AND service.name == "ledger-service"
| STATS events = COUNT(*), first_seen = MIN(@timestamp), last_seen = MAX(@timestamp)
        BY error.kind,
           db.connection_pool.size,
           db.connection_pool.available
| SORT events DESC

One failure mode, pool_timeout, across all 956 events, spanning the four minutes of saturation, with db.connection_pool.size at 8 and db.connection_pool.available at 0. The message behind it, which the next query shows in full:

connection pool exhausted, no connection available after 250ms

Expanding the trace where all three errors share a millisecond shows the chain intact without the timestamps:

FROM logs-*.otel-*
| WHERE trace.id == "f9b32dca62c1e142537413a047fb9b69"
| KEEP @timestamp, service.name, log.level, message, upstream.service
| SORT @timestamp ASC

upstream.service reads (null) for ledger-service, ledger-service for payment-gateway, and payment-gateway for checkout-api.

Add trace IDs and structured fields to your error logs

These queries need three things from your logging, and only one costs any effort:

• trace.id on every record. OpenTelemetry auto-instrumentation supplies it for logs emitted inside an active span. Access logs and background threads fall outside that scope. The Werkzeug access logs in this lab arrived with no trace context and were unusable for correlation.
• Structured fields instead of formatted strings. error.kind as a field lets you count failure modes. The same words inside a sentence do not.
• The upstream dependency named in the error. One key, upstream.service, on the errors your service logs when a dependency fails.

The pattern this lab reproduces is common in production incidents: a component absorbing overflow from an upstream failure often logs more errors than the component that caused it. The same ranking problem applies to automated investigation, where trace-scoped evidence is what makes a root cause claim checkable against specific records and queries.

Conclusion: what the logs proved and the metrics could not

Metrics reported three services broken at 19.1%. Error volume reported catalog-api, which was healthy. Grouping the same 4,688 log records by trace.id isolated 956 cascades, identified ledger-service as their origin, and returned the message that explained the failure.

Two techniques produced that result. Ordering by timestamp needs no code changes and was right 955 times out of 956, with the miss caused by millisecond ties. Reading a declared upstream.service field needs one extra key in your error logs and was right on every trace.

Neither technique needs a service dependency model, and neither one survives contact with logs that have no trace.id. That field is the whole investment.

Resources

• Companion notebook that replays the incident and runs every query in this article.
• ES|QL queries for debugging LLM latency, cost and GPU saturation for more query patterns over OpenTelemetry data.
• EDOT Python configuration reference to control which signals your services export.

Try root cause analysis from logs on your own data

• Run the step 2 query against your own logs-* data and check how many failing traces contain more than one erroring service.
• Audit one service's error logs for whether they name the dependency that failed.

This is an alert noise reduction exercise done with queries. Kibana writes every alert into a hidden index and keeps it there after recovery, so we put three questions to that alert history with ES|QL. Was this one incident or many, where did it start, and which rule produced more noise than signal? Then we change the grouping key on the noisy rule and replay the same failure, and the same query comes back with one episode that lasts seven minutes.

Kibana stamps every alert document with a rule revision, so the before and after of that tuning change sit in one index. At the end, an Agent Builder agent answers the same questions over the same alert history.

Everything below was tested on Elasticsearch and Kibana 9.5.2 on Elastic Cloud, with OpenTelemetry Collector Contrib 0.157.0.

Prerequisites

• Elasticsearch and Kibana 9.5 or later, on Elastic Cloud or self-managed.
• Kibana privileges to create and edit Elasticsearch query rules and Observability custom threshold rules, plus read access to the .alerts-* indices.
• OpenTelemetry Collector Contrib 0.157.0 or later, and an Elasticsearch API key with ingest privileges for the OTLP endpoint.
• Agent Builder enabled, for the final section only. Everything before it is plain ES|QL and works without it.

Why alert history beats alert notifications

A notification answers one short-lived question: who should look at this right now?

Most of the questions that make alerting better arrive later, and they are questions about data:

• Did 18 alert episodes represent 18 problems, or one incident evaluated from 18 angles?
• Which service showed the first symptom, and did the failure follow the dependency chain?
• Which rules produce short, repetitive episodes that nobody can act on?
• Did last month's threshold change actually reduce noise?

Inside Elastic, this framing has a name: alerts as data.

The idea is that every alert and every state transition is a fact in time, stored with explicit semantics, so correlation becomes a query across streams rather than a bespoke feature.

It also matches what shows up in real operations channels: a single infrastructure failure can open dozens of alerts within seconds, and flapping alerts with miscalibrated thresholds are a familiar source of page fatigue.

Both problems are hard to solve from notifications and straightforward to study from preserved alert history.

What Kibana alerting stores in every alert document

Kibana alerting rules write alerts-as-data documents into hidden indices that you can query like any other index.

The two aliases used in this article are .alerts-stack.alerts-default for Elasticsearch query rules and .alerts-observability.threshold.alerts-default for the Observability custom threshold rule.

The unit of storage is what we will call an episode: one document that covers one alert instance from the moment it becomes active until it recovers.

The fields that matter for investigation:

| Field | What it tells you | | :---- | :---- | | kibana.alert.instance.id | The grouping-key value this episode belongs to | | kibana.alert.start, kibana.alert.end | When the episode opened and recovered | | kibana.alert.duration.us | How long it stayed active | | kibana.alert.status | active or recovered | | kibana.alert.flapping, kibana.alert.flapping_history | Whether Kibana marked it as rapidly changing state | | kibana.alert.rule.name, kibana.alert.rule.revision | Which rule, and which version of it | | kibana.alert.grouping | The structured grouping values, for example service.name | | kibana.alert.reason | The human-readable condition at trigger time | | kibana.alert.url | A deep link back to the rule's data view |

Two properties make the rest of the article possible.

First, a new episode for the same instance creates a new document, so counting documents counts episodes.

Alert documents and notifications are decoupled: whether an episode also pages anyone depends on the actions attached to the rule and on their frequency, which can be every check, only on status changes, or a summary on a custom interval.

Our three rules have no actions attached, so nobody was paged during the lab; everything we count from here on is an alert document.

Second, grouping fields with ECS names such as service.name are copied to the top level of the alert document, which means alert history can join against other indices.

One caveat before you query: alert indices are wide.

The alerts index on this deployment maps roughly 1,400 fields, so every ES|QL query in this article projects a narrow set of columns with KEEP, and you should do the same, especially in tools that feed an LLM.

Because these are hidden system indices, the Discover editor cannot always introspect their fields, so it may flag the query with a warning or even an error badge while still running it and returning rows. Trust the result count in the toolbar, not the badge.

One more orientation note: Kibana screenshots below render timestamps in the browser's local time (UTC-5 in this lab), while the query result tables in the text use UTC.

The lab: how one bad deploy produces an alert storm

To generate an honest storm, we need services that fail in a chain, not a single threshold breach.

The scenario: fraud-scorer 2.3.0 ships with slow model loading, the payments queue backs up, payment-gateway requests start timing out, and customers see failed checkouts on payments-web.

Every service writes structured JSON logs, and an OpenTelemetry Collector tails them with a filelog receiver and ships them over OTLP/HTTP to the deployment's Elastic OpenTelemetry endpoint using an API key.

receivers:
  filelog/payment_gateway:
    include:
      - ${env:LAB_DIR}/events/payment-gateway.jsonl
    start_at: end
    resource:
      service.name: payment-gateway
      service.namespace: payments
      service.version: 5.1.2
      cloud.region: us-central1
    operators:
      - id: parse_json
        type: json_parser
        parse_from: body
      - id: parse_severity
        type: severity_parser
        parse_from: attributes.severity_text
      - id: use_message_as_body
        type: move
        from: attributes.message
        to: body

exporters:
  otlphttp/elasticsearch:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      Authorization: ApiKey ${env:ES_API_KEY}

A transform processor routes everything into one data stream, logs-payments.stormlab-default, and the records land with service.name, log.level, custom string attributes under labels.*, and numeric attributes under numeric_labels.*.

The full collector configuration for all three services, and a notebook that reproduces the whole lab, are in the companion repository.

We also load one small index that will do a lot of work later: a service catalog.

PUT payments_service_catalog
{
  "settings": { "index.mode": "lookup" },
  "mappings": {
    "properties": {
      "service.name": { "type": "keyword" },
      "team":         { "type": "keyword" },
      "tier":         { "type": "keyword" },
      "depends_on":   { "type": "keyword" }
    }
  }
}

Three documents map each service to its owning team, tier, and upstream dependency: fraud-scorer depends on payments-queue, payment-gateway on fraud-scorer, and payments-web on payment-gateway.

POST payments_service_catalog/_bulk
{ "index": {} }
{ "service.name": "fraud-scorer",    "team": "risk-ml",       "tier": "backend",  "depends_on": "payments-queue" }
{ "index": {} }
{ "service.name": "payment-gateway", "team": "payments-core", "tier": "edge",     "depends_on": "fraud-scorer" }
{ "index": {} }
{ "service.name": "payments-web",    "team": "storefront",    "tier": "frontend", "depends_on": "payment-gateway" }

index.mode: lookup is what later allows LOOKUP JOIN from ES|QL.

How grouping keys decide how many alerts you get

The storm is observed by three alerting rules, and their grouping keys are the real subject of this article.

| Rule | Rule type | Grouping key | Window | Threshold | | :---- | :---- | :---- | :---- | :---- | | gateway-5xx-per-endpoint-status | Elasticsearch query rule (ES|QL mode) | labels.endpoint, labels.status_code, cloud.region | 1 minute | 3 or more errors | | fraud-scorer-queue-lag | Elasticsearch query rule (ES|QL mode) | service.name, cloud.region | 1 minute | Max lag of 30 seconds or more | | payments-error-rate-per-service | Observability custom threshold rule | service.name | 2 minutes | More than 5 ERROR documents |

The first rule is deliberately miscalibrated in a way we have all shipped at some point: it groups gateway errors by endpoint and status code with a one-minute window and a low threshold.

FROM logs-payments.stormlab*
| WHERE service.name == "payment-gateway" AND log.level == "ERROR"
| STATS error_count = COUNT(*) BY labels.endpoint, labels.status_code, cloud.region
| WHERE error_count >= 3

It runs every minute as an Elasticsearch query rule in ES|QL mode, and each returned row becomes one alert instance, so /api/payments,503,us-central1 and /api/payments/confirm,504,us-central1 alert separately.

The second rule watches the first domino directly: consumer lag on the fraud queue, reported by the service in numeric_labels.queue_lag_seconds.

FROM logs-payments.stormlab*
| WHERE service.name == "fraud-scorer" AND numeric_labels.queue_lag_seconds IS NOT NULL
| STATS max_lag_seconds = MAX(numeric_labels.queue_lag_seconds) BY service.name, cloud.region
| WHERE max_lag_seconds >= 30

The third rule is an Observability custom threshold rule: more than five ERROR documents in two minutes, grouped by service.name.

It represents the sane default: one stable alert instance per affected service. It also demonstrates that different rule types feed the same investigation because all of them write alerts-as-data documents.

All three rules carry the tag alerts-as-data-v2, which is how every query below isolates them.

Tags on rules are not cosmetic: kibana.alert.rule.tags is stored on every alert document, so a tag is effectively a queryable dataset label for your alert history.

Replaying the incident: 18 alert episodes in seven minutes

We replay the incident: three minutes of healthy baseline, the 2.3.0 deploy, queue lag climbing past the threshold, six minutes of rotating gateway 5xx bursts, customer-visible errors, then a rollback and recovery.

The Alerts page during the storm looks like the pager felt.

Notice the table, not just the count.

The same instance IDs appear both as Active and as Recovered within minutes of each other, which is churn you can see but not yet measure.

When the dust settles, the preserved history can be queried in Discover's ES|QL mode.

The raw material of the storm, the error logs themselves, confirms the cascade shape:

FROM logs-payments.stormlab*
| WHERE labels.incident_id == "payments-brownout-20260722" AND log.level == "ERROR"
| STATS errors = COUNT(*) BY minute = DATE_TRUNC(1 minutes, @timestamp), service.name
| SORT minute

![Discover ES|QL view of error counts per minute and service: fraud-scorer errors first, then payment-gateway at seven per minute, then payments-web]]3

Now we leave the logs behind and interrogate only the alert history.

How to tell if an alert storm was one incident or many

FROM .alerts-*
| WHERE kibana.alert.rule.tags == "alerts-as-data-v2"
| STATS episodes = COUNT(*),
        alert_instances = COUNT_DISTINCT(kibana.alert.instance.id),
        first_episode = MIN(kibana.alert.start),
        last_episode = MAX(kibana.alert.start)
  BY rule = kibana.alert.rule.name
| SORT episodes DESC

| rule | episodes | alert_instances | first_episode | last_episode | | :---- | ----: | ----: | :---- | :---- | | gateway-5xx-per-endpoint-status | 14 | 7 | 16:36:38 | 16:42:38 | | payments-error-rate-per-service | 3 | 3 | 16:36:41 | 16:37:41 | | fraud-scorer-queue-lag | 1 | 1 | 16:35:35 | 16:35:35 |

Eighteen episodes, one incident, and every one of them opened inside a seven-minute window.

The gateway rule alone produced 14 of the 18 episodes while tracking only 7 distinct instances, which means half of its episodes were re-opens of something that had just recovered.

That single row is the quantified version of a complaint that usually stays anecdotal: this rule fires more than it helps.

How to find which service failed first with ES|QL

Because the grouping fields were ECS names, each episode carries service.name at the top level, and we can join the alert history against the service catalog.

FROM .alerts-*
| WHERE kibana.alert.rule.tags == "alerts-as-data-v2" AND service.name IS NOT NULL
| STATS first_symptom = MIN(kibana.alert.start) BY service.name
| LOOKUP JOIN payments_service_catalog ON service.name
| KEEP first_symptom, service.name, team, tier, depends_on
| SORT first_symptom

| first_symptom | service.name | team | tier | depends_on | | :---- | :---- | :---- | :---- | :---- | | 16:35:35 | fraud-scorer | risk-ml | backend | payments-queue | | 16:36:41 | payment-gateway | payments-core | edge | fraud-scorer | | 16:37:41 | payments-web | storefront | frontend | payment-gateway |

The alert order matches the dependency chain exactly: the first symptom belongs to team risk-ml, the edge degraded 66 seconds later, and customers noticed a minute after that.

There is a subtler deduction available in this table.

The earliest alerting service depends on payments-queue, which never alerted at all, so the most likely true origin sits one hop upstream of the first visible symptom, and that is where a post-incident review should start reading logs.

Also note which rule is absent from this analysis: the noisy gateway rule grouped by endpoint and status code, so its episodes carry no service.name and cannot participate in the topology join.

A grouping key is the data contract for your future investigations.

How to score which alerting rule is noisiest

A noise scorecard query condenses rule quality into one table.

FROM .alerts-*
| WHERE kibana.alert.rule.tags == "alerts-as-data-v2"
| EVAL minutes_active = COALESCE(kibana.alert.duration.us, 0) / 60000000.0
| STATS episodes = COUNT(*),
        alert_instances = COUNT_DISTINCT(kibana.alert.instance.id),
        flapping_episodes = SUM(CASE(kibana.alert.flapping == true, 1, 0)),
        median_minutes_active = MEDIAN(minutes_active)
  BY rule = kibana.alert.rule.name, revision = kibana.alert.rule.revision
| SORT episodes DESC

| rule | revision | episodes | instances | flapping | median minutes active | | :---- | ----: | ----: | ----: | ----: | ----: | | gateway-5xx-per-endpoint-status | 0 | 14 | 7 | 0 | 1.0 | | payments-error-rate-per-service | 0 | 3 | 3 | 0 | 7.0 | | fraud-scorer-queue-lag | 0 | 1 | 1 | 0 | 9.0 |

Read the last column first.

A median active time of one minute means the typical episode from the gateway rule was already resolving itself before the on-call could even sign in, while the two well-grouped rules produced episodes long enough to represent the actual incident.

The flapping column is Kibana's own attempt at the same problem, and it is damage control.

The real fix is upstream, in the rule definition, and the alert history has just told us exactly what to change: the grouping key manufactures instances, and the one-minute window manufactures re-opens.

Alert noise reduction: fixing the rule and proving it worked

The rule's own detail page already summarizes the problem: 14 alerts in 24 hours, every one of them from a single incident, produced by the definition on the right.

We update the rule rather than replace it, changing the grouping to service.name, widening the window to five minutes, and raising the threshold.

FROM logs-payments.stormlab*
| WHERE service.name == "payment-gateway" AND log.level == "ERROR"
| STATS error_count = COUNT(*) BY service.name, cloud.region
| WHERE error_count >= 10

Updating a rule in place matters for one specific reason: Kibana stamps every alert document with kibana.alert.rule.revision, so the before and after of a tuning change are separable in the same index with no bookkeeping on our side.

Then we replay a compressed version of the same failure: same deploy, same lag ramp, same rotating gateway bursts, same customer impact, rollback after seven minutes.

The scorecard query runs unchanged, and now the revision column earns its place:

| rule | revision | episodes | instances | flapping | median minutes active | | :---- | ----: | ----: | ----: | ----: | ----: | | gateway-5xx-per-endpoint-status | 0 | 14 | 7 | 0 | 1.0 | | payments-error-rate-per-service | 0 | 6 | 3 | 0 | 6.0 | | fraud-scorer-queue-lag | 0 | 2 | 1 | 0 | 8.0 | | gateway-5xx-per-endpoint-status | 1 | 1 | 1 | 0 | 7.0 |

The unchanged rules now show both incidents in their rows, six and two episodes, which is exactly what stable rules should do.

The gateway rule splits into two rows because the revision changed, and the contrast is the entire argument of this article in four numbers.

Same failure shape, one episode instead of fourteen, and the median episode now lasts seven minutes instead of one, so it spans the incident instead of a fraction of it.

This is the part of alerting work that usually relies on faith, and here it is a query result: the tuning change is verified against the same incident pattern, in the same index, by the same ES|QL.

Nothing about this loop is specific to our lab rule.

Any rule that tags its alerts can be scored, tuned, and re-scored this way, which turns alert tuning from an opinion into a measurement.

Querying alert history with an Agent Builder agent

Every query above is deterministic, which makes them ideal tools for an AI agent.

We register three ES|QL tools in Agent Builder, each one a parameterized version of a query from this article with an explicit KEEP projection, and wire them into an agent called Alert Historian.

POST kbn:/api/agent_builder/tools
{
  "id": "alert_noise_scorecard",
  "type": "esql",
  "description": "Noise scorecard per alerting rule and revision: episodes, distinct instances, flapping episodes, median minutes active.",
  "configuration": {
    "query": "FROM .alerts-* | WHERE kibana.alert.rule.tags == \"alerts-as-data-v2\" AND DATE_DIFF(\"hours\", kibana.alert.start, NOW()) <= ?lookback_hours | EVAL minutes_active = COALESCE(kibana.alert.duration.us, 0) / 60000000.0 | STATS episodes = COUNT(*), alert_instances = COUNT_DISTINCT(kibana.alert.instance.id), flapping_episodes = SUM(CASE(kibana.alert.flapping == true, 1, 0)), median_minutes_active = MEDIAN(minutes_active) BY rule = kibana.alert.rule.name, revision = kibana.alert.rule.revision | KEEP rule, revision, episodes, alert_instances, flapping_episodes, median_minutes_active | SORT episodes DESC",
    "params": {
      "lookback_hours": {
        "type": "integer",
        "description": "How many hours of alert history to analyze, for example 6"
      }
    }
  }
}

The agent's instructions define the vocabulary (an episode is one alert document), the method (timeline first, then propagation, then scorecard), and the honesty rules (report numbers and timestamps from tool output, never invent alerts).

We ask it one question: what happened in the last three hours, was it one incident or many, which team owns the first symptom, and did the gateway rule tuning reduce noise?

The agent called all three tools and got the structure right: it separated the history into two storms with a quiet gap, one per failure wave, rather than blending them into one event.

It attributed the first symptom to team risk-ml, reproduced the upstream deduction about the queue that never alerted, and then answered the tuning question with the revision table.

Its verdict on the fix reads like a review comment backed by data: fourteen episodes at revision 0 against one stable episode at revision 1, with the median episode going from one minute to seven.

The agent is not doing anything we could not do by hand.

That is precisely the point: because alerts are durable, structured data with explicit semantics, a language model can reason over patterns and histories instead of reacting to a single notification, and every number it states is traceable to a tool call you can rerun.

Running alert history queries in production

A few things to decide before you rely on alert history in production.

Retention is a learning-window decision

Alert indices follow an ILM policy, and the default keeps far more than the few hours this lab needed, but if you want quarter-over-quarter rule quality trends, align the policy with that ambition deliberately.

Scope alert index read access narrowly

Alert documents embed rule parameters, reasons, and deep links, so treat .alerts-* read privileges like any other operational dataset and grant the narrowest index patterns that work, for example only the observability alert aliases.

Grouping keys are a data contract

Use ECS field names in BY clauses when they exist, keep cardinality bounded, and never put secrets or personal data into grouping values, because they become kibana.alert.instance.id and live for the retention window.

Project columns aggressively with KEEP

With around 1,400 mapped fields in the alert indices, an unprojected query is unpleasant for humans and actively harmful inside LLM tools, so end every reusable query with KEEP.

Alert storms also stress your automation

If a rule triggers a workflow per alert, fourteen episodes mean fourteen executions doing the same work, which is another argument for pointing automation at the aggregated alert dataset rather than at each notification.

Where to start with your own noisiest rule

The lab in this article is intentionally small, and every piece of it generalizes.

Score your noisiest real rule with the scorecard query, fix its grouping or window, and let kibana.alert.rule.revision tell you whether the fix held.

Join your alert history against whatever catalog you already have; a CMDB export in a lookup index is enough, and the propagation order becomes a query.

If you use Agent Builder, wrap your two or three most-used investigation queries as tools, and the next storm review starts with a conversation instead of a wall of notifications.

The companion notebook builds everything used here: the catalog index, the log data stream, the three rules, both incident replays, the queries, and the Agent Builder tools and agent. It runs for about 25 minutes because the rules evaluate once a minute and the episode durations are the data.

A notification tells you something broke.

The alert history tells you how your detection is performing, and it is already sitting in an index waiting to be queried.

The simpler path is hot → frozen: one day on SSD for live troubleshooting, then Index Lifecycle Management (ILM) moves everything else to the frozen tier as searchable snapshots on blob storage. No warm tier. No cold tier. Frozen indices are partially mounted and read-only. You do not keep indexing into them. You do search them in Kibana and ES|QL, without a restore-then-query workflow.

Elasticsearch maps fields dynamically at ingest, so you do not need a rigid schema before you send logs, metrics, traces, or security logs. The same two-tier ILM policy can cover those data streams. Layer Elastic Streams on top when you need per-stream processing, drops, aggregation, downsampling, or retention. For the companion cost walkthrough, see Not every log deserves 90 days: per-stream retention in Elastic Streams.

When four data tiers is more than log storage needs

Traditional Elastic Cloud Hosted deployments often follow a ladder like this:

Each transition requires ILM policy tuning, node sizing for warm and cold hardware profiles, and replica management across tiers. Frozen still depends on a snapshot repository, but you pass through warm and cold first.

At hundreds of terabytes or multiple petabytes per day, those middle tiers become expensive to operate and hard to explain to finance teams. You pay for spinning disks and replica shards on data that nobody queries at second-level latency anymore.

How hot-to-frozen log storage works

A two-tier hot/frozen architecture collapses the ladder:

| Phase | Storage | Purpose | | --- | --- | --- | | Hot (1 day) | SSD, high IO | Active incidents, alerting, SLO burn rates, ongoing ingest | | Frozen | Blob plus cache nodes | Audit, compliance, historical search (read-only searchable snapshots) |

After the first 24 hours, ILM converts indices to searchable snapshots on the found-snapshots repository that Elastic Cloud Hosted provisions against cloud object storage. Data stays queryable. You omit warm and cold entirely.

The hot-to-frozen pattern matches how Elastic documents large time-series Observability and Security deployments that do not require updates after ingest. New events continue to land on hot. Aged backing indices move to frozen.

Elastic benchmarked a related hot-frozen pattern on ECH with 105 TB of logs over 90 days. Frozen-tier Discover queries stayed within single-digit seconds at p99.9 for most tasks, and holding one day in hot plus the rest in frozen cost roughly 16x less than keeping the full dataset on hot nodes alone. See Measuring searchable snapshots performance.

What petabyte-scale log storage costs on Elastic Cloud Hosted

The figures below are an illustrative example based on a telco-scale workload model, not a quote or pricing commitment from Elastic. Your costs will vary with ingest volume, retention, query patterns, cloud region, hardware profile, and licensing tier. Use Elastic Cloud Hosted pricing and your account team for project-specific estimates.

How does Elastic Cloud Hosted pricing differ from Serverless?

Elastic Cloud Hosted (ECH) bills primarily on the memory footprint of the resources you run: hot and frozen tier node capacity (GB RAM-hour), plus snapshot storage and data transfer. Even with a two-tier hot-to-frozen ILM policy, you still size and pay for the Elasticsearch nodes and frozen cache tier that execute queries against blob-backed data.

Elastic Cloud Serverless bills on ingest volume and stored data, not provisioned cluster RAM. That model removes node sizing, capacity planning, and most lifecycle administration. You pay for what you send and what you keep.

This example models ECH only. Serverless is not included in the tables below because Observability Serverless does not yet support writing aged data out to S3 or blob storage for long-term frozen retention. When that capability ships, the same hot-to-blob pattern on Serverless should reduce administration further while keeping ingest-plus-storage pricing. Until then, ECH with hot-to-frozen ILM is the path for petabyte-scale audit and compliance on blob.

The example assumes 1-day hot, then ILM to blob (frozen searchable snapshots) for long-term retention. Treat the per-PB figures as a planning block for mixed telemetry, not as a claim that 1 PB of raw logs occupies the same disk as 1 PB of raw metrics. Time series data streams store metrics far more efficiently than a regular data stream. Calculations use published ECH capacity, snapshot storage, and data transfer rates with Enterprise licensing.

Log storage cost at 1 PB per day without Streams

At 1 PB/day ingest in this example, ECH alone is modeled at $178K/month ($2.14M/year) using published rates from elastic.co/pricing/cloud-hosted.

Scaling linearly:

| Daily ingest | ECH monthly | ECH annual | | --- | --- | --- | | 1 PB/day | $178K | $2.14M | | 2 PB/day | $357K | $4.28M | | 3 PB/day | $535K | $6.42M | | 4 PB/day | $713K | $8.56M | | 5 PB/day | $891K | $10.70M | | 6.5 PB/day | $1.16M | $13.90M | | 8.45 PB/day | $1.51M | $18.07M |

Even without Streams optimization, the two-tier model is built for petabyte-per-day scale: short hot retention, blob-backed frozen search, and no warm or cold hardware profiles. Competitor TCO depends on ingest mix, retention length, and whether other vendors bill long-term searchability as premium hot storage. Compare published ECH rates to your current quotes rather than treating any single percentage as a universal saving.

How Elastic Streams cuts log retention costs by 28%

Adding Elastic Streams controls (drop, aggregate, downsample, retention) on top of the same hot-to-frozen ILM foundation reduces the example run rate by roughly 28%:

| Daily ingest | Streams optimized monthly | Streams optimized annual | | --- | --- | --- | | 1 PB/day | $127K | $1.52M | | 2 PB/day | $254K | $3.04M | | 3 PB/day | $380K | $4.56M | | 4 PB/day | $507K | $6.08M | | 5 PB/day | $634K | $7.60M | | 6.5 PB/day | $824K | $9.88M | | 8.45 PB/day | $1.07M | $12.85M |

At 5 PB/day with 30% year-over-year growth, the three-year comparison looked like this:

| Year | Daily volume | ECH annual | Streams optimized annual | | --- | --- | --- | --- | | Year 1 | 5 PB/day | $10.70M | $7.60M | | Year 2 | 6.5 PB/day | $13.90M | $9.88M | | Year 3 | 8.45 PB/day | $18.07M | $12.85M |

In this example, that is $12.3M lower over three years with Streams optimization compared to ECH alone. Use that delta as a sensitivity check, then adjust for telemetry mix. Metrics in a TSDS, downsampled series, and verbose logs do not share one storage multiplier.

When Serverless adds blob-tier retention for observability data, rerun this comparison on Serverless pricing. Ingest-plus-storage billing plus zero node administration should make Serverless the simpler operational choice for long-retention workloads at scale.

ILM policy example: one day hot, frozen retention for compliance

On ECH, attach a policy that rolls daily on hot and transitions to frozen searchable snapshots. Omit warm, cold, and delete phases when audit requirements call for indefinite retention on blob.

PUT _ilm/policy/obs-hot-frozen
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "frozen": {
        "min_age": "1d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      }
    }
  }
}

The searchable_snapshot action in the frozen phase mounts a read-only, partially mounted snapshot (partial-*). Keep ingest on hot. Point historical Discover and ES|QL at frozen.

Apply the policy through index templates, Kibana Index Management, or the Streams Retention tab. Frozen tier nodes cache frequently queried blob regions locally. First access to a new time range may be slower; repeat queries hit cache and perform closer to hot-tier latency.

For regulated workloads, pair indefinite frozen retention with Streams AI Partitioning so compliance-heavy streams (payments audit, security events) inherit longer lifetimes while verbose debug streams get shorter DSL delete policies on child streams.

Why Elasticsearch data tiers work for petabyte-scale log storage

Among managed observability platforms, Elastic Cloud Hosted is the only option we are aware of that combines:

• Petabyte-scale telemetry ingest on one platform for logs, metrics, traces, security logs, and more.
• Two-tier ILM from hot to frozen without warm or cold hardware profiles.
• Searchable snapshots that query blob storage directly, with no rehydration step.
• Published-rate TCO modeling you can run against ECH pricing before you talk to finance.

Competing observability SaaS tools often charge premium rates for long hot retention, bill per telemetry type, or move aged data to archives that are not searchable with the same query tools you use for live incidents. At petabyte-per-day ingest, those pricing models widen the gap. Elastic built tiering into the search engine itself, which is why Discover, ES|QL, and APM views work across hot and frozen data transparently.

How to size hot and frozen nodes for petabyte-scale log storage

Exact node counts depend on daily ingest, query concurrency, and hardware profile. A practical starting ratio from Elastic benchmarks:

• Hot nodes sized for one day of ingest plus indexing headroom.
• Frozen cache nodes sized for concurrent query patterns, not total blob volume.

Before production rollout:

1. Measure daily telemetry ingest (logs, metrics, traces, security logs).
2. Pilot the one-day hot policy on a representative data stream.
3. Run Discover and ES|QL queries against frozen data to validate cache sizing.
4. Apply Streams drops and partitioning to reduce volume entering hot.

Keep alert evaluation and SLO burn-rate queries on hot. Plan incident workflows that start on the last 24 hours, then expand into frozen for historical context. Frozen queries can show higher latency on first access after cache eviction.

Searchable snapshots and the frozen tier require appropriate Elastic Cloud capabilities. Review Elastic Cloud pricing and your deployment tier before enabling frozen at scale.

How to move your log retention policy to hot-to-frozen

• Replace warm and cold phases with a hot-to-frozen ILM policy on observability data streams.
• Model TCO at your ingest rate using published ECH pricing, then add Streams optimization scenarios.
• Send OpenTelemetry telemetry to wired streams and apply Streams TCO controls.
• Read the searchable snapshots documentation and data tiers guide.

Petabyte observability does not require four storage tiers or choosing between cost and searchability. One day on hot, everything else in frozen searchable snapshots on blob, and Streams dials on top: that is how Elastic Cloud Hosted keeps full history queryable at a TCO finance teams can approve.

The controls work the same on Elastic Cloud Serverless, Elastic Cloud Hosted (ECH), and self-managed clusters. Elasticsearch uses dynamic mapping (schema-on-write with automatic type detection), so the same Streams controls apply to logs, metrics, traces, and any other signal you ingest over OpenTelemetry or native shippers, with no fixed schema required up front.

UI note (Stack 9.5+): the Streams Retention tab was renamed Data lifecycle. This article uses Data lifecycle for current GA. Screenshots captured on earlier builds may still show the old label; the controls themselves are unchanged. See Configure data lifecycle with Streams.

Why observability data storage costs get out of hand

Most teams start with one retention policy for everything. That is simple until:

• Debug logs from a load generator sit next to payment audit events in the same data stream.
• High-cardinality metrics keep growing while only a handful of dimensions matter after a week.
• Trace volume spikes during deploys, but you still pay to store every span at full resolution for 90 days.

The usual fixes (sampling everything globally, turning off logging, or maintaining brittle routing rules in ingest pipelines) trade away visibility for cost. Streams takes a different path: identify patterns first, then apply the right storage policy to each pattern.

Observability data retention controls in Elastic Streams

Streams sits on top of Elasticsearch data streams. Each stream gets its own Data lifecycle tab, processing pipeline, and (for wired streams) a place in a parent/child tree. That structure is what turns generic "reduce observability spend" goals into concrete settings you can audit.

Streams-native controls (configure these in the Streams UI):

| Control | What it does | Typical signal | | --- | --- | --- | | Drop | Stop indexing matched documents before they hit storage | Noisy debug logs, health checks | | Retention (DSL or ILM) | Delete or tier data after a defined age via the Data lifecycle tab | Short-lived app logs vs compliance logs | | Downsample | Replace high-resolution metrics with fixed intervals (from Elastic 9.4) | Infrastructure metrics after 7 days |

Adjacent Elasticsearch capabilities that compound the savings (not separate Streams actions, but important for TCO):

| Capability | What it does | Typical signal | | --- | --- | --- | | TSDS storage efficiency | Dimension-based routing, _tsid/@timestamp sorting, and synthetic _source improve compression; identical _tsid + timestamp data points are rejected | OTel or Prometheus metrics | | Ingest processors / transforms | Processors reshape or drop individual documents before indexing; transforms pivot already-indexed data into summary destinations on a schedule | High-volume counters, enrichment | | ILM tiering | Move older data to warm, cold, or frozen storage (exposed in Streams Data lifecycle on ECH/on-prem) | Long-lived logs and metrics |

On Elastic Cloud Serverless, you also benefit from storage optimizations built into the platform (for example, LogsDB index mode for logs and efficient TSDB storage for metrics). Streams helps you reduce what you store by routing and trimming data before Serverless bills reflect the volume.

On ECH and self-managed deployments, you get the full Elasticsearch lifecycle toolkit: ILM phases, data stream lifecycle (DSL), searchable snapshots, and downsampling. Streams exposes those policies in the UI so you are not hunting through JSON policy files.

Which control to use

Drop, retention and downsampling solve different problems, and the cost of each is paid at a different point.

| Control | What you still pay for | What you give up | Best fit | | --- | --- | --- | --- | | Drop | Nothing. Matched documents are never indexed. | The data, permanently. No reindex, no recovery. | Health checks, load generator output, debug namespaces | | Retention (DSL) | Full storage until the age limit, then nothing. | Any query beyond the retention window. | Short-lived application logs, anything with a compliance floor | | Retention (ILM tiering) | A lower cost per GB as data moves to warm, cold, or frozen. | Query speed on older data. Frozen needs searchable snapshots. | Long-lived logs and metrics on ECH or self-managed | | Downsample | One data point per interval instead of every sample. | Resolution. Original samples are replaced, not kept alongside. | Infrastructure metrics and other high cardinality metrics after the first week |

Step 1: Find log patterns with AI Partitioning

You cannot right-size retention until you know what is in the firehose. Streams AI Partitioning analyzes incoming documents and proposes child streams based on field patterns, service names, log shapes, or plain-English instructions.

A common starting point is the wired logs.otel stream created when you send OpenTelemetry logs to the Managed OTLP endpoint. The same idea applies to metrics and traces wired streams as your deployment ingests them.

How to run AI Partitioning

1. Open Observability > Streams and select a parent wired stream.
2. Run AI Partitioning on a representative time range.
3. Review suggested child streams (for example per team, service, or log level).
4. Refine partitions in natural language if the first pass merges things you want separated.
5. Apply partitions so each child stream inherits processing and retention from the parent, with per-child overrides.

Once partitioned, a payments API stream can keep 90 days while a notifications worker keeps 7. You set that in the UI instead of maintaining reroute processors in Logstash or the OTel Collector.

For a full walkthrough, see One OTLP endpoint, three teams, zero routing rules.

Step 2: Log filtering in Streams instead of your telemetry pipeline

Partitioning tells you who the data belongs to. Processing tells you what to keep.

In the Streams Processing tab you can:

• Add conditional blocks scoped to one child stream or log shape.
• Use AI-generated Grok patterns to parse samples instead of hand-writing regex.
• Add a drop processor for documents that should never be indexed.

This preview-first workflow matters for TCO. You inspect matched, parsed, and dropped samples before anything goes live. That is safer than a pipeline change that silently discards the wrong field.

Example: drop load-generator logs inside a Kubernetes containers stream while keeping application errors. The Kubernetes logs processing walkthrough shows the full flow with the Dropped tab preview.

Drops are the most direct control when the data has zero long-term value. You pay nothing to store what never gets indexed.

Step 3: Set log retention per stream in the Data lifecycle tab

After partitioning, open the Data lifecycle tab (formerly Retention prior to Stack 9.5) on each child stream.

You will see:

• Current storage size and projected ingestion rates.
• Lifecycle phase breakdown when ILM is active (hot, warm, cold, frozen).
• Options to switch between DSL (simple age-based retention) and ILM (tiered storage).

Wired streams support inheritance: set retention on a parent stream and let children inherit it, or override specific children that need longer or shorter lifetimes. Classic streams can follow existing index template policies while still giving you Streams visibility.

For DSL and ILM editing in the UI, see How Streams simplifies retention management and Configure data lifecycle with Streams.

Step 4: Downsample high cardinality metrics and roll up time series

Logs are often about retention length and drops. Metrics are often about resolution over time.

Starting in Elastic 9.4, you can configure downsampling directly in the Streams Data lifecycle tab for both ILM-backed and DSL-managed streams. Pick a phase, set a fixed_interval, and preview the timeline before you save.

Example ILM policy with hot and cold downsampling

{
  "phases": {
    "hot": {
      "min_age": "0ms",
      "actions": {
        "rollover": { "max_age": "1d" },
        "downsample": {
          "fixed_interval": "5m",
          "wait_timeout": "1d"
        }
      }
    },
    "cold": {
      "min_age": "4d",
      "actions": {
        "downsample": {
          "fixed_interval": "10m",
          "wait_timeout": "1d"
        }
      }
    },
    "delete": {
      "min_age": "30d",
      "actions": { "delete": {} }
    }
  }
}

How TSDS reduces metrics storage

Elasticsearch Time Series Data Streams (TSDS) cuts storage for OTel and Prometheus metrics through dimension-based routing (each time series lands on one shard), index sorting by _tsid and @timestamp that co-locates repeated values for better compression, and synthetic _source. TSDS also rejects duplicate data points with an identical _tsid and timestamp. Pair TSDS with downsampling for metrics that only need minute-level granularity after the first week.

See Configure downsampling directly in Elastic Streams for UI steps and shared-policy warnings.

Storage tiers by deployment type

Observability data retention on Elastic Cloud Serverless

Serverless removes cluster sizing work, but you still control what lands in storage. Streams partitioning, drops, and per-stream retention directly affect ingest volume and retention duration, which drives Serverless observability costs.

Platform-level optimizations (LogsDB for logs, columnar TSDB improvements for metrics) stack on top of the policies you set in Streams. You get efficient storage and intentional data lifecycle management.

If you only need log analytics at a lower price point, Logs Essentials on Serverless is another option. Streams TCO controls still apply when you need per-team retention on Complete.

Storage tiers on Elastic Cloud Hosted and self-managed

ECH and on-prem deployments add storage tiering: hot SSD, warm, cold, frozen, and searchable snapshots. ILM moves streams across tiers automatically. DSL offers a simpler delete-after-N-days model when you do not need tiering.

The same Streams UI and APIs apply. The difference is you also choose hardware profiles and snapshot repositories behind the policies.

How retention differs for logs, metrics and traces

Elasticsearch does not require a fixed schema at ingest. OpenTelemetry logs, metrics, and traces can land in wired streams, get partitioned by AI on whatever fields appear in your data, and pick up stream-specific retention.

That matters for TCO because observability signals are not isolated cost lines:

• Logs: drop noise, shorten retention on verbose services, keep audit trails longer.
• Metrics: use TSDS storage efficiency, downsample after incident windows pass.
• Traces: partition by service or environment, align retention with sampling policies.

You are not maintaining three different cost systems. You are applying the same Streams primitives to each signal type.

Suggested observability data retention workflow

1. Ingest to wired streams (for example logs.otel, metrics, or traces OTel endpoints).
2. Partition with AI so each team or service gets its own stream.
3. Drop known noise (health checks, load tests, debug namespaces).
4. Set retention per stream based on compliance and debugging needs.
5. Downsample or tier metrics and high-volume logs on ECH/on-prem.
6. Review the Data lifecycle tab monthly using ingestion graphs to catch drift.

How much storage can you actually save?

Exact savings depend on your workload. Teams that previously kept everything for 30 days often see the largest wins from:

• Dropping 20–40% of debug or duplicate logs at ingest.
• Shortening retention on high-volume, low-value streams.
• Downsampling metrics after the first few days.

Combine those controls with Elasticsearch storage modes (LogsDB, TSDS) for compounding effects. Measure before and after using the storage size metric on each stream's Data lifecycle tab and cluster-level _stats where you have access.

Try per-stream retention on one noisy stream

• Enable wired streams and send sample OTel data to logs.otel.
• Run AI Partitioning on a busy parent stream.
• Set one aggressive retention policy on a noisy child stream and compare storage after 48 hours.
• Read the Streams documentation for API options (Upsert stream, ingest settings) when you want to automate policies in GitOps.

Observability TCO is not about collecting less data by default. It is about storing the right data for the right duration. Streams gives you the map, the scissors, and the dial.

If you run Observability on Elastic Cloud Hosted (ECH), this will feel familiar. Cross-cluster search (CCS) has long enabled teams to query across remote clusters, and it's a mainstay in large scale ECH environments. CPS takes the same idea to Serverless, and makes linking even simpler: no topology mapping, certificate configuration or per connection credentials. You link projects in the Cloud console and Elastic manages the rest.

What is cross-project search?

Serverless organizes your data into projects: Elasticsearch, Observability, or Security, each with its own scaling, permissions, and lifecycle. That isolation is a feature: it preserves data locality where needed, simplifies cost attribution, and lets teams own their own data. The tradeoff, historically, has been visibility. If your logs live in one project and your APM data lives in another, or if every business unit runs its own Observability project, getting a unified view has typically meant duplicating data into a shared store.

CPS removes that tradeoff. You designate an origin project (the project you search from) and link it to other projects you want included in that search. Once linked, an unqualified query like FROM logs* or a Discover session automatically spans the origin plus every linked project; no special syntax, no remote index prefixes. Kibana's project picker lets anyone narrow a query to just the current project or expand it to everything in scope, and administrators can set a default scope per Kibana space so a local incident-response space stays local by default while a global operations space starts broad. Access is enforced per project using your existing Elastic Cloud roles, so linking a project for search doesn't quietly grant broader permissions than a user already has.

What's the difference between cross-project search and cross-cluster search?

Cross-cluster search (ECH, self-managed)

Cross-project search (Serverless)

Where you link

Remote cluster connections, configured per cluster

Link projects in the Elastic Cloud console

Credentials

Certificates or API keys you issue and rotate

Managed by Elastic

Network setup

Network topology you map and maintain

None to configure

Query syntax

Remote cluster name referenced in the query

Unqualified query, no remote index prefix

Scope control

Set per query

Project picker, per-space defaults, and tag-based routing

Where CCS asks administrators to configure remote cluster connections, exchange certificates or API keys, and reference remote cluster names in every query, CPS asks you to link projects in the UI and query the way you already do. It's the same underlying idea: remote search without moving data, adapted to a control plane where infrastructure is meant to disappear.

Why cross-project search matters for Observability teams

Cross-project search matters most the moment your observability data stops living in one place. Observability organizations typically split their data by team, environment, region, or customer, usually for good reasons: isolating noisy neighbors, controlling blast radius, meeting data residency requirements, or giving each team ownership over its own ingest and retention. The problem shows up during an incident or a cross-cutting investigation, when someone needs to see across those boundaries without waiting for a data migration or standing up a consolidated cluster.

A few examples of where CPS changes the equation:

Keep observability data in its region and still search globally

Teams operating in multiple regions or clouds often need to keep observability data in the region it was generated, whether for regulatory reasons or to limit cross-region egress. CPS lets each region keep its own Observability project while a central project links to all of them, so an SRE team can run one query to check whether an anomaly in one region is also showing up elsewhere, without pulling all that data into one place first.

Isolate environments without losing the incident view

Splitting production, staging, and per-service or per-team projects makes scaling and cost predictable, and it limits how far a bad deploy or a runaway job can spread. It also means an on-call engineer used to have to check several places during an incident. With CPS, on-call can work from a single origin project and pull in whichever linked environments matter to the investigation, then narrow back down once things are resolved.

Central teams monitoring many business units or customers

Platform or Observability Center of Excellence (CoE) teams that operate on behalf of many business units, or Managed Service Providers (MSPs) monitoring several customer environments, can give each unit or customer its own project, with its own retention, access, and cost, while the central team links across all of them for cross-cutting reporting and anomaly detection. Each project owner keeps control of their own data; the center gets the aggregate view it needs.

Avoid ingesting the same data twice

When two teams need the same data for different purposes, the old options were duplicate ingestion pipelines or a shared, less isolated deployment. With linked projects, data is ingested once and queried from wherever it's needed, which keeps both storage costs and pipeline maintenance down.

This applies across solution types too: CPS links Elasticsearch, Observability, and Security projects interchangeably, so a Security team and an Observability team can each keep their own project, with their own retention and access controls, while querying the same underlying logs. A security analyst investigating an alert can query infrastructure or service context from a linked Observability project in Discover, dashboards, or ES|QL, and an SRE debugging an incident can query related signals from a linked Security project the same way, all without either team re-ingesting the other's data.

What does cross-project search support in Elastic Observability?

At GA, CPS is available across Elasticsearch, Observability, and Security project types (Observability and Security projects require the Complete tier). For Elastic Observability specifically, application and infrastructure monitoring, alerting rules, SLOs, machine learning anomaly detection (including APM anomaly detection), and Synthetics work across linked projects, alongside the platform-level capabilities every project gets: Discover, dashboards, and ES|QL. Project routing lets you target a query to a subset of linked projects using predefined tags (alias, cloud provider, region) or your own custom tags, combined with boolean logic, so "all EU projects" or "every environment except staging" can be expressed once and reused.

This release also raises the default number of projects you can link to a single origin, well beyond the tech preview limit, with room to go higher for larger deployments. Any existing project can now act as an origin rather than only newly created ones. For programmatic access, you only need a Cloud API key to call CPS from your own tooling. A new dedicated cloud role can now grant those keys for programmatic access.

A few nuances are worth knowing before you design around CPS:

• Alerting: Alerting rules defined in the origin project can evaluate data across linked projects. The alerts they generate stay local to the project where the alerting rule lives.
• SLOs: SLOs don't follow the project picker in the Kibana header. Instead, you set scope in the SLO create and edit form, choosing this project, all linked projects, or a specific subset, since changing scope rebuilds the SLO's underlying computation. You can't view SLOs created in linked projects from the current project, but you can create an SLO here based on data that lives in those projects.
• Synthetics: Overview, monitor details, certificates, errors, and journeys honor the Kibana header project picker and show linked-project monitors as read-only. Creating, editing, and managing monitors stay in the origin project, and Synthetics status and TLS rules still evaluate origin monitors only.
• Streams: Not CPS-aware yet; Streams stay scoped to the project where they were created.

How do you set up cross-project search?

Setting up cross-project search takes a few minutes:

1. From your origin project's settings in the Elastic Cloud console, open Link projects.
2. Select the projects you want to link, individually or in bulk.
3. Save.
4. Open Discover, run an ES|QL query, or build a dashboard as you normally would: linked project data is included automatically, and the project picker lets you narrow or widen the scope at any time.

For the full configuration walkthrough, including linking topologies (hub-and-spoke, shared data project, or full mesh) and how to scope project routing with tags and expressions, see the cross-project search documentation and the Observability-specific guide.

Cross-project search pricing

CPS pricing has two components: a charge for data retained in each linked project you query, and a charge for data egress generated by cross-project query traffic, tracked as its own line item so you can see what CPS is adding to your transfer costs. There's no separate per-link fee. Because actual cost depends on your data volumes, retention settings, and how many projects you link, use the Elastic Cloud Serverless pricing estimator to model costs for your own environment against your own ingest and retention numbers.

Start using cross-project search

If you're already running Elastic Observability on Serverless, cross-project search is available in your project settings today: link your projects and run your first cross-project query in Discover. If you haven't tried Elastic Cloud Serverless yet, start a free trial and see how far a single query can reach.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic, Elasticsearch, and associated marks are trademarks, logos, or registered trademarks of Elasticsearch B.V. in the United States and other countries. All other company and product names are trademarks, logos, or registered trademarks of their respective owners.

Why small log files were skipped

Filestream needs a stable identifier for every file it reads so that it can track how far it has read from each file. In earlier versions, we used filesystem metadata such as inode numbers. This was not enough. For example, filesystems such as ext4 can reuse an inode number after a file is deleted, leading to a new file being mistaken for the old one. This is why we introduced the fingerprint identity and made it the default in 9.0.

Fingerprinting means files are identified by their content, not their metadata. The first 1024 bytes (by default) are hashed and used as the unique identity of the file. A registry keeps track of all IDs we encounter beyond the agent's process lifetime. Restart your machine, change the underlying device, host the files over the network, it doesn't matter; Elastic Agent will remember.

The major downside is that we can't hash a file's first 1024 bytes if it doesn't have this many bytes to begin with. Imagine a critical Kubernetes pod that panics on startup and its small log output is the reason the panic never reaches your Kibana dashboard.

The main way we've dealt with this until now is through configuration: instead of 1024 bytes, we could hash 512. Or 256. Or even less. However, each step towards reducing the threshold leaves less file content available to distinguish one file from another. Distinct files are therefore more likely to share the same fingerprint material, making the resulting file identity less stable. This is where the new growing fingerprint can shine instead.

How growing fingerprint splits file identity into two phases

With the introduction of the growing fingerprint mode, filestream splits a file's identity into two phases based on its size. While a file is smaller than the 1024-byte fingerprint threshold, it's in "growing" mode. Filestream identifies it with a hash of everything the file contains so far. When the file grows past the threshold, it gets promoted to a static fingerprint, the one it would have had pre-9.5.

What about file-identity collisions? Two files with identical contents do share a growing fingerprint, but only for as long as they stay identical: the moment their contents diverge, so do their fingerprints.

Let's take a closer look inside the internals. Let's start with a simple 13-byte text file:

mkdir -p /tmp/demo
echo 'Hello world!' > /tmp/demo/file.txt

and start Filebeat, the component inside Elastic Agent that collects log files, with a minimal configuration:

cat >/tmp/demo/filebeat.yml <<YAML
path.home: /tmp/demo

filebeat.inputs:
  - type: filestream
    id: growing-demo
    paths: [ "/tmp/demo/file.txt" ]

output.file:
  path: /tmp/demo/out
logging.level: debug
YAML

filebeat -c /tmp/demo/filebeat.yml

Phase 1: the growing fingerprint of a 13-byte file

After a few moments, filestream will pick up our new file. Digging through the debug logs (vastly reduced here for simplicity), we see:

{"log.level":"debug","log.logger":"input.filestream.prospector","message":"A new file /tmp/demo/file.txt has been found"}

We can also directly check the registry write-ahead log (WAL) filestream writes to data/registry/filebeat/log.json:

{
  "k": "filestream::growing-demo::fingerprint::f04f9d5ad8c21a115cd207488c6aaadeb4d7db581ea675f48c7f5d23e038c805",
  "v": {
    "cursor": {
      "eof": false,
      "offset": 13
    },
    "meta": {
      "fingerprint_len": 13,
      "identifier_name": "fingerprint",
      "source": "/tmp/demo/file.txt"
    }
  }
}

The k field is the registry key filestream uses to identify this file state. Its final segment is the fingerprint hash, which we can reproduce from the file contents:

$ xxd -p /tmp/demo/file.txt | tr -d '\n' | sha256sum
f04f9d5ad8c21a115cd207488c6aaadeb4d7db581ea675f48c7f5d23e038c805  -

Still in phase 1: the fingerprint changes as the file grows

Let's write more lines to the file:

for i in {1..70}; do echo "Hello $i" >> /tmp/demo/file.txt; done

Filestream detects the write while the file remains below the threshold:

{"log.logger":"input.filestream.prospector","message":"File /tmp/demo/file.txt has been updated","operation":"write"}

In the registry, the larger file now has a new fingerprint and a 634-byte fingerprint_len:

{
  "k": "filestream::growing-demo::fingerprint::cb8e361971e7ba91e343d1b5bd8e6560be0bdadbbed2b2f89a6c50099e972d63",
  "v": {
    "cursor": {
      "eof": false,
      "offset": 634
    },
    "meta": {
      "fingerprint_len": 634,
      "identifier_name": "fingerprint",
      "source": "/tmp/demo/file.txt"
    }
  }
}

The updated key ends in the SHA-256 hash of the hexadecimal fingerprint material:

$ xxd -p /tmp/demo/file.txt | tr -d '\n' | sha256sum
cb8e361971e7ba91e343d1b5bd8e6560be0bdadbbed2b2f89a6c50099e972d63  -

Phase 2: promotion to a static 1024-byte fingerprint

Let's now grow the file beyond the 1024-byte threshold:

for i in {1..500}; do echo "More messages $i" >> /tmp/demo/file.txt; done

Once the file crosses the threshold, filestream reports another write:

{"log.logger":"input.filestream.prospector","message":"File /tmp/demo/file.txt has been updated","operation":"write"}

The registry key changes one final time as the growing fingerprint is promoted to its static form:

{
  "k": "filestream::growing-demo::fingerprint::c8e252d76d621a02161f92e7a59d1617e3079db666cefd6cfcab106329082d1e",
  "v": {
    "cursor": {
      "eof": false,
      "offset": 9526
    },
    "meta": {
      "identifier_name": "fingerprint",
      "source": "/tmp/demo/file.txt"
    }
  }
}

The promoted fingerprint hashes the first 1024 bytes. It is exactly what a pre-9.5 Filebeat would have computed for this file:

$ head -c 1024 /tmp/demo/file.txt | sha256sum
c8e252d76d621a02161f92e7a59d1617e3079db666cefd6cfcab106329082d1e

How filestream matches a growing fingerprint to an existing file

During each scan, filestream resolves a file's identity in this order:

1. Look for an exact registry key matching the file's current fingerprint. This part has not changed.
2. If no key is found, search the growing entries for a shorter fingerprint, trying the same path first. A prefix match means filestream saw the same file earlier, when it contained fewer bytes.
3. Move the existing registry state to the new key, preserving the read offset.

You might have noticed from the registry entries above that we don't store the raw file contents. That was our initial approach, but it quickly exploded the size of the registry. Instead, we use the SHA-256 hash as the registry key and store fingerprint_len, which records the number of raw bytes the fingerprint represents. This reduces the storage required per entry by nearly 100×.

When filestream looks for a prefix match, it first groups the growing fingerprints by length. It then streams the scanned file's hexadecimal fingerprint material through SHA-256, snapshotting the digest at each group length. Filestream compares each snapshot only with stored hashes for the same length. A match means that the stored fingerprint is a prefix of the scanned file.

Growing fingerprint benchmarks: memory and CPU at 10,000 log files

So what does all this extra work cost? We benchmark Elastic Agent 9.5 ingesting thousands of above-threshold and sub-threshold files. To make the benchmark fair, we compare the growing fingerprint against the workaround of reducing the fingerprint size to 128 bytes. Otherwise, we would be comparing doing something against doing nothing and nothing would always win.

The benchmarks run on an r8id.2xlarge EC2 instance (8 vCPUs, Ubuntu 24.04) and report the median of five runs. Memory numbers are the Elastic Agent's anonymous memory as reported by the kernel via cgroups.

At 10,000 files, the two modes compare as follows.

| Metric | Condition | 128-byte static | Growing | Difference | |---|---|---|---|---| | Peak memory | Files above 1 KiB | 1,057 MiB | 1,099 MiB | +4% | | Peak memory | Files below 1 KiB | 941 MiB | 1,012 MiB | +8% | | Peak memory | Files crossing the threshold | 968 MiB | 1,016 MiB | +5% | | Steady-state memory | Files above 1 KiB | 291 MiB | 290 MiB | none | | Steady-state memory | Files below 1 KiB | 270 MiB | 322 MiB | +52 MiB | | CPU, one core | Files above 1 KiB, idle | 4.1% | 4.1% | none | | CPU, one core | Files below 1 KiB, idle | 3.9% | 4.9% | +1.0pp | | CPU, one core | Files crossing the threshold | 16.8% | 25.5% | +8.7pp | | Registry size at rest | Files above and below 1 KiB | 8.6 MB | 8.6 MB | none |

Memory use at 100, 1,000 and 10,000 log files

At 100 and 1,000 files, 128-byte static and growing fingerprints differ by only a few MiB. At 10,000 files the peaks separate by a few percent, but the only lasting difference is for sub-threshold files: above the threshold both modes stabilize within a MiB of each other.

The most demanding case is when the fingerprints are still changing. We created 10,000 sub-threshold files and appended 2,000 lines/s across them until every file crossed the threshold. That repeatedly exercises the prefix-lookup path and moves registry state to new keys. The extra memory this costs is released once the files are past the threshold and their harvesters close: both modes settle under 300 MiB, within a few MiB of each other:

CPU use with and without writes

The CPU impact is larger: every scan re-hashes a sub-threshold file's full contents instead of a fixed 128-byte prefix. That costs one extra percentage point of a core with no writes coming in, and almost nine under the changing-fingerprint workload above:

What changes in 9.6

We are already working on improving these numbers. On current main (future 9.6 release), the same benchmarks peak at less than half the memory, and the growing fingerprint's steady-state overhead, in both memory and CPU, is roughly halved.

Upgrading and rolling back

Growing mode ships enabled by default in 9.5 through file_identity.fingerprint.growing. Set it to false to restore the pre-9.5 behavior. There is no need for a migration on upgrade; existing registry entries keep working and files that were previously too small get picked up on the next scan.

Rolling back is also possible, but you go back to pre-9.5 behaviour: small files are ignored and when they cross the threshold, they get re-ingested.

What growing fingerprint means for your log files

Elastic Agent 9.5 ingests log files smaller than 1024 bytes by default, with no configuration and no waiting for the file to grow. Shared identities between identical small files resolve themselves as soon as their contents diverge, which a static fingerprint cannot do. The cost stays small: at realistic node scales the growing fingerprint is indistinguishable from the old workarounds, and at a 10,000-file workload it costs a few percent of memory and single-digit percent of one CPU core. Remember when upgrading that small files that were invisible before count as new, however long they have been sitting in your paths. If the trade doesn't work for you, file_identity.fingerprint.growing: false brings back the old behavior.

References

• How to choose file identity for filestream covers all identity options and their trade-offs.
• Growing fingerprint reference documents the setting and the rollback caveat.
• PR #50566 is the implementation and PR #51784 the registry optimization; PR #51863 and PR #51914 trimmed per-scan allocations and fingerprint retention in 9.5.2.
• Optimizations coming up in 9.6: PR #51278 parks idle harvesters instead of keeping one goroutine per open file, PR #51694 scans each file in a single pass, and PR #52476 normalizes published events in place. PR #52103, which replaces the multiline timeout goroutine with a read deadline, is not exercised by these benchmarks.

The Elastic Supabase Integration handles Supabase monitoring for every one of them, with nothing to deploy. No agent and no collector. Metrics come from Supabase's Prometheus-compatible Metrics API, logs arrive through Supabase Log Drains, and both sit in Elasticsearch where you can query them together. Four dashboards, a full set of alert rules, and four SLO templates install once data arrives. Elastic is a listed metrics monitoring partner on Supabase's observability docs.

What Supabase monitoring covers, layer by layer

The Supabase Integration covers the full stack of what Supabase actually is: a database plus a connection pooler plus an auth service plus a REST API layer plus real-time subscriptions plus file storage, all running on a node you don't directly manage.

The integration collects two signal types: metrics scraped from Supabase's Prometheus-compatible Metrics API, and logs forwarded through Supabase Log Drains. But what's interesting is the depth of each; this isn't "CPU and error rate." It goes all the way from the node your project runs on down to individual PostgreSQL statements, connection pool wait times, and per-service request counts.

Infrastructure and node metrics cover the compute layer Supabase runs on: CPU utilisation, memory availability, disk I/O, filesystem usage, and network traffic. On Supabase's smaller compute tiers, CPU and disk I/O are the most commonly exhausted resources. Their exhaustion cascades quietly into slower queries and connection pool pressure before anything explicitly fails.

PostgreSQL internals go deep: database size, active connections versus the max_connections ceiling, transaction commit and rollback counts, cache hit ratios, WAL volume, replication lag, bgwriter activity, and statement-level statistics from pg_stat_statements. This is the data you need to understand whether slowdowns are a schema problem, a connection problem, or a capacity problem.

Connection pooling covers both Supavisor (Supabase's shared pooler) and PgBouncer (the dedicated pooler). Pool occupancy, idle connections, and client wait times are the signals that show you whether connection exhaustion is approaching before clients start getting refused.

Application services: Auth (GoTrue), PostgREST, and Realtime emits their own metrics. GoTrue tracks running status and authentication request rates. PostgREST exposes database pool availability and timeout counts. Realtime tracks active client subscriptions. These are the layers users actually interact with, and their health is distinct from PostgreSQL's health.

Logs from every Supabase service land in a single data stream (logs-supabase.otel-*), with a service filter on the dashboard to focus on any individual component. When a PostgREST error rate spikes, the logs tell you exactly what requests failed and why.

All metrics land in metrics-supabase.metrics.otel-* with their original Prometheus names preserved.

How to connect Supabase metrics and logs to Elastic

The integration has two setup paths that run in parallel: metrics and logs.

For metrics:

1. In Kibana, go to Management → Integrations → Supabase and install the Supabase integration.
2. From your Supabase project, copy the Project URL under Settings → API.
3. Copy your service_role key from the same page.

The integration uses those to authenticate against Supabase's Prometheus endpoint and scrape metrics on a schedule.

For logs, Elastic operates a managed endpoint that receives log drain payloads. In Kibana, go to Add data → More → Supabase and copy the Supabase logs endpoint URL. It follows this pattern:

https://<managed-endpoint>/inputs/supabase/_default_/v1/logs

1. Create an API key there.
2. Go to your Supabase project's Settings → Log Drains and create a drain.
3. Choose OpenTelemetry (OTLP) as the destination.
4. Set the endpoint URL.
5. Choose http/protobuf as the protocol.
6. Enable Gzip.
7. Add the header:

Authorization: ApiKey <your-key>

For the full setup walkthrough, refer to the Supabase (OpenTelemetry) Integration docs.

What ships with Supabase monitoring in Elastic

Once data is flowing, dashboards, alert rule templates, and SLO templates install automatically via the Supabase OpenTelemetry Assets package.

Four Supabase dashboards in Kibana

The Node & Infrastructure dashboard is where resource exhaustion shows up first. CPU saturation, memory pressure, disk I/O utilisation, filesystem usage, and network throughput – all at the node level. On smaller compute tiers, this dashboard often tells the story before PostgreSQL metrics do.

The Databases & PostgreSQL dashboard covers the database layer in depth: connection counts versus the max_connections limit, transaction throughput, cache hit ratios, WAL volume, replication lag, PgBouncer pool status, and bgwriter activity. This is where you diagnose whether a slowdown is a schema problem, a connection problem, or a capacity problem.

The Services & API dashboard covers the application layer: Auth (GoTrue) running status and request rates, PostgREST database pool availability and timeouts, Realtime subscription counts, and Go runtime metrics. A PostgREST pool timeout spike or a GoTrue outage shows up here before users start filing support tickets.

The Service Logs dashboard brings together log streams from every Supabase service with a service filter to focus on any single component. When an error rate spikes on the Services dashboard, this is where you read what actually happened.

Alert rule templates for Supabase failure modes

Pre-built alert rules ship with the Assets package, covering Supabase's specific failure modes in a way generic infrastructure alerts don't. They cover:

• Service availability – PostgreSQL, Auth (GoTrue), and PgBouncer each have dedicated rules, because each failure has a different blast radius. PostgreSQL down means everything fails. Auth down means no new logins. PgBouncer down means clients fall back to direct connections and can exhaust the connection budget within minutes.
• Connection budget exhaustion – the most common way a Supabase project fails. Once max_connections is hit, new clients are refused outright.
• Long running transactions – which hold locks, block vacuum, and pin WAL simultaneously.
• PostgreSQL deadlocks – any deadlock in the window is worth surfacing. These are application bugs, not capacity problems, and they show up as pg_stat_database_deadlocks_total increasing.
• Replication slot lag – building quietly toward disk exhaustion.
• Disk I/O saturation and memory pressure – infrastructure signals that precede failures at the database and service layer.
• PostgREST pool timeouts – each one a user visible 504 error.
• Metrics collection health – fires when the scraper itself starts failing, because a silent scraper means every other alert goes quiet.

SLO templates for Supabase

Four SLO templates ship with the Assets package.

| SLO template | Target | Window | What it catches | |---|---|---|---| | Database availability | 99.5% | Rolling 30 days | PostgreSQL down, which takes everything else with it | | Mean query latency | 99.5% | Rolling 30 days | Average execution time above 50ms, the closest project-wide signal for a slow API | | REST API connection pool timeouts | 99.5% | Rolling 30 days | PostgREST pool exhaustion, where every timeout is a user visible HTTP 504 | | Transaction commit ratio | 99.5% | Rolling 30 days | Fewer than 95% of transactions committing |

Start monitoring your Supabase project

If you're on Supabase Pro, Team, or Enterprise and already using Elastic Cloud, both signal types are available to you today. The integration covers the full stack (infrastructure, database internals, application services, and logs), so you can start with whatever is most pressing and layer in the rest from there.

The integration is available on Elastic Cloud Serverless and Elastic Cloud Hosted. Head to Management → Integrations → Supabase in Kibana to get started, or follow the Supabase (OpenTelemetry) Integration docs.

This post builds that pipeline from a real debugging session and ends with Agent Builder diagnosing a Jellyfin playback failure from the enriched logs. It also covers the alternatives, journald and the OpenTelemetry Collector, and when to prefer them.

How rootless Podman logs differ from Docker logs

The Docker integration reads Docker's format, paths and socket, and rootless Podman uses different ones for all three. Each produced a distinct symptom during our debugging session.

The log format is CRI, not Docker JSON

Podman's default file-based log driver is k8s-file, and even --log-driver=json-file is just an alias for it.

Instead of Docker's one-JSON-object-per-line format, Podman writes the format used by Kubernetes container runtimes (CRI): a timestamp, the stream name, a partial/full flag, and then the message.

An example follows:

A parser expecting Docker JSON breaks on this immediately.

Where rootless Podman stores container logs

Rootful Docker writes logs under /var/lib/docker/containers/<container-id>/. Rootless Podman stores everything under the user's home instead:

Any integration with a hardcoded /var/lib/... path finds nothing.

The Podman API socket is per-user

In rootless Podman there is no /var/run/docker.sock. Instead, it exposes a Docker-compatible API on a user-scoped socket, typically /run/user/<uid>/podman/podman.sock.

Where Elastic Agent runs: host install vs containerized

This walkthrough assumes Elastic Agent is installed directly on the host, not in a container. In our case, the Agent is Fleet-managed and installed directly on a Ubuntu host. It is not running in a container. That matters because the paths in the Agent Policy are resolved by the Agent process itself: a containerized Agent only sees its own mount namespace, so the host directories above would have to be volume-mounted into the Agent container and referenced by their in-container paths. Running as root with default privileges is also what lets the Agent read files under another user's home directory.

How the rootless Podman log pipeline fits together

The setup we ended up with is simple once all the pieces are known:

1. Elastic Agent tails the raw CRI log files from the rootless user's container storage.
2. A container parser with format: cri strips the timestamp, stream, and flag prefix from each line.
3. A processor extracts the container ID from a fixed position in the log file path.
4. The add_docker_metadata processor resolves that ID against the rootless Podman socket and attaches container name, image, and labels.
5. Enriched events are shipped to Elasticsearch.

Let's now build the log ingestion pipeline step by step.

Step 1: Confirm the Podman log driver

First, check which log driver Podman uses on your host, because the default varies by distribution and containers.conf:

If it prints journald, either switch to the file-based driver or jump to the journald alternative at the end of this post. To make k8s-file the default for all containers of this user, set it in ~/.config/containers/containers.conf:

You can also set it per container with podman run --log-driver=k8s-file. Remember that json-file is accepted but silently behaves as k8s-file: do not expect Docker-formatted JSON from it. Containers pick up the driver at creation time, so you will need to recreate any containers that were started with a different driver.

Step 2: Enable the rootless Podman API socket

The metadata enrichment in step 4 needs the Podman API. Enable the user-scoped socket as the user that runs the containers:

The enable-linger call keeps the user's systemd instance, and with it the socket, alive when the user is not logged in. Verify the socket answers Docker-compatible API calls:

You should get a JSON array describing the running containers. This compatibility layer is exactly why a processor named add_docker_metadata will work against Podman later.

Step 3: Collect rootless Podman logs with a Custom Logs integration

In Fleet, add the Custom Logs (Filestream) integration to your agent policy instead of the Docker integration.

Set the file path pattern to the rootless storage location:

Then, in the advanced options, configure the parser so the agent understands the CRI format:

The container parser removes the <timestamp> <stream> <flag> prefix, reassembles partial lines (the P flag marks a message that was split), and stores the original timestamp and stream in the event.

For standalone agents, the equivalent input configuration looks like this:

At this point you should see clean log messages in Discover, but without any container context: no name, no image, no labels. A raw container ID buried in log.file.path is all you have, which makes the data hard to filter and nearly impossible to correlate.

Step 4: Enrich with container metadata using add_docker_metadata

The add_docker_metadata processor enriches our logs with information such as the container name. It extracts a container ID from the log file path, queries the Docker-compatible API for that container, and attaches the container's metadata to each event.

Add it to the integration's processors field (or under processors: in a standalone input):

The match_source_index value deserves an explanation, as it is not a commonly used one. The processor splits the log file path on /, discards the empty leading element, and picks the component at the given index as the container ID.

For the rootless Podman path, the indices work out like this:

Index

Component

0

home

1

<user-id>

2

.local

3

share

4

containers

5

storage

6

overlay-containers

7

<container-id>

8

userdata

The default value of 4 exists because Docker's path is /var/lib/docker/containers/<container-id>/..., where index 4 lands on the ID. However, as you can see from the table above, in our case the proper value is 7.

Note that the correct index depends on the depth of the home directory. /home/<user-id>/... puts the ID at index 7, but a nonstandard home location shifts it. Count the components of your actual path to verify whether 7 is the right value for you.

Once the ID resolves correctly, the processor calls the Podman socket and each event gains the familiar container.* fields. Here is a real document from a Jellyfin container managed with podman-compose:

The metadata is more complete than it first appears: alongside container.id and container.image.name, every label on the container arrives too, including the com_docker_compose_* and io_podman_compose_* labels set by podman-compose and the standard org_opencontainers_image_* labels baked into the image. Filtering all logs of a compose project is now one query on container.labels.com_docker_compose_project.

Step 5: Validate the rootless Podman log pipeline

Generate some traffic and check Discover:

A healthy pipeline produces events where:

• message contains only the application log line, with no CRI prefix.
• @timestamp matches the timestamp Podman wrote.
• log.file.path points into overlay-containers.
• container.id is the full 64-character ID, and container.name and container.image.name are populated.

If metadata is missing but messages parse fine, test the socket with the curl command from step 2 and re-check match_source_index. If messages still carry a timestamp prefix, the CRI parser is not applied; verify the parser YAML made it into the integration policy.

Alternatives for collecting rootless Podman logs

The file-based approach above yields the same ECS container.* fields as the Docker integration, but three alternatives are worth weighing first.

Approach

Metadata

ECS container.*

Throughput

Best when

Custom Logs + add_docker_metadata

Name, image, and all labels from the Podman socket

Native

High, plain file tailing

You want the same fields the Docker integration produces

journald log driver

Container ID, name, and image as journal fields

Needs a rename step

Lower, rate limited by RateLimitBurst

Containers run as systemd services via Quadlet

Docker integration on the compat socket

None for logs

Metrics datasets only

Not applicable to logs

You only need metrics, validated per dataset

OpenTelemetry Collector (filelog)

container.id via a regex operator, names and images need custom work

Partial

High, plain file tailing

You are standardizing on EDOT

Collect Podman logs with the journald log driver

Podman integrates with systemd natively, and on many distributions journald is already the default log driver. With --log-driver=journald, Podman writes each log line to the journal and attaches CONTAINER_ID_FULL, CONTAINER_NAME, and the image name as structured journal fields.

You can collect these with the Custom Journald logs integration. The metadata comes for free, with no socket, no path counting, and no match_source_index.

However, journald applies rate limiting that can drop bursts from chatty containers unless you raise RateLimitBurst. Throughput is lower than plain file tailing. The container fields also arrive with journald's naming, so you need a rename step (an ingest pipeline or processors) to get ECS-style container.* fields. Rootless containers also write to the per-user journal rather than the system one. An agent running as root still sees those entries, because reading the default journal as root includes user journals, but enable journald persistence (Storage=persistent in journald.conf) if you want container logs to survive a reboot.

If your containers are managed as systemd services via Quadlet, this is a natural fit.

Point the Docker integration at the Podman compat socket (metrics only)

Since Podman exposes a Docker-compatible API, it is tempting to point the Docker integration's host at unix:///run/user/<uid>/podman/podman.sock.

• Metrics datasets can work, because they only talk to the API.
• Logs do not, because the integration reads Docker's JSON log files from Docker's paths, and rootless Podman uses neither. Treat this as a partial option for metrics, and validate each dataset you enable.

Collect Podman logs with the OpenTelemetry Collector

If you are standardizing on OpenTelemetry, for instance by using EDOT (Elastic Distributions of OpenTelemetry) the Collector's filelog receiver reads the same files with its container operator, which auto-detects CRI-style formats:

Set add_metadata_from_filepath: false because that option expects Kubernetes pod log paths, which don't match Podman's layout.

Here you will miss out on some data enrichment: there is no Podman equivalent of the k8sattributes processor, so container names and labels are not attached automatically.. You can extract container.id from the file path with a regex operator, but resolving it to names and images requires custom work. The contrib Collector does include a podman_stats receiver for container metrics over the same socket, so a Collector-based setup covers metrics well and logs with reduced metadata.

Parsing application logs from Podman containers

The Custom Logs pipeline above ships raw log lines without application-level parsing. For instance, Nginx logs will end up in your cluster plainly, without proper processing. If you'd like to enable proper processing, you can use Ingest Pipelines to "redirect" specific containers (e.g. nginx) to the Ingest Pipeline defined by the Elastic Integration, and get full parsing of that data "for free".

Root cause analysis on Podman logs with Agent Builder

Was it worth the effort? Absolutely! Getting clean, metadata-rich container logs into Elasticsearch lets an LLM investigate issues against the data directly, and Elastic Agent Builder is the quickest way to see that in action.

Agent Builder, available on Elastic Cloud Serverless and the Enterprise tier for Elastic Cloud Hosted and self-managed, provides a chat interface in Kibana, backed by an LLM, with built-in tools to explore your indices and run ES|QL queries against them.

Here is a real example from the host we just configured. Among other containers, it runs Jellyfin, a media server, as a rootless Podman container. One day, playback of a title fails instantly on an Android TV client: the screen goes black and drops back to the menu, with no error apart from a cryptic "Playback error".

In Discover, filtering on container.name: "jellyfin" narrows the view to a few hundred verbose .NET log lines around the playback attempts. The histogram shows a burst of activity at each failed attempt, but the actual cause is buried somewhere in the noise.

Instead of scanning the messages manually, open the Agent Builder panel directly from Discover and ask it to investigate the issue. The agent queries the log data stream on its own, reconstructs the timeline, and returns a diagnosis:

In this run it worked out that the client requested playback six times. Jellyfin tried to convert a 4K Dolby Vision/HEVC video to H.264 through VA-API hardware transcoding while burning in a subtitle stream, and FFmpeg exited with code 187 each time before producing the first HLS segment. That is why playback stopped at zero milliseconds. It also stated its uncertainty honestly: the available logs don't contain FFmpeg's stderr, so it proposed concrete steps to discriminate between a broken transcoding combination and missing GPU access, including checking whether the container can reach the render device /dev/dri/renderD128.

The pipeline we just built made each step of this investigation possible. The agent can slice by container.name only because add_docker_metadata resolved it from the Podman socket, the message field is queryable because the CRI parser stripped the prefixes, and the timeline is trustworthy because @timestamp comes from Podman, not from ingestion time. Without that work, the LLM would face a pile of prefixed raw lines, likely resulting in worse performance and higher token usage.

Conclusion: three settings for rootless Podman logs

Rootless Podman is Docker-compatible where it matters the most: the API. For log collection it uses its own format, file paths, and socket location, which is what the Elastic Docker integration reads.

Once you know that, the fix is three configuration decisions: parse cri instead of Docker JSON, tail the user's overlay-containers storage, and point add_docker_metadata at the per-user Podman socket with a corrected match_source_index.

The same recipe applies to any rootless user on the host; only the home path and UID in the socket change.

If you want to try this workflow end to end to collect your Podman logs, spin up an Elastic Cloud trial, install the Elastic Agent on your host, and you can go from raw CRI files to fully enriched container logs in one integration policy. If you want to push this further, from a one-off chat to automated investigations that open cases with evidence attached, see automated root cause analysis with Elastic Agent Builder.

Three ES|QL tools and about two dozen lines of query text turn AI root cause analysis into something you can check. This article builds them in Elastic Agent Builder and wires them to an agent whose report names the tool behind every number, and a trace.id and document _id behind every root cause claim.

What you need to reproduce this root cause analysis

• Elasticsearch and Kibana 9.5, self-managed or on Elastic Cloud.
• A generative AI connector for Agent Builder. Ours is Anthropic Claude Sonnet 4.6.
• Python 3.12 with opentelemetry-distro[otlp] and the Flask and requests instrumentations.
• An Elasticsearch API key with write access for OTLP ingest and Kibana access for the Agent Builder APIs.

Use the companion notebook if you want to reproduce the use case in this article.

The demo environment: four Python services on OpenTelemetry

Four Python services run on one host. checkout-api handles the customer request and calls pricing-api, which calls fx-rates for a currency quote. A fourth service, search-api, serves product search and sits outside that call path.

Each service exports OTLP straight to the Elasticsearch native OTLP endpoint, with no collector in between. Logs land in logs-generic.otel-default and spans in traces-generic.otel-default. Two checkout-api processes run on different versions at once, which one of the tools below depends on.

Starting a service is one command, with the OTLP endpoint and the service identity in the environment:

export OTEL_EXPORTER_OTLP_ENDPOINT="${ES_URL}/_otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=ApiKey ${ES_API_KEY}"
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true

OTEL_SERVICE_NAME=checkout-api \
OTEL_RESOURCE_ATTRIBUTES="service.version=2026.07.26.2,deployment.environment=production" \
  opentelemetry-instrument python services/checkout_api.py

Why the loudest service is not the root cause

At 09:55:33Z, fx-rates starts refusing about a third of its requests because its cached FX snapshot is older than the max_age it enforces. pricing-api turns each refusal into a 500, and checkout-api returns a 500 to the customer.

Two unrelated things happen in the same window. Release 2026.07.26.2 of checkout-api rolls out at 09:55, the minute the errors start, and 120 milliseconds after fx-rates breaks, search-api starts timing out during a scheduled reindex of its catalog.

One failure propagates, and the two other signals share only the clock:

The on-call view is four unhealthy services and a fresh deploy. This query gives the failure rate per service over the window:

FROM traces-generic.otel-default
| WHERE kind == "Server"
| EVAL failed = CASE(attributes.http.status_code >= 500, 1, 0)
| STATS failures = SUM(failed), requests = COUNT(*)
    BY service = resource.attributes.service.name
| EVAL failure_rate_pct = ROUND(100.0 * failures / requests, 1)
| SORT failure_rate_pct DESC

Nothing in the result ranks the candidates. search-api has the highest failure rate at 36.4%, and it is the one service with no involvement in the failing checkouts.

Automated root cause analysis with the default Elastic AI Agent

Before writing any tools, we gave the incident to the default Elastic AI Agent to see how far the built-in observability skills get on their own:

checkout-api started returning HTTP 500s to customers today. Investigate the
window 2026-07-26T09:50:00.000Z to 2026-07-26T10:02:00.000Z and tell me the root
cause. Context you have from the deploy log: checkout-api release 2026.07.26.2
rolled out at 09:55 UTC, and the on-call channel also reported search-api
timeouts starting at 09:55 UTC.

It got the cascade right. It named fx-rates as the origin, quoted the stale-snapshot message, and separated the search-api timeouts as a different problem. Then it ranked its hypotheses:

Hypothesis 2 claims the release restarted fx-rates without refreshing its snapshot, or lowered its max_age. Neither appears in the telemetry: the release was on checkout-api, and fx-rates reports service.version 2026.07.19.1 across all 7,374 of its spans.

The agent had a timestamp: the deploy and the first error share a minute. Across three runs, it connected them with a different invented mechanism each time:

• a restart that dropped the snapshot
• a new call path that was not exercised before the release
• a previously tolerant code path that stopped tolerating stale data

The numbers have the same problem. The impact summary in that same answer reports the failure rate as "~38%", dividing the 2,043 failures by the 5,331 successes instead of the 7,374 requests, with no query next to the number for a reader to check.

The rest of this article closes that gap with tools.

The 16 built-in observability tools in Agent Builder

Agent Builder ships with tools that cover most of the exploration work, so check the built-in tools reference before writing anything. The catalog is under Agent Builder > Manage components > Tools:

Sixteen of the built-in tools are observability tools, organized around investigation steps instead of index operations:

| Tool | What it answers | | :---- | :---- | | observability.get_logs | What is the log volume and shape for this filter, with samples and message categories | | observability.get_traces | What documents belong to these traces, grouped by trace.id | | observability.get_service_topology | Which dependencies does this service have, with error rate per connection | | observability.get_apm_correlations | Which attributes are over-represented in the slow or failing transactions | | observability.run_log_rate_analysis | Which fields or patterns correlate with a change in log throughput | | observability.get_log_change_points | Which message categories spiked, dipped or shifted, and when |

One platform tool matters here too. platform.streams.investigation_progress_report is what an agent calls to publish its hypothesis list while it works, with a status per hypothesis, a conclusion, and an explicit list of what the data could not settle:

{
  "summary": "string",
  "hypotheses": [
    {
      "candidate": "string",
      "confidence": 0.0,
      "status": "investigating | dismissed | confirmed",
      "reason": "string"
    }
  ],
  "conclusion": "string",
  "gaps_found": ["string"]
}

Three ES|QL tools for AI root cause analysis

Agent Builder's built-in observability tools are shaped for showing the landscape. Ruling a candidate out needs one number that answers one question, and two questions in this incident have no built-in tool. A third returns the document IDs that make a claim checkable.

How to tell one incident from two coincident failures (rca_failure_shapes)

Two services failing in the same minute belong to the same incident only if they appear in the same requests. This tool groups error logs by trace, collapses each trace to the set of services that erred inside it, then counts each shape:

FROM logs-generic.otel-default
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND severity_text == "ERROR" AND trace_id IS NOT NULL
| STATS services = VALUES(resource.attributes.service.name) BY trace_id
| EVAL failure_shape = MV_CONCAT(MV_SORT(services), " + ")
| STATS traces = COUNT(*) BY failure_shape
| SORT traces DESC
| LIMIT 20

MV_CONCAT(MV_SORT(...)) matters here. Grouping directly by a multivalue field makes ES|QL expand it, one row per service, which loses the combination. Collapsing the set to a single string first keeps the shape intact.

Two clusters come back, and they share no traces. The three checkout services appear together in 2,043 traces, search-api errs alone in 1,324, and that zero overlap settles the coincidence in one row.

How to rule out a deploy as the root cause (rca_version_split)

A release that caused the failures makes the version carrying it fail at a materially higher rate than the version it replaced. Server spans carry both service.version and the response status, so one query decides it:

FROM traces-generic.otel-default
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND resource.attributes.service.name == ?service AND kind == "Server"
| EVAL failed = CASE(attributes.http.status_code >= 500, 1, 0)
| STATS failures = SUM(failed), requests = COUNT(*)
    BY version = resource.attributes.service.version
| EVAL failure_rate_pct = ROUND(100.0 * failures / requests, 1)
| SORT version

27.2% against 28.2%, on roughly 3,700 requests per version, is a difference within noise. The version that was already running fails as often as the one that shipped, which rules the release out.

The tool depends on both versions serving at once. With an instantaneous and total rollout, no query separates a broken new version from something else breaking at the same moment.

How to return document IDs an agent can cite (rca_evidence_sample)

METADATA _id on an ES|QL source command returns the Elasticsearch document ID, so a claim can point at a specific record:

FROM logs-generic.otel-default METADATA _id, _index
| WHERE @timestamp >= TO_DATETIME(?start) AND @timestamp <= TO_DATETIME(?end)
  AND severity_text == "ERROR" AND resource.attributes.service.name == ?service
| KEEP @timestamp, _id, _index, trace_id,
       attributes.error.kind, attributes.upstream.service, body.text
| SORT @timestamp DESC
| LIMIT 5

Each tool is registered with one POST kbn:/api/agent_builder/tools call carrying the query and its typed parameters. Registered, rca_failure_shapes looks like this; the agent fills start and end at call time:

Write the tool description as the decision it supports

The model reads the tool description to decide when to call the tool, so the description does more work than the name.

We describe rca_failure_shapes as the way to test whether two services that broke together are one failure or two. That phrasing got it called whenever a question mentioned a second failing service.

The description also outweighs the agent instructions. An agent with these three tools and one line of instruction, "You are an SRE assistant, help the user find the root cause of production incidents", still ruled out the deploy and the coincidence every time.

Wiring the tools into an incident root cause analysis agent

The agent gets the three custom tools, four built-in observability tools, and platform.streams.investigation_progress_report. Its instructions are five numbered steps:

1. Scope. Establish the affected service, the failure window, and the size of the
   symptom before you name any cause. State the window as an explicit ISO 8601
   range and reuse that same range in every tool call.

2. Enumerate. Write down at least three candidate causes before you test any of
   them. Include the candidate a human on call would reach for first, such as a
   recent deploy or another service that started failing at the same minute.

3. Refute. For each candidate, state the observation that would prove it wrong,
   then run the query that produces that observation. A candidate is dismissed
   when the refuting observation is present, not when a different candidate
   looks better.

4. Cite. Every number you report must name the tool that returned it. Every claim
   about a root cause must carry at least one trace_id and at least one document
   _id. A claim with no citation is not a finding, it is a guess.

5. Report. Put anything the available data cannot settle in gaps_found rather
   than resolving it with reasoning.

Two rules follow the steps. Each one blocks a mistake we saw in the default agent's answers:

• Do not rank services by error volume and call the loudest one the cause.
• Do not treat time correlation as causation. Two services that start failing in the same minute belong to the same incident only if they appear in the same traces.

The finished agent, with Elastic capabilities switched off so everything it produces comes from these eight tools:

Running the investigation: 11 tool calls, 72 seconds

We gave the new agent the same question, word for word. It opened with a progress report listing its candidates, then ran the refuting queries in parallel batches:

Eleven tool calls and 72 seconds later, the report opens with a citation table:

Three layers, three verbatim messages, three document IDs, and one trace ID shared by all of them. The origin message states the mechanism: StaleQuoteError: fx snapshot age 1215s exceeds max_age 900s (provider=ecb-eod).

The dismissals come next, each with the observation that ruled it out:

| Candidate | Refuting observation | Tool | | :---- | :---- | :---- | | checkout-api release 2026.07.26.2 | Both versions fail at the same rate, 27.2% against 28.2%, on roughly 3,700 requests each | rca_version_split | | search-api timeouts | Zero shared traces. 2,043 traces contain the three checkout services, 1,324 contain search-api alone | rca_failure_shapes | | Database failure | No database errors in the checkout-api logs | rca_evidence_sample |

The report closes with a gaps section:

The first gap covers the question the default agent answered with an invented mechanism. The snapshot was already 1,127 seconds old at the first error, so the feed stopped refreshing before the window opened, and no log in this system records when. The next place to look is the feed ingestion job.

Verifying the agent's citation by hand in ES|QL

The report gives a trace ID, so open it:

FROM logs-generic.otel-default
| WHERE trace_id == "a131d0cf7343195c0a7a14f6a98da6b7"
| KEEP @timestamp, resource.attributes.service.name,
       attributes.upstream.service, body.text
| SORT @timestamp ASC

Three records, one millisecond apart, in the order the report claimed. fx-rates errs first and names no upstream, because it is the origin. pricing-api names fx-rates, and checkout-api names pricing-api.

Add upstream.service to your error logs

upstream.service is a logging convention that OpenTelemetry does not provide: each service writes it on the error it emits when a dependency fails. With that one key, error records order themselves into a call chain, and the ordering survives the clock skew and sub-millisecond hops that break timestamp sorting.

Two other fields carry the rest of the weight: trace.id arrives free from auto-instrumentation as long as the log is emitted inside an active span, and a structured error.kind lets you count failure modes without matching on message text.

Extracting knowledge indicators from your streams

Knowledge indicators are facts Elastic extracts from your raw data with LLM models. It picks out things like the underlying infrastructure and the dependencies between services.

You find this option under Streams > {your_stream} > Significant events:

Click Generate and the indicators are created for you:

What three ES|QL tools changed about the agent's answer

Three ES|QL tools and about two dozen lines of query text turned an agent's answer into a report that names its candidates, shows the observation that ruled each one out, and carries a record ID behind every number.

Next steps

• Run the companion notebook to reproduce the incident, the tools, and the agent in your own cluster.
• Write the query that rules out your team's most common wrong answer, the recent deploy or the loudest service, and register it as a tool whose description states the decision it supports.
• Audit one service's error logs for trace.id, a structured error.kind, and a declared upstream.service. The last two are a one-line change in your logging.
• Register the failure modes your team already knows about as significant events, so the next investigation starts with a history instead of a blank window.
• For a different shape of Agent Builder investigation, read From five dashboards to one prompt, which scores APM service health with five ES|QL tools.

Why managing observability dashboards by hand breaks down at scale

Large organizations often end up with hundreds of dashboards. Teams build similar panels and maintain them using the Kibana UI.

When a small change comes in (a panel rename, a field fix, a new error threshold), there is no easy way to apply it across all of them. You either open each dashboard and edit it in the UI one by one, or you export the NDJSON, run a string replace, and re-import it.

Dashboards are code now

Elastic ships a typed Kibana Dashboards API and a native elasticstack_kibana_dashboard Terraform resource. You define a dashboard in an HCL file and then manage versions and changes as if it was regular code.

Golden signals dashboard: one definition for every team

The platform team owns a standard dashboard built on the four golden signals: latency, traffic, errors, and saturation. Every team should get that standard, and some teams add a panel or two of their own.

We want one definition of the standard, each team's dashboard generated from it, and a single change that reaches every team.

Prerequisites

• An Elastic Cloud deployment or self-managed cluster running Elastic 9.4 or newer, or an Elastic Cloud Serverless project
• Terraform installed
• An Elasticsearch API key

The full Terraform configuration, the seed script, and the captured terraform plan outputs used in this article are available in the companion repo.

Configure the Elastic Terraform provider

Create a provider.tf next to the rest of your Terraform files:

terraform {
  required_providers {
    elasticstack = {
      source  = "elastic/elasticstack"
      version = "~> 0.11"
    }
  }
}

variable "elasticsearch_endpoint" {
  type = string
}

variable "elasticsearch_api_key" {
  type      = string
  sensitive = true
}

variable "kibana_endpoint" {
  type = string
}

variable "kibana_api_key" {
  type      = string
  sensitive = true
}

provider "elasticstack" {
  elasticsearch {
    endpoints = [var.elasticsearch_endpoint]
    api_key   = var.elasticsearch_api_key
  }
  kibana {
    endpoints = [var.kibana_endpoint]
    api_key   = var.kibana_api_key
  }
}

Provide your credentials through a local terraform.tfvars file (and add it to .gitignore so the keys never reach the repo):

elasticsearch_endpoint = "https://...es.region.cloud.es.io"
elasticsearch_api_key  = "..."
kibana_endpoint        = "https://...kb.region.cloud.es.io"
kibana_api_key         = "..."

You can use the same API key for both elasticsearch_api_key and kibana_api_key as long as it has dashboard write privileges in the target space.

Then initialize the working directory:

terraform init

Define a single-team Kibana dashboard in HCL

Start with a baseline dashboard for a single team. Panels sit on a 48-column grid, and each one is a Lens visualization configured inline. Use config_json for KPI tiles (it exposes secondary metrics and value coloring) and xy_chart_config for time-series charts.

Add the baseline resource to a new dashboards.tf:

resource "elasticstack_kibana_dashboard" "golden_signals" {
  title            = "Golden Signals - payments"
  description      = "Latency, traffic, errors"
  query            = { language = "kql", text = "" }
  refresh_interval = { pause = false, value = 60000 }
  time_range       = { from = "now-15m", to = "now" }

  panels = [
    {
      type = "vis"
      grid = { x = 0, y = 0, w = 12, h = 5 }
      config_json = jsonencode({
        type        = "metric"
        data_source = {
          type  = "esql"
          query = "FROM logs-payments-* | STATS `5xx errors` = COUNT(CASE(status >= 500, 1, null))"
        }
        metrics = [{ type = "primary", column = "5xx errors" }]
      })
    },
    # More panels follow the same shape: other metric tiles, xy_chart_config line charts, and a breakdown datatable. See the companion repo for the full file.
  ]
}

Each panel sets a type and grid position, then picks one chart kind. KPI tiles serialize the whole Lens config into config_json; the ES|QL query lives under data_source and the metric column is referenced by name in metrics[*].column. The dashboard time picker already scopes ES|QL panels, so the query needs no explicit @timestamp range filter.

Preview Kibana dashboard changes with terraform plan

Run terraform plan to see what Terraform will create:

terraform plan

The plan output lists the new elasticstack_kibana_dashboard.golden_signals resource and every attribute it will set: the top-level dashboard fields and one entry per panel with its grid position, chart kind, and data source.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # elasticstack_kibana_dashboard.golden_signals will be created
  + resource "elasticstack_kibana_dashboard" "golden_signals" {
      + description      = "Latency, traffic, errors"
      + title            = "Golden Signals - payments"
      + query            = { language = "kql", text = "" }
      + refresh_interval = { pause = false, value = 60000 }
      + time_range       = { from = "now-15m", to = "now" }
      + panels           = [
          # Every panel described in full: KPI tiles (config_json),
          # line charts (xy_chart_config), and the breakdown datatable.
        ]
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Reviewing the plan is your last check before anything ships to Kibana.

Don't apply yet. The next section extends the file with per-team dashboards, and then a single terraform apply ships everything.

Generate per-team observability dashboards from a shared panel library

On top of the baseline, each team gets the standard set of panels, with the option to add a few of their own. Hardcoding one resource per team does not scale. Instead, define a panel library and a teams map as locals, then build the dashboards with for_each. Each library entry describes a chart kind, a title, and the data it needs; the resource emits the right Lens block (config_json for metric tiles, xy_chart_config for line charts) based on chart_type.

Replace the contents of dashboards.tf with:

locals {
  panel_library = {
    errors = {
      chart_type     = "metric"
      title          = "Error rate"
      esql_query_tpl = "FROM {idx} | STATS `5xx errors` = COUNT(CASE(status >= 500, 1, null))"
      esql_column    = "5xx errors"
    }
    saturation = {
      chart_type     = "metric"
      title          = "Saturation (CPU)"
      # Saturation reads from the metrics TSDB, so this query is not parameterized by {idx}.
      esql_query_tpl = "TS metrics-payments-* | STATS avg_cpu = AVG(cpu.pct)"
      esql_column    = "avg_cpu"
    }
    latency = {
      chart_type = "xy"
      title      = "Latency p95"
      x_json     = jsonencode({
        operation          = "date_histogram"
        field              = "@timestamp"
        suggested_interval = "auto"
      })
      y_json = jsonencode({
        operation  = "percentile"
        field      = "duration_ms"
        percentile = 95
      })
    }
    # ... more entries (traffic, cart_value) in the companion repo.
  }

  teams = {
    payments = {
      index  = "logs-payments-*"
      panels = ["errors", "saturation", "latency", "traffic"]
    }
    checkout = {
      index  = "logs-checkout-*"
      panels = ["errors", "cart_value", "latency", "traffic"]
    }
  }
}

resource "elasticstack_kibana_dashboard" "golden_signals" {
  for_each         = local.teams
  title            = "Golden Signals - ${each.key}"
  description      = "Latency, traffic, and errors for the ${each.key} service"
  query            = { language = "kql", text = "" }
  refresh_interval = { pause = false, value = 60000 }
  time_range       = { from = "now-15m", to = "now" }

  sections = [
    {
      title     = "KPIs"
      grid      = { y = 0 }
      collapsed = false
      panels = [
        for i, p in [for q in each.value.panels : q if local.panel_library[q].chart_type == "metric"] : {
          type        = "vis"
          grid        = { x = (i % 4) * 12, y = 0, w = 12, h = 5 }
          config_json = jsonencode({ ... }) # one metric tile per panel; see the companion repo for the full config
        }
      ]
    },
    {
      title     = "Trends"
      grid      = { y = 1 }
      collapsed = false
      panels = [
        for i, p in [for q in each.value.panels : q if local.panel_library[q].chart_type == "xy"] : {
          type       = "vis"
          grid       = { x = (i % 3) * 16, y = 0, w = 16, h = 10 }
          vis_config = { by_value = { xy_chart_config = { ... } } }
        }
      ]
    },
    # A third "Breakdown" section holds the request-by-status datatable. See the companion repo.
  ]
}

Adding a team is one entry in teams. Adding a panel to every team is one entry in panel_library and one reference per team. The full config (data source ES|QL queries, metrics, layers, axis defaults, and legend placement) lives in dashboards.tf.

The saturation panel queries the metrics data stream with the ES|QL TS command, which is designed for TSDB. For the query to work, data streams matching metrics-payments-* must use time_series mode, so the configuration also ships an index template (metrics_tsdb.tf) that enables that.

Apply dashboards as code to Kibana with terraform apply

Run terraform plan to confirm both team dashboards (payments and checkout) will be created then apply:

terraform apply

Open Kibana and you'll see one Golden Signals dashboard per team, each backed by its own index pattern.

Dashboards as code in the GitOps loop: review changes in pull requests

Dashboards are now an artifact in version control, like the rest of your infrastructure.

You edit the library or a team's selection, open a pull request, your reviewer reads the terraform plan diff and sees which dashboards change.

For example, say you tighten the "critical error" threshold from status >= 500 to status >= 503 in panel_library.errors.esql_query_tpl. Running terraform plan shows the change reaching both teams at once:

Note: Full output in terraform-plan-update.txt.

A single edit to panel_library.errors propagates to every team that references it. After the PR merges, it's time to run terraform apply.

After the apply finishes, refresh the dashboards in Kibana and the new threshold is in effect:

Detect dashboard drift and roll back with git

If someone edits a dashboard using the UI, the next terraform plan shows the difference, because the code and the live state no longer match.

To see this in action, open Golden Signals - payments in Kibana, rename the Latency p95 panel to Latency p95 (EDITED), and save the dashboard.

Then run terraform plan:

Terraform reads the panel title from the live dashboard, compares it against the code, and proposes reverting the UI rename. You decide whether to keep the change (update the code to match) or revert it by running terraform apply.

You can commit the new version, or rollback one or many versions using git.

Replaying the earlier example: if you reopen the PR that changed panel_library.errors to broaden the error threshold and add a clearer title, git diff dashboards.tf shows the entire intent in two lines:

Every team that references errors picks up the new threshold on the next terraform apply, and reverting that commit rolls the change back across all of them at once.

Wrap up

Managing Kibana observability dashboards by hand does not scale past a few teams. With the Kibana Dashboards API and Terraform, you define a standard once, compose each team's dashboard from a shared library, and review every change in a pull request. One edit reaches every team, and you can roll back by reverting a commit.

The proposed file structure is only one of many ways you can organize your dashboards depending on how much information they share.

Next steps

• Kibana Dashboards as code with Terraform
• elasticstack_kibana_dashboard resource reference
• Elastic Stack Terraform provider documentation
• Kibana Dashboards API documentation
• ES|QL reference

Why multi-team log routing needs structure

Multi-team Elasticsearch deployments typically converge on a single shared index, which works until teams need different retention periods, sharding settings or processor pipelines.

Before Streams, you had to set up your ingestion scripts to send data to different indices, or data streams, or use the reroute processor to define the data destination based on some field name.

With AI Partitioning, you let the data arrive first. Then the AI analyzes what showed up, reviews the suggestions it proposes, refines them as needed, and applies them. The result is a set of wired child streams that inherit retention, processors, and schema from the parent, while still allowing you to override any of those per child.

What you need before using Streams AI Partitioning

Before running the example, three things need to be in place:

1. Wired Streams enabled. On Elastic Cloud Serverless and Elastic Cloud Hosted 9.4+, wired streams are on by default. If you upgraded from an earlier version, open the Streams app and confirm the toggle is on under Settings.
2. The Elastic Managed LLM connector. Go to Stack Management > Connectors > Create connector > Elastic Managed LLM. Keep in mind the following considerations:
3. This connector ships preconfigured and does not require an external account or API key.
4. Any generative AI connector works with the feature.
5. Note that Elastic Managed LLMs incur a cost per million tokens for input and output.
6. The account you use needs the manage_inference cluster privilege (the built-in inference_admin role grants it).
7. The Managed OTLP endpoint URL and an API key. Open Cloud Console > Manage > Application endpoints > Ingest. Copy the Managed OTLP endpoint URL and generate an API key from that same panel.

Once these are ready, open Observability > Streams and confirm a logs.otel wired stream is listed. That stream is the parent we will partition.

If Streams does not appear in the sidebar, your Kibana space may be using a solution view other than Observability. You can change it in Stack Management > Spaces > edit your space > set Solution view to Observability.

Generating multi-team log data

Our example uses three apps produced by teams from the same fictional company:

• payments-api: structured JSON with transaction_id and amount_cents. Sensitive data, long retention needs.
• checkout-web: JSON with cart_id and customer_id. Mostly INFO and ERROR.
• notifications-worker: less structured, with recipient and channel. High volume.

We use a Python script with the OpenTelemetry Python SDK to emit logs for all three teams over OTLP directly to the Managed OTLP endpoint. The full code, including setup and execution, is in the companion notebook.

Each team is defined with a service name, a set of message templates, and a function that generates team-specific attributes:

TEAMS = {
    "payments": {
        "service": "payments-api",
        "messages": [
            ("INFO", "charge captured tx={tx} amount_cents={amt}"),
            ("ERROR", "charge declined tx={tx} reason=insufficient_funds"),
            ("INFO", "refund issued tx={tx} amount_cents={amt}"),
        ],
        "extra": lambda: {
            "transaction_id": f"tx_{random.randint(10000, 99999)}",
            "amount_cents": random.randint(100, 50000),
        },
    },
    "checkout": {
        "service": "checkout-web",
        "messages": [
            ("INFO", "cart updated cart={cart} customer={cust}"),
            ("INFO", "checkout started cart={cart} customer={cust}"),
            ("ERROR", "checkout failed cart={cart} stage=address_validation"),
        ],
        "extra": lambda: {
            "cart_id": f"c_{random.randint(1000, 9999)}",
            "customer_id": f"u_{random.randint(100, 999)}",
        },
    },
    "notifications": {
        "service": "notifications-worker",
        "messages": [
            ("INFO", "email queued recipient={rcp} channel=email"),
            ("INFO", "sms queued recipient={rcp} channel=sms"),
            ("ERROR", "webhook failed recipient={rcp} channel=webhook status=503"),
        ],
        "extra": lambda: {
            "recipient": f"+1555{random.randint(1000000, 9999999)}",
            "channel": random.choice(["email", "sms", "webhook"]),
        },
    },
}

Setting elasticsearch.index to logs.otel as a resource attribute routes the data into the wired streams root instead of the default OTLP data stream.

def setup_provider():
    resource = Resource.create({"elasticsearch.index": "logs.otel"})
    provider = LoggerProvider(resource=resource)
    provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
    set_logger_provider(provider)
    handler = LoggingHandler(level=logging.INFO, logger_provider=provider)
    root = logging.getLogger()
    root.setLevel(logging.INFO)
    root.addHandler(handler)
    return provider

Run the notebook to emit 115 records with an uneven split across teams.

Open Observability > Streams > logs.otel and switch to the Partitioning tab. You should see the ingested data in the preview panel, with attributes like team, service.name, and the team-specific fields visible in the columns.

How Streams AI Partitioning proposes child streams

Streams AI Partitioning analyzes up to 1,000 documents from the parent stream, identifies attribute clustering and cardinality distribution, then proposes child streams, keyed on whichever field best separates the data logically (the ML approach behind this analysis is detailed in automated log parsing in Streams).

For the data we just emitted, the AI proposed three child streams keyed on attributes.service.name:

Each suggestion shows a Streamlang condition and the percentage of sampled documents that would match. The AI picked service.name because it is a standard OpenTelemetry attribute and a natural identifier for any single workload.

This is a reasonable first proposal, but it is worth thinking about what happens as the deployment grows. Right now there are three services because there are three teams. Tomorrow, Payments might add a refunds-api and a fraud-detector. Each new service would mechanically create another child stream, and over time you would end up with dozens of partitions for what is really just three organizational boundaries.

Elastic's partitioning recommendations prefer logical groupings like team or technology type, and aim for tens of partitions rather than hundreds. A team-keyed partitioning is more stable because Payments stays one child stream no matter how many services that team operates.

Refining Streams AI Partitioning suggestions in natural language

After reviewing the AI's initial suggestions in Streams AI Partitioning, click Modify suggestions to open a free-text prompt.

After submitting, the AI regenerates the suggestions. Now the three cards are keyed on attributes.team instead of service.name:

Select all three and click Accept selected. A confirmation dialog shows the streams that will be created, each with its WHERE attributes.team equals <team> condition.

Click Create all streams. The Partitioning tab now shows the three child streams as part of the logs.otel parent:

Every new document arriving at the OTLP endpoint will be routed into the correct child according to these conditions. You can open any child stream to verify its data. For example, logs.otel.checkout shows only checkout logs:

How do you set per-team log retention in Elasticsearch Streams?

After Streams AI Partitioning creates child streams, each one can have its own lifecycle configuration independently of the parent. Because wired streams use a parent-child hierarchy, every child inherits retention, processors, and schema from the parent by default. You only need to override the partitions you need to change.

Open the child stream logs.otel.payments and go to the Retention tab. Click Edit retention method, select Custom period, and set it to 90 days.

Do the same for the other teams with the retention that fits their needs:

| Stream | Retention | Rationale | | :---- | :---- | :---- | | logs.otel.payments | 90 days | Sensitive financial data, compliance requirements | | logs.otel.checkout | 30 days | Useful for debugging, no long-term need | | logs.otel.notifications | 7 days | High volume, low value after delivery confirmation |

Conclusion: from shared index to per-team streams, without routing rules

A shared Elastic deployment with several teams shipping logs is the normal starting point. Organizing it used to mean writing routing rules upfront or maintaining separate index templates and ILM policies by hand.

With Streams AI Partitioning, the workflow is different: you let the data arrive, let the AI read what showed up, refine the suggestions in natural language when they need adjusting, and accept.

The result is a set of child streams that inherit everything from the parent while giving each team its own retention and processing, without any manual template management.

Next steps

• Try the companion notebook to generate your own multi-team data.
• Read How Streams in Elastic Observability Simplifies Retention Management for a deeper look at the retention model.
• Read Streams Processing: Stop Fighting with Grok to explore the parsing side of Streams when teams need different processors.
• Read Introducing Streams for Observability for the broader investigation story Streams is part of.

Getting SNMP data into Elasticsearch unlocks rich visibility into your network — interface utilization, routing health, L2 forwarding, and more. The path there involves a few familiar steps: choosing which MIBs to walk, mapping OIDs to human-readable field names, configuring SNMP v2c or v3 authentication, accommodating vendor-specific MIB extensions, and tuning the pipeline to gracefully handle device timeouts across large inventories. With a solid template in place, what used to be a bespoke Logstash project becomes a repeatable, shareable setup that any engineer on the team can pick up and extend.

The plugin includes a Logstash pipeline template that handles the common cases out of the box. It walks IF-MIB (interface counters and status), IP-MIB (ARP tables and IP address assignments), BRIDGE-MIB (MAC address forwarding tables), BGP4-MIB (BGP peer sessions), and OSPF-MIB (OSPF neighbor adjacencies) per target device on a configurable poll interval. You add your device list and authentication details, start Logstash, and data begins flowing into Elasticsearch.

The template also handles the operational annoyances that trip people up: poll timeouts, missing OID branches on devices that don't support a given MIB, and batching walks across large device inventories.

Structuring SNMP data in Elasticsearch: schema design

Once SNMP data lands in Elasticsearch, the next problem is structure. Interface counters like ifInOctets and ifOperStatus map to ECS concepts reasonably well. They're host-level metrics with direct equivalents in host.network.* fields. But the data network engineers actually need for troubleshooting is relational, and this plugin offers a way to view those relationships.

A BGP peer session has a state, a remote AS number, an uptime, and an update count. An OSPF adjacency has a neighbor router ID, an area, a priority, and a state machine position. A MAC table entry records which physical switch port a given MAC address was learned on. None of these have ECS field definitions, and stuffing them into generic event.* or observer.* fields loses the semantic meaning that makes the data useful.

The plugin takes an opinionated approach: use ECS where it fits, extend with clear namespaces where it doesn't. Interface data maps to ECS-aligned fields. Routing protocol and L2 forwarding data goes into purpose-built namespaces (bgp_peer.*, ospf_neighbor.*, arp.*, mac_table.*) with field names that match the concepts operators already think in. If you know what bgpPeerState means on a router CLI, bgp_peer.state in Elasticsearch is immediately familiar. If you already collect SNMP data in a homegrown schema, the plugin's templates and ingest pipeline will complement it rather than replace it. The new fields are additive, so you can adopt them at your own pace!

| Data Type | Key Fields | ECS Namespace | | --- | --- | --- | | BGP Peer Session | State, Remote AS, Uptime, Update Count | bgp_peer.* | | OSPF Adjacency | Neighbor Router ID, Area, Priority, State | ospf_neighbor.* | | MAC Table Entry | Switch Port, Learned MAC Address | mac_table.* | | ARP Entry | IP-to-MAC Mapping | arp_table.* |

An ingest pipeline (snmp-device-enrichment) handles classification at index time. It parses each device's sysDescr string to assign a normalized device.type (router, switch, firewall, access point) and device.vendor, so every downstream consumer (dashboards, ES|QL queries, alerting rules, the topology view) gets consistent device metadata without custom parsing. The pipeline recognizes common vendors out of the box and is extensible for environments with less common hardware.

The result is SNMP data you can query like any other structured data in Elasticsearch. "Show me every BGP session not in Established state" is a filter, not a regex exercise. "Which Cisco switches have interfaces that are admin-up but oper-down" is a KQL query, not a script.

Visualising SNMP network topology in Kibana

Dashboards excel at answering "what are the numbers?" A topological view answers a complementary question: "what's connected to what?" Network engineers think in topology: upstream and downstream relationships, path diversity, and blast radius of a link failure. A spatial, graph-based view brings that mental model directly into Kibana, sitting alongside the charts and data tables operators already rely on.

The plugin adds an interactive topology graph to Kibana's Observability navigation. It reads the structured SNMP data from Elasticsearch, builds an adjacency graph from ARP, MAC table, BGP, and OSPF relationships, and renders it as a force-directed layout you can zoom, pan, and rearrange. Nodes are devices, edges are discovered relationships, and clicking any device opens a flyout with its interface table, ARP neighbors, and routing protocol sessions.

How do you set up SNMP network topology monitoring in Kibana?

The plugin is nearly ready to go out of the box, only a few assets need installation. Here's what the setup looks like:

1. Install the plugin zip on a self-managed Kibana instance (bin/kibana-plugin install file:///path/to/networkTopology-<version>.zip).

2. Apply the index templates and ingest pipeline. Click through the template installation in the plugin's Setup tab. A few button clicks and the schema is in place.

3. Deploy the Logstash pipeline. Add your device targets, authentication details, or other configuration to the included template and start it. If you're using Logstash Centralized Pipeline Management, push it from Kibana, no SSH required.

Data hits Elasticsearch on the next poll cycle, the ingest pipeline classifies and enriches them, and the topology view populates. Start to finish, you're looking at minutes, not hours or days of trial and error.

A sample data generator is included for teams that want to evaluate the plugin before connecting to live infrastructure; spin up a Docker development environment and explore the full feature set with a realistic multi-site network.

Before you can rely on most tools to trigger a meaningful alert, you have to do the heavy lifting of telling them exactly what to watch. You have to write the rules, specify what a "normal" baseline looks like, and manually define your service catalog. We're working to change that dynamic at Elastic, and the first major building block is now in place: a system designed to simply read your logs and figure out what's inside them on its own.

Consider what happens when an alert fires today. Your on-call engineer opens an investigation, and the first few minutes are inevitably burned reconstructing basic facts. They have to figure out which services are involved, how those services connect to one another, what error patterns are typical, and which queries they actually need to run to dig deeper. An AI agent faces this exact same cold start problem. Without prior knowledge of your system's architecture, an agent has to read through hundreds of log lines just to establish baseline context that really should already be available.

This blank slate is the default state of most observability setups. You only know what you've explicitly configured. When new services spin up and start writing logs, they sit there without rules until someone takes the time to write them. When architectural dependencies shift, your topology map quietly goes stale unless you've done an exceptional job instrumenting all your services. If an error pattern fires every day but nobody wrote a specific rule to catch it, it remains invisible.

Knowledge Indicators (KI) are our way of closing this gap. When you run extraction against a log stream, Elastic analyzes the raw data and returns structured facts about your environment. It identifies which services are running, the underlying infrastructure they rely on, how they depend on each other, and the log schemas they're using. It even generates a set of ES|QL queries for conditions that might be worth alerting on. Rather than a static configuration, this knowledge accumulates over time, automatically expires when a service disappears, and feeds directly into downstream capabilities like Rules, topology maps, AI agent investigations, and dashboards.

Topology graph generated from dependency KIs: service nodes, dependency edges, and detected error conditions.

The Extraction Pipeline

When designing this system, our primary goal was to eliminate the need for prior context. There should be no mandatory schemas, no service catalogs tied to specific properties, and no predefined static assets that would need to be maintained. We asked ourselves a simple question: if you handed a sample of raw logs to an engineer who had never seen the system before, what could they deduce just by looking?

That thought experiment became our core approach. The system samples a small batch of logs from a stream, processes them through a combination of LLM analysis and deterministic code generators, and accumulates its findings across multiple rounds, entirely configuration-free.

Imagine hiring a room full of junior SREs with one specific job: read these log lines and report their observations, not to fix anything or trigger alarms, just to notice things. "This looks like an nginx server," or "This database is PostgreSQL," or "Service A is calling Service B over HTTP." That's essentially what our extraction job is doing continuously across your streams.

To see how this works in practice, take a look at this single line from an nginx access log:

192.168.1.45 - - [31/Mar/2026:14:23:01 +0000] "POST /api/v2/claims HTTP/1.1" 200 1247 "-" "claim-intake/1.4.2"

From just this string, the pipeline extracts three distinct facts:

• Entity: claim-intake (identifiable as a service from the User-Agent)
• Version: 1.4.2 (extracted from the User-Agent string)
• Technology: nginx (the web server fielding the request)
• Schema: Combined Log Format

Similarly, consider this Java service log:

2026-03-31T14:23:03.412Z INFO fraud-check --- [nio-8080-exec-3] c.e.FraudCheckService : Calling upstream POST http://policy-lookup:8081/v1/policy latency=142ms status=200

Here, the extraction identifies:

• Entity: fraud-check (a Spring Boot service)
• Dependency: fraud-check → policy-lookup (via an outbound HTTP call)
• Technology: Java, Spring Boot

Pull twenty lines like these from across your stream, and you quickly build a working, accurate picture of your system architecture.

To ensure this process never blocks ingestion, extraction runs entirely as a background task. You can trigger it on demand from the stream detail view or the Significant Events Discovery UI, but the goal is to have it running by default without requiring attention.

The pipeline itself runs multiple iterations, each time fetching a small sample of documents. We use a mix of random and already-excluded documents to ensure we discover the full scope of the system. KIs found in one iteration are fed back as exclusions into the next, so each round focuses on what the previous one missed—ensuring quieter, less-represented services aren't crowded out by noisier ones.

Extraction pipeline: biased document sampling feeds a parallel LLM pass and four deterministic generators. Results are merged and deduplicated before storage.

Once sampled, the documents are sent to an LLM. We use a system prompt that instructs the model to identify a few specific types of features, which we plan to extend over time:

| Type | What it captures | |------|------------------| | Entity | Distinct system components: services, applications, jobs | | Infrastructure | Environment context: Kubernetes, cloud provider, OS | | Technology | Languages, frameworks, libraries, databases | | Dependency | Relationships between components | | Schema | Log format conventions: ECS, OTel, custom |

The LLM returns its findings, delivering newly identified traits alongside any intentionally ignored ones (like user-excluded false positives). To be accepted, every feature must include stable identifying properties and cite direct evidence from the sampled logs. The LLM also assigns a confidence score from 0–100 for each KI, so any downstream use of that KI knows how much to trust it.

In parallel, a set of deterministic code-based generators independently analyze the data to produce statistical summaries, log samples, pattern clusters, and error-specific features. Because these are computed rather than inferred, they always receive a confidence score of 100.

Finally, the LLM results and computed features are merged and deduplicated. Known KIs reuse their existing UUIDs, new discoveries get fresh ones, and any user-excluded features are quietly dropped server-side. Surviving KIs are saved with an active status and an expiration date set for seven days out.

Knowledge Indicators tab showing 84 KIs across streams, with type, confidence (1–5 stars), and stream columns.

What a Knowledge Indicator Contains

Knowledge Indicators fall into two categories: Feature KIs and Query KIs.

Feature KIs are descriptive. They explain the contents of the stream: what services are running, the infrastructure housing them, their dependencies, and the active tech stack.

Query KIs are actionable. They are ready-to-run ES|QL queries targeting notable conditions like connection exhaustion, out-of-memory errors, or fatal exceptions. Each comes with a severity score from 0 to 100, and when promoted to Rules, they fire Events.

Feature KIs carry a full data model:

• type / subtype: the category of the fact (Entity, Infrastructure, Technology, Dependency, Schema)
• title / description: a human-readable summary
• properties: stable key-value pairs used to deduplicate findings across multiple runs
• confidence: 0–100. LLM-identified KIs score based on evidence quality. Deterministic KIs always score 100.
• evidence: 2–5 supporting log excerpts that justify the KI's existence
• filter: an optional StreamLang condition scoping the KI to specific documents

A dependency KI looks like this:

{
  "type": "dependency",
  "subtype": "service_dependency",
  "title": "api_gateway → inference_service",
  "description": "Service-to-service HTTP dependency from api_gateway to inference_service, observed in request logs",
  "properties": {
    "source": "api_gateway",
    "target": "inference_service",
    "protocol": "http"
  },
  "confidence": 85,
  "evidence": [
    "service.name=api_gateway http.url=/v1/inference peer.service=inference_service",
    "upstream=inference_service:8080 request=POST /v1/inference 200"
  ],
  "filter": { "field": "service.name", "eq": "api_gateway" },
  "status": "active",
  "expires_at": "2026-04-09T00:00:00Z"
}

Query KIs take a simpler shape, focusing solely on the title, severity score, and the executable query:

{
  "kind": "query",
  "title": "PostgreSQL connection slot exhaustion",
  "description": "Fires when Postgres runs out of available connection slots",
  "severity_score": 90,
  "esql": {
    "query": "FROM logs-* | WHERE service.name == \"postgres\" AND message : \"remaining connection slots\""
  }
}

The properties field is what keeps Feature KIs stable across multiple pipeline runs. The dependency KI for api_gateway → inference_service records the source, target, and protocol as fixed pairs. The next time extraction runs, Elastic recognizes this existing relationship and updates the KI's last_seen timestamp rather than creating a duplicate.

KI detail panel showing type, subtype, properties, confidence, evidence, and expiry date for a service dependency.

The Foundation for Intelligent Observability

So what can we do with all of this? These KIs serve as the contextual foundation for Elastic's more advanced capabilities. From just these extracted KIs, we can automatically generate active Rules to surface interesting signals, without a human engineer writing a single line of configuration. More on this particular capability in the next post in this series.

85 auto-generated Rules from 84 KIs, with impact ratings and event occurrence sparklines.

As a user or an agent, the dependency KIs automatically construct an infrastructure graph—inferred entirely from log data, not from distributed tracing or any manual configuration. During an incident, this graph is invaluable for assessing blast radius. If a specific database goes down, the topology map immediately shows you exactly which upstream services are about to fail, without maintaining a manual service catalog.

Service dependency graph extracted from KIs, showing services, databases, and infrastructure components.

This context changes how an AI agent handles an incident. Instead of starting from scratch, the agent initiates its investigation using your system's actual topology and known failure modes. Based on the KIs, it identifies the relevant streams, runs the applicable queries, and formulates a specific hypothesis. In our example, it already knows that api_gateway relies on inference_service, and it knows that connection slot exhaustion is a high-severity failure mode for your Postgres instance.

This extracted knowledge doesn't have to be perfect to be useful. Because LLMs are inherently non-deterministic, a KI might occasionally be slightly off, but it still gives the agent a significant head start. The agent can cross-reference the KI against live logs and self-correct on the fly. The real benefit is simply not having to reconstruct basic facts during a critical outage. KIs also drive AI-generated dashboard suggestions and inform Grok pattern generation whenever you introduce new streams.

Self-Cleaning and Scalable

Maintaining this knowledge base is entirely hands-off. KIs auto-expire after 7 days if they aren't observed in subsequent extraction runs. If you decommission a service, its associated KIs simply fade away without any manual cleanup. If the service comes back online later, the KIs are re-extracted. Users can also mark individual feature KIs as false positives, and the system carries those exclusions forward into future runs to prevent re-identification.

Because we scoped KI extraction as a specific classification task, looking at around 20 log samples to identify services, infrastructure, and dependencies, it doesn't require a large frontier model to run. A fast, cost-effective model handles this without multi-step reasoning.

You Shouldn't Have to Tell Your Tools What to Watch

The fundamental promise of observability is to help you understand your systems. For far too long, the burden of teaching the tool how those systems actually work has fallen on the engineers operating them.

The next post in this series looks at what agents do with that context: why every agent that investigates your system without KIs re-learns the same things from scratch on every incident, and what changes when it doesn't have to.

NOTE: These capabilities are available behind a feature flag in Serverless Observability projects. Turn on observability:streamsEnableSignificantEvents by searching for it in the Kibana advanced settings page.

Frequently asked questions

What are Knowledge Indicators in Elasticsearch Streams? Knowledge Indicators (KIs) are structured facts extracted from raw log streams: service names, infrastructure components, service-to-service dependencies, and tech stack details. Elastic extracts them automatically by sampling log lines, without requiring schemas, service catalogs, or manual configuration.

How does Elastic build a service topology map from logs alone? The extraction pipeline samples log lines and identifies dependency relationships, such as an outbound HTTP call from one service to another. These dependency KIs are used to construct a topology graph that shows which services depend on which, entirely inferred from log data, without distributed tracing or any manual input.

Why does an AI agent need Knowledge Indicators before investigating an incident? Without KIs, an AI agent starts every investigation from scratch: it has to read hundreds of log lines just to establish which services exist and how they relate. KIs give the agent a pre-built map of your system, including services, known failure modes, and relevant queries, so it can begin reasoning about the actual incident immediately.

Do I need to configure anything for Knowledge Indicator extraction to work? No. The pipeline requires no schema definitions, no service catalog, and no predefined rules. It samples a small set of log lines from a stream, analyzes them through a combination of LLM inference and deterministic generators, and accumulates findings automatically.

How accurate are LLM-extracted KIs compared to computed ones? Computed (deterministic) KIs always receive a confidence score of 100 because they are derived from statistical analysis rather than inference. LLM-extracted KIs receive scores from 0 to 100 based on the quality of evidence found in the sampled logs. Rules, agent investigations, and topology maps can all use this score to weight their decisions.

What happens when a service is decommissioned? KIs carry a 7-day expiration. If a service stops appearing in subsequent extraction runs, its KIs expire and are removed automatically. No manual cleanup required. If the service comes back, the KIs are re-extracted on the next run.

How does this compare to service discovery via distributed tracing? Distributed tracing requires instrumented services and a trace collector. Knowledge Indicator extraction requires nothing beyond existing log streams: no SDK, no agent, no schema. For environments with partial or no tracing coverage, KI extraction provides topology and dependency information that tracing would otherwise miss.

When processing fails, data lands in the failure store

When a Streams pipeline fails (a Grok pattern doesn't match, a field type conflicts with the mapping), the documents that caused the failure are written to the failure store. The failure store is a set of backing indices attached to your data stream. It scales the same way as any other data stream, so it can absorb everything that fails. It's enabled by default for logs as of Elasticsearch 9.2.

The Data quality tab gives you insights into the quality of your stream and into documents in the failure store. When failures are accumulating, you'll see a rising count of failed documents along with the error type and a sample of the messages that triggered it.

The Data quality tab showing a rising failure count, error type, and a sample of the documents that triggered it.

A Grok expression mismatch (illegal_argument_exception) is sending documents to the failure store. The raw log line doesn't match the expected pattern. The documents aren't dropped. They're in the failure store, ready to debug against.

Processing: Switch the sample source to the failure store

Start by navigating to the Processing tab.

By default, the editor samples from recent live documents. Switch the sample source to Failure store instead: it loads the exact documents that failed, the unmodified originals before any Streams processing ran. You're iterating against the actual failures.

Change the sample source dropdown from the default to Failure store.

The sample source dropdown with the Failure store option selected.

The editor loads up to 100 documents from the failure store and runs them through the current pipeline. You can see exactly where parsing breaks down.

The pipeline editor loaded with documents from the failure store instead of recent live samples.

Fix the processor against the actual failures

With the failure store documents loaded as samples, iterate on the processor. The editor shows you the result against the actual failed documents in real time.

In this example, the pipeline was originally built to parse HTTP access logs:

DELETE /api/v1/auth/logout from 26.72.241.177 - Status: 200 - Response time: 38ms - Request ID: req_24363339 - Location: São Paulo, BR - Device: desktop
HEAD /api/v1/notifications from 20.94.145.254 - Status: 202 - Response time: 60ms - Request ID: req_74513322 - Location: Tokyo, JP - Device: mobile

The original Grok pattern matched those:

%{WORD:http.method} %{URIPATH:uri.path}

A second log type started flowing in. Cache hits and external API calls arrived in a different format:

cache_hit: Cache hit for key: config
external_api_call: External API call completed - latency: 1695ms - Duration: 598ms

The original pattern doesn't match these at all. Every one goes straight to the failure store. With the failure store loaded as the sample source, the problem is immediately obvious: the editor shows the parse failing on lines that start with a word followed by a colon, not an HTTP method followed by a path.

The fix is a second pattern to handle the new format:

%{WORD:event.type}: %{GREEDYDATA:message}

Add it to the processor, and the editor immediately shows both log types parsing correctly against the failure store samples.

When the sample view shows all fields extracting correctly and the parse rate hits 100%, the fix is ready.

Both log types parsing correctly after adding the second Grok pattern. Parse rate at 100%.

No guessing — the editor confirms the fix before you save.

Watch the failure count drop

Save the updated pipeline. New documents are now processed with the corrected pipeline. Switch back to the Data quality tab and watch the failure count.

The failure count dropping as new documents are processed by the corrected pipeline.

The count drops as the fixed pipeline handles new incoming data correctly. The remaining documents in the failure store are the pre-fix failures. They'll clear out as retention ages them off.

The fix applies to new documents only. Documents already in the failure store aren't automatically reprocessed; each was processed by the pipeline version active when it arrived. If you need them in your main stream, that's a separate step.

The recovery loop

Open Data quality, switch to the failure store, fix the processor, save. The whole thing takes a few minutes at most.

No re-ingestion from source. No shipper-level dead letter queue to operate. If you haven't checked the Data quality tab for your streams recently, it's worth a look. There might be failures sitting there that a one-line fix would clear.

For a deeper look at what the Data quality tab shows and how to configure the failure store, see Elastic Observability: Streams Data Quality and Failure Store Insights.

• Elasticsearch 9.3+ (or Elastic Cloud Serverless)

• Elasticsearch API KEY and Kibana URL

• An application instrumented with Elastic APM: the RUM agent for frontend interactions (populates traces-apm-*) and the APM agent for backend errors (populates logs-apm.error-*

• Cursor (version 2.6+) installed

The problem with two worlds

Application logs and code are two separate worlds that don't talk to each other. If you want to apply log insights into the application you have to analyze the logs, and then come back to the editor and apply your findings.

The Model Context Protocol (MCP) changes this. MCP is an open standard that lets AI clients like Cursor connect to external tools and data sources through a standardized interface. Instead of your IDE only knowing about your local code, it can also talk to your Elasticsearch cluster, query your APM data, and reason about production behavior alongside your source files.

Elastic ships a built-in MCP server as part of Agent Builder. You define tools in Kibana, expose them via the MCP endpoint, and any MCP-compatible client can call them. Cursor supports MCP natively, which means you can set this up in minutes.

What we're building

We're working with an eCommerce search app instrumented with Elastic APM. The RUM JS agent tracks filter click interactions from the browser, stored in traces-apm-default. The Node.js APM agent captures backend errors, stored in logs-apm.error-default.

Two situations come up during development:

• Use case 1: The product team wants to simplify the search page. There are six filters but we don't know which ones users actually click. We need usage data to decide which to keep.

• Use case 2: Users report intermittent 500 errors on search. The errors are not constant and started two days ago. We need the error details to find the root cause.

To bring that data into Cursor, we'll build two Agent Builder tools in Kibana and connect them via the Elastic Agent Builder MCP Server:

• get_filter_usage: queries traces-apm-default for filter click events and returns a usage breakdown by filter name

• get_recent_errors: queries logs-apm.error-default for the most recent error groups for a given service, including the exception message and stack trace culprit

For a deeper look at the overall architecture, see the Agent Builder reference guide.  

Setting up the Elastic MCP Server  

Step 1: Create the Agent Builder tools

We create both tools via the Kibana Agent Builder API. Each tool is an ES|QL query with a name and description that Cursor uses to decide when to call it. The full implementation of the tools is in the following notebook.

Tool 1: get_filter_usage

The product team needs to know which filters users actually click before deciding which ones to remove. The query reads RUM interaction events from traces-apm-default and groups them by filter name:

    {
      "id": "get_filter_usage",
      "type": "esql",
      "description": "Returns the usage count for each search filter in the ecommerce-search-ui service, sorted by most used first.",
      "configuration": {
        "query": "FROM traces-apm-default | WHERE service.name == \"ecommerce-search-ui\" | WHERE transaction.type == \"user-interaction\" | WHERE labels.filter_name IS NOT NULL | STATS count = COUNT(*) BY labels.filter_name | SORT count DESC"
      }
    }

Tool 2: get_recent_errors

For the error debugging use case, we need to surface the most frequent recent errors for a service, along with where in the code they originate. STATS ... BY groups errors by their fingerprint (grouping_key), surfaces the exception message and the line of code that caused it (culprit), and ranks by frequency:

    {
      "id": "get_recent_errors",
      "type": "esql",
      "description": "Returns the most frequent error groups for ecommerce-search-ui, ranked by occurrence count, with the exception message and code location.",
      "configuration": {
        "query": "FROM logs-apm.error-default | WHERE service.name == \"ecommerce-search-ui\" | WHERE processor.name == \"error\" | STATS count = COUNT(*) BY error.grouping_key, error.exception.0.message, error.culprit | SORT count DESC | LIMIT 5"
      }
    }

Both tools are created with POST /api/agent_builder/tools. You can learn more about the Kibana API endpoints for Elastic Agent Builder here.

Step 2: Connect to Cursor

Open ~/.cursor/mcp.json and add the Elastic server. For detailed information, see the Cursor documentation. The Agent Builder MCP endpoint uses Server-Sent Events (SSE) transport, so we connect via mcp-remote, a lightweight bridge that Cursor invokes as a local process:

    {
      "mcpServers": {
        "elastic-agent-builder": {
          "command": "npx",
          "args": [
            "-y",
            "mcp-remote",
            "https://YOUR_KIBANA_URL/api/agent_builder/mcp",
            "--header",
            "Authorization: ApiKey YOUR_API_KEY"
          ]
        }
      }
    }

Replace YOUR_KIBANA_URL and YOUR_API_KEY with your values.

Restart Cursor, open the Agent panel, and confirm that get_filter_usage and get_recent_errors appear in the available tools list. 

Use case 1: Data-driven UI optimization

The eCommerce search page has six filters: category, manufacturer, price range, customer gender, day of week, and region. The product team wants to simplify the UI by removing filters that users don't use as much. Rather than guessing, we ask Cursor to check.

When you type a prompt in Cursor's Agent panel, the model sees the name and description of every connected MCP tool. It matches your intent to the best-fitting tool and calls it automatically. This is why the description field we set in Step 1 matters: it's what the model reads to decide which tool answers your question. If you are interested in learning more about Cursor’s MCP tools management, read the following documentation.

Open a Cursor chat and ask: "Show me how often each search filter is used." Cursor calls the tool and returns something like:

The category and manufacturer filters get most of the clicks. The bottom three filters (customer_gender, day_of_week, region) are rarely used.

Ask Cursor to act on this: "Based on this data, simplify the SearchFilters component. Keep the top 3 filters visible, collapse the others under a 'More filters' toggle."

Cursor opens src/components/SearchFilters.jsx, reads the current implementation, and proposes the change.

Before: 

After: 

The entire loop took one chat prompt. The decision was backed by production data, not a team discussion about what users probably care about.

Use case 2: Production error debugging

A bug report comes in: intermittent 500 errors on the search endpoint. The errors started appearing two days ago but they're not constant. The developer opens Cursor and asks: "Show me what errors ecommerce-search-ui is throwing."

Cursor calls the tool and returns the error groups:

The error message is explicit: category is a text field and can't be used in terms of aggregation. The correct field is category.keyword. 

With APM data available alongside your code, the debugging session becomes a conversation: you describe the symptom, the agent pulls the relevant logs, and you work through what's happening together. You can ask follow-up questions, check whether the error correlates with a recent deployment, or ask which endpoints are most affected, all within the same context where you'll make the fix. If you want to go further, Elastic also provides pre-built observability tools in Agent Builder that you can use alongside custom tools like the ones we created here. For a complementary approach to AI-driven observability, see how to monitor web AI agents with OpenLIT and Elastic.

Conclusion

What we covered:

• How to create Agent Builder tools in Kibana that wrap APM data queries

• How to connect the Elastic Agent Builder MCP Server to Cursor in three lines of JSON

• Using production telemetry to make a UI decision backed by real usage data

• Debugging a production error from the same window where you fix it

These two use cases are a starting point. The same pattern works for any data you have in Elasticsearch: performance metrics, A/B test results, audit logs, feature flag usage, user session data. Define the Agent Builder tool, connect it via MCP, and it becomes part of your development context in Cursor. For other examples of what's possible, see automating synthetic monitoring with MCP and agentic CI/CD deployment gates.

Next steps

• Elastic Agent Builder MCP server documentation

• Model Context Protocol specification

• Elastic Agent Builder overview

In benchmarks, LogsDB brought a dataset from 162.7 GB down to 39.4 GB — a 76% reduction in storage. You can explore the full nightly benchmark results at elasticsearch-benchmarks.elastic.co.

In this tutorial you'll reproduce the experiment yourself using Kibana Dev Tools and an Apache logs dataset. You'll create two identical indices, ingest the same documents into both, and measure the storage difference with the _stats API. By the end, you'll see a 44% reduction on your test data — and understand exactly why production numbers push even higher.

Already on Elasticsearch 9.2+? Any data stream with a logs- prefix already uses LogsDB by default. Jump to What about your existing logs? to verify your setup.

Want the full picture? For the engineering history behind these savings — how Lucene doc values, synthetic _source, index sorting, and ZSTD were developed and stacked over twelve years — see Elasticsearch over the years: how LogsDB cuts index size by up to 75%.

Prerequisites

• Elasticsearch 8.17+ cluster, Elastic Cloud deployment, or Serverless
• Kibana with Dev Tools access
• Some logs
• Basic familiarity with running API calls in Kibana Dev Tools

How LogsDB saves storage

LogsDB stacks three mechanisms to achieve its storage reduction:

• Index sorting — documents are sorted by host.name then @timestamp, grouping similar log lines so compression codecs find far more repeated patterns. Sorting alone accounts for roughly 30% of the savings.
• ZSTD compression with delta/GCD/run-length encoding — best_compression switches from LZ4 to Zstandard and applies numeric codecs to each doc values column. The standard index in this tutorial uses LZ4, so part of what you're measuring is the full package LogsDB delivers automatically.
• Synthetic _source — Elasticsearch skips storing the raw JSON blob entirely and reconstructs _source on demand from doc values, adding another 20–40% of savings on top.

Synthetic _source trade-offs: Field ordering in returned documents may differ from the original, and some edge cases around multi-value array fields behave differently. For most log analytics workloads these differences are invisible, but check the synthetic _source documentation before enabling it in latency-sensitive applications.

For a deep dive into the architecture behind each mechanism, see Elasticsearch over the years: how LogsDB cuts index size by up to 75%.

Let's now walk through the steps you can take to enable LogsDB and measure the storage savings.

Step 1: Collect logs with Elastic Agent

The recommended way to ingest Apache logs into Elasticsearch is through Elastic Agent with the Apache integration. It handles collection, parsing, ECS field mapping, and routing automatically.

Browse all available integrations in the Elastic integrations catalog.

Once the Agent is collecting logs and routing them to logs-apache.access-*, move to the next step.

Step 2: Create the two indices

All commands in this tutorial are run in Kibana Dev Tools.

Create one standard index and one LogsDB index with identical field mappings. The only difference is "index.mode": "logsdb".

Standard index:

PUT /apache-standard
{
  "mappings": {
    "properties": {
      "@timestamp":                  { "type": "date" },
      "host.name":                   { "type": "keyword" },
      "http.request.method":         { "type": "keyword" },
      "url.path":                    { "type": "keyword" },
      "http.version":                { "type": "keyword" },
      "http.response.status_code":   { "type": "integer" },
      "http.response.bytes":         { "type": "integer" },
      "http.request.referrer":       { "type": "keyword" },
      "user_agent.original":         { "type": "keyword" }
    }
  }
}

LogsDB index:

PUT /apache-logsdb
{
  "settings": {
    "index.mode": "logsdb"
  },
  "mappings": {
    "properties": {
      "@timestamp":                  { "type": "date" },
      "host.name":                   { "type": "keyword" },
      "url.path":                    { "type": "keyword" },
      "http.request.method":         { "type": "keyword" },
      "http.version":                { "type": "keyword" },
      "http.response.status_code":   { "type": "integer" },
      "http.response.bytes":         { "type": "integer" },
      "http.request.referrer":       { "type": "keyword" },
      "user_agent.original":         { "type": "keyword" }
    }
  }
}

That single "index.mode": "logsdb" line activates all three storage mechanisms. Elasticsearch enables these additional settings behind the scenes — you don't set any of them manually:

{
  "index.sort.field":              ["host.name", "@timestamp"],
  "index.sort.order":              ["asc", "desc"],
  "index.codec":                   "best_compression",
  "index.mapping.ignore_malformed": true,
  "index.mapping.ignore_above":    8191
}

Step 3: Reindex the logs

Use the _reindex API to copy the same documents into both test indices:

POST /_reindex
{
  "source": { "index": "logs-apache.access-*" },
  "dest":   { "index": "apache-standard" }
}

POST /_reindex
{
  "source": { "index": "logs-apache.access-*" },
  "dest":   { "index": "apache-logsdb" }
}

Both indices now hold identical documents, so the storage comparison in the next step reflects only the index mode difference.

Step 4: Force merge for a fair comparison

Before measuring, force merge both indices to a single segment:

POST /apache-standard/_forcemerge?max_num_segments=1

POST /apache-logsdb/_forcemerge?max_num_segments=1

These calls block until the merge finishes. Wait for both responses before continuing.

Why this matters: Elasticsearch writes data into multiple Lucene segments before merging them in the background. Measuring mid-merge gives artificially inflated numbers because each segment is compressed independently. Forcing a single segment shows the real steady-state storage footprint you'd see in a mature production index.

Only run _forcemerge on indices that are no longer being written to. Force merging an index that is still receiving writes is resource-intensive and can impact ingestion performance. In production, you can use Index Lifecycle Management (ILM) to automate force merges as part of the warm or cold phase, once an index is rolled over and no longer actively ingested into.

Step 5: Measure the difference

GET /apache-standard/_stats?filter_path=indices.*.primaries.store

GET /apache-logsdb/_stats?filter_path=indices.*.primaries.store

The filter_path parameter keeps the response focused. Look for primaries.store.size_in_bytes in each response.

In our test with Apache log records, the results were:

| Index | Documents | Size | |------------------|-----------|----------| | apache-standard | 111,818 | 15.37 MB | | apache-logsdb | 111,818 | 8.6 MB | | Reduction | | 44% |

To put this in perspective: at 1 TB of log data, LogsDB brings that down to around 560 GB. That's 450 GB saved without any changes to your queries. At production scale with billions of documents and synthetic _source enabled, savings push to 76% — taking 162.7 GB down to 39.4 GB in our benchmark.

Visualize in Kibana

To see the storage difference visually, open Kibana and go to Management → Stack Management → Index Management. You'll see both indices listed with their current sizes side by side.

Why Kibana shows larger numbers than _stats: Kibana Index Management displays the total index size including all replica shards. The _stats query above uses primaries to report primary shards only. The ratio between the two indices remains the same either way.

What about your existing logs?

Elasticsearch 9.2+ (already enabled by default)

Since 9.2, any data stream matching the logs-* naming pattern automatically uses LogsDB. You're likely already saving storage without any configuration change.

Verify your existing data streams:

GET /.ds-logs-*/_settings?filter_path=*.settings.index.mode

If you see "index.mode": "logsdb" in the responses, you're already getting the savings.

Elasticsearch 8.x or 9.0–9.1 (enable per data stream via index template)

For earlier versions, enable LogsDB on a data stream by updating its index template. This affects all new indices created from that template — existing indices are not changed, so the transition is safe and gradual.

Option A — Update an existing template:

PUT _index_template/logs-myapp-template
{
  "index_patterns": ["logs-myapp-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "index.mode": "logsdb"
    }
  },
  "priority": 200
}

Option B — Check and patch an existing integration template:

First, find the template managing your data stream:

GET _index_template/logs-apache*

Then add the index.mode setting to the template.settings block using a PUT _index_template/<name> call with the full template body including your addition.

After updating the template, the next index rollover will use LogsDB. Trigger a rollover immediately if you don't want to wait:

POST /logs-myapp-default/_rollover

Upgrading from 8.x to 9.0+: Existing data streams are not changed automatically. Only new rollovers will use LogsDB. There is no data loss and no reindexing required — the savings accumulate as new indices roll over.

What about query performance?

LogsDB does not significantly impact query performance for typical log analytics workloads. The index sorting by host.name and @timestamp can actually improve range query and aggregation performance on those fields, since matching documents are stored adjacently. Queries that don't filter on those fields perform comparably to a standard index.

For indexing throughput data across releases, see the performance section of the companion article.

Conclusion

LogsDB activates with a single "index.mode": "logsdb" setting and delivers measurable storage savings immediately: 44% in our hands-on test, and 76% (162.7 GB → 39.4 GB) in production benchmarks with synthetic _source. On Elasticsearch 9.2+, logs-* data streams already use LogsDB by default. For 8.x or earlier 9.x clusters, a one-line index template change enables it on your next rollover with no data loss and no reindexing required.

Next steps

• LogsDB index mode documentation
• Configuring a logs data stream
• LogsDB GA announcement
• LogsDB and TSDS performance benchmarks

The write overhead

Lucene, the library underneath, keeps multiple structures for every indexed document:

• The inverted index maps terms to documents. This is what makes text search fast.
• _source stores the original JSON blob, returned when you fetch a document.
• Doc values store field values in columns for sorting and aggregation.
• Points / BKD trees index numeric and date fields for range queries.

The inverted index earns its keep: it's what lets you search a billion log lines by keyword in milliseconds, and there's no cheaper way to build that capability. _source exists to give you back exactly what you indexed: search results and GET requests return this blob directly. The problem is that it stores the full event even though the same field values are already available through doc values and the other structures.

Take a log event with fields like host.name, @timestamp, http.response.status_code, and duration_ms. The entire event is serialized as JSON in _source. The same field values are also written into doc values columns, indexed into the inverted index, and stored in BKD trees for range queries. Same data, multiple structures, each with its own on-disk footprint.

For a search engine where you need fast retrieval across all dimensions, that overhead is a reasonable tradeoff. For logs, where you rarely need the raw JSON and almost never do relevance-ranked search, much of it is pure waste.

One write, four on-disk structures: _source (the raw JSON blob), the inverted index, doc values columns, and BKD / points trees for numeric range queries. The same field values end up in multiple places.

Why columnar storage matters for compression

Doc values are the key to everything LogsDB does. Unlike _source, which stores entire documents as blobs, doc values store each field as a separate column across all documents in a Lucene segment.

Picture a segment with a million log events. The _source representation is a million JSON blobs, one per event, each containing all fields jumbled together. The doc values representation is a set of columns: one column of a million timestamps, one column of a million host names, one column of a million status codes, and so on.

Row-oriented _source keeps all fields for each document in one blob — doc0 through doc5 each carry host.name, @timestamp, status, duration_ms, and more jumbled together. Column-oriented doc values restructure the same data so all host.name values sit in one column, all timestamps in another, all status codes in another. Compression codecs can then run on each contiguous column independently.

That columnar layout is what makes per-column compression possible. When all values of http.response.status_code sit in a contiguous column, Lucene can apply codecs that exploit patterns in the sequence.

Delta encoding stores differences between adjacent values instead of full values. GCD encoding finds a common factor and divides everything down. Run-length encoding collapses repeats. Lucene picks the codec per segment and re-evaluates when segments merge.

Four sorted @timestamps from the same host, compressed in four stages. RAW: four 32-bit integers, 128 bits total. DELTA: store differences instead of full values — base stays, deltas +100, +200, +300 take 59 bits. GCD: divide out the common factor of 100, leaving 1, 2, 3 at 39 bits. BIT-PACK: pack those three small integers into contiguous bit storage, 9 bits freed.

But here's the catch: these codecs only work well when adjacent documents have correlated values. Consider the @timestamp column.

If logs arrive from dozens of hosts interleaved randomly, the timestamps in the column jump around. The delta between adjacent values might be +3 seconds, then -47 seconds, then +120 seconds. Delta encoding can't do much with that.

Now consider what happens if you sort by host.name and @timestamp before writing to the segment. All logs from host-A land in a contiguous run, followed by all logs from host-B, and so on. Within each host's run, the timestamps are monotonically increasing and the deltas are predictable.

Four timestamps from the same host might look like 1706745600, +100s, +200s, +300s. Delta encoding shrinks those to a base value plus three small integers.

GCD encoding finds that 100, 200, 300 are all divisible by 100 and stores 1, 2, 3 instead. Bit-packing then fits those three values into a handful of bits. The same pattern applies to fields like host.name, service.name, or http.response.status_code: within a sorted run, long stretches of identical values collapse to near nothing under run-length encoding.

Five hosts — api-01, api-02, db-01, web-01, web-02 — scattered randomly in arrival order (left). Sorting by host.name groups them into five contiguous blocks of eight (center). Run-length encoding collapses each block to a single (value, count) pair — 5 pairs stored instead of 40, the remaining slots freed (right).

Elasticsearch never sorted by default. Documents landed in arrival order, compressed with DEFLATE. We left a lot on the table.

How we got here: 2012–2026

Not all of the individual techniques in LogsDB were designed for logs. They were built over twelve years to solve different problems, and LogsDB is what happens when you stack them.

The foundation (2012–2017). Lucene 4.0 introduced doc values in 2012. By Elasticsearch 5.0 in 2016, they were on by default for all keyword and numeric fields. Lucene 7.0 added sparse doc values, so fields that only appear in some documents don't waste space on every document in the segment. That fixed a significant force-merge bloat problem (up to 10× on sparse fields) and set up the storage model everything else depends on.

Dense encoding reserves an 8-byte slot per document regardless of presence. Sparse encoding stores only documents that have a value at 12 bytes each (value + doc ID). For error_code with 2 of 16 docs populated (12% fill), sparse is 81% smaller: 24 B vs 128 B. For request_path at 88% fill, sparse is larger: 168 B vs 128 B. Lucene picks per field; sparse wins below ~67% fill.

Incremental wins (2020–2021). Two smaller changes targeted observability workloads. Dictionary-based stored fields compression deduplicated repetitive string metadata for about a 10% win.

The match_only_text field type dropped term frequencies and positions from the inverted index. Term frequencies are what BM25 uses to score documents by relevance — how often a term appears in a document relative to the rest of the corpus. For log search that signal is meaningless: you don't care whether "timeout" appeared twice or seven times in a log line, you just want to find it. Positions are similar: they're stored so Elasticsearch can do exact phrase matching, but the position data is expensive and phrase queries on logs are rare enough that the tradeoff is worth it. When you do run a phrase query on a match_only_text field, it still works — it just falls back to a slower path that rescores candidates rather than using stored positions directly.

text stores each term with its frequency and every position it appears at. match_only_text keeps only the doc IDs — enough to find the document, nothing more. The timeout term appears twice in this message (positions 1 and 4), which is exactly the kind of data that gets dropped.

Dropping frequencies and positions cuts the inverted index for a text field by roughly 40%. The overall index impact in 2021 was only ~10%, which sounds like a poor return on a 40% field-level reduction. The reason is where storage was going at the time: _source was stored in full for every document as a raw JSON blob, doc values were uncompressed and unsorted, and nothing was using ZSTD. The message field's inverted index was a small slice of a much larger, poorly-compressed whole. As the next five years of work addressed those other structures, the same 40% field-level savings became a meaningful fraction of a much smaller total.

Neither change was decisive on its own, but they established that log-specific storage optimization was worth pursuing.

The TSDB turning point (April 2023). This is where the story really starts. We shipped synthetic _source and index sorting for time series metrics in Elasticsearch 8.7.

Synthetic source changes the write-and-read contract. At write time, we skip storing the raw JSON blob entirely. At read time, when a query needs to return the original document, we reconstruct it by reading each field's value out of doc values and stored fields and assembling them back into JSON. The result is functionally equivalent to the original _source (with minor differences like field ordering), but we never stored the blob.

Index sorting groups documents by dimension fields and timestamp before writing to disk. Together, synthetic source and index sorting cut metrics storage by up to 70%.

That result told us something important: the same architecture could work for logs.

Without LogsDB, Elasticsearch writes every log event twice: once as a raw _source blob on disk, once into doc values columns. LogsDB skips the blob entirely. At read time, a GET <index>/_doc/1 request gathers field values from doc values and assembles the document on the fly.

The TSDB codec (2024). In 8.13 and 8.14, we built a custom doc values codec with run-length encoding optimized for sorted consecutive values, PFOR-delta encoding, and cyclic ordinal encoding for multi-valued dimensions. The numbers were striking: kubernetes.pod.name doc values dropped from 110 MB to 7.25 MB in one benchmark. We extended coverage to all numeric and keyword types including ip, scaled_float, and unsigned_long.

LogsDB Tech Preview (August 2024). In 8.15, we combined everything into index.mode: logsdb: host-first sorting, synthetic _source, ZSTD compression, and the TSDB numeric codecs. One decision mattered more than expected: sort order. Sorting by host.name first, then @timestamp, delivers up to ~40% storage reduction. Sorting by timestamp first gives ≤10%. The host-first ordering co-locates documents that share field values, which is exactly what the numeric codecs need.

ZSTD and GA (November–December 2024). In 8.16, we switched best_compression from DEFLATE to ZSTD permanently (level 3, blocks up to 2,048 documents or 240 kB, native bindings via Panama FFI on JDK 21+). ZSTD gave us ~12% smaller stored fields and ~14% higher indexing throughput at the same time, which almost never happens. LogsDB went GA in 8.17.

At GA, we claimed up to 65% storage reduction.

Routing and recovery (April 2025). In 8.18, route_on_sort_fields started routing documents to shards by sort field values instead of _id. Without this optimization, Elasticsearch hashes the _id to pick a shard, so logs from the same host scatter across all shards. With routing on sort fields, logs with similar host.name values land on the same shard. This co-locates similar documents at the shard level, not just within segments, adding ~20% storage reduction at a 1–4% ingest penalty. Routing on sort fields requires auto-generated _id.

Data stream .ds-logs-nginx-default-00001 with six hosts across three shards. STANDARD (hashed by _id): all host colors scattered randomly. ROUTED (route_on_sort_fields): same-host logs land on the same shard, but remain in arrival order within it. ROUTED + SORTED (host-first sort): each shard contains contiguous blocks of a single host — the combination that lets numeric codecs and RLE reach their full potential.

We also switched peer recovery to synthetic source reconstruction, eliminating the duplicate _recovery_source blob. In 9.0, logs-*-* indices default to LogsDB.

Nightly synthetic source benchmark, December 2024. Index size written drops 39% — from ~279 GB to ~171 GB — the day peer recovery switches from copying the raw _recovery_source blob to reconstructing documents from doc values.

Merge and recovery overhaul: 9.1 (July 2025). We fully eliminated the recovery source. Peer recovery uses batched synthetic reconstruction, cutting write I/O by ~50% and boosting median indexing throughput ~19% over the 8.17 baseline. We replaced up to four separate doc values merge passes with a single pass, cutting background merge CPU by up to 40%. And we swapped _seq_no's BKD tree for Lucene doc value skippers, halving _seq_no storage.

pattern_text and Failure Store: 9.2–9.3 (October 2025–February 2026). In 9.2, we shipped pattern_text as a Tech Preview: a new field type that decomposes log messages into static templates and dynamic variable parts. A log line like Session opened for user alice from 10.0.1.42 via TLS gets split into the template Session opened for user {} from {} via TLS (stored once, as a template ID) and the variables alice, 10.0.1.42 (stored per document). For logs with high template repetition, this cuts message field storage by up to 50%. A companion template_id sub-field lets you sort by template, and the LogsDB setting index.logsdb.default_sort_on_message_template enables this automatically. pattern_text went GA in 9.3.

TEXT stores each log message as a full string per document — eight copies of near-identical blobs. PATTERN_TEXT decomposes them: the shared template Session opened for user {} from {} via TLS is stored once with ID T0, and only the variable columns (user, ip) are stored per document — alice/10.0.1.42, bob/10.0.1.87, carol/10.0.2.11, and so on.

pattern_text does come with an indexing CPU cost: decomposing each message into template and variables takes more work at write time than storing a raw string. Whether that tradeoff makes sense depends on your dataset and your priorities.

If your log messages follow highly repetitive patterns (structured application logs, Kubernetes events, access logs), the storage wins are large and the CPU overhead is bounded. If your messages are free-form or low-repetition, the compression gains shrink while the CPU cost stays roughly the same.

For data you keep for months or years, the cumulative storage reduction usually makes it worthwhile. For high-cardinality, rapidly changing messages where storage isn't the constraint, it may not be.

9.3 also brought compression for binary doc values, making wildcard field types significantly more storage-efficient. Internally, wildcard fields store an inverted index of trigrams in a binary doc values column; that column is now compressed with Zstandard instead of being stored raw. In one benchmark, a URL field dropped from 2.92 GB to 1.12 GB, more than 60% compression. If you use wildcard fields heavily, the gain is automatic with no mapping changes needed.

Also in 9.3, skip lists for @timestamp and host.name became available as an opt-in for LogsDB. Skip lists let Elasticsearch jump ahead in a doc values column without reading every entry, which speeds up time-range queries on large segments. Other index modes have skip lists disabled by default; in LogsDB you can enable them selectively for the fields you range-query most.

Also in 9.3, the Failure Store became enabled by default for logs-*-* data streams. Failed documents (mapping conflicts, ingest pipeline errors) now land in dedicated ::failures indices instead of being rejected, which means LogsDB's strict synthetic source requirements are less likely to cause silent data loss during migration.

Performance, not just storage

LogsDB started as a storage optimization, and the early releases came with a throughput cost — sorting, synthetic source reconstruction, and ZSTD all add work at write time. Over two years of releases, we clawed that back. Indexing throughput is now on par with what users had before enabling LogsDB. You get the storage reduction without giving up the ingest rate you were used to.

Throughput (teal) has climbed from ~25k to ~35k docs/s since the Tech Preview. Storage on disk (blue) has dropped from ~65 GB to ~36 GB on the same benchmark dataset. Both curves move in the right direction, driven by the same layered releases: ZSTD in 8.16, routing optimization in 8.18, the merge and recovery overhaul in 9.1. Live numbers at elasticsearch-benchmarks.elastic.co.

The two trends compound each other. Less storage means fewer segments to merge, which frees CPU for indexing. Synthetic source reconstruction is cheaper to compute than it is to store and replicate the raw blob. Each release that shrank the index also reduced background I/O, which fed back into throughput.

The practical result: if you were running standard Elasticsearch for log ingestion two years ago, the throughput you had then is roughly what LogsDB delivers now — with a 50–75% smaller index alongside it.

How to enable it

As of 9.0, logs-*-* data streams default to LogsDB automatically. If your data streams match that pattern, you're already using it.

Want a hands-on walkthrough? Cut Elasticsearch log storage costs by 76% with LogsDB walks through creating two indices, reindexing, and measuring the difference with the _stats API — including version-specific enable instructions for 8.x clusters.

For other index patterns, set it in your template:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "index.mode": "logsdb"
    }
  }
}

Synthetic _source turns on automatically with index.mode: logsdb.

For the routing optimization (8.18+), add one more setting:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "index.mode": "logsdb",
      "index.logsdb.route_on_sort_fields": true
    }
  }
}

This routes shards by sort field values instead of _id, adding ~20% storage reduction at a 1–4% ingestion penalty. It requires at least two sort fields beyond @timestamp and auto-generated _id.

Switching an existing index to LogsDB requires a reindex. So does rolling back. There's no in-place conversion, so try it on new data streams first.

Storage improves further as segments merge — freshly written data compresses well, but merged segments compress even better.

What's next

Elasticsearch still carries some structural overhead from its search engine roots. _id and _seq_no are two examples: both consume meaningful disk space (on small documents they can account for more than half the index size), but neither is essential for log analytics workloads.

We've already taken the first step for TSDB: PR #144026 eliminated stored _id bytes from TSDB indices by reconstructing the field on the fly from doc values, the same approach synthetic _source uses. We're exploring the same direction for LogsDB.

9.4 and beyond. The architecture still has room to improve, and we're on it.

For the full reference, see the logs data stream documentation.

In addition to its queuing capabilities, Kafka can serve as a central processing middleware for data pre-processing and enrichment. This is particularly useful when such operations are impractical to perform directly downstream due to specific business or technical requirements or constraints.

For instance, integrating Kafka with stream processing engines like KsqlDB or Materialize, allows for advanced stream processing tasks, including SQL-based joins across topics and streams to enrich data at scale in real-time. The enriched datasets can then be ingested into Elasticsearch for further processing at subsequent stages.

Despite these benefits, adopting Kafka or similar queuing systems is arguably conditional. These systems introduce additional costs and complexity to the overall platform implementation and maintenance. They may also add processing overhead, delay data flow to the downstream, and risk becoming bottlenecks if not correctly sized or optimized to align with other pipeline components.

This article provides guidance for troubleshooting ingestion bottlenecks in data pipelines built with Kafka and Elastic. Identifying and fixing such issues can be sometimes challenging, particularly when multiple changes are made across multiple systems aspects at the same time, which often increases the number of variables in play. This commonly results in a longer process and inconsistent results.

Consider the below Security Operations Center (SOC) platform, where data is ingested from various sources via Elastic Agent. The data is queued and pre-processed in a Kafka cluster before being pulled by Logstash and forwarded to Elastic Security. In this environment, delays at any stage of the pipeline can result in critical security events going undetected by Elastic Security, emphasizing the importance of a well-optimized data pipeline.

Implement lag and throughput monitoring

Ingestion bottlenecks usually materialize as limited throughput and event lags, which often correlate. Monitoring these two indicators is important to measure the impact of tuning attempts. 

Tip: With the anomaly detection features of machine learning you can use the Logs Anomalies page to detect and inspect log anomalies and the log partitions where the log anomalies occur.

End-to-end lag monitoring can be broken down into the various stages of the pipeline. The incremental improvements across those stages would collectively contribute to a significant reduction in the end-to-end lag:

A) Ingest lag between the source and Kafka: This lag is the time difference between the real event-time, which is typically extracted from the event itself or added by the event producer (Elastic Agent for example), and the Kafka record timestamp, which can be added to the Logstash events via event decoration in the Kafka input plugin. 

In most cases, this lag is influenced by the write performance of the Kafka cluster and network latency between the event source and Kafka. In some cases, the lag may also appear due to time configuration mismatches that make it look like there's a lag when there really isn't.

B) Ingest lag between Kafka and Logstash: This lag is the time difference between the Kafka record's timestamp and the execution timestamp of the first filter in the Logstash pipeline. If your pipelines are using a persistent queue, note that this duration also includes the time spent in the PQ.

The below Ruby filter adds the current-time to the event in the `logstash.start` field to use for comparison later.

ruby {
 code => "event.set(logstash.start, Time.now());"
}

The primary factors contributing to ingestion lag include the consumption performance of the Kafka cluster, the Logstash input performance, data skew across the different topic partitions, and most importantly, the backpressure propagation to the Logstash input plugin, because Logstash does not fetch new events from the Kafka topic as quickly as they become available, when it is busy processing the events that it has already fetched. 

Network latency and reduced size of TCP read buffer (SO_RCVBUF) on the Logstash host can also throttle Logstash from fetching the data from Kafka at the required rate.

Consumer lag serves as an effective indicator of this issue and can be viewed on Kafka's consumer group metrics. It is calculated as the difference between the log-end offset (the offset of the most recently produced message) and the current offset (the last committed offset by the consumer) for each partition.

$KAFKA_HOME/bin/kafka-consumer-groups.sh&nbsp; --bootstrap-server <server:port> --describe --group <group_id>

GROUP                 TOPIC             PARTITION  CURRENT-OFFSET  LOG-END-OFFSET   LAG
logstash-cg-soc-1     windows-events    0          4498            17309            12811
logstash-cg-soc-1     windows-events    1          4470            17213            12743
...

C) Ingest lag in the Logstash processing: This lag is the time difference between the first and last Logstash filters. To calculate this lag, an additional filter can be added at the end of the pipeline to record the `logstash.end` timestamp in the same way the `logstash.start` field was added before. The primary factors contributing to this lag are the filters efficiency of processing, which is primarily affected by the complexity and optimization of the transformations they perform, access to external services for data loading which might require network, limited number of the pipeline’s workers and small batch size, and the amount of resources available for Logstash – particularly when running on virtual environments with resources contention.

D) Ingest lag between Logstash and Elasticsearch: This lag is the time difference between the last applied Logstash filter in the pipeline, and the timestamp when the event is ingested in Elasticsearch. The ECS field `event.ingested` is automatically added by the Elastic integrations to record this value. For custom sources, the field should be added via an ingest pipeline:

{
    "processors": [
      {
        "set": {
          "field": "event.ingested",
          "value": "{{_ingest.timestamp}}"
        }
      }
…

If the data is undergoing heavy processing in Elasticsearch before indexing, it also pays to analyze the performance of each ingest processor in the pipeline to pinpoint and optimize the heaviest ones. Ingest pipelines monitoring dashboard can help streamline this process.

The primary factors contributing to this phase’s lag are usually the Logstash output configuration like a small number of pipeline workers and batch size, slow indexing actions (like upserts), network latency, and how fast the Elasticsearch cluster can run the ingest pipelines and index the data. You can find more techniques about this last point here.

Visualizing these stages in Kibana helps identify the most throttled areas and analyze the impact of various parameter adjustments across the entire data pipeline during the tuning process.

Isolate and fix the bottleneck

Identifying the source of the bottleneck can be challenging without a systematic approach to isolating the behavior of each component and stage of the pipeline. To make the investigation approach more consistent, it is important to keep the source data consistent as well. One approach can be to use a dedicated topic with a replicated production workload, and repeat the test using different consumer groups.

Below is a set of benchmarks that can be driven while monitoring the event lag and the pipeline throughput. The best achieved results from each of the tuning exercises can be used as a basis for the next one.

First benchmark: Kafka input, no filters, null output

This benchmark is aimed at assessing the throughput of the Kafka input in isolation, excluding the downstream impacts of the Logstash filters and outputs. Use the sink plugin in the output section to discard the events without incurring IO overhead and get a theoretical maximum reading speed.

This test is better performed with and without a persistence queue to isolate the additional overhead at this stage. 

It is helpful to use a unique consumer group_id for this test instead of the default `logstash`. Otherwise, this null-output pipeline might consume and drop events that should be processed by other pipelines.

input {
 kafka {
   ...
 }
}
filter {
}
output {
  sink { }
}

If the throughput from this test closely matches the original pipeline, then most probably you have a closed valve upstream and consuming the events is definitely a bottleneck. 

Note that the maximum throughput is significantly impacted by the Kafka cluster's ability to handle consumer requests and network latency. The maximum throughput is also bound by the rate of events that is flowing into the Kafka topic once the consumer group has caught up with the topic.

A few things might be considered in this exercise: 

• Match consumers count to partitions count: Ideally, the total number of consumer threads across all the pipelines that share the same consumer group_id, should be equal to the number of topic partitions for a perfect balance. Each Kafka topic-partition can be assigned to at-most one consumer within a consumer group at a time. So if you have more consumer threads than your topic partitions, some of those threads will not be assigned a partition. Partition-replicas do not count, as consumer threads consume messages from the leader partitions, not directly from replicas. Exceeding 1:1 ratio may also introduce unnecessary computational overhead in Logstash without any gains in read throughput. Incrementally increasing the partition count in the topic can potentially improve the throughput. Kafka 4.0 introduces early access to KIP932, which bypasses this 1:1 mapping requirement using share groups implementing a queuing semantic to the consumption model. The Share Groups are not supported in Logstash yet.

• Tune the input parameters for maximum throughput: Increasing max.poll.records, fetch.max.bytes, and receive.buffer.bytes can enhance performance. The TCP read buffer size is rarely an issue but can also be significantly important.  This setting is bound by the net.core.rmem_max value.

• Use fast disks with enough space if using persistent queues: The queue sits between the input and filter stages in the same process. The I/O performance of the storage directly impacts the input throughput. When the queue is full, Logstash puts back pressure on the inputs to stall the data flow.

Second benchmark: Kafka input, filters, no outputs

This benchmark helps measure the impact of the filters on the input throughput using the best achieved input configuration from the first exercise. It quantifies the throttling effect on the input stream only caused by the events processing. Note that some filter plugins are also IO-bound, like the plugins that use the network to enrich the events.

input {
 kafka {
   ...
 }
}
filter {
...
}
output {
}

To increase the number of simultaneously processed events by the filters, try increasing the number of pipeline workers and the pipeline batch size, particularly if the pipeline worker\_utilization flow metric is near 100 and Logstash is not spending all available CPU. Increasing the workers number past the number of available processors can also yield better results as some of the filter plugins may spend significant time in an I/O wait state like external lookups. 

Increasing the number of workers past the number of available processors can also improve performance, as some filter plugins may spend considerable time in an I/O wait state, such as during external lookups. This also makes a more efficient use of the Logstash host resources.

Optimizing the pipeline filters is the most effective approach to resolving this bottleneck. It can significantly reduce latency and increase the throughput regardless of the pipeline input configuration and Logstash resources. The per-plugin worker_utilization and worker_millis_per_event flow metrics are very useful in identifying where most of the resources are being spent, and consequently, where these improvements should focus first.

Optimizing pipeline filters is the most effective way to address this bottleneck. it can significantly reduce latency and boost throughput, regardless of the pipeline's input configuration or available resources. The per-plugin worker_utilization and worker_millis_per_event flow metrics are useful for finding which plugins are spending the most resources, and the optimization efforts should focus on those plugins first. Some general best practices that can usually make improvements are utilizing anchors for Grok plugins, switching to faster plugins like dissect whenever possible, optimizing Ruby filters code, eliminating unnecessary parsing, and improving the network-based enrichments. 

Source: do you grok

In some cases, optimizing the pipeline may require a complete redesign of the ingestion workflow or the pipeline itself!

Third benchmark: Kafka input, no filters, Elasticsearch output

This benchmark helps quantify the throttling effect of the Elasticsearch output on the input throughput. The test can be divided into two phases: the first phase uses raw logs to isolate the impact of Elasticsearch indexing, while the second phase assesses the impact of ingest pipelines.

In case a pipeline is using multiple outputs, note that, by default, a pipeline is blocked if any single output is blocked. This behavior is important in guaranteeing at-least-once delivery of data, but can cause the outputs to perform at the rate of the most clogged one.

input {
 kafka {
   ...
 }
}
filter {
}
output {
 Elasticsearch {
   ...
 }
}

To increase throughput, consider progressively increasing the number of pipeline workers and the pipeline batch size. Prior guidance about the worker\_utilization flow metric applies here too although availability of CPU plays a smaller role since this output is mostly IO-bound.  Also keep looking for the Elasticsearch Output's rejection rates (e.g.: response code 429 `es_rejected_execution_exception` indicating explicit back-pressure) as a signal that the Elasticsearch cluster is busy processing other batches.

The Logstash output tries to send batches of events to the Elasticsearch Bulk API in a single request. However, if a batch exceeds 20 MB, the plugin splits it into multiple bulk requests. 

If the Elasticsearch cluster is behind a proxy or API gateway, it's important to adjust the proxy limits to allow Logstash requests with large payloads to pass through to the Elasticsearch cluster. By default, most proxy servers have a much smaller maximum size for HTTP request payloads, which should be tuned in this case to accommodate larger requests. To identify potential issues, look for error code 413 in your proxy logs, as this indicates that the size of the Logstash request has exceeded the maximum payload size the proxy is configured to handle.

On the Elasticsearch cluster, tune your ingest pipelines efficiency following the same general best practices discussed above for the Logstash pipelines. Also, tune for the indexing speed by using faster hardware, less index refreshes, auto-generated IDs, and consider increasing the number of primary shards to enhance indexing parallelism if you have multiple nodes. Beware that excessively increasing this number can negatively impact the search performance.

Finally, keep in mind that the Elasticsearch output plugin is mostly IO-bound, which means that your network latency and bandwidth significantly reduce the rate at which data is transferred and hence your output throughput.

Reassemble your pipeline

After tuning the pipeline in each of the previous phases separately, put all the parts together again to assess the real throughput and latency of the reassembled pipeline. At this last step, you should have reached the best performance from your Logstash host as well, and you can progressively add more instances to reach the ultimate latency and throughput you are aiming for for a specific topic or data source.  

Example

Below is an example of the configuration required on Logstash and Elasticsearch to implement the architecture above.

Logstash pipeline:

input {
 kafka {
   bootstrap_servers => "<server>:<port>"
   topics => ["<topic-id>"]
   group_id => "<consumer-group-id>"
   decorate_events => "extended"
   auto_offset_reset => "earliest"
   codec => json {
   }
 }
}


filter {
 ruby {
   code => "event.set('[logstash][start]', Time.now());"
 }


 mutate {
   add_field => {
     "[kafka][timestamp]" => "%{[@metadata][kafka][timestamp]}"
     "[kafka][offset]" => "%{[@metadata][kafka][offset]}"
     "[kafka][consumer_group]" => "%{[@metadata][kafka][consumer_group]}"
     "[kafka][topic]" => "%{[@metadata][kafka][topic]}"
   }
 }


 date {
   match => ["[kafka][timestamp]", "UNIX", "UNIX_MS"]
   target => "[kafka][timestamp]"
 }
 ...
 ruby {
   code => "event.set('[logstash][end]', Time.now());"
 }
}


output {
 elasticsearch {
   hosts => "hosts"
   api_key => "api_key"
   data_stream => true
   ssl => true
 }
}

Create an ingest pipeline for lag calculation. Note that when using Elastic integrations, the ECS fields: \ \*.end\, \ \*.start\, \ \*.timestamp\ are automatically mapped as a date.

PUT _ingest/pipeline/calculate_ingest_lag
{
  "processors": [
    {
      "set": {
        "field": "event.ingested",
        "value": "{{_ingest.timestamp}}",
        "ignore_failure": true
      }
    },
    {
      "script": {
        "lang": "painless",
        "if": "ctx['@timestamp'] != null && ctx?.kafka?.timestamp != null && ctx?.logstash?.start != null && ctx?.logstash?.end != null && ctx?.event?.ingested != null",
        "source": """ 
  ctx.lag_in_millis = [:];
              ctx.lag_in_millis.src_kfk = Duration.between(ZonedDateTime.parse(ctx['@timestamp']), ZonedDateTime.parse(ctx['kafka']['timestamp'])).toMillis(); 
              ctx.lag_in_millis.kfk_ls = Duration.between(ZonedDateTime.parse(ctx['kafka']['timestamp']), ZonedDateTime.parse(ctx['logstash']['start'])).toMillis();
              ctx.lag_in_millis.within_ls  = Duration.between(ZonedDateTime.parse(ctx['logstash']['start']), ZonedDateTime.parse(ctx['logstash']['end'])).toMillis();
              ctx.lag_in_millis.ls_es = Duration.between(ZonedDateTime.parse(ctx['logstash']['end']), ZonedDateTime.parse(ctx['event']['ingested'])).toMillis(); 
              ctx.lag_in_millis.end_end = Duration.between(ZonedDateTime.parse(ctx['@timestamp']), ZonedDateTime.parse(ctx['event']['ingested'])).toMillis();     
        """
      }
    }
  ]
}

Use the pipeline to add the lag calculation to your Elastic integrations

PUT _ingest/pipeline/logs-system.integration@custom
{
  "processors": [
    {
      "pipeline": {
        "name": "calculate_ingest_lag",
        "ignore_missing_pipeline": true,
        "description": "add ingest lag calculation to elastic_agent integration"
      }
    }
  ]
}

Kibana Dashboard and Alerts

Using the metrics mentioned above along with the Log Rate ML job, you can set up Kibana alerts to trigger when with anomalous changes in throughput or delays or simply when delays exceed defined thresholds.

Time to try it out

Start your free 14-day trial of Elastic Cloud to experience the latest version of Elastic. Also, make sure to take advantage of the Elastic threat detection training to set yourself up for success.

The Design Problem in Log Processing

We rarely talk about how projects actually begin. 

How do you design something that doesn't fully exist yet?

How do you align AI capabilities, system constraints, real user pains into one coherent experience?

Streams gave us that challenge.

Logs are one of the richest signals in observability - but also one of the messiest. Streams is an agentic AI-powered solution that rethinks how teams work with logs to enable fast incident investigation and resolution. 

Streams uses AI to partition and parse raw logs, extract relevant fields, reduce schema management overhead, and surface significant events like critical errors and anomalies.

This led us to make logs investigation-ready from the start, and not force the Site Reliability Engineer to fight their data. But in order to enable such experience, we had to carefully rethink a core concept and step in the process - Processing.

Designing Processing UX in Elastic Streams

Logs are powerful, but only if they are structured correctly. Today, a user would onboard logs via Elastic Agent, using a custom integration, extract something as simple as an IP field by:

• Write GROK patterns
• Create pipelines
• Manage mappings
• Test transformation
• Iterate repeatedly

What sounds simple requires 20+ steps — and deep expertise most teams shouldn’t need. Our goal became simple: make this dramatically simpler.

Our early design question was:

“ Can we reduce this experience to 2 meaningful steps instead of 20 technical ones?”

That question shaped how we approached the Stream UX.

The Foundation

Before we jumped into designing the UI in Kibana, we defined a core mental model. 

A Stream is a collection of documents stored together that share:

• Retention
• Configuration
• Mappings
• Processing rules
• Lifecycle behaviour

The key design principle:

“A Stream should contain data that behaves consistently.”

Why Does Data Consistency Matter?

We started with an example to test our thinking. Take Nginx access and error logs.

Access logs describe request/response events:

192.168.1.10 - - [16/Feb/2026:12:32:10 +0000] "GET /api/orders/123 HTTP/1.1" 200 532 "-" "Mozilla/5.0"

Error logs describe diagnostic events:

2026/02/16 12:32:10 [error] 2719#2719: *342 connect() failed (111: Connection refused) while connecting to upstream…

If both live in the same Streams that might cause:

• Processing logic conflicts
• Field divergence
• Mapping conflicts
• Investigations would be fundamentally harder

That insight clarified something critical: 

“Processing isn’t just about extracting fields. It’s about protecting consistency.”

Making Complexity Manageable

The ingest ecosystem isn’t small, simple, or hypothetical. Real pipelines use dozens of processors — from common ones like rename, set, convert, and append, to niche types like urldecode and network_direction.

The UI had to support both high-frequency actions and long-tail edge cases without losing structure. Currently Elasticsearch supports over 40 different ingest processors. We had to make sure our interface could handle the different types.

We introduced a clear, nested structure for pipeline steps. Users could create, reorder, edit, or remove individual steps or grouped ones with confidence. The nested drag and drop capability was also added as a pattern in our EUI library.

This gave us the context and foundation to work on integrating those concepts into a model that would be definitive for everything in Streams.

Page Archetypes

Processing is powerful - and risky. Changing a parsing condition or step might affect:

• Field availability
• Search behaviour
• Alerts
• AI Insights
• Investigations

So we asked ourselves how do we make something so powerful and important, safe for the user? The answer led to a core page archetype:

Create > Preview > Confirm

This wasn’t a UI pattern added later. It emerged directly from our concept work and understanding what users would have to deal with.

To support this archetype and core idea, we also introduced a split-screen structure.

Left: Build

This is where users would:

• Add processing steps
• Define conditions
• Apply rules
• Leverage AI suggestions both as a whole pipeline creation or individual steps like a GROK processor

It remained focused, intentional and structured.

Right Preview

This is where users would:

• See real life log samples
• See extracted fields in context
• Immediate feedback on changes, with insights about the matched and unmatched percentage of documents
• Optional drilldown side panel on the right

The preview panel became the anchor of confidence. This was not about visual symmetry, but to reinforce experimentation, control over errors and decrease the level of mistakes. Knowing that users might want to switch their focus from interaction to detailed preview, we introduced the resizeable function to both panels, and unlocked more flexiblity and control over the use cases.

AI Automation

Streams is agentic and AI powered. That added another layer of complexity for the design, but also another opportunity to unlock even more power and insights from users' log data. 

AI introduced a new tension: how do you accelerate processing without turning it into a black box?

We established a few guardrails:

• Clear, concise suggestions
• Visible impact through matched document metrics
• Inspectability
• Alignment with the Create → Preview → Confirm model

Processing UX became the bridge between automation and human in the loop. Log data is one of the most powerful investigation signals. Every design decision reinforced that belief.

What We Learned

Designing for the future does not start with screens. It starts with:

• Edge case testing
• Clear mental models
• Strong and guiding principles
• Behavioral consistency
• Scalable and stress-tested archetypes

We know that in order for a user to be able unlock insightful discoveries from their logs, they would need to process and manage their data effectively. We knew we were shaping their entire observability foundation. 

Processing is about trust, control, and scalable data management.

Trust enables investigation speed.

Investigation speed enables resilience.

Learn more

Sign up for an Elastic trial at cloud.elastic.co, and trial Elastic's Serverless offering which will allow you to play with all of the Streams functionality. You want to know more about Streams? Check some of the links below:

Read about Reimagining streams

Read about Retention management

Look at the Streams website

Check the Streams documentation

However, the landscape of log management is shifting. By combining the industry-standard ingestion of OpenTelemetry (OTel) with the AI-driven capabilities of Elastic Streams, we can change how we monitor Windows infrastructure. This approach isn't just moving data. We are also using Large Language Models (LLMs) to understand it.

The Challenge with Traditional Windows Logging

Windows generates a massive variety of logs: System, Security, Application, Setup, and Forwarded Events. Within those categories, you have thousands of Event IDs. Historically, getting this data into an observability platform involved installing proprietary agents and configuring complex pipelines to strip out the XML headers and format the messages.

Once the data was ingested, we can try to figure out what "bad" looked like. You had to know in advance that Event ID 7031 indicated a service crash, and then write a specific alert for it. If you missed a specific Event ID or if the format changed, your monitoring went dark.

Step 1: Ingestion via OpenTelemetry

The first step in modernizing this workflow is adopting OpenTelemetry. The OTel collector has matured significantly and now offers robust support for Windows environments. By installing the collector directly on Windows servers, you can configure receivers to tap into the event log subsystems.

The beauty of this approach is standardization. You aren't locked into a vendor-specific shipping agent. The OTel collector acts as a universal router, grabbing the logs and sending them to your observability backend in this case, the Elastic logs index designed to handle high-throughput streams.

The key thing to pay attention to in this configuration is how we add this transform statement:

transform/logs-streams:
  log_statements:
    - context: resource
      statements:
        - set(attributes["elasticsearch.index"], "logs")

This works with the vanilla opentelemetry collector and when the data arrives in Elastic, it tells Elastic to use the new wired streams feature which enables all the downstream AI features we discuss in later steps.

Checkout my example configuration here

Step 2: AI-Driven Partitioning

Once the data arrives, the next challenge is organization. Dumping all Windows logs into a single logs-* index is a recipe for slow queries and confusion. In the past, we split indices based on hardcoded fields. Now, we can use AI to "fingerprint" the data.

This process involves analyzing the incoming stream to identify patterns. The system looks at the structure and content of the logs to determine their origin. For example, it can distinguish between a Windows Security Audit log and a Service Control Manager log purely based on the data shape.

The result is automatic partitioning. The system creates separate, optimized "buckets" or streams for each data type. You get a clean separation of concerns, Security logs go to one stream, File Manager logs to another, without having to write a single conditional routing rule. This partitioning is crucial for performance and for the next phase of the process: analysis.

Step 3: Significant Events and LLM Analysis

Once your data is partitioned (e.g., into a dedicated Service Control Manager stream), you can apply GenAI models to analyze the semantic meaning of that stream.

In a traditional setup, the system sees text strings. In an AI-driven setup, the system understands context. When an LLM analyzes the Service Control Manager stream, it identifies what that system is responsible for. It knows that this specific component manages the starting and stopping of system services.

Because the model understands the purpose of the log stream, it can generate suggestions for what constitutes a "Significant Event." It doesn't need you to tell it to look for crashes; it knows that for a Service Manager, a crash is a critical failure.

From Passive Storage to Proactive Suggestions

The workflow effectively automates the creation of detection rules. The LLM scans the logs and generates a list of potential problems relevant to that specific dataset, such as:

• Service Crashes: High severity anomalies where background processes terminate unexpectedly.
• Startup/Boot Failures: Critical errors preventing the OS from reaching a stable state.
• Permission Denials: Security-relevant events regarding service interactions.

It bubbles these up as suggested observations. You can review a list of potential issues, see the severity the AI has assigned to them (e.g., Critical, Warning), and with a single click, generate the query required to find those logs.

Conclusion

The combination of OpenTelemetry for standardized ingestion and AI-driven Streams for analysis turns the chaotic flood of Windows logs into a structured, actionable intelligence source. We are moving away from the era of "log everything, look at nothing" to an era where our tools understand our infrastructure as well as we do.

The barrier to effective monitoring is no longer technical complexity. Whether you are tracking security audits or debugging boot loops, leveraging LLMs to partition and analyze your streams is the new standard for observability.

Try Streams today

Elastic has been leading the charge with AIOps, especially in the recent 9.2 update of Elastic Observability with Streams. The conversation around AIOps has shifted dramatically as we move through the year. DevOps and SRE teams aren't asking whether they need AIOps, they're asking how to leverage it more effectively to stay ahead of exponentially growing complexity.

The current challenge of AIOps is that modern cloud-native environments generate massive volumes of telemetry data that are magnitudes larger than past environments. But here's what many teams overlook: logs are the richest source of operational intelligence you have. Logs are able to tell you exactly what happened and why, while metrics only tell you something is wrong, and traces only tell you where. The problem is that most organizations are drowning in logs. Microservices, such as user authentications or inventories, serverless functions, and Kubernetes generate millions of log entries daily. Without AI and machine learning, finding meaningful patterns in this data takes too much time and energy.

Log Intelligence Improvement: What's New in 2025

Historically in observability, unlocking your log intelligence included long manual effort that required not only parsing through logs, but also structuring those logs. Elastic Observability has drastically changed how teams extract value from logs. Observability is not just simple signal analysis - modern tools need to have proactive, log-driven investigations. At Elastic, this modernity is Streams.

Streams, a new release from Elastic, is a collection of AI-driven tools that identify significant events in parsed raw logs by enriching logs with meaningful fields. With Streams, SREs can maximize the value of their data, their logs, and their systems. With system reliability as the goal, Streams helps to reduce pipeline management overhead and accelerates observability analysis. And it takes nearly no time to set up!

Here is how Streams powers the Elastic Observability capabilities available now.

Advanced Log Rate Analysis

Log rate analysis can go far beyond only detecting spikes. Elastic's machine learning automatically identifies when log volumes deviate from expected baselines, then contextualizes these changes within your broader system performance. When your application suddenly generates more error logs, Elastic’s AIOps doesn't just alert you, it also determines whether it's a critical issue requiring immediate attention or just a temporary anomaly.

This matters to your analysis because not all log spikes are equal. A 10x increase in DEBUG logs might indicate verbose logging accidentally enabled in production. A 2x increase in ERROR logs could signal a cascading failure. Log rate analysis distinguishes between these scenarios automatically, giving your team the context needed to respond appropriately.

Intelligent Log Categorization with Streams

This is where AIOps shines with log data. Streams uses machine learning algorithms in order to automatically classify and group similar log patterns, dramatically reducing noise. Instead of manually parsing millions of entries, the system identifies common structures, groups related events, and surfaces the categories that matter most.

Logs are unstructured by nature, making them difficult to analyze at scale. Streams corrals chaotic log streams into organized, queryable patterns. Instantly, you can see that 80% of your errors fall into three categories, helping you prioritize where to focus remediation efforts. This approach helps you reduce noise and accelerate analysis, allowing teams to act on insights faster.

Multi-Dimensional Anomaly Detection

Anomaly detection now simultaneously examines relationships between logs, metrics, and traces. A slight increase in response time might not trigger an alert by itself, but when correlated with unusual log patterns and memory consumption changes, the system recognizes it as an early warning sign.

Logs contain a myriad of contextual information that metrics and traces can't capture: stack traces, user IDs, transaction details, error messages, etc. By correlating log anomalies with other signals, you get the full picture of what's happening in your system. This whole holistic view enables teams to catch issues earlier, as well as understand their full impact across the stack.

Enhanced Root Cause Analysis Powered by Significant Events

When an issue occurs, Elastic's Streams accelerates root cause analysis through AI-assisted parsing of logs and bringing about “Significant events.” Significant event queries can be defined by AI or manually, depending on if you know what logs you are looking for or not. Then, Elastic’s AIOps traces the problem through your entire stack using these events, as well as enriched log data combined with distributed tracing. This system is able to correlate failed transactions with specific log entries, deployment events, and infrastructure changes. This helps you understand not just what broke, but why and when.

Streams makes the analysis of your logs quick and automatic by going across your entire distributed system within seconds, grabbing relevant log entries such as stack traces, state information, error messages, and more. What used to require hours of manual investigation and deduction now happens automatically, freeing you and your team from tedious detective work and enabling faster resolution. 

Logs in Action: Real-World Impact

Let's look at how these capabilities work together in practice. Imagine your payment processing service is experiencing intermittent failures - only 0.5% of transactions, but enough to concern your team. Traditional monitoring shows everything is mostly okay, but customers are still complaining.

Without Streams, an SRE might initially run some broad queries, manually sift through thousands of logs, struggle to connect all the dots, and ultimately not understand the correlation between the errors and recent system changes. 

With Elastic Streams and AIOps, many of these potential problems are instantly mitigated:

• Streams automatically parse the payment service, adding connection timeouts to a new category of significant events

• Log rate analysis with Streams reveal that this significant event category has been slowly growing over the past month, showing growth of the timeouts from a small number of occurrences into a larger amount

• Elastic’s built-in anomaly detection correlates these significant events with deployment data, and identifies that they started appearing after a recent load balancer configuration

• Root analysis pinpoints the exact database connection pool setting that is too restrictive for peak load by tracing affected transactions through previously enriched logs

What usually takes 4-8 hours of manual log analysis is resolved in minutes, with Elastic automatically highlighting the relevant log entries that tell the complete story. This is the power of AIOps and Streams as applied to log intelligence.

The Power of Unified Log Intelligence

What sets Elastic apart is treating logs as a priority in your observability strategy. Elastic provides comprehensive log ingestion that centralizes petabytes of logs from across your infrastructure with flexible parsing and enrichment. The platform uses purpose-built machine learning models that understand log patterns, not generic algorithms retrofitted for log analysis.

Logs don't exist in isolation, which is why Elastic correlates log data with metrics, traces, and business events to provide complete context. And because log volumes can be massive, Elastic's tiered storage approach means you can retain years of logs for compliance and historical analysis without breaking the budget.

Why Logs Matter More Than Ever

Logs have become the cornerstone of effective AIOps for three critical reasons.

First off, logs capture what metrics can't. A metric tells you the CPU is at 80%, but a log tells you which process is consuming resources and why. This level of detail is essential for understanding not just that something is wrong, but what specifically is causing the problem.

Second, logs provide business context. Error messages contain user IDs, transaction ldetails, and business logic failures that help you understand customer impact. When you're troubleshooting an issue, knowing which customers are affected and what they were trying to do is invaluable for prioritizing your response.

Third, logs enable true root cause analysis. Stack traces, error messages, and application state captured in logs are essential for understanding the why behind every incident. Without this information, teams are left guessing at root causes rather than definitively identifying and fixing them.

The teams winning with AIOps in 2025 aren't just monitoring metrics, they're extracting intelligence from their logs at scale, turning operational data into actionable insights.

Transform Your Log Strategy Today

Every hour your team spends manually searching through logs is an hour they're not spending on innovation. Every incident that could have been prevented through intelligent log analysis represents both technical debt and business risk.

Elastic Observability provides the foundation you need to unlock the intelligence hidden in your logs. With automatic categorization, anomaly detection, and ML-powered analysis, you can start seeing value immediately. Check out this recent article to get started with Elastic Streams and Observability today!

What's in data quality

At-a-glance summary

The summary card shows:

• Degraded documents - Documents that contain the _ignored field - see this for more info.
• Failed documents - Documents that were rejected at ingestion due to mapping conflicts or pipeline failures.

The overall quality score (Good, Degraded, Poor) is automatically calculated based on the percentage of degraded and failed documents.

Trends over time

The tab includes a time-series chart so you can track how degraded and failed documents are accumulating over time. Use the date picker to zoom into a specific range and understand when problems are spiking.

Quality issues table

A detailed table lists the types of issues affecting your stream. For each issue, you can:

• See which fields are causing problems.
• Review counts of affected documents.
• Filter by issues that have not been solved yet (Current issues only).
• Open a flyout to dive deeper into the cause of the issue and learn how to fix it.

Monitoring degraded documents

A degraded document is one that contains the _ignored field, which means one or more of its fields were ignored during indexing. One of the reasons could be that their values didn’t match the expected mappings. While the rest of the document is still indexed, a high number of degraded documents can affect query results, dashboards, and overall observability accuracy.

To help keep these issues under control, the Data quality tab provides visibility into the percentage of degraded documents in your stream.

Set up a rule to stay ahead of issues

You can use the Create rule button above the Degraded docs chart to define an alert that notifies you when the percentage of degraded documents crosses a certain threshold. This makes it easy to proactively monitor for mapping mismatches and ensure your data continues to meet quality expectations.

For more information on how to configure this rule, see Degraded docs rule conditions.

Handling failed documents with the failure store

Failure store is a special index that captures documents rejected during ingestion. Instead of losing this data, the failure store retains it in a dedicated ::failures index, allowing you to inspect the problematic documents, understand what went wrong, and fix the underlying issues.

In Data Quality tab, the failed documents are only visible if your stream has a failure store enabled, for checking failure store documents you are required to have at least read_failure_store privileges. If the failure store is not enabled, you’ll see an “Enable failure store” link that opens a modal to configure it and set the retention period. For enabling failure store you are required to have manage_failure_store privileges over the specific data stream. For further information about failure store security you can refer to Searching failures.

Once enabled, you can edit the failure store configuration or disable it at any time using the Edit button above the failed docs chart.

The failure store can also be configured in the Streams Retention tab - see this article for more information.

Technical implementation

Under the hood, the Data quality tab builds on the existing Dataset quality plugin - the same one that powers the Dataset quality page in Stack Management. However, instead of working in the context of datasets following the Data stream naming scheme, it’s now tailored specifically for streams.

To determine the quality of a stream, the UI sends three ES|QL query server requests:

1. All documents (including failures):

 FROM myStream, myStream::failures | STATS doc_count = COUNT(*)

1. Failed documents only:

 FROM myStream::failures | STATS failed_doc_count = COUNT(*)

1. Degraded documents:

FROM myStream METADATA _ignored | WHERE _ignored IS NOT NULL | STATS degraded_doc_count = COUNT(*)

The results of these queries are then used to calculate the percentages of failed and degraded documents. The overall data quality is determined using simple thresholds:

• Good: Both percentages are 0%
• Degraded: Any percentage is greater than 0% but less than 3%
• Poor: Any percentage is above 3%

For managing the failure store, Streams uses the Update data stream options API with the failure_store parameter to configure and update the failure store settings, including enabling the store and setting the retention period.

Why you’ll love this

The new Data quality tab gives you:

• Visibility into ingestion problems without digging into logs
• A clear breakdown of degraded vs. failed documents
• Insights into which fields are ignored and why
• Tools to capture and troubleshoot failed docs with the failure store

By surfacing data quality issues directly in the Streams UI, we’re making it easier to keep your data flowing reliably and to ensure your analytics are built on a strong foundation.

Try it out today

The data quality feature is available in Elastic Observability on Serverless, and coming soon for self-managed and Elastic Cloud users.

Sign up for an Elastic trial at cloud.elastic.co, and trial Elastic's Serverless offering which will allow you to play with all of the Streams functionality.

For more information on Streams:

Read about Reimagining streams

Look at the Streams website

Read the Streams documentation

Streams changes that. It introduces a clear, unified way to manage how long your data lives, whether you’re using DSL or ILM. You can visualize ingestion, understand where data sits across tiers, and adjust retention with confidence, applying updates to a single stream without worrying about unintended changes elsewhere, all from a single view.

Walkthrough: Exploring the Retention Tab

Retention management lives in the Retention tab of each stream. This is your control panel for understanding how much data you’re storing, how quickly it’s growing, and how your lifecycle policies are applied. It’s also where you can monitor and configure the Failure store, which tracks and retains documents that failed to be ingested.

Metrics at a glance

At the top of the view, you’ll find an overview of key metrics:

• Storage size: the total data volume currently held by the stream.
• Ingestion averages: calculated from the selected time range, Streams extrapolates both daily and monthly averages to give you a sense of long-term trends.

This combination of near-real-time and projected values helps you quickly spot when ingestion is ramping up and whether your retention policy aligns with it.

Ingestion over time

Below the metrics, a graph shows ingestion volume over time. This information is approximated based on the number of documents over time, multiplied by the average document size in the backing index. 

Visualizing lifecycle phases

When an ILM policy is effective, the retention view becomes more visual. Streams displays a phase breakdown (hot, warm, cold, frozen) showing the data volume stored in each phase. This gives you a clear sense of how your data is distributed across the storage tiers and whether your lifecycle is doing what you expect.

Failure store

A failure store is a secondary set of indices inside a data stream, dedicated to storing documents that failed to be ingested. Within the Retention tab, you can toggle the Failure store on or off, and configure its own retention period. We’ll cover Failure store and Data quality in more detail in this article.

Updating Retention

Beyond visualizing your retention, Streams makes it easy to change how it’s managed.

Switching between DSL and ILM

You can freely switch a stream between DSL and ILM management, or update a DSL retention with just a few clicks. Streams takes care of updating the lifecycle settings at the data stream level, ensuring consistent retention across all existing backing indices, not just new ones.

Whether you prefer the simplicity of DSL or the fine-grained tiering of ILM, you can move between the two seamlessly.

Clicking “Edit data retention” opens a modal that allows you to update the stream’s configuration. From there you can update the ILM policy or set a custom retention period via DSL.

You can set a custom period, or pick an Indefinite retention for your data.

You can also update streams’ lifecycle via the Upsert stream or the Update ingest stream settings Kibana APIs.

Inherit or defer: different strategies for different stream types

Classic streams

For classic streams, you can default to the existing index template’s retention. Retention isn’t managed by Streams in this case, it follows the lifecycle configuration defined in the template just as it normally would.

This option is useful if you’re onboarding existing data streams and want to keep their lifecycle behavior intact while still benefiting from Streams’ visibility and monitoring features.

Wired streams

Wired streams live in a tree structure, and that hierarchy allows an inheritance model.

A child stream can inherit the lifecycle of its nearest ancestor that has a concrete policy (ILM or DSL). This keeps your configuration lean and consistent since you can set a single lifecycle at a higher level in the tree and let Streams automatically apply it to all relevant descendants.

If that ancestor’s lifecycle is later updated, Streams cascades the change down to all children that inherit it, so everything stays in sync.

In the figure below, we set a different retention for logs.prod and logs.staging environments. The child partitions of these environments automatically inherit the configuration.

How it works under the hood

When you apply or update a lifecycle, Streams calls Elasticsearch’s /_data_stream/_settings. This is a new API we’ve added in 8.19 / 9.1 for this purpose. 

This API is key to keeping retention consistent:

1. It applies the lifecycle directly at the data stream level, overriding any configuration from cluster settings or index templates.
2. It propagates the retention update to all existing backing indices, not just new ones, so retention remains uniform across your historical and future data.

By centralizing lifecycle management at the data stream level and applying a consistent configuration across the backing indices, we remove the ambiguity that used to exist between template-level and index-level configurations. You always know which retention policy is actually in effect, and you can see it directly in the UI.

Wrapping Up

With Streams, retention management becomes clear and consistent. You can visualize ingestion, switch between DSL and ILM, or inherit policies across streams, all without diving into templates or manual index settings.

By unifying retention into a single view, Streams turns lifecycle management into something simple, predictable, and transparent.

Sign up for an Elastic trial at cloud.elastic.co, and trial Elastic's Serverless offering which will allow you to play with all of the Streams functionality.

Additionally, check out:

Read about Reimagining streams

Look at the Streams website

Read the Streams documentation

Observability, the broken promise

To see why the current model fails, let’s look at the all-too-familiar challenge every SRE dreads: knowing a problem exists but needing to spend valuable time just trying to find where to even start the investigation.

Imagine you get a Slack message from the support team: "a few high-value customers are reporting their payments are failing." You have no shortage of alerts, but most are just flagging symptoms. You don’t know where to start. You decide to check the logs to see if there is anything obvious, starting with the systems that have the high CPU alert.

You spend a few minutes searching and grep-ing through terabytes of logs for affected customer IDs, trying to piece together the problem. Nothing. You worry that you aren’t getting all the logs to reveal the problem, so you turn on more logging in the application. Now you’re knee-deep in data, desperately trying to find patterns, errors, or other "hints" that will give you a clue as to the why.

Finally, one of the broader log queries hits on an error code associated with an impacted customer ID. This is the first real clue. You pivot your search to this new error code and after an hour of digging, you finally uncover the error message. You've finally found the why, but it was a stressful, manual hunt that took far too much time and impacted dozens more customers.

This incident perfectly illustrates the broken promise of modern observability: The complete failure of the investigation process. Investigations are a manual, reactive process that SREs are forced into every day. At Elastic, we believe metrics, traces, and logs are all essential, but their roles, and the workflow between them, must be fundamentally re-imagined for effective investigations.

Observability is about having the clearest understanding possible of the what, where, and why. Metrics are essential for understanding the what. They are the heartbeat of your system, powering the dashboards and alerts that tell you when a threshold has been breached, like high CPU utilization or error rates. But they are aggregates; they show the symptom, rarely the root cause. Traces are good at identifying the where. They map the journey of a request through a distributed system, pinpointing the specific microservice or function where latency spikes or an error originates. Yet, their effectiveness hinges on complete and consistent code instrumentation, a constant dependency on development teams that can leave you with critical visibility gaps. Logs tell you the why. They contain all the rich, contextual, and unfiltered truth of an event. If we can more proactively and efficiently extract information from logs, we can greatly improve our overall understanding of our environments.

Challenges of logs in modern environments

While logs are in the standard toolbox, they have been neglected. SREs using today’s solutions deal with several major problems:

• First, due to their unstructured nature, it’s very difficult to parse and manage logs so that they’re useful. As a result, many SRE teams spend a lot of time building and maintaining complex pipelines to help manage this process. 

• Second, logs can get expensive at high volume, which leads teams to drop them on the floor to control costs, throwing away valuable information in the process. Consequently, when an incident occurs, you waste precious time hunting for the right logs, and manually correlating across services.

• Finally, nobody has built a log solution that proactively works to find the important signals in logs and to surface those critical whys to you when you need them. As a result, log-based investigations are too painful and slow.

Why are we here? As applications became more complex, log volume became unmanageable. Instead of solving this with automation, the industry took a shortcut: it gave up on getting the most out of logs and prioritized more manageable but less informative signals.

This decision is the origin of the broken, reactive model. It forced observability into a manual loop of 'observing' alerts, rather than building automation that could help us truly understand our systems to improve how we root cause and resolve issues. This has transformed SREs from investigators into full-time data wranglers, wrestling with Grok patterns and fragile ETL scripts instead of solving outages. 

Introducing Streams to rethink how you use logs for investigations

Streams is an agentic AI solution that simplifies working with logs to help SRE teams rapidly understand the why behind an issue for faster resolution. The combination of Elasticsearch and AI is turning manual management of noisy logs into automated workflows that identify patterns, context, and meaning, marking a fundamental shift in observability.

Log everything in any format

By applying the Elasticsearch platform for context engineering to bring together retrieval and AI-driven parsing to keep up with schema changes, we are reimagining the entire log pipeline.  

Streams ingests raw logs from all your sources to a single destination. It then uses AI to partition incoming logs into their logical components and parses them to extract relevant fields for an SRE to validate, approve, or modify. Imagine a world where you simply point your logs to a single endpoint, and everything just works. Less wrestling with Grok patterns, configuring processors, and hunting for the right plugin. All of which significantly reduces the complexity. Streams is a big step towards realizing that vision.

As a result, SREs are freed from managing complex ingestion pipelines, allowing them to spend less time on data wrangling and more time preventing service disruptions.

Solve incidents faster with Significant Events 

Significant Events, a capability within Streams, uses AI to automatically surface major errors and anomalies, enabling you to be proactive in your investigations. So, instead of just combing through endless noise, you can focus on the events that truly matter, such as startup and shutdown messages, out-of-memory errors, internal server failures, and other significant signals of change. These events act as actionable markers, giving SREs early warning and clear focus to begin an investigation before service impact.

With this new foundation, logs will become your primary signal for investigation. The panicked, manual search for a needle in a digital haystack is about to be over. Significant Events acts like a smart metal detector that sifts through the chaos and only beeps when it finds issues, helping you to easily ignore all that hay and find the "needle" faster. 

Now imagine the same scenario we started with. Instead of starting a frantic, time-consuming grep through terabytes of logs. Streams has already done the heavy lifting. Its AI-driven analysis has detected a new, anomalous pattern that began before your support team even knew about it and automatically surfaced it as a significant event. Rather than you hunting for a clue, the clue finds you. 

With a single click, you have the why: a Java out-of-memory error in a specific service component. This is your starting point. You find the root cause in under two minutes and begin remediation. The customer impact is stopped, the dev team gets the specific error, and the problem is contained before it can escalate. In this case, metrics and traces were unhelpful in finding the why. The answer was waiting in the logs all along.

This ideal outcome is possible because you can both afford to keep every log and instantly find the signal within them. Elastic's cost-efficient architecture with powerful compression, searchable snapshots, and data tiering makes full retention a reality. From there, Streams automatically surfaces the significant event, ensuring that the answer is never lost in the noise.

Elastic is the only company that provides an AI-driven log-first approach to elevate your observability signals and make it dramatically faster and easier to get to why. This is built on our decades of leadership in search, relevance, and powerful analytics that provides the foundation for understanding logs at a deep, semantic level.

The vision for Streams 

The partitioning, parsing, and Significant Events you see today is just the starting point. The next step in our vision is to use the Significant Events to automatically generate critical SRE artifacts. Imagine Streams creating intelligent alerts, on-the-fly investigation dashboards, and even data-driven SLOs based only on the events that actually impact service health. From there, the goal is to use AI to drive automated Root Cause Analysis (RCA) directly from log patterns and generate remediation runbooks, turning a multi-hour hunt into an instant resolution recommendation.

Once this AI-drive log foundation is in place, our vision for Streams expands to become a unified intelligence layer that operates across all your telemetry data. It’s not just about making each signal better in isolation, but about understanding the context and relationships between them to solve complex problems. 

For metrics, Streams won’t just alert you to a single metric spike but detect a correlated anomaly across multiple, seemingly unrelated metrics e.g. p99 latency for a specific service, rise in garbage collection time, transaction success rate.

Similarly, for traces it identifies a new, unexpected service call (e.g., a new database or an external API) appears in a critical transaction path after a deployment or identifies specific span is suddenly responsible for a majority of errors across all traces, even if the overall error rate hasn't breached a threshold.

The goal is not to have separate streams for logs, metrics, and traces, but to weave them into a single narrative that automatically correlates all three signals. Ultimately, Streams is about fundamentally changing the goal from human led data gathering exercise to proactive, AI-driven resolution.

For more on Streams:

Read the Streams launch blog

Look at the Streams website

Anomaly Detection

A production environment is ingesting millions of log lines per minute. Elastic’s AIOps jobs continuously profile normal log throughput and content without any manual rules. When log volume or message structure deviates beyond learned baselines, the platform automatically fires a high‑fidelity anomaly alert. Because the models are unsupervised, they adapt to changing traffic patterns and flag both sudden spikes (e.g., 10× error surge) and rare new log categories.

In addition to looking directly for Log Spikes, Elastic trains seasonal/univariant models to predict expected event counts per bucket and applies statistical tests to classify outliers. Simultaneously, log categorization clusters similar messages with cosine similarity on token embeddings, making it trivial to identify a previously unseen error string.

Investigating Alerts: Automated Pattern Analysis

Clicking the alert reveals more than a timestamp. Elastic’s ML job already correlates the spike with the dominant new log pattern ERROR 1114 (HY000): table "orders" is full and surfaces example lines. Instead of grep‑driven hunting, engineers get an immediate hypothesis about what subsystem is failing and why.

If deeper context is needed, the builtin Elastic AI Assistant can be invoked directly from the alert. Thanks to Retrieval‑Augmented Generation (RAG) over your telemetry, the assistant explains the anomaly in plain language, references the exact log events, and proposes next steps without hallucinating.

AI‑Assisted Root Cause Verification

From within the same chat, you might ask, “Using lens create a single graph of all http response status codes =400 from logs-nginx.access-default over the last 3 hours..”  The assistant translates that intent into an ES|QL aggregation, retrieves the data, and renders a bar chart with no DSL knowledge required. If there are a number of errors with a status code above 400, you’ve validated that end‑users are impacted.

Global Impact Analysis with Enriched Logs

Structured log enrichment (e.g., GeoIP, user ID, service tags) lets the assistant answer business questions on the fly. A query like “What are the top 10 source.geo.country_name with http.response.status.code>=400 over the last 3 hours. Use logs-nginx.access-default. Provide counts for each country name.” surfaces whether the incident is regional or global.

Quantifying Business Impact

Technical metrics alone rarely sway executives. Suppose historical data shows the application normally processes $1,000 in transactions per minute. The assistant can combine that baseline with real‑time failure counts to estimate revenue loss. Presenting financial impact alongside error graphs sharpens prioritization and justifies extraordinary remediation steps.

Pinpointing Infrastructure & Ownership

Every log is automatically enriched with Kubernetes, cloud, and custom metadata. A single question “Which pod and cluster emit the ‘table full’ error, and who owns it?” returns the full information about the pod, namespace and owner as shown below.

Immediate, accurate routing replaces frantic Slack threads, cutting minutes (or hours) off of downtime.

Some of the magic happening here is because we can put instructions in the Elastic AI Assistants knowledge base to guide the AI assistant. For example this simple entry in the knowledge base is what allows the assistant to populate the response in the previous screenshot.

``markdown ## Kubernetes Information Query Instructions If asked about Kubernetes pod, namespace, cluster, location, or owner run the "query" tool. 1. Use the index logs-mysql.error-default unless another log location is specified. 2. Include the following fields in the query:    - Pod: agent.name    - Namespace: data_stream.namespace    - Cluster Name: orchestrator.cluster.name    - Cloud Provider: cloud.provider    - Region: cloud.region    - Availability Zone: cloud.availability_zone    - Owner: cloud.account.id` 3. Use the ES|QL query format:    esql    FROM logs-mysql.error-default    | KEEP agent.name, data_stream.namespace, orchestrator.cluster.name, cloud.provider, cloud.region, cloud.availability_zone, cloud.account.id     4. Ensure the query is executed within the appropriate time range and context. 

## Leveraging Institutional Knowledge with RAG

Elastic can index runbooks, GitHub issues, and wikis alongside telemetry. Asking “Find documentation on fixing a full orders table” retrieves and summarizes a prior runbook that details archiving old rows and adding a partition. Grounding remediation in proven procedures avoids guesswork and accelerates fixes.

![](https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/bltba6ebcc6e2f451ad/6a7f01b233fa8ae5f62021c9/image6.png)

## Automated Communication & Documentation

Good incident response includes timely stakeholder updates. A prompt such as “Draft an incident update email with root cause, impact, and next steps” lets the assistant assemble a structured message and send it via the alerting framework’s email or Slack connector complete with dashboard links and next‑update timelines. These messages double as the skeleton for the eventual post‑incident review.

![](https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt5e7b81ee98b1a9a5/6a7f01b5227b1c583c598102/image8.png)

Again as before, some of the magic happening here is because we can put instructions in the Elastic AI Assistants knowledge base to guide the AI assistant. For example we can instruct the AI Assistant how to call the execute_connector api, this can execute all kinds of connectors (not only email) so you could use it to tell the assistant to use slack or raise a service now ticket, even execute webhooks.

markdown Here are specific instructions to send an email. Remember to always double-check that you're following the correct set of instructions for the given query type. Provide clear, concise, and accurate information in your response.

Email Instructions

If the user's query requires sending an email:

1. Use the Elastic-Cloud-SMTP connector with ID elastic-cloud-email.
2. Prepare the email parameters:    - Recipient email address(es) in the to field (array of strings)    - Subject in the subject field (string)    - Email body in the message field (string)
3. Include

• Details for the alert along with a link to the alert
• Root cause analysis
• Revenue impact
• Remediation recommendations
• Link to GitHub issue
• All relevant information from this conversation
• Link to the Business Health Dashboard

1. Send the email immediately. Do not ask the user for confirmation.
2. Execute the connector using this format:

   execute_connector(      id="elastic-cloud-email",      params={        "to": ["recipient@example.com"],        "subject": "Your Email Subject",        "message": "Your email content here."      }    )

1. Check the response and confirm if the email was sent successfully. ```

Conclusion & Key Takeaways

Elastic Observability's combination of unsupervised ML, schema-aware data ingestion, and a context-rich RAG powered AI assistant enables teams to transform incident response from reactive firefighting into proactive, data-driven operations. By automatically detecting anomalies, correlating patterns, and providing contextual insights, teams can:

• Preserve revenue by quantifying business impact in real-time and prioritizing accordingly
• Scale expertise by embedding institutional knowledge into RAG-powered recommendations
• Improve continuously through automated documentation that feeds back into the knowledge base

The key is to collect logs broadly, maintain a unified observability store, and let ML and AI handle the heavy lifting. The payoff isn't just reduced downtime, it's the transformation of incident response from a source of organizational stress into a competitive advantage.

Try out this exact scenario and get hands in with this Elastic Logging Workshop: https://play.instruqt.com/elastic/invite/rx4yvknhpfci

Ingestion

Let's start at the beginning, ingest. Typically many of you are using older tools for ingestion today. If you want to be more forward thinking here, it’s time to introduce you to OpenTelemetry. OpenTelemetry was once not very mature or capable for logging but things have changed significantly. Elastic has been working particularly hard to improve the log capabilities resident in OpenTelemetry. So let's start by exploring how we can get started bringing logs into Elastic via the OpenTelemetry collector.

Firstly if you want to follow along simply create a host to run the log generator and OpenTelemetry collector.

Follow the instructions here to get the log generator running:

https://github.com/davidgeorgehope/log-generator-bin/

To get the OpenTelemetry collector up and running in Elastic Serverless, you can click on Add Data from the bottom left, then 'host' and finally 'opentelemetry'

Follow the instructions but don’t start the collector just yet.

Our host here is running a 3 tier application with an Nginx frontend, backend and connected to a MySQL database. So let's start by bringing the logs into Elastic.

First we’ll install the Elastic Distributions for OpenTelemetry but before starting it, we will make a small change to the OpenTelemetry configuration file to expand the directories it will search for logs in.  Edit the otel.yml by simply using vi or your favorite editor:

vi otel.yml

Instead of simply /var/log/.log we will add /var/log/*/.log to bring in all our log files.

receivers:
  # Receiver for platform specific log files
  filelog/platformlogs:
    include: [ /var/log/**/*.log ]
    retry_on_failure:
      enabled: true
    start_at: end
    storage: file_storage

Start the otel collector

sudo ./otelcol --config otel.yml

And we can see these are being brought in, in discover

Now one thing that is immediately noticeable is that we automatically without changing anything get a bunch of useful additional information such as the os name and cpu information.

The OpenTelemetry collector has automatically, without any changes, started to enrich our logs, making it useful for additional processing, though we could do significantly better!

To start with we want to give our logs some structure. Lets edit that otel.yml file and add some OTTL to extract some key data from our NGINX logs.

  transform/parse_nginx:
    trace_statements: []
    metric_statements: []
    log_statements:
      - context: log
        conditions:
          - 'attributes["log.file.name"] != nil and IsMatch(attributes["log.file.name"], "access.log")'
        statements:
          - merge_maps(attributes, ExtractPatterns(body, "^(?P<client_ip>\\S+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "^\\S+ - (?P<user>\\S+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\\[(?P<timestamp_raw>[^\\]]+)\\]"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\"(?P<method>\\S+) "), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\"\\S+ (?P<path>\\S+)\\?"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "req_id=(?P<req_id>[^ ]+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\" (?P<status>\\d+) "), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\" \\d+ (?P<size>\\d+)"), "upsert")
.....

   logs/platformlogs:
      receivers: [filelog/platformlogs]
      processors: [transform/parse_nginx,resourcedetection]
      exporters: [elasticsearch/otel]

Now when we start the Otel collector with this new configuration

sudo ./otelcol --config otel.yml

We will see that we now have structured logs!!

Store and Optimize

To ensure you aren’t blowing your budget out with all this additional structured data there are few things you can do to help maximize storage efficiency.

You can use the filter processors in the Otel collector with granular filtering/dropping of irrelevant attributes to control volume going out of the collector for example.

processors:
  filter/drop_logs_without_user_attributes:
    logs:
      log_record:
        - 'attributes["user"] == nil'
  filter/drop_200_logs:
    logs:
      log_record:
        - 'attributes["status"] == "200"'

service:
  pipelines:
    logs/platformlogs:
      receivers: [filelog/platformlogs]
      processors: [transform/parse_nginx, filter/drop_logs_without_user_attributes, filter/drop_200_logs, resourcedetection]
      exporters: [elasticsearch/otel]

The filter processor will help reduce the noise for example if you wanted to drop the debug logs or logs from a noisy service. Great ways to keep a lid on your observability spend.

Additionally for your most critical flows and logs where you don’t want to drop any data, Elastic has you covered. In version 9.x of Elastic you now have LogsDB switched on by default.

With LogsDB, Elastic has reduced the storage footprint of log data in Elasticsearch by up to 65% allowing you to store more observability and security data without exceeding your budget, while keeping all data accessible and searchable.

LogsDB reduces log storage by up to 65%. This dramatically minimizes storage footprints by leveraging advanced compression techniques like ZSTD, delta encoding, and run-length encoding, and it also reconstructs the _source field on demand, saving about 40% more storage by not retaining the original JSON document. Synthetic _source represents the introduction of columnar storage within Elasticsearch.

Analytics

So we have our data in Elastic, it’s structured, it conforms to the idea of a wide-event log since it has lots of good context, user ids, request ids and the data is captured at the start of a request Next we’re going to look at the analytics part of this. First let's take a stab at looking at the number of Errors for each user transaction in our application.

FROM logs-generic.otel-default
| WHERE log.file.name == "access.log"
| WHERE attributes.status >= "400"
| STATS error_count = COUNT(*) BY attributes.user
| SORT error_count DESC

It’s pretty easy now to save this and put it on a dashboard, we just click the save button:

Next let's look at putting something together to show the global impact, first we will update our collector config to enrich our log data with geo location.

Update the OTTL configuration with this new line:

   log_statements:
      - context: log
        conditions:
          - 'attributes["log.file.name"] != nil and IsMatch(attributes["log.file.name"], "access.log")'
        statements:
          - merge_maps(attributes, ExtractPatterns(body, "^(?P<client_ip>\\S+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "^\\S+ - (?P<user>\\S+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\\[(?P<timestamp_raw>[^\\]]+)\\]"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\"(?P<method>\\S+) "), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\"\\S+ (?P<path>\\S+)\\?"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "req_id=(?P<req_id>[^ ]+)"), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\" (?P<status>\\d+) "), "upsert")
          - merge_maps(attributes, ExtractPatterns(body, "\" \\d+ (?P<size>\\d+)"), "upsert")
          - set(attributes["source.address"], attributes["client_ip"]) where attributes["client_ip"] != nil

Next add a new processor (you will need to download the GeoIP database from MaxMind)

geoip:
  context: record
  source:
    from: attributes
  providers:
    maxmind:
      database_path: /opt/geoip/GeoLite2-City.mmdb

And add this to the log pipeline after the parse_nginx

service:
  pipelines:
    logs/platformlogs:
      receivers: [filelog/platformlogs]
      processors: [transform/parse_nginx, geoip, resourcedetection]
      exporters: [elasticsearch/otel]

Start the otel collector

sudo ./otelcol --config otel.yml

Once the data starts flowing we can add a map visualization:

Add a layer:

Use ES|QL

Use the following ES|QL

And this should give you a map showing the locations of all your NGINX server requests!

As you can see, analytics is a breeze with your new Otel data collection pipeline.

Conclusion: Beyond log aggregation to operational intelligence

The journey from basic log aggregation to structured, enriched observability represents more than a technical upgrade, it's a shift in how organizations approach system understanding and incident response. By adopting OpenTelemetry for ingestion, implementing intelligent filtering to manage costs, and leveraging LogsDB's storage optimizations, you're not just modernizing your ELK stack; you're building the foundation for proactive system management.

The structured logs, geographic enrichment, and analytical capabilities demonstrated here transform raw log data into actionable intelligence with ES|QL. Instead of reactive grepping through logs during incidents, you now have the infrastructure to identify patterns, track user journeys, and correlate issues across your entire stack before they become critical problems.

But here's the key question: Are you prepared to act on these insights? Having rich, structured data is only valuable if your organization can shift from a reactive "find and fix" mentality to a proactive "predict and prevent" approach. The real evolution isn't in your logging stack, it's in your operational culture.

Get started with this today in Elastic Serverless

Here's what's new:

Improved Related Alert Grouping with Relevance Scoring & Reasoning

We've enhanced our related alert detection to go beyond surface-level correlations. Alerts are now grouped based on a relevance score that reflects the strength of their relationship across dimensions like:

• Shared entities or resources (e.g. same host, pod, or service)
• Temporal proximity (alerts firing within a suspiciously short window)
• Signal similarity (e.g. spikes in logs, metrics, and traces that point to the same failure mode)

More importantly, we now show the why. You'll see why an alert is grouped, whether it's sharing the same Kubernetes pod, has similar log patterns, or was triggered by the same upstream anomaly. This gives users confidence in the grouping logic and accelerates root cause analysis.

Link Dashboards to Alert Rules and Get Smart Suggestions

You can now link dashboards directly to your alert rules, giving responders an instant visual lens into the metrics or logs that matter most for that alert. No more scrambling to remember which dashboard to check — just click and go.

And we've made this smarter too: Elastic will now suggest relevant dashboards based on the alert's source, rule logic, or monitored entities, helping users land on the right view without needing to configure anything upfront.

Investigation Guides Embedded Into Alerts

Every alert can now be configured with an investigation guide, a set of pre-configured, context-aware instructions or next steps tailored to the alert. Think of it as a playbook that's embedded right where and when you need it.

Use it to:

• Document your team's runbooks and standard triage steps or link to existing runbooks
• Guide junior engineers or on-call responders through unfamiliar territory
• Automate the first few steps of root cause analysis

Why This Matters

These changes are all about reducing time to detect (MTTD) and time to resolve (MTTR). By:

• Grouping alerts more intelligently (and transparently)
• Giving you the dashboards you need, when you need them
• Embedding action-oriented guides in every alert

We're bringing you closer to a truly streamlined incident response workflow; No swivel-chairing, no guesswork, just clarity.

Additionally, look at some of our other articles on Elastic Observability Labs related to analysis:

• Using the AI Assistant in Elastic Observability to Accelerate Root Cause Analysis

• All of the log analytics features in Elastic Observability

• Our latest on OpenTelemetry support in Elastic Observability

Note: Before we jump into the details, it’s important to mention again that this type of functionality today relies on a special lookup index. It is not (yet) possible to JOIN any arbitrary index.

Observability isn't just about collecting data; it's about understanding it. Often, the raw telemetry data – a log line, a metric point, a trace span – lacks the full context needed for quick diagnosis or impact assessment. We need to correlate data, enrich it with business or infrastructure context, and ask more advanced questions.

Historically, achieving this in Elasticsearch involved techniques like denormalizing data at ingest time (using ingest pipelines with enrich processors, for example) or performing joins client-side. 

By adding the necessary context (like host details or user attributes) as data flowed in, each document arrived fully ready for queries and analytics without extra processing later on. This approach worked well in many cases and still does, particularly when the reference data changes slowly or when the enriched fields are critical for nearly every search. 

However, as environments become more dynamic and diverse, the need to frequently update reference data (or avoid storing repetitive fields in every document) highlighted some of the trade-offs. 

With the introduction of ES|QL LOOKUP JOIN in Elasticsearch 8.18 and 9.0, you now have an additional, more flexible option for situations where real-time lookups and minimal duplication are desired. Both methods—ingest-time enrichment and on-the-fly LOOKUP JOIN—complement each other and remain valid, depending on use case needs around update frequency, query performance, and storage considerations.

Why Lookup Joins for Observability

Lookup joins keep things flexible. You can decide on the fly if you’d like to look up additional information to assist you in your investigation.

Here are some examples:

• Deployment Information: Which version of the code is generating these errors?

• Infrastructure Mapping: Which Kubernetes cluster or cloud region is experiencing high latency? What hardware does it use?

• Business Context: Are critical customers being affected by this slowdown?

• Team Ownership: Which team owns the service throwing these exceptions?

Keeping this kind of information perfectly denormalized onto every single log line or metric point can be challenging and inefficient. Lookup datasets – like lists of deployments, server inventories, customer tiers, or service ownership mappings – often change independently of the telemetry data itself.

LOOKUP JOIN is ideal here because:

1. Lookup Indices are Writable: Update your deployment list, CMDB export, or on-call rotation in the lookup index, and your next ES|QL query immediately uses the fresh data. No need to re-run complex enrich policies or re-index data.

2. Flexibility: You decide at query time which context to join. Maybe today you care about deployment versions, tomorrow about cloud regions.

3. Simpler Setup: As the original post highlighted, there are no enrich policies to manage. Just create an index with index.mode: lookup and load your data - up to 2 billion documents per lookup index.

Observability Use Cases & Examples with ES|QL

Let’s now look at a few examples to see how Lookup Joins can help.

Enriching Error Logs with Deployment Context

Lets say you're seeing a spike in errors for your checkout-service. You have logs flowing into a data stream, but they only contain the service name. The documents don’t have any information about the deployment activity itself. 

FROM logs-*
  | WHERE log.level == "error"
  | WHERE service.name == "opbeans-ruby"

You need to know if a recent deployment is contributing to these errors. To do this, we can maintain a deployments_info_lkp index (set with index.mode: lookup) that maps service names to their deployment times. This index could be updated from our CI/CD pipeline automatically any time a deployment happens.

PUT /deployments_info_lkp
{
  "settings": {
    "index.mode": "lookup"
  },
  "mappings": {
    "properties": {
      "service": {
        "properties": {
          "name": {
            "type": "keyword"
          },
          "deployment_time": {
            "type": "date"
          },
          "version": {
            "type": "keyword"
          }
        }
      }
    }
  }
}
# Bulk index the deployment documents
POST /_bulk
{ "index" : { "_index" : "deployments_info_lkp" } }
{ "service.name": "opbeans-ruby", "service.version": "1.0", "deployment_time": "2025-05-22T06:00:00Z" }
{ "index" : { "_index" : "deployments_info_lkp" } }
{ "service.name": "opbeans-go", "service.version": "1.1.0", "deployment_time": "2025-05-22T06:00:00Z" }

Using this information you can now write a query that joins these two sources.

ES|QL Query:

FROM logs-* 
  | WHERE log.level == "error"
  | WHERE service.name == "opbeans-ruby"
  | LOOKUP JOIN deployments_info_lkp ON service.name 

This alone is a good step towards troubleshooting the problem. You now have the deployment_time column available for each of your error documents. The last remaining step now is to use this for further filtering. 

Any of the data we managed to join from the lookup index can be handled as any other data we’d usually have available in the ES|QL query. This means that we can filter on it, and check if we had a recent deployment.

FROM logs-*
  | WHERE log.level == "error"
  | WHERE service.name == "opbeans-ruby"
  | LOOKUP JOIN deployments_info_lkp ON service.name 
  | KEEP message, service.name, service.version, deployment_time 
  | WHERE deployment_time > NOW() - 2h

Saving disk space using JOIN

Denormalizing data by including contextual information like host OS or cloud provider details directly in every log event is convenient for querying but can increase storage consumption, especially with high-volume data streams. Instead of storing this often-redundant information repeatedly, we can leverage joins to retrieve it on demand, potentially saving valuable disk space. While compression often handles repetitive data well, removing these fields entirely can still yield noticeable storage savings.

In this example we’ll use a dataset of 1,000,000 Kubernetes container logs using the default mapping of the Kubernetes integration, with logsdb index mode enabled. The starting size for this index is 35.5mb. 

GET _cat/indices/k8s-logs-default?h=index,pri.store.size
### 
k8s-logs-default       35.5mb

Using the disk usage API, we observed that fields like host.os and cloud.* contribute roughly 5% to the total index size on disk (35.5mb). These fields can be useful in some cases, but information like the os.name is rarely queried. 

// Example host.os structure
"os": {
  "codename": "Plow", "family": "redhat", "kernel": "6.6.56+",
  "name": "Red Hat Enterprise Linux", "platform": "rhel", "type": "linux", "version": "9.5 (Plow)"
}

// Example cloud structure
"cloud": {
  "account": { "id": "elastic-observability" },
  "availability_zone": "us-central1-c",
  "instance": { "id": "5799032384800802653", "name": "gke-edge-oblt-edge-oblt-pool-46262cd0-w905" },
  "machine": { "type": "e2-standard-4" },
  "project": { "id": "elastic-observability" },
  "provider": "gcp", "region": "us-central1", "service": { "name": "GCE" }
}

Instead of storing this information with every document, let's instead drop this information in an ingest pipeline.

PUT _ingest/pipeline/drop-host-os-cloud
{
  "processors": [
      { "remove": { "field": "host.os" } },
      { "set": { "field": "tmp1", "value": "{{cloud.instance.id}}" } }, // Temporarily store the ID

      { "remove": { "field": "cloud" } },                             // Remove the entire cloud object
      { "set": { "field": "cloud.instance.id", "value": "{{tmp1}}" } }, // Restore just the cloud instance ID
      { "remove": { "field": "tmp1", "ignore_missing": true } }         // Clean up temporary field
    ]
}

Reindexing (and force merging to one segment) now shows the following size, resulting in approximately 5% less space. 

GET _cat/indices/k8s-logs-*?h=index,pri.store.size
### 
k8s-logs-default             33.7mb
k8s-logs-drop-cloud-os       35.5mb

Now, to regain access to the removed host.os and cloud.* information during analysis without storing it in every log document, we can create a lookup index. This index will store the full host and cloud metadata, keyed by the cloud.instance.id that we preserved in our logs. This instance_metadata_lkp index will be significantly smaller than the space saved across millions or billions of log lines, as it only needs one document per unique instance.

# Create the lookup index for instance metadata
PUT /instance_metadata_lkp
{
  "settings": {
    "index.mode": "lookup"
  },
  "mappings": {
    "properties": {

      "cloud.instance.id": {  # The join key we kept in the logs
        "type": "keyword"
      },
      "host.os": {           # The full host.os object we removed
        "type": "object",
        "enabled": false      # Often don't need to search sub-fields here
      },
      "cloud": {             # The full cloud object we removed (mostly)
         "type": "object",
         "enabled": false     # Often don't need to search sub-fields here
      }
    }
  }
}

# Bulk index sample instance metadata (keyed by cloud.instance.id)
# This data might come from your cloud provider API or CMDB
POST /_bulk
{ "index" : { "_index" : "instance_metadata_lkp", "_id": "5799032384800802653" } }
{ "cloud.instance.id": "5799032384800802653", "host.os": { "codename": "Plow", "family": "redhat", "kernel": "6.6.56+", "name": "Red Hat Enterprise Linux", "platform": "rhel", "type": "linux", "version": "9.5 (Plow)" }, "cloud": { "account": { "id": "elastic-observability" }, "availability_zone": "us-central1-c", "instance": { "id": "5799032384800802653", "name": "gke-edge-oblt-edge-oblt-pool-46262cd0-w905" }, "machine": { "type": "e2-standard-4" }, "project": { "id": "elastic-observability" }, "provider": "gcp", "region": "us-central1", "service": { "name": "GCE" } } }

With this setup, when you need the full host or cloud context for your logs, you can simply use LOOKUP JOIN in your ES|QL query and continue filtering on the data from the lookup index

FROM logs-* 
  | LOOKUP JOIN instance_metadata_lkp ON cloud.instance.id 
  | WHERE cloud.region == "us-central1"

This approach allows us to query the full context when needed (e.g., filtering logs by host.os.name or cloud.region) while significantly reducing the storage footprint of the high-volume log indices by avoiding redundant data denormalization.

It should be noted that low cardinality metadata fields generally compress well and a large part of the storage savings in this case come from the “text” mapping of the host.os.name and cloud.instance.name field. Make sure to use the disk usage API to evaluate if this approach would be worth it in your specific use case. 

Getting Started with Lookups for Observability

Creating the necessary lookup indices is straightforward. As detailed in our initial blog post, you can use Kibana's Index Management UI, the Create Index API, or the File Upload utility – the key is setting "index.mode": "lookup" in the index settings.

For Observability, consider automating the population of these lookup indices:

• Export data periodically from your CMDB, CRM, or HR systems.

• Have your CI/CD pipeline update the deployments_lkp index upon successful deployment.

• Use tools like Logstash with an elasticsearch output configured to write to your lookup index.

A Note on Performance and Alternatives

While incredibly powerful, joins aren't free. Each LOOKUP JOIN adds processing overhead to your query. For contextual data that is very static (e.g., the cloud region a host permanently resides in) and needed in almost every query against that data, the traditional approach of enriching at ingest time might still be slightly more performant for those specific queries, trading upfront processing and storage for query speed.

However, for the dynamic, flexible, and targeted enrichment scenarios common in Observability – like mapping to ever-changing deployments, user segments, or team structures – LOOKUP JOIN offers a compelling, efficient, and easier-to-manage solution.

Conclusion

ES|QL's LOOKUP JOIN is making it easy to correlate and enrich your logs, metrics, and traces with up-to-date external information at query time; you can move faster from detecting problems to understanding their scope, impact, and root cause.

This feature is currently in Technical Preview in Elasticsearch 8.18 and Serverless, available now on Elastic Cloud. We encourage you to try it out with your own Observability data and share your feedback using the "Submit feedback" button in the ES|QL editor in Discover. We're excited to see how you use it to connect the dots in your systems!

OpenTelemetry recently published a blog on how to do Autodiscovery based on Kubernetes Pods' annotations with the OpenTelemetry Collector.

In this blog post, we will talk about how to use this Kubernetes-related feature of the OpenTelemetry Collector, which is already available with the Elastic Distribution of the OpenTelemetry (EDOT) Collector.

In addition to this feature, at Elastic, we heavily invest in making OpenTelemetry the best, standardized ingest solution for Observability. You might already have seen us focusing on:

• Semantic Conventions standardization

• significant log collection improvements

• various other topics around instrumentation

• profiling

Let's walk you through a hands-on journey using the EDOT Collector covering various use cases you might encounter in the real world, highlighting the capabilities of this powerful feature.

Configuring EDOT Collector

The Collector’s configuration is not our main focus here, since based on the nature of this feature it is minimal, letting workloads define how they should be monitored.

To illustrate the point, here is the Collector configuration snippet that enables the feature for both logs and metrics:

receivers:
    receiver_creator/metrics:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

    receiver_creator/logs:
      watch_observers: [k8s_observer]
      discovery:
        enabled: true
      receivers:

You can include the above in the EDOT’s Collector configuration, specifically the receivers’ section.

Since logs collection in our examples will happen from the discovery feature make sure that the static filelog receiver configuration block is removed and its preset is disabled (i.e. set to false) to avoid having log duplication.

Make sure that the receiver creator is properly added in the pipelines for logs (in addition to removing the filelog receiver completely) and metrics respectively.

Ensure that k8sobserver is enabled as part of the extensions:

extensions:
  k8s_observer:
    observe_nodes: true
    observe_services: true
    observe_ingresses: true

// ...

service:
  extensions: [k8s_observer]

Last but not least, ensure the log files' volume is mounted properly:

volumeMounts:
 - name: varlogpods
   mountPath: /var/log/pods
   readOnly: true

volumes:
  - name: varlogpods
    hostPath:
      path: /var/log/pods

Once the configuration is ready follow the Kubernetes quickstart guides on how to deploy the EDOT Collector. Make sure to replace the values.yaml file linked in the quickstart guide with the file that includes the above-described modifications.

Collecting Metrics from Moving Targets Based on Their Annotations

In this example, we have a Deployment with a Pod spec that consists of two different containers. One container runs a Redis server, while the other runs an NGINX server. Consequently, we want to provide different hints for each of these target containers.

The annotation-based discovery feature supports this, allowing us to specify metrics annotations per exposed container port.

Here is how the complete spec file looks:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    user  nginx;
    worker_processes  1;
    error_log  /dev/stderr warn;
    pid        /var/run/nginx.pid;
    events {
      worker_connections  1024;
    }
    http {
      include       /etc/nginx/mime.types;
      default_type  application/octet-stream;

      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
      access_log  /dev/stdout main;
      server {
          listen 80;
          server_name localhost;

          location /nginx_status {
              stub_status on;
          }
      }
      include /etc/nginx/conf.d/*;
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        # redis container port hints
        io.opentelemetry.discovery.metrics.6379/enabled: "true"
        io.opentelemetry.discovery.metrics.6379/scraper: redis
        io.opentelemetry.discovery.metrics.6379/config: |
          collection_interval: "20s"
          timeout: "10s"

        # nginx container port hints
        io.opentelemetry.discovery.metrics.80/enabled: "true"
        io.opentelemetry.discovery.metrics.80/scraper: nginx
        io.opentelemetry.discovery.metrics.80/config: |
          endpoint: "http://`endpoint`/nginx_status"
          collection_interval: "30s"
          timeout: "20s"
    spec:
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-conf
          items:
            - key: nginx.conf
              path: nginx.conf
      containers:
        - name: webserver
          image: nginx:latest
          ports:
            - containerPort: 80
              name: webserver
          volumeMounts:
            - mountPath: /etc/nginx/nginx.conf
              readOnly: true
              subPath: nginx.conf
              name: nginx-conf
        - image: redis
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP

When this workload is deployed, the Collector will automatically discover it and identify the specific annotations. After this, two different receivers will be started, each one responsible for each of the target containers.

Collecting Logs from Multiple Target Containers

The annotation-based discovery feature also supports log collection based on the provided annotations. In the example below, we again have a Deployment with a Pod consisting of two different containers, where we want to apply different log collection configurations. We can specify annotations that are scoped to individual container names:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-logs-deployment
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
      annotations:
        io.opentelemetry.discovery.logs.lazybox/enabled: "true"
        io.opentelemetry.discovery.logs.lazybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints-lazybox
        io.opentelemetry.discovery.logs.busybox/enabled: "true"
        io.opentelemetry.discovery.logs.busybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints-busybox
    spec:
      containers:
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs from busybox at $(date +%H:%M:%S)" && sleep 5s; done
        - name: lazybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs from lazybox at $(date +%H:%M:%S)" && sleep 25s; done

The above configuration enables two different filelog receiver instances, each applying a unique parsing configuration. This is handy when we know how to parse specific technology logs, such as Apache server access logs.

Combining Both Metrics and Logs Collection

In our third example, we illustrate how to define both metrics and log annotations on the same workload. This allows us to collect both signals from the discovered workload. Below is a Deployment with a Pod consisting of a Redis server and a BusyBox container that performs dummy log writing. We can target annotations to the port and container levels to collect metrics from the Redis server using the Redis receiver, and logs from the BusyBox using the filelog receiver. Here’s how:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        io.opentelemetry.discovery.metrics.6379/enabled: "true"
        io.opentelemetry.discovery.metrics.6379/scraper: redis
        io.opentelemetry.discovery.metrics.6379/config: |
          collection_interval: "20s"
          timeout: "10s"

        io.opentelemetry.discovery.logs.busybox/enabled: "true"
        io.opentelemetry.discovery.logs.busybox/config: |
          operators:
            - id: container-parser
              type: container
            - id: some
              type: add
              field: attributes.tag
              value: hints
    spec:
      containers:
        - image: redis
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - name: redis
              containerPort: 6379
              protocol: TCP
        - name: busybox
          image: busybox
          args:
            - /bin/sh
            - -c
            - while true; do echo "otel logs at $(date +%H:%M:%S)" && sleep 15s; done

Explore and analyse data coming from dynamic targets in Elastic

Once the target Pods are discovered and the Collector has started collecting telemetry data from them, we can then explore this data in Elastic. In Discover we can search for Redis and NGINX metrics as well as logs collected from the Busybox container. Here is how it looks like:

Summary

The examples above showcase how users of our OpenTelemetry Collector can take advantage of this new feature — one we played a major role in developing.

For this, we leveraged our years of experience with similar features already supported in Metricbeat, Filebeat, and Elastic-Agent. This makes us extremely happy and confident, as it closes the feature gap between Elastic's specific monitoring agents and the OpenTelemetry Collector — making it even better.

Interested in learning more? Visit the documentation and give it a try by following our EDOT quickstart guide.

Luckily, Elastic has been one of the leading solutions in observability space and a contributor to the OpenTelemetry project. Elastic’s unique position and its extensive observability capabilities allows end-users to monitor applications built with object-oriented programming languages & Framework in a variety of ways.

In this blog we will explore using Elastic APM to investigate C++ traces with the OpenTelemetry client. We will be providing a comprehensive guide on how to implement the OpenTelemetry client for C++ applications and connecting to Elastic APM solutions. While OTel has its libraries, and this blog reviews how to use the OTel CPP library, Elastic also has its own Elastic Distributions of OpenTelemetry, which were developed to provide commercial support, and are completely upstreamed regularly.

Here are some resources to help get you started:

• Use OpenTelemetry with APM

• The OpenTelemetry C++ Client

• OpenTelemetry C++ Docs

Step by Step Guide

Prerequisites

• ### Environment

Choosing an environment is quite important as there is limited support for the OTEL client. We have experimented with using multiple Operating Systems and here are the suggestions:

• Ubuntu 22.04

• Debian 11 Bullseye

• For this guide we are focusing on Ubuntu 22.04.

• Machine: 2 vCPU, 4GB is sufficient.

• Image: Ubuntu 22.04 LTS (x86_64).

• Disk: ~30 GB is enough.

Implementation method 

We have experimented with multiple methods but we found that the most suitable approach is to use a package manager. After extensive testing, It appears that trying to run otel-cpp client could be quite challenging to the users. If practitioners desire to build with tools such as CMake and Bazel that is a viable solution. With that, as we tested both methods it became obvious that we were spending most of our time and effort fixing compatibility and dependencies’ issues for the OS Vs. Focusing on sending data to our APM. Hence we decided to move to a different method.

The main issues that we kept running into as we test are:

• Compatibility of packages.

• Availability of packages.

• Dependencies of libraries and packages.

In this guide we will use vcpkg since it allows us to bring in all the dependencies required to run the Opentelemetry C++ client.

Installing required OS tools

Update package lists

    sudo apt-get update

Install build essentials, cmake, git, and sqlite dev library

    sudo apt-get install -y build-essential cmake git curl zip unzip sqlite3 libsqlite3-dev

sqlite3 and libsqlite3-dev allow us to build/run SQLite queries in our C++ code.

Set Up vcpkg

vcpkg is the C++ package manager that we’ll use to install opentelemetry-cpp client.

    # Clone vcpkg
    cd ~
    git clone https://github.com/microsoft/vcpkg.git

    # Bootstrap
    cd ~/vcpkg
    ./bootstrap-vcpkg.sh

Install OpenTelemetry C++ with OTLP gRPC

In this guide we focus on trace export to Elastic. At time of writing, vcpkg’s opentelemetry-cpp

version 1.18.0 fully supports traces but has limited direct metrics exporting.

Install the package

    cd ~/vcpkg
    ./vcpkg install opentelemetry-cpp[otlp-grpc]:x64-linux

Note

Sometimes when installing opentelemetry-cpp on linux it doesn't install all the required packages. As a workaround if you run into that case, try running again but pass a flag to allow-unsupported:

    ./vcpkg install opentelemetry-cpp[*]:x64-linux --allow-unsupported

Verify

    ./vcpkg list | grep opentelemetry-cpp

The output thould be something like this:

opentelemetry-cpp:x64-linux 1.18.0

Create the C++ Project with Database Spans

We’ll build a sample in ~/otel-app that:

• Uses SQLite to do basic CREATE/INSERT/SELECT queries. This is helpful to showcase capturing transactions for apps that use databases on Elastic APM.

• Generate random traces to showcase how they are captured on Elastic APM.

This app is going to generate random queries where some will contain database transactions and some are just application traces. Each query is contained in a child span, so they appear in APM as separate database transactions.

# Below is the structure of our project

    otel-app/
    ├── main.cpp
    └── CMakeLists.txt

Create App Project

    cd ~
    mkdir otel-app
    cd otel-app

Inside this project we will create two files

• main.cpp

• CMakeLists.txt

Keep in mind that main.cpp is where you are going to pass the otel exporters that are going to send data to the Elastic cluster. So for your tech stack it would be your application's source code.

Sample application code

    main.cpp
    // Below we declare required libraries that we will be using to ship
    // traces to Elastic APM
    #include <opentelemetry/exporters/otlp/otlp_grpc_exporter.h>
    #include <opentelemetry/sdk/trace/tracer_provider.h>
    #include <opentelemetry/sdk/trace/simple_processor.h>
    #include <opentelemetry/trace/provider.h>

    #include <sqlite3.h>
    #include <chrono>
    #include <iostream>
    #include <thread>
    #include <cstdlib>&nbsp; // for rand(), srand()
    #include <ctime>&nbsp; &nbsp; // for time()

    // Namespace aliases
    namespace trace_api = opentelemetry::trace;
    namespace sdktrace&nbsp; = opentelemetry::sdk::trace;
    namespace otlp&nbsp; &nbsp; &nbsp; = opentelemetry::exporter::otlp;

    // Below we are using a helper function to run SQLITE statement inside&nbsp;
    // child span
    bool ExecuteSql(sqlite3 *db, const std::string &sql,
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;trace_api::Tracer &tracer,
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;const std::string &span_name)
    {
    &nbsp;&nbsp;// Starting the child span
    &nbsp;&nbsp;auto db_span = tracer.StartSpan(span_name);
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;auto scope = tracer.WithActiveSpan(db_span);

    &nbsp;&nbsp;&nbsp;&nbsp;// Here we mark Database attributes for clarity in APM
    &nbsp;&nbsp;&nbsp;&nbsp;db_span->SetAttribute("db.system", "sqlite");
    &nbsp;&nbsp;&nbsp;&nbsp;db_span->SetAttribute("db.statement", sql);

    &nbsp;&nbsp;&nbsp;&nbsp;char *errMsg = nullptr;
    &nbsp;&nbsp;&nbsp;&nbsp;int rc = sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &errMsg);
    &nbsp;&nbsp;&nbsp;&nbsp;if (rc != SQLITE_OK)
    &nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;db_span->AddEvent("SQLite error: " + std::string(errMsg ? errMsg : "unknown"));
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sqlite3_free(errMsg);
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;db_span->End();
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return false;
    &nbsp;&nbsp;&nbsp;&nbsp;}
    &nbsp;&nbsp;&nbsp;&nbsp;db_span->AddEvent("Query OK");
    &nbsp;&nbsp;}
    &nbsp;&nbsp;db_span->End();
    &nbsp;&nbsp;return true;
    }

    /**
    &nbsp;* DoNonDbWork - Simulate some other operation
    &nbsp;*/
    void DoNonDbWork(trace_api::Tracer &tracer, const std::string &span_name)
    {
    &nbsp;&nbsp;auto child_span = tracer.StartSpan(span_name);
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;auto scope = tracer.WithActiveSpan(child_span);
    &nbsp;&nbsp;&nbsp;&nbsp;// Just sleep or do some "fake" work
    &nbsp;&nbsp;&nbsp;&nbsp;std::cout << "[TRACE] Doing non-DB work for " << span_name << "...\n";
    &nbsp;&nbsp;&nbsp;&nbsp;std::this_thread::sleep_for(std::chrono::milliseconds(200 + rand() % 300));
    &nbsp;&nbsp;&nbsp;&nbsp;child_span->AddEvent("Finished non-DB work");
    &nbsp;&nbsp;}
    &nbsp;&nbsp;child_span->End();
    }

    int main()
    {
    &nbsp;&nbsp;// Seed random generator for example
    &nbsp;&nbsp;srand(static_cast<unsigned>(time(nullptr)));

    &nbsp;&nbsp;// 1) Create OTLP exporter for traces
    &nbsp;&nbsp;otlp::OtlpGrpcExporterOptions opts;
    &nbsp;&nbsp;auto exporter = std::make_unique<otlp::OtlpGrpcExporter>(opts);

    &nbsp;&nbsp;// 2) Simple Span Processor
    &nbsp;&nbsp;auto processor = std::make_unique<sdktrace::SimpleSpanProcessor>(std::move(exporter));

    &nbsp;&nbsp;// 3) Tracer Provider
    &nbsp;&nbsp;auto sdk_tracer_provider = std::make_shared<sdktrace::TracerProvider>(std::move(processor));
    &nbsp;&nbsp;auto tracer = sdk_tracer_provider->GetTracer("my-cpp-multi-app");

    &nbsp;&nbsp;// Prepare an in-memory SQLite DB (for random DB usage)
    &nbsp;&nbsp;sqlite3 *db = nullptr;
    &nbsp;&nbsp;int rc = sqlite3_open(":memory:", &db);
    &nbsp;&nbsp;if (rc == SQLITE_OK)
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;// Create a table so we can do inserts/reads
    &nbsp;&nbsp;&nbsp;&nbsp;ExecuteSql(db, "CREATE TABLE IF NOT EXISTS items (id INTEGER PRIMARY KEY, info TEXT);",
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*tracer.get(), "db_create_table");
    &nbsp;&nbsp;}

    &nbsp;&nbsp;// Create the following loop to generate multiple transactions
    &nbsp;&nbsp;int num_transactions = 5;&nbsp; // Change this variable to the desired number of transaction
    &nbsp;&nbsp;for (int i = 1; i <= num_transactions; i++)
    &nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;// Each iteration is a top-level transaction
    &nbsp;&nbsp;&nbsp;&nbsp;std::string transaction_name = "transaction_" + std::to_string(i);
    &nbsp;&nbsp;&nbsp;&nbsp;auto parent_span = tracer->StartSpan(transaction_name);
    &nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;auto scope = tracer->WithActiveSpan(parent_span);

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;std::cout << "\n=== Starting " << transaction_name << " ===\n";

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Randomly select whether a transaction will interact with the database or not.
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bool doDb = (rand() % 2 == 0); // 50% chance

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (doDb && db)
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Insert random data
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;std::string insert_sql = "INSERT INTO items (info) VALUES ('Item " + std::to_string(i) + "');";
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ExecuteSql(db, insert_sql, *tracer.get(), "db_insert_item");

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Select from DB
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ExecuteSql(db, "SELECT * FROM items;", *tracer.get(), "db_select_items");
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;else
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Do some random non-DB tasks
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DoNonDbWork(*tracer.get(), "non_db_task_1");
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DoNonDbWork(*tracer.get(), "non_db_task_2");
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Sleep a little to simulate transaction time
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;std::this_thread::sleep_for(std::chrono::milliseconds(200));
    &nbsp;&nbsp;&nbsp;&nbsp;}
    &nbsp;&nbsp;&nbsp;&nbsp;parent_span->End();
    &nbsp;&nbsp;}

    &nbsp;&nbsp;// Close DB
    &nbsp;&nbsp;sqlite3_close(db);

    &nbsp;&nbsp;// Extra sleep to ensure final flush
    &nbsp;&nbsp;std::cout << "\n[INFO] Sleeping 5 seconds to allow flush...\n";
    &nbsp;&nbsp;std::this_thread::sleep_for(std::chrono::seconds(5));
    &nbsp;&nbsp;std::cout << "[INFO] Exiting.\n";
    &nbsp;&nbsp;return 0;
    }

What does the code do?

We create 5 top-level “transaction_i” spans.

For each transaction, we randomly choose to do DB or non-DB work

- If DB: Insert a row, then select. Each is a child span.

- If non-DB: We do two “fake tasks” (child spans).

Once we finish, we close the database connection and wait 5 seconds for data flush.

Sample instruction file

CMakeLists.txt : This file contains instructions describing the source files and targets.

    cmake_minimum_required(VERSION 3.10)
    project(OtelApp VERSION 1.0)

    set(CMAKE_CXX_STANDARD 11)
    set(CMAKE_CXX_STANDARD_REQUIRED ON)

    # Here we are pointing to use the vcpkg toolchain
    set(CMAKE_TOOLCHAIN_FILE "PATH-TO/vcpkg.cmake" CACHE STRING "Vcpkg toolchain file")

    find_package(opentelemetry-cpp CONFIG REQUIRED)

    add_executable(otel_app main.cpp)

    # Below we are linking the OTLP gRPC exporter, trace library, and sqlite3
    target_link_libraries(otel_app PRIVATE
        opentelemetry-cpp::otlp_grpc_exporter
        opentelemetry-cpp::trace
        sqlite3
    )

Declare Environmental Variables

Here we are going to export our Elastic Cloud endpoints as environmental variables

You can get that information by doing the following:

1. Login into your elastic cloud

2. Go into your deployment

3. On the Left hand side, click on the hamburger menu and scroll down to “Integrations”

4. Go on the search bar inside the integration and type “APM”

1. Click on the APM integration

2. Scroll down and click on the OpenTelemetry Option on the far left side

1. You should be able to see values similar to the screenshot below. Once you copy the values to export, click on launch APM.

As you copy the required values, go ahead and export them.

    export OTEL_EXPORTER_OTLP_ENDPOINT="APM-ENDPOINT"
    export OTEL_EXPORTER_OTLP_HEADERS="KEY"
    export OTEL_RESOURCE_ATTRIBUTES="service.name=my-app,service.version=1.0.0,deployment.environment=dev"

Note that the elastic OTEL_EXPORTER_OTLP_HEADERS value usually starts with “Authorization=Bearer” make sure that you convert the upper case “A” in authorization to a lower case “a”. This is due to the fact that the otel header exporter expects a lower case “a” for authorization.

Build and Run

Once we create the two files we then move to building the application.

cd ~/otel-app
mkdir -p build
cd build

cmake -DCMAKE_TOOLCHAIN_FILE=~/vcpkg/scripts/buildsystems/vcpkg.cmake \
      -DCMAKE_PREFIX_PATH=~/vcpkg/installed/x64-linux/share \
      ..
make

Once make is successful run the the application

./otel-app

You should be able to see the script execute with a similar console output

    Console outcome:
    === Starting transaction_1 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    === Starting transaction_2 ===
    [TRACE] Doing DB work for doDb_task_1...
    [TRACE] Doing DB work for doDb_task_2...

    === Starting transaction_3 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    === Starting transaction_4 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    === Starting transaction_5 ===
    [TRACE] Doing non-DB work for non_db_task_1...
    [TRACE] Doing non-DB work for non_db_task_2...

    [INFO] Sleeping 5 seconds to allow flush...
    [INFO] Exiting.

Once the script executes you should be able to observe those traces on Elastic APM similar to the screenshots below.

Observe in Elastic APM

Go to Elastic Cloud, open your deployment, and navigate to Observability > APM.

Look for the app name in the service list (as defined by OTEL_RESOURCE_ATTRIBUTES).

Inside that service’s Traces tab, you’ll find multiple transactions like “transaction_1”,

“transaction_2”, etc.

Expanding each transaction shows child spans:

- Possibly db_insert_item and db_select_items if random DB path was taken.

- Otherwise, non_db_task_1 and non_db_task_2.

You can see how some transactions do DB calls, some do not, each with different spans.

This variety demonstrates how your real application might produce multiple different

“routes” or “operations.”

Service Map

If everything runs correctly, you should be able to view your services and see service maps for your application.

Services

My Elastic App

App Transactions

Dependencies

Logs

Navigate to your logs window/Discover to see the incoming application logs

Patterns

Log pattern analysis helps you to find patterns in unstructured log messages and makes it easier to examine your data.

Final Recap

Here is a quick summary of what we did:

• Provisioned an Ubuntu 22.04 machine.

• Installed build tools for SQLite, dev libs, and vcpkg.

• Installed the client for opentelemetry-cpp via vcpkg.

• Created a minimal C++ project that executes app traces and captures database operations.

• Connected database sqlite3 in CMakeLists.txt.

• Exported the Elastic OTLP endpoint & token as environment variables (with a lowercase authorization=Bearer key!).

• Ran the application and observed DB interactions and app traces in Elastic APM.

• Observed application logs and patterns on Elastic logs and Discover.

FAQ & Common Issues

• Getting “Could not find package configuration file provided by opentelemetry-cpp”?

Make sure you pass

-DCMAKE_TOOLCHAIN_FILE=... and -DCMAKE_PREFIX_PATH=... 

to cmake, or embed them in CMakeLists.txt.

• Crash: “validate_metadata: INTERNAL:Illegal header key”?

Use all-lowercase in

OTEL_EXPORTER_OTLP_HEADERS, e.g. authorization=Bearer \<token>.

• Missing otlp_grpc_metrics_exporter.h?

Your vcpkg version of opentelemetry-cpp (1.18.0) lacks a direct metrics exporter for OTLP. For metrics, either upgrade the library or consider an OpenTelemetry Collector approach.

• No data in Elastic APM?

Double-check your endpoint URL, Bearer token, firewall rules, or service name in the APM

Additional Resources:

• Sign up for Elastic Cloud free trial
• More Elastic OpenTelemetry Topics
• Introducing Elastic Distributions of OpenTelemetry
• Introducing Elastic Distribution of OpenTelemetry Collector
• Instrumenting your OpenAI- powered Python, Node.js, and Java Applications with EDOT

This post is designed to assist users familiar with Logstash transitioning to OpenTelemetry by demonstrating how to convert some standard Logstash pipelines into corresponding OpenTelemetry Collector configurations.

What is OpenTelemetry Collector and why should I care?

OpenTelemetry is an open-source framework that ensures vendor-agnostic data collection, providing a standardized approach for the collection, processing, and ingestion of observability data. Elastic is fully committed to this principle, aiming to make observability truly vendor-agnostic and eliminating the need for users to re-instrument their observability when switching platforms.

By embracing OpenTelemetry, you have access to these benefits:

• Unified Observability: By using the OpenTelemetry Collector, you can collect and manage logs, metrics, and traces from a single tool, providing holistic observability into your system's performance and behavior. This simplifies monitoring and debugging in complex, distributed environments like microservices.
• Flexibility and Scalability: Whether you're running a small service or a large distributed system, the OpenTelemetry Collector can be scaled to handle the amount of data generated, offering the flexibility to deploy as an agent (running alongside applications) or as a gateway (a centralized hub).
• Open Standards: Since OpenTelemetry is an open-source project under the Cloud Native Computing Foundation (CNCF), it ensures that you're working with widely accepted standards, contributing to the long-term sustainability and compatibility of your observability stack.
• Simplified Telemetry Pipelines: The ability to build pipelines using receivers, processors, and exporters simplifies telemetry management by centralizing data flows and minimizing the need for multiple agents.

In the next sections, we will explain how OTEL Collector and Logstash pipelines are structured, and we will clarify how the steps for each option are used.

OTEL Collector Configuration

An OpenTelemetry Collector Configuration has different sections:

• Receivers: Collect data from different sources.
• Processors: Transform the data collected by receivers
• Exporters: Send data to different collectors
• Connectors: Link two pipelines together
• Service: defines which components are active
• Pipelines: Combine the defined receivers, processors, exporters, and connectors to process the data
• Extensions are optional components that expand the capabilities of the Collector to accomplish tasks not directly involved with processing telemetry data (e.g., health monitoring)
• Telemetry where you can set observability for the collector itself (e.g., logging and monitoring)

We can visualize it schematically as follows:

We refer to the official documentation Configuration | OpenTelemetry for an in-depth introduction in the components.

Logstash pipeline definition

A Logstash pipeline is composed of three main components:

• Input Plugins: Allow us to read data from different sources
• Filters Plugins: Allow us to transform and filter the data
• Output Plugins: Allow us to send the data

Logstash also has a special input and a special output that allow the pipeline-to-pipeline communication, we can consider this as a similar concept to an OpenTelemetry connector.

Logstash pipeline compared to Otel Collector components

We can schematize how Logstash Pipeline and OTEL Collector pipeline components can relate to each other as follows:

Enough theory! Let us dive into some examples.

Convert a Logstash Pipeline into OpenTelemetry Collector Pipeline

Example 1: Parse and transform log line

Let's consider the below line:

2024-09-20T08:33:27: user frank accessed from 89.66.167.22:10592 path /blog with error 404

We will apply the following steps:

1. Read the line from the file /tmp/demo-line.log.
2. Define the output to be an Elasticsearch datastream logs-access-default.
3. Extract the @timestamp, user.name, client.ip, client.port, url.path and http.status.code.
4. Drop log messages related to the SYSTEM user.
5. Parse the date timestamp with the relevant date format and store it in @timestamp.
6. Add a code http.status.code_description based on known codes' descriptions.
7. Send data to Elasticsearch.

Logstash pipeline

input {
    file {
        path => "/tmp/demo-line.log" #[1]
        start_position => "beginning"
        add_field => { #[2]
            "[data_stream][type]" => "logs"
            "[data_stream][dataset]" => "access_log"
            "[data_stream][namespace]" => "default"
        }
    }
}

filter {
    grok { #[3]
        match => {
            "message" => "%{TIMESTAMP_ISO8601:[date]}: user %{WORD:[user][name]} accessed from %{IP:[client][ip]}:%{NUMBER:[client][port]:int} path %{URIPATH:[url][path]} with error %{NUMBER:[http][status][code]}"
        }
    }
    if "_grokparsefailure" not in [tags] {
        if [user][name] == "SYSTEM" { #[4]
            drop {}
        }
        date { #[5]
            match => ["[date]", "ISO8601"]
            target => "[@timestamp]"
            timezone => "UTC"
            remove_field => [ "date" ]
        }
        translate { #[6]
            source => "[http][status][code]"
            target => "[http][status][code_description]"
            dictionary => {
                "200" => "OK"
                "403" => "Permission denied"
                "404" => "Not Found"
                "500" => "Server Error"
            }
            fallback => "Unknown error"
        }
    }
}

output {
    elasticsearch { #[7]
        hosts => "elasticsearch-enpoint:443"
        api_key => "${ES_API_KEY}"
    }
}

OpenTelemtry Collector configuration

receivers:
  filelog: #[1]
    start_at: beginning
    include:
      - /tmp/demo-line.log
    include_file_name: false
    include_file_path: true
    storage: file_storage 
    operators:
    # Copy the raw message into event.original (this is done OOTB by Logstash in ECS mode)
    - type: copy
      from: body
      to: attributes['event.original']
    - type: add #[2]
      field: attributes["data_stream.type"]
      value: "logs"
    - type: add #[2]
      field: attributes["data_stream.dataset"]
      value: "access_log_otel" 
    - type: add #[2]
      field: attributes["data_stream.namespace"]
      value: "default"

extensions:
  file_storage:
    directory: /var/lib/otelcol/file_storage

processors:
  # Adding  host.name (this is done OOTB by Logstash)
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        os.type:
          enabled: false

  transform/grok: #[3]
    log_statements:
      - context: log
        statements:
        - 'merge_maps(attributes, ExtractGrokPatterns(attributes["event.original"], "%{TIMESTAMP_ISO8601:date}: user %{WORD:user.name} accessed from %{IP:client.ip}:%{NUMBER:client.port:int} path %{URIPATH:url.path} with error %{NUMBER:http.status.code}", true), "insert")'

  filter/exclude_system_user:  #[4]
    error_mode: ignore
    logs:
      log_record:
        - attributes["user.name"] == "SYSTEM"

  transform/parse_date: #[5]
    log_statements:
      - context: log
        statements:
          - set(time, Time(attributes["date"], "%Y-%m-%dT%H:%M:%S"))
          - delete_key(attributes, "date")
        conditions:
          - attributes["date"] != nil

  transform/translate_status_code:  #[6]
    log_statements:
      - context: log
        conditions:
        - attributes["http.status.code"] != nil
        statements:
        - set(attributes["http.status.code_description"], "OK")                where attributes["http.status.code"] == "200"
        - set(attributes["http.status.code_description"], "Permission Denied") where attributes["http.status.code"] == "403"
        - set(attributes["http.status.code_description"], "Not Found")         where attributes["http.status.code"] == "404"
        - set(attributes["http.status.code_description"], "Server Error")      where attributes["http.status.code"] == "500"
        - set(attributes["http.status.code_description"], "Unknown Error")     where attributes["http.status.code_description"] == nil

exporters:
  elasticsearch: #[7]
    endpoints: ["elasticsearch-enpoint:443"]
    api_key: ${env:ES_API_KEY}
    tls:
    logs_dynamic_index:
      enabled: true
    mapping:
      mode: ecs

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers:
        - filelog
      processors:
        - resourcedetection/system
        - transform/grok
        - filter/exclude_system_user
        - transform/parse_date
        - transform/translate_status_code
      exporters:
        - elasticsearch

These will generate the following document in Elasticsearch

{
    "@timestamp": "2024-09-20T08:33:27.000Z",
    "client": {
        "ip": "89.66.167.22",
        "port": 10592
    },
    "data_stream": {
        "dataset": "access_log",
        "namespace": "default",
        "type": "logs"
    },
    "event": {
        "original": "2024-09-20T08:33:27: user frank accessed from 89.66.167.22:10592 path /blog with error 404"
    },
    "host": {
        "hostname": "my-laptop",
        "name": "my-laptop",
     },
    "http": {
        "status": {
            "code": "404",
            "code_description": "Not Found"
        }
    },
    "log": {
        "file": {
            "path": "/tmp/demo-line.log"
        }
    },
    "message": "2024-09-20T08:33:27: user frank accessed from 89.66.167.22:10592 path /blog with error 404",
    "url": {
        "path": "/blog"
    },
    "user": {
        "name": "frank"
    }
}

Example 2: Parse and transform a NDJSON-formatted log file

Let's consider the below json line:

{"log_level":"INFO","message":"User login successful","service":"auth-service","timestamp":"2024-10-11 12:34:56.123 +0100","user":{"id":"A1230","name":"john_doe"}}

We will apply the following steps:

1. Read a line from the file /tmp/demo.ndjson.
2. Define the output to be an Elasticsearch datastream logs-json-default
3. Parse the JSON and assign relevant keys and values.
4. Parse the date.
5. Override the message field.
6. Rename fields to follow ECS conventions.
7. Send data to Elasticsearch.

Logstash pipeline

input {
    file {
        path => "/tmp/demo.ndjson" #[1]
        start_position => "beginning"
        add_field => { #[2]
            "[data_stream][type]" => "logs"
            "[data_stream][dataset]" => "json"
            "[data_stream][namespace]" => "default"
        }
    }
}

filter {
  if [message] =~ /^\{.*/ {
    json { #[3] & #[5]
        source => "message"
    }
  }
  date { #[4]
    match => ["[timestamp]", "yyyy-MM-dd HH:mm:ss.SSS Z"]
    remove_field => "[timestamp]"
  }
  mutate {
    rename => { #[6]
      "service" => "[service][name]"
      "log_level" => "[log][level]"
    }
  }
}


output {
    elasticsearch { # [7]
        hosts => "elasticsearch-enpoint:443"
        api_key => "${ES_API_KEY}"
    }
}

OpenTelemtry Collector configuration

receivers:
  filelog/json: # [1]
    include: 
      - /tmp/demo.ndjson
    retry_on_failure:
      enabled: true
    start_at: beginning
    storage: file_storage 
    operators:
     # Copy the raw message into event.original (this is done OOTB by Logstash in ECS mode)
    - type: copy
      from: body
      to: attributes['event.original']
    - type: add #[2]
      field: attributes["data_stream.type"]
      value: "logs"      
    - type: add #[2]
      field: attributes["data_stream.dataset"]
      value: "otel" #[2]
    - type: add
      field: attributes["data_stream.namespace"]
      value: "default"     


extensions:
  file_storage:
    directory: /var/lib/otelcol/file_storage

processors:
  # Adding  host.name (this is done OOTB by Logstash)
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]
      resource_attributes:
        os.type:
          enabled: false

  transform/json_parse:  #[3]
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - merge_maps(attributes, ParseJSON(body), "upsert")
        conditions: 
          - IsMatch(body, "^\\{")


  transform/parse_date:  #[4]
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(time, Time(attributes["timestamp"], "%Y-%m-%d %H:%M:%S.%L %z"))
          - delete_key(attributes, "timestamp")
        conditions: 
          - attributes["timestamp"] != nil

  transform/override_message_field: [5]
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(body, attributes["message"])
          - delete_key(attributes, "message")

  transform/set_log_severity: # [6]
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(severity_text, attributes["log_level"])          

  attributes/rename_attributes: #[6]
    actions:
      - key: service.name
        from_attribute: service
        action: insert
      - key: service
        action: delete
      - key: log_level
        action: delete

exporters:
  elasticsearch: #[7]
    endpoints: ["elasticsearch-enpoint:443"]
    api_key: ${env:ES_API_KEY}
    tls:
    logs_dynamic_index:
      enabled: true
    mapping:
      mode: ecs

service:
  extensions: [file_storage]
  pipelines:
    logs/json:
      receivers: 
        - filelog/json
      processors:
        - resourcedetection/system    
        - transform/json_parse
        - transform/parse_date        
        - transform/override_message_field
        - transform/set_log_severity
        - attributes/rename_attributes
      exporters: 
        - elasticsearch

These will generate the following document in Elasticsearch

{
    "@timestamp": "2024-10-11T12:34:56.123000000Z",
    "data_stream": {
        "dataset": "otel",
        "namespace": "default",
        "type": "logs"
    },
    "event": {
        "original": "{\"log_level\":\"WARNING\",\"message\":\"User login successful\",\"service\":\"auth-service\",\"timestamp\":\"2024-10-11 12:34:56.123 +0100\",\"user\":{\"id\":\"A1230\",\"name\":\"john_doe\"}}"
    },
    "host": {
        "hostname": "my-laptop",
        "name": "my-laptop",
     },
    "log": {
        "file": {
            "name": "json.log"
        },
        "level": "WARNING"
    },
    "message": "User login successful",
    "service": {
        "name": "auth-service"
    },
    "user": {
        "id": "A1230",
        "name": "john_doe"
    }
}

Conclusion

In this post, we showed examples of how to convert a typical Logstash pipeline into an OpenTelemetry Collector pipeline for logs. While OpenTelemetry provides powerful tools for collecting and exporting logs, if your pipeline relies on complex transformations or scripting, Logstash remains a superior choice. This is because Logstash offers a broader range of built-in features and a more flexible approach to handling advanced data manipulation tasks.

What's Next?

Now that you've seen basic (but realistic) examples of converting a Logstash pipeline to OpenTelemetry, it's your turn to dive deeper. Depending on your needs, you can explore further and find more detailed resources in the following repositories:

• OpenTelemetry Collector: Learn about the core OpenTelemetry components, from receivers to exporters.
• OpenTelemetry Collector Contrib: Find community-contributed components for a wider range of integrations and features.
• Elastic's opentelemetry-collector-components: Dive into Elastic's extensions for the OpenTelemetry Collector, offering more tailored features for Elastic Stack users.

If you encounter specific challenges or need to handle more advanced use cases, these repositories will be an excellent resource for discovering additional components or integrations that can enhance your pipeline. All these repositories have a similar structure with folders named receiver, processor, exporter, connector, which should be familiar after reading this blog. Whether you are migrating a simple Logstash pipeline or tackling more complex data transformations, these tools and communities will provide the support you need for a successful OpenTelemetry implementation.

The prevalence of high-entropy logs in distributed systems has significantly raised the risk of PII (Personally Identifiable Information) seeping into our logs, which can result in security and compliance issues. This 2-part blog delves into the crucial task of identifying and managing this issue using the Elastic Stack. We will explore using NLP (Natural Language Processing) and Pattern matching to detect, assess, and, where feasible, redact PII from logs being ingested into Elasticsearch.

In Part 1 of this blog, we covered the following:

• Review the techniques and tools we have available to manage PII in our logs
• Understand the roles of NLP / NER in PII detection
• Build a composable processing pipeline to detect and assess PII
• Sample logs and run them through the NER Model
• Assess the results of the NER Model

In Part 2 of this blog, we will cover the following:

• Apply the redact regex pattern processor and assess the results
• Create Alerts using ESQL
• Apply field-level security to control access to the un-redacted data
• Production considerations and scaling
• How to run these processes on incoming or historical data

Reminder of the overall flow we will construct over the 2 blogs:

All code for this exercise can be found at: https://github.com/bvader/elastic-pii.

Part 1 Prerequisites

This blog picks up where Part 1 of this blog left off. You must have the NER model, ingest pipelines, and dashboard from Part 1 installed and working.

• Loaded and configured NER Model
• Installed all the composable ingest pipelines from Part 1 of the blog
• Installed dashboard

You can access the complete solution for Blog 1 here. Don't forget to load the dashboard, found here.

Applying the Redact Processor

Next, we will apply the redact processor. The redact processor is a simple regex-based processor that takes a list of regex patterns and looks for them in a field and replaces them with literals when found. The redact processor is reasonably performant and can run at scale. At the end, we will discuss this in detail in the production scaling section.

Elasticsearch comes packaged with a number of useful predefined patterns that can be conveniently referenced by the redact processor. If one does not suit your needs, create a new pattern with a custom definition. The Redact processor replaces every occurrence of a match. If there are multiple matches, they will all be replaced with the pattern name.

In the code below, we leveraged some of the predefined patterns as well as constructing several custom patterns.

        "patterns": [
          "%{EMAILADDRESS:EMAIL_REGEX}",      << Predefined
          "%{IP:IP_ADDRESS_REGEX}",           << Predefined
          "%{CREDIT_CARD:CREDIT_CARD_REGEX}", << Custom
          "%{SSN:SSN_REGEX}",                 << Custom
          "%{PHONE:PHONE_REGEX}"              << Custom
        ]

We also replaced the PII with easily identifiable patterns we can use for assessment.

In addition, it is important to note that since the redact processor is a simple regex find and replace, it can be used against many "secrets" patterns, not just PII. There are many references for regex and secrets patterns, so you can reuse this capability to detect secrets in your logs.

The code can be found here for the following two sections of code.

# Add the PII redact processor pipeline
DELETE _ingest/pipeline/logs-pii-redact-processor
PUT _ingest/pipeline/logs-pii-redact-processor
{
  "processors": [
    {
      "set": {
        "field": "redact.proc.successful",
        "value": true
      }
    },
    {
      "set": {
        "field": "redact.proc.found",
        "value": false
      }
    },
    {
      "set": {
        "if": "ctx?.redact?.message == null",
        "field": "redact.message",
        "copy_from": "message"
      }
    },
    {
      "redact": {
        "field": "redact.message",
        "prefix": "<REDACTPROC-",
        "suffix": ">",
        "patterns": [
          "%{EMAILADDRESS:EMAIL_REGEX}",
          "%{IP:IP_ADDRESS_REGEX}",
          "%{CREDIT_CARD:CREDIT_CARD_REGEX}",
          "%{SSN:SSN_REGEX}",
          "%{PHONE:PHONE_REGEX}"
        ],
        "pattern_definitions": {
          "CREDIT_CARD": """\d{4}[ -]\d{4}[ -]\d{4}[ -]\d{4}""",
          "SSN": """\d{3}-\d{2}-\d{4}""",
          "PHONE": """(\+\d{1,2}\s?)?1?\-?\.?\s?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}"""
        },
        "on_failure": [
          {
            "set": {
              "description": "Set 'error.message'",
              "field": "failure",
              "value": "REDACT_PROCESSOR_FAILED",
              "override": false
            }
          },
          {
            "set": {
              "field": "redact.proc.successful",
              "value": false
            }
          }
        ]
      }
    },
    {
      "set": {
        "if": "ctx?.redact?.message.contains('REDACTPROC')",
        "field": "redact.proc.found",
        "value": true
      }
    },
    {
      "set": {
        "if": "ctx?.redact?.pii?.found == null",
        "field": "redact.pii.found",
        "value": false
      }
    },
    {
      "set": {
        "if": "ctx?.redact?.proc?.found == true",
        "field": "redact.pii.found",
        "value": true
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "failure",
        "value": "GENERAL_FAILURE",
        "override": false
      }
    }
  ]
}

And now, we will add the logs-pii-redact-processor pipeline to the overall process-pii pipeline

# Updated Process PII pipeline that now call the NER and Redact Processor pipeline
DELETE _ingest/pipeline/process-pii
PUT _ingest/pipeline/process-pii
{
  "processors": [
    {
      "set": {
        "description": "Set true if enabling sampling, otherwise false",
        "field": "sample.enabled",
        "value": true
      }
    },
    {
      "set": {
        "description": "Set Sampling Rate 0 None 10000 all allows for 0.01% precision",
        "field": "sample.sample_rate",
        "value": 1000
      }
    },
    {
      "set": {
        "description": "Set to false if you want to drop unsampled data, handy for reindexing hostorical data",
        "field": "sample.keep_unsampled",
        "value": true
      }
    },
    {
      "pipeline": {
        "if": "ctx.sample.enabled == true",
        "name": "logs-sampler",
        "ignore_failure": true
      }
    },
    {
      "pipeline": {
        "if": "ctx.sample.enabled == false || (ctx.sample.enabled == true && ctx.sample.sampled == true)",
        "name": "logs-ner-pii-processor"
      }
    },
    {
      "pipeline": {
        "if": "ctx.sample.enabled == false || (ctx.sample.enabled == true &&  ctx.sample.sampled == true)",
        "name": "logs-pii-redact-processor"
      }
    }
  ]
}

Reload the data as described in the Reloading the logs. If you have not generated the logs the first time, follow the instructions in the Data Loading Appendix

Go to Discover and enter the following into the KQL bar sample.sampled : true and redact.message: REDACTPROC and add the redact.message to the table and you should see something like this.

And if you did not load the dashboard from Blog Part 1 at already, load it, it can be found here using the Kibana -> Stack Management -> Saved Objects -> Import.

It should look something like this now. Note that the REGEX portions of the dashboard are now active.

Checkpoint

At this point, we have the following capabilities:

• Ability to sample incoming logs and apply this PII redaction
• Detect and Assess PII with the NER/NLP and Pattern Matching
• Assess the amount, type and quality of the PII detections

This is a great point to stop if you are just running all this once to see how it works, but we have a few more steps to make this useful in production systems.

• Clean up the working and unredacted data
• Update the Dashboard to work with the cleaned-up data
• Apply Role Based Access Control to protect the raw unredacted data
• Create Alerts
• Production and Scaling Considerations
• How to run these processes on incoming or historical data

Applying to Production Systems

Cleanup working data and update the dashboard

And now we will add the cleanup code to the overall process-pii pipeline.

In short, we set a flag redact.enable: true that directs the pipeline to move the unredacted message field to raw.message and the move the redacted message field redact.messageto the message field. We will "protect" the raw.message in the following section.

NOTE: Of course you can change this behavior if you want to completely delete the unredacted data. In this exercise we will keep it and protect it.

In addition we set redact.cleanup: true to clean up the NLP working data.

These fields allow a lot of control over what data you decide to keep and analyze.

The code can be found here for the following two sections of code.

# Updated Process PII pipeline that now call the NER and Redact Processor pipeline and cleans up 
DELETE _ingest/pipeline/process-pii
PUT _ingest/pipeline/process-pii
{
  "processors": [
    {
      "set": {
        "description": "Set true if enabling sampling, otherwise false",
        "field": "sample.enabled",
        "value": true
      }
    },
    {
      "set": {
        "description": "Set Sampling Rate 0 None 10000 all allows for 0.01% precision",
        "field": "sample.sample_rate",
        "value": 1000
      }
    },
    {
      "set": {
        "description": "Set to false if you want to drop unsampled data, handy for reindexing hostorical data",
        "field": "sample.keep_unsampled",
        "value": true
      }
    },
    {
      "pipeline": {
        "if": "ctx.sample.enabled == true",
        "name": "logs-sampler",
        "ignore_failure": true
      }
    },
    {
      "pipeline": {
        "if": "ctx.sample.enabled == false || (ctx.sample.enabled == true && ctx.sample.sampled == true)",
        "name": "logs-ner-pii-processor"
      }
    },
    {
      "pipeline": {
        "if": "ctx.sample.enabled == false || (ctx.sample.enabled == true &&  ctx.sample.sampled == true)",
        "name": "logs-pii-redact-processor"
      }
    },
    {
      "set": {
        "description": "Set to true to actually redact, false will run processors but leave original",
        "field": "redact.enable",
        "value": true
      }
    },
    {
      "rename": {
        "if": "ctx?.redact?.pii?.found == true && ctx?.redact?.enable == true",
        "field": "message",
        "target_field": "raw.message"
      }
    },
    {
      "rename": {
        "if": "ctx?.redact?.pii?.found == true && ctx?.redact?.enable == true",
        "field": "redact.message",
        "target_field": "message"
      }
    },
    {
      "set": {
        "description": "Set to true to actually to clean up working data",
        "field": "redact.cleanup",
        "value": true
      }
    },
    {
      "remove": {
        "if": "ctx?.redact?.cleanup == true",
        "field": [
          "ml"
        ],
        "ignore_failure": true
      }
    }
  ]
}

Reload the data as described here in the Reloading the logs.

Go to Discover and enter the following into the KQL bar sample.sampled : true and redact.pii.found: true and add the following fields to the table

message,raw.message,redact.ner.found,redact.proc.found,redact.pii.found

You should see something like this

We have everything we need to move forward with protecting the PII and Alerting on it.

Load up the new dashboard that works on the cleaned-up data

To load the dashboard, go to Kibana -> Stack Management -> Saved Objects and import the pii-dashboard-part-2.ndjson file that can be found here.

The new dashboard should look like this. Note: It uses different fields under the covers since we have cleaned up the underlying data.

You should see something like this

Apply Role Based Access Control to protect the raw unredacted data

Elasticsearch supports role-based access control, including field and document level access control natively; it dramatically reduces the operational and maintenance complexity required to secure our application.

We will create a Role that does not allow access to the raw.message field and then create a user and assign that user the role. With that role, the user will only be able to see the redacted message, which is now in the message field, but will not be able to access the protected raw.message field.

NOTE: Since we only sampled 10% of the data in this exercise the non-sampled message fields are not moved to the raw.message, so they are still viewable, but this shows the capability you can apply in a production system.

The code can be found here for the following section of code.

# Create role with no access to the raw.message field
GET _security/role/protect-pii
DELETE _security/role/protect-pii
PUT _security/role/protect-pii
{
  "cluster": [],
  "indices": [
    {
      "names": [
        "logs-*"
      ],
      "privileges": [
        "read",
        "view_index_metadata"
      ],
      "field_security": {
        "grant": [
          "*"
        ],
        "except": [
          "raw.message"
        ]
      },
      "allow_restricted_indices": false
    }
  ],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": [
        "all"
      ],
      "resources": [
        "*"
      ]
    }
  ],
  "run_as": [],
  "metadata": {},
  "transient_metadata": {
    "enabled": true
  }
}

# Create user stephen with protect-pii role
GET _security/user/stephen
DELETE /_security/user/stephen
POST /_security/user/stephen
{
  "password" : "mypassword",
  "roles" : [ "protect-pii" ],
  "full_name" : "Stephen Brown"
}

Now log into a separate window with the new user stephen with the protect-pii role. Go to Discover and put redact.pii.found : true in the KQL bar and add the message field to the table. Also, notice that the raw.message is not available.

You should see something like this

Create an Alert when PII Detected

Now, with the processing of the pipelines, creating an alert when PII is detected is easy. To review Alerting in Kibana in detail if needed

NOTE: Reload the data if needed to have recent data.

First, we will create a simple ES|QL query in Discover.

The code can be found here.

FROM logs-pii-default
| WHERE redact.pii.found == true
| STATS pii_count = count(*)
| WHERE pii_count > 0

When you run this you should see something like this.

Now click the Alerts menu and select Create search threshold rule, and will create an alert to alert us when PII is found.

Select a time field: @timestamp Set the time window: 5 minutes

Assuming you loaded the data recently when you run Test it should do something like

pii_count : 343 Alerts generated query matched

Add an action when the alert is Active.

For each alert: On status changes Run when: Query matched

Elasticsearch query rule {{rule.name}} is active:

- PII Found: true
- PII Count: {{#context.hits}} {{_source.pii_count}}{{/context.hits}}
- Conditions Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}

Add an Action for when the Alert is Recovered.

For each alert: On status changes Run when: Recovered

Elasticsearch query rule {{rule.name}} is Recovered:

- PII Found: false
- Conditions Not Met: {{context.conditions}} over {{rule.params.timeWindowSize}}{{rule.params.timeWindowUnit}}
- Timestamp: {{context.date}}
- Link: {{context.link}}

When all setup it should look like this and Save

\ \

You should get an Active alert that looks like this if you have recent data. I sent mine to Slack.

Elasticsearch query rule pii-found-esql is active:
- PII Found: true
- PII Count:  374
- Conditions Met: Query matched documents over 5m
- Timestamp: 2024-10-15T02:44:52.795Z
- Link: https://mydeployment123.aws.found.io:9243/app/management/insightsAndAlerting/triggersActions/rule/7d6faecf-964e-46da-aaba-8a2f89f33989

And then if you wait you will get a Recovered alert that looks like this.

Elasticsearch query rule pii-found-esql is Recovered:
- PII Found: false
- Conditions Not Met: Query did NOT match documents over 5m
- Timestamp: 2024-10-15T02:49:04.815Z
- Link: https://mydeployment123.kb.us-west-1.aws.found.io:9243/app/management/insightsAndAlerting/triggersActions/rule/7d6faecf-964e-46da-aaba-8a2f89f33989

Production Scaling

NER Scaling

As we mentioned Part 1 of this blog of this blog, NER / NLP Models are CPU-intensive and expensive to run at scale; thus, we employed a sampling technique to understand the risk in our logs without sending the full logs volume through the NER Model.

Please review the setup and configuration of the NER model from Part 1 of the blog.

We chose the base BERT NER model bert-base-NER for our PII case.

To scale ingest, we will focus on scaling the allocations for the deployed model. More information on this topic is available here. The number of allocations must be less than the available allocated processors (cores, not vCPUs) per node.

The metrics below are related to the model and configuration from Part 1 of the blog.

• 4 Allocations to allow for more parallel ingestion
• 1 Thread per Allocation
• 0 Byes Cache, as we expect a low cache hit rate Note If there are many repeated logs, cache can help, but with timestamps and other variations, cache will not help and can even slow down the process
• 8192 Queue

GET _ml/trained_models/dslim__bert-base-ner/_stats
.....
           "node": {
              "0m4tq7tMRC2H5p5eeZoQig": {
.....
                "attributes": {
                  "xpack.installed": "true",
                  "region": "us-west-1",
                  "ml.allocated_processors": "5", << HERE 
.....
            },
            "inference_count": 5040,
            "average_inference_time_ms": 138.44285714285715, << HERE 
            "average_inference_time_ms_excluding_cache_hits": 138.44285714285715,
            "inference_cache_hit_count": 0,
.....
            "threads_per_allocation": 1,
            "number_of_allocations": 4,  <<< HERE
            "peak_throughput_per_minute": 1550,
            "throughput_last_minute": 1373,
            "average_inference_time_ms_last_minute": 137.55280407865988,
            "inference_cache_hit_count_last_minute": 0
          }
        ]
      }
    }

There are 3 key pieces of information above:

• "ml.allocated_processors": "5" The number of physical cores / processors available

• "number_of_allocations": 4 The number of allocations which is maximum 1 per physical core. Note: we could have used 5 allocations, but we only allocated 4 for this exercise

• "average_inference_time_ms": 138.44285714285715 The averages inference time per document.

The math is pretty straightforward for throughput for Inferences per Min (IPM) per allocation (1 allocation per physical core), since an inference uses a single core and a single thread.

Then the Inferences per Min per Allocation is simply:

IPM per allocation = 60,000 ms (in a minute) / 138ms per inference = 435

When then lines up with the Total Inferences per Minute

Total IPM = 435 IPM / allocation * 4 Allocations = ~1740

Suppose we want to do 10,000 IPMs, how many allocations (cores) would I need?

Allocations = 10,000 IPM / 435 IPM per allocation = 23 Allocation (cores rounded up)

Or perhaps logs are coming in at 5000 EPS and you want to do 1% Sampling.

IPM = 5000 EPS * 60sec * 0.01 sampling = 3000 IPM sampled

Then

Number of Allocators = 3000 IPM / 435 IPM per allocation = 7 allocations (cores rounded up)

Want Faster! Turns out there is a more lightweight NER Model distilbert-NER model that is faster, but the tradeoff is a little less accuracy.

Running the logs through this model results in an inference time nearly twice as fast!

"average_inference_time_ms": 66.0263959390863

Here is some quick math: $IPM per allocation = 60,000 ms (in a minute) / 61ms per inference = 983

Suppose we want to do 25,000 IPMs, how many allocations (cores) would I need?

Allocations = 25,000 IPM / 983 IPM per allocation = 26 Allocation (cores rounded up)

Now you can apply this math to determine the correct sampling and NER scaling to support your logging use case.

Redact Processor Scaling

In short, the redact processor should scale to production loads as long as you are using appropriately sized and configured nodes and have well-constructed regex patterns.

Assessing incoming logs

If you want to test on incoming logs data in a data stream. All you need to do is change the conditional in the logs@custom pipeline to apply the process-pii to the dataset you want to. You can use any conditional that fits your condition.

Note: Just make sure that you have accounted for the proper scaling for the NER and Redact processors they were described above in Production Scaling

    {
      "pipeline": {
        "description" : "Call the process_pii pipeline on the correct dataset",
        "if": "ctx?.data_stream?.dataset == 'pii'", <<< HERE
        "name": "process-pii"
      }
    }

So if for example your logs are coming into logs-mycustomapp-default you would just change the conditional to

        "if": "ctx?.data_stream?.dataset == 'mycustomapp'",

Assessing historical data

If you have a historical (already ingested) data stream or index you can run the assessment over them using the _reindex API>

Note: Just make sure that you have accounted for the proper scaling for the NER and Redact processors they were described above in Production Scaling

There are a couple of extra steps: The code can be found here.

1) First we can set the parameters to ONLY keep the sampled data as there is no reason to make a copy of all the unsampled data. In the process-pii pipeline, there is a setting sample.keep_unsampled, which we can set to false, which will then only keep the sampled data

    {
      "set": {
        "description": "Set to false if you want to drop unsampled data, handy for reindexing hostorical data",
        "field": "sample.keep_unsampled",
        "value": false <<< SET TO false
      }
    },

2) Second, we will create a pipeline that will reroute the data to the correct data stream to run through all the PII assessment/detection pipelines. It also sets the correct dataset and namespace

DELETE _ingest/pipeline/sendtopii
PUT _ingest/pipeline/sendtopii
{
  "processors": [
    {
      "set": {
        "field": "data_stream.dataset",
        "value": "pii"
      }
    },
    {
      "set": {
        "field": "data_stream.namespace",
        "value": "default"
      }
    },
    {
      "reroute" : 
      {
        "dataset" : "{{data_stream.dataset}}",
        "namespace": "{{data_stream.namespace}}"
      }
    }
  ]
}

3) Finally, we can run a _reindex to select the data we want to test/assess. It is recommended to review the _reindex documents before trying this. First, select the source data stream you want to assess, in this example, it is the logs-generic-default logs data stream. Note: I also added a range filter to select a specific time range. There is a bit of a "trick" that we need to use since we are re-routing the data to the data stream logs-pii-default. To do this, we just set "index": "logs-tmp-default" in the _reindex as the correct data stream will be set in the pipeline. We must do that because reroute is a noop if it is called from/to the same datastream.

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "logs-generic-default",
    "query": {
      "bool": {
        "filter": [
          {
            "range": {
              "@timestamp": {
                "gte": "now-1h/h",
                "lt": "now"
              }
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-tmp-default",
    "pipeline": "sendtopii"
  }
}

Summary

At this point, you have the tools and processes need to assess, detect, analyze, alert and protect PII in your logs.

The end state solution can be found here:.

In Part 1 of this blog, we accomplished the following.

• Reviewed the techniques and tools we have available for PII detection and assessment
• Reviewed NLP / NER role in PII detection and assessment
• Built the necessary composable ingest pipelines to sample logs and run them through the NER Model
• Reviewed the NER results and are ready to move to the second blog

In Part 2 of this blog, we covered the following:

• Redact PII using NER and redact processor
• Apply field-level security to control access to the un-redacted data
• Enhance the dashboards and alerts
• Production considerations and scaling
• How to run these processes on incoming or historical data

So get to work and reduce risk in your logs!

Data Loading Appendix

Code

The data loading code can be found here:

https://github.com/bvader/elastic-pii

$ git clone https://github.com/bvader/elastic-pii.git

Creating and Loading the Sample Data Set

$ cd elastic-pii
$ cd python
$ python -m venv .env
$ source .env/bin/activate
$ pip install elasticsearch
$ pip install Faker

Run the log generator

$ python generate_random_logs.py

If you do not changes any parameters, this will create 10000 random logs in a file named pii.log with a mix of logs that containe and do not contain PII.

Edit load_logs.py and set the following

# The Elastic User 
ELASTIC_USER = "elastic"

# Password for the 'elastic' user generated by Elasticsearch
ELASTIC_PASSWORD = "askdjfhasldfkjhasdf"

# Found in the 'Manage Deployment' page
ELASTIC_CLOUD_ID = "deployment:sadfjhasfdlkjsdhf3VuZC5pbzo0NDMkYjA0NmQ0YjFiYzg5NDM3ZDgxM2YxM2RhZjQ3OGE3MzIkZGJmNTE0OGEwODEzNGEwN2E3M2YwYjcyZjljYTliZWQ="

Then run the following command.

$ python load_logs.py

Reloading the logs

Note To reload the logs, you can simply re-run the above command. You can run the command multiple time during this exercise and the logs will be reloaded (actually loaded again). The new logs will not collide with previous runs as there will be a unique run.id for each run which is displayed at the end of the loading process.

$ python load_logs.py

The prevalence of high-entropy logs in distributed systems has significantly raised the risk of PII (Personally Identifiable Information) seeping into our logs, which can result in security and compliance issues. This 2-part blog delves into the crucial task of identifying and managing this issue using the Elastic Stack. We will explore using NLP (Natural Language Processing) and Pattern matching to detect, assess, and, where feasible, redact PII from logs that are being ingested into Elasticsearch.

In Part 1 of this blog, we will cover the following:

• Review the techniques and tools we have available to manage PII in our logs
• Understand the roles of NLP / NER in PII detection
• Build a composable processing pipeline to detect and assess PII
• Sample logs and run them through the NER Model
• Assess the results of the NER Model

In Part 2 of this blog of this blog, we will cover the following:

• Redact PII using NER and the redact processor
• Apply field-level security to control access to the un-redacted data
• Enhance the dashboards and alerts
• Production considerations and scaling
• How to run these processes on incoming or historical data

Here is the overall flow we will construct over the 2 blogs:

All code for this exercise can be found at: https://github.com/bvader/elastic-pii.

Tools and Techniques

There are four general capabilities that we will use for this exercise.

• Named Entity Recognition Detection (NER)
• Pattern Matching Detection
• Log Sampling
• Ingest Pipelines as Composable Processing

Named Entity Recognition (NER) Detection

NER is a sub-task of Natural Language Processing (NLP) that involves identifying and categorizing named entities in unstructured text into predefined categories such as:

• Person: Names of individuals, including celebrities, politicians, and historical figures.
• Organization: Names of companies, institutions, and organizations.
• Location: Geographic locations, including cities, countries, and landmarks.
• Event: Names of events, including conferences, meetings, and festivals.

For our use PII case, we will choose the base BERT NER model bert-base-NER that can be downloaded from Hugging Face and loaded into Elasticsearch as a trained model.

Important Note: NER / NLP Models are CPU-intensive and expensive to run at scale; thus, we will want to employ a sampling technique to understand the risk in our logs without sending the full logs volume through the NER Model. We will discuss the performance and scaling of the NER model in part 2 of the blog.

Pattern Matching Detection

In addition to using an NER, regex pattern matching is a powerful tool for detecting and redacting PII based on common patterns. The Elasticsearch redact processor is built for this use case.

Log Sampling

Considering the performance implications of NER and the fact that we may be ingesting a large volume of logs into Elasticsearch, it makes sense to sample our incoming logs. We will build a simple log sampler to accomplish this.

Ingest Pipelines as Composable Processing

We will create several pipelines, each focusing on a specific capability and a main ingest pipeline to orchestrate the overall process.

Building the Processing Flow

Logs Sampling + Composable Ingest Pipelines

The first thing we will do is set up a sampler to sample our logs. This ingest pipeline simply takes a sampling rate between 0 (no log) and 10000 (all logs), which allows as low as ~0.01% sampling rate and marks the sampled logs with sample.sampled: true. Further processing on the logs will be driven by the value of sample.sampled. The sample.sample_rate can be set here or "passed in" from the orchestration pipeline.

The command should be run from the Kibana -> Dev Tools

The code can be found here for the following three sections of code.

# logs-sampler pipeline - part 1
DELETE _ingest/pipeline/logs-sampler
PUT _ingest/pipeline/logs-sampler
{
  "processors": [
    {
      "set": {
        "description": "Set Sampling Rate 0 None 10000 all allows for 0.01% precision",
        "if": "ctx.sample.sample_rate == null",
        "field": "sample.sample_rate",
        "value": 10000
      }
    },
    {
      "set": {
        "description": "Determine if keeping unsampled docs",
        "if": "ctx.sample.keep_unsampled == null",
        "field": "sample.keep_unsampled",
        "value": true
      }
    },
    {
      "set": {
        "field": "sample.sampled",
        "value": false
      }
    },
    {
      "script": {
        "source": """ Random r = new Random();
        ctx.sample.random = r.nextInt(params.max); """,
        "params": {
          "max": 10000
        }
      }
    },
    {
      "set": {
        "if": "ctx.sample.random <= ctx.sample.sample_rate",
        "field": "sample.sampled",
        "value": true
      }
    },
    {
      "drop": {
         "description": "Drop unsampled document if applicable",
        "if": "ctx.sample.keep_unsampled == false && ctx.sample.sampled == false"
      }
    }
  ]
}

Now, let's test the logs sampler. We will build the first part of the composable pipeline. We will be sending logs to the logs-generic-default data stream. With that in mind, we will create the logs@custom ingest pipeline that will be automatically called using the logs data stream framework for customization. We will add one additional level of abstraction so that you can apply this PII processing to other data streams.

Next, we will create the process-pii pipeline. This is the core processing pipeline where we will orchestrate PII processing component pipelines. In this first step, we will simply apply the sampling logic. Note that we are setting the sampling rate to 100, which is equivalent to 10% of the logs.

# Process PII pipeline - part 1
DELETE _ingest/pipeline/process-pii
PUT _ingest/pipeline/process-pii
{
  "processors": [
    {
      "set": {
        "description": "Set true if enabling sampling, otherwise false",
        "field": "sample.enabled",
        "value": true
      }
    },
    {
      "set": {
        "description": "Set Sampling Rate 0 None 10000 all allows for 0.01% precision",
        "field": "sample.sample_rate",
        "value": 1000
      }
    },
    {
      "set": {
        "description": "Set to false if you want to drop unsampled data, handy for reindexing hostorical data",
        "field": "sample.keep_unsampled",
        "value": true
      }
    },
    {
      "pipeline": {
        "if": "ctx.sample.enabled == true",
        "name": "logs-sampler",
        "ignore_failure": true
      }
    }
  ]
}

Finally, we create the logs logs@custom, which will simply call our process-pii pipeline based on the correct data_stream.dataset

# logs@custom pipeline - part 1
DELETE _ingest/pipeline/logs@custom
PUT _ingest/pipeline/logs@custom
{
  "processors": [
    {
      "set": {
        "field": "pipelinetoplevel",
        "value": "logs@custom"
      }
    },
        {
      "set": {
        "field": "pipelinetoplevelinfo",
        "value": "{{{data_stream.dataset}}}"
      }
    },
    {
      "pipeline": {
        "description" : "Call the process_pii pipeline on the correct dataset",
        "if": "ctx?.data_stream?.dataset == 'pii'", 
        "name": "process-pii"
      }
    }
  ]
}

Now, let's test to see the sampling at work.

Load the data as described here Data Loading Appendix. Let's use the sample data first, and we will talk about how to test with your incoming or historical logs later at the end of this blog.

If you look at Observability -> Logs -> Logs Explorer with KQL filter data_stream.dataset : pii and Breakdown by sample.sampled, you should see the breakdown to be approximately 10%

At this point we have a composable ingest pipeline that is "sampling" logs. As a bonus, you can use this logs sampler for any other use cases you have as well.

Loading, Configuration, and Execution of the NER Pipeline

Loading the NER Model

You will need a Machine Learning node to run the NER model on. In this exercise, we are using Elastic Cloud Hosted Deployment on AWS with the CPU Optimized (ARM) architecture. The NER inference will run on a Machine Learning AWS c5d node. There will be GPU options in the future, but today, we will stick with CPU architecture.

This exercise will use a single c5d with 8 GB RAM with 4.2 vCPU up to 8.4 vCPU

Please refer to the official documentation on how to import an NLP-trained model into Elasticsearch for complete instructions on uploading, configuring, and deploying the model.

The quickest way to get the model is using the Eland Docker method.

The following command will load the model into Elasticsearch but will not start it. We will do that in the next step.

docker run -it --rm --network host docker.elastic.co/eland/eland \
  eland_import_hub_model \
  --url https://mydeployment.es.us-west-1.aws.found.io:443/ \
  -u elastic -p password \
  --hub-model-id dslim/bert-base-NER --task-type ner

Deploy and Start the NER Model

In general, to improve ingest performance, increase throughput by adding more allocations to the deployment. For improved search speed, increase the number of threads per allocation.

To scale ingest, we will focus on scaling the allocations for the deployed model. More information on this topic is available here. The number of allocations must be less than the available allocated processors (cores, not vCPUs) per node.

To deploy and start the NER Model. We will do this using the Start trained model deployment API

We will configure the following:

• 4 Allocations to allow for more parallel ingestion
• 1 Thread per Allocation
• 0 Byes Cache, as we expect a low cache hit rate
• 8192 Queue

# Start the model with 4 Allocators x 1 Thread, no cache, and 8192 queue
POST _ml/trained_models/dslim__bert-base-ner/deployment/_start?cache_size=0b&number_of_allocations=4&threads_per_allocation=1&queue_capacity=8192

You should get a response that looks something like this.

{
  "assignment": {
    "task_parameters": {
      "model_id": "dslim__bert-base-ner",
      "deployment_id": "dslim__bert-base-ner",
      "model_bytes": 430974836,
      "threads_per_allocation": 1,
      "number_of_allocations": 4,
      "queue_capacity": 8192,
      "cache_size": "0",
      "priority": "normal",
      "per_deployment_memory_bytes": 430914596,
      "per_allocation_memory_bytes": 629366952
    },
...
    "assignment_state": "started",
    "start_time": "2024-09-23T21:39:18.476066615Z",
    "max_assigned_allocations": 4
  }
}

The NER model has been deployed and started and is ready to be used.

The following ingest pipeline implements the NER model via the inference processor.

There is a significant amount of code here, but only two items of interest now exist. The rest of the code is conditional logic to drive some additional specific behavior that we will look closer at in the future.

1. The inference processor calls the NER model by ID, which we loaded previously, and passes the text to be analyzed, which, in this case, is the message field, which is the text_field we want to pass to the NER model to analyze for PII.

2. The script processor loops through the message field and uses the data generated by the NER model to replace the identified PII with redacted placeholders. This looks more complex than it really is, as it simply loops through the array of ML predictions and replaces them in the message string with constants, and stores the results in a new field redact.message. We will look at this a little closer in the following steps.

The code can be found here for the following three sections of code.

The NER PII Pipeline

# NER Pipeline
DELETE _ingest/pipeline/logs-ner-pii-processor
PUT _ingest/pipeline/logs-ner-pii-processor
{
  "processors": [
    {
      "set": {
        "description": "Set to true to actually redact, false will run processors but leave original",
        "field": "redact.enable",
        "value": true
      }
    },
    {
      "set": {
        "description": "Set to true to keep ml results for debugging",
        "field": "redact.ner.keep_result",
        "value": true
      }
    },
    {
      "set": {
        "description": "Set to PER, LOC, ORG to skip, or NONE to not drop any replacement",
        "field": "redact.ner.skip_entity",
        "value": "NONE"
      }
    },
    {
      "set": {
        "description": "Set to PER, LOC, ORG to skip, or NONE to not drop any replacement",
        "field": "redact.ner.minimum_score",
        "value": 0
      }
    },
    {
      "set": {
        "if": "ctx?.redact?.message == null",
        "field": "redact.message",
        "copy_from": "message"
      }
    },
    {
      "set": {
        "field": "redact.ner.successful",
        "value": true
      }
    },
    {
      "set": {
        "field": "redact.ner.found",
        "value": false
      }
    },
    {
      "inference": {
        "model_id": "dslim__bert-base-ner",
        "field_map": {
          "message": "text_field"
        },
        "on_failure": [
          {
            "set": {
              "description": "Set 'error.message'",
              "field": "failure",
              "value": "REDACT_NER_FAILED"
            }
          },
          {
            "set": {
              "field": "redact.ner.successful",
              "value": false
            }
          }
        ]
      }
    },
    {
      "script": {
        "if": "ctx.failure_ner != 'REDACT_NER_FAILED'",
        "lang": "painless",
        "source": """String msg = ctx['message'];
          for (item in ctx['ml']['inference']['entities']) {
              if ((item['class_name'] != ctx.redact.ner.skip_entity) && 
                (item['class_probability'] >= ctx.redact.ner.minimum_score)) {  
                    msg = msg.replace(item['entity'], '<' + 
                    'REDACTNER-'+ item['class_name'] + '_NER>')
              }
          }
          ctx.redact.message = msg""",
        "on_failure": [
          {
            "set": {
              "description": "Set 'error.message'",
              "field": "failure",
              "value": "REDACT_REPLACEMENT_SCRIPT_FAILED",
              "override": false
            }
          },
          {
            "set": {
              "field": "redact.successful",
              "value": false
            }
          }
        ]
      }
    },

    {
      "set": {
        "if": "ctx?.ml?.inference?.entities.size() > 0", 
        "field": "redact.ner.found",
        "value": true,
        "ignore_failure": true
      }
    },
    {
      "set": {
        "if": "ctx?.redact?.pii?.found == null",
        "field": "redact.pii.found",
        "value": false
      }
    },
    {
      "set": {
        "if": "ctx?.redact?.ner?.found == true",
        "field": "redact.pii.found",
        "value": true
      }
    },
    {
      "remove": {
        "if": "ctx.redact.ner.keep_result != true",
        "field": [
          "ml"
        ],
        "ignore_missing": true,
        "ignore_failure": true
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "failure",
        "value": "GENERAL_FAILURE",
        "override": false
      }
    }
  ]
}

The updated PII Processor Pipeline, which now calls the NER Pipeline

# Updated Process PII pipeline that now call the NER pipeline
DELETE _ingest/pipeline/process-pii
PUT _ingest/pipeline/process-pii
{
  "processors": [
    {
      "set": {
        "description": "Set true if enabling sampling, otherwise false",
        "field": "sample.enabled",
        "value": true
      }
    },
    {
      "set": {
        "description": "Set Sampling Rate 0 None 10000 all allows for 0.01% precision",
        "field": "sample.sample_rate",
        "value": 1000
      }
    },
    {
      "set": {
        "description": "Set to false if you want to drop unsampled data, handy for reindexing hostorical data",
        "field": "sample.keep_unsampled",
        "value": true
      }
    },
    {
      "pipeline": {
        "if": "ctx.sample.enabled == true",
        "name": "logs-sampler",
        "ignore_failure": true
      }
    },
    {
      "pipeline": {
        "if": "ctx.sample.enabled == false || (ctx.sample.enabled == true && ctx.sample.sampled == true)",
        "name": "logs-ner-pii-processor"
      }
    }
  ]
}

Now reload the data as described here in Reloading the logs

Results

Let's take a look at the results with the NER processing in place. In the Logs Explorer with KQL query bar, execute the following query data_stream.dataset : pii and ml.inference.entities.class_name : ("PER" and "LOC" and "ORG" )

Logs Explorer should look something like this, open the top message to see the details.

NER Model Results

Lets take a closer look at what these fields mean.

Field: ml.inference.entities.class_name\ Sample Value: [PER, PER, LOC, ORG, ORG]\ Description: An array of the named entity classes that the NER model has identified.

Field: ml.inference.entities.class_probability\ Sample Value: [0.999, 0.972, 0.896, 0.506, 0.595]\ Description: The class_probability is a value between 0 and 1, which indicates how likely it is that a given data point belongs to a certain class. The higher the number, the higher the probability that the data point belongs to the named class. This is important as in the next blog we can decide a threshold that we will want to use to alert and redact on.' You can see in this example it identified a LOC as an ORG, we can filter this out / find them by setting a threshold.

Field: ml.inference.entities.entity\ Sample Value: [Paul Buck, Steven Glens, South Amyborough, ME, Costco]\ Description: The array of entities identified that align positionally with the class_name and class_probability.

Field: ml.inference.predicted_value\ Sample Value: [2024-09-23T14:32:14.608207-07:00Z] log.level=INFO: Payment successful for order #4594 (user: [Paul Buck](PER&Paul+Buck), david59@burgess.net). Phone: 726-632-0527x520, Address: 3713 [Steven Glens](PER&Steven+Glens), [South Amyborough](LOC&South+Amyborough), [ME](ORG&ME) 93580, Ordered from: [Costco](ORG&Costco)\ Description: The predicted value of the model.

PII Assessment Dashboard

Lets take a quick look at a dashboard built to assess PII the data.

To load the dashboard, go to Kibana -> Stack Management -> Saved Objects and import the pii-dashboard-part-1.ndjson file that can be found here:

https://github.com/bvader/elastic-pii/blob/main/elastic/blog-part-1/pii-dashboard-part-1.ndjson

More complete instructions on Kibana Saved Objects can be found here.

After loading the dashboard, navigate to it and select the right time range and you should see something like below. It shows metrics such as sample rate, percent of logs with NER, NER Score Trends etc. We will examine the assessment and actions in part 2 of this blog.

Summary and Next Steps

In this first part of the blog, we have accomplished the following.

• Reviewed the techniques and tools we have available for PII detection and assement
• Reviewed NLP / NER role in PII detection and assessment
• Built the necessary composable ingest pipelines to sample logs and run them through the NER Model
• Reviewed the NER results and are ready to move to the second blog

In the upcoming Part 2 of this blog of this blog, we will cover the following:

• Redact PII using NER and redact processor
• Apply field-level security to control access to the un-redacted data
• Enhance the dashboards and alerts
• Production considerations and scaling
• How to run these processes on incoming or historical data

Data Loading Appendix

Code

The data loading code can be found here:

https://github.com/bvader/elastic-pii

$ git clone https://github.com/bvader/elastic-pii.git

Creating and Loading the Sample Data Set

$ cd elastic-pii
$ cd python
$ python -m venv .env
$ source .env/bin/activate
$ pip install elasticsearch
$ pip install Faker

Run the log generator

$ python generate_random_logs.py

If you do not changes any parameters, this will create 10000 random logs in a file named pii.log with a mix of logs that containe and do not contain PII.

Edit load_logs.py and set the following

# The Elastic User 
ELASTIC_USER = "elastic"

# Password for the 'elastic' user generated by Elasticsearch
ELASTIC_PASSWORD = "askdjfhasldfkjhasdf"

# Found in the 'Manage Deployment' page
ELASTIC_CLOUD_ID = "deployment:sadfjhasfdlkjsdhf3VuZC5pbzo0NDMkYjA0NmQ0YjFiYzg5NDM3ZDgxM2YxM2RhZjQ3OGE3MzIkZGJmNTE0OGEwODEzNGEwN2E3M2YwYjcyZjljYTliZWQ="

Then run the following command.

$ python load_logs.py

Reloading the logs

Note To reload the logs, you can simply re-run the above command. You can run the command multiple time during this exercise and the logs will be reloaded (actually loaded again). The new logs will not collide with previous runs as there will be a unique run.id for each run which is displayed at the end of the loading process.

$ python load_logs.py

To illustrate this idea, we will use the Sample flight data dataset. Now, let’s compare two visualizations ranking the top 10 destination countries per total number of flights. Which visualization has a higher impact?

If you chose the second one, you may be wondering how this was done with the Kibana Lens editor. While preparing for the certification last year, I found a way to achieve this result. The secret is using two different layers and some magic formulas. This post will explain how math in Lens formulas helps create two data-color visualizations.

We will start with the first example that emphasizes only the highest value of the dataset we are focusing on. The second example describes how to highlight other high values (as shown in the illustration above).

[Note: the tips explained in this blog post can be applied from v 7.15]

Only the highest value

To understand how math helps to separate high values from common ones, let’s start with this first example: emphasizing only the highest value.

We start with a bar horizontal chart:

We need to identify the highest value of the scope we are currently examining. We will use one proper overall_* function: the overall_max(), a pipeline function (equivalent to a pipeline aggregation in Query DSL). 

In our example, we group the flights by country(destination). This means we count the number of flights for each DestCountry (= 1 bucket). The overall_max() will select which bucket has the highest value. 

The math trick here is to divide the number of flights per bucket by the maximum value found among all buckets. Only one bucket will return 1: the bucket matching the max value found by overall_max(). All the other buckets will return a value < 1 and >0. We use floor() to ensure any 0.xxx values are rounded to 0. 

Now, we can multiple it with a count() and we have our formula for the 1st layer!

Layer 1: count()*floor(count()/overall_max(count()))

From here, in Lens Editor, we duplicate the layer to adjust the formula of the second layer containing the rest of the data. We need to append another count() followed by the minus operator to the formula. This is the other trick. In this layer, we just need to ensure the highest value is not represented, which will happen only once. It is when count() = overall_max(), which is = 1 when we divide them.

Layer 2: count() - count()*floor(count()/overall_max(count()))

To achieve a nice merge of these two layers, we need to do the following adjustments in both:

• select bar horizontal stacked

• Vertical axis: change”Rank by” to Custom and ensure Rank function is “Count”

Here is the final setup of the two layers:

Layer 1: count()*floor(count()/overall_max(count()))

Layer 2: count() - count()*floor(count()/overall_max(count()))

This visualization also works well for time series data where you need to quickly highlight which time period (12h in the example below) had the highest number of flights:\

Above the surface

Building on what we have done earlier, we can extend the approach to get other high values above the surface. Let’s see which formula we used to create the visualization in the introduction:

For this visualization, we used a property of the round() function. This function brings in only the values greater than 50% of the highest value.

Let's duplicate our first visualization and swap out the floor() function with round().

Layer 1: count()*round(count()/overall_max(count()))

Layer 2: count() - count()*round(count()/overall_max(count()))

It was an easy fix.\ What if we want to extend the first layer further by adding more high values?\ For instance, we would like all the values above the average.

To do this, we use overall_average() as a new reference value instead of the overall_max () reference to separate the eligible values in Layer 1.

As we are comparing against the average value among all the buckets, the division might return values greater than 1.

Here, the clamp() function nicely solves this issue. 

According to the formula reference, clamp() "limits the value from a minimum to maximum". Combining clamp() and floor() ensures that there are only two possible output values: either the minimum value ( 0 ) or the maximum value ( 1 ) given as parameters.

Applied to our flights dataset, it highlights the country destinations that have more flights than the average:

Layer 1: count()*clamp(floor(count()/overall_average(count())),0,1)

Layer 2: count() - count()*clamp(floor(count()/overall_average(count())),0,1)

It also opens up options for using other dynamic references. For instance, we could place all the values greater than 60% of the highest above the surface ( > 0.6*overall_max(count())). We can tune our formula as follow: 

count()*clamp(floor(count()/(0.6*overall_max(count()) ) ),0,1)

Conclusion

In the first part, we have seen the main tips allowing us to create a two-color histogram:

• Two layers: one for the highest value and one for the remaining values

• Visualization type: bar horizontal/vertical stacked

• To separate the data we use a formula where only the highest value return 1 otherwise 0

Then in the second part, we have seen how we can extend this principle to embrace more high values above the surface. This approach can be summarized as follows:

• Start with layer 1 focusing on the high value: count()*\

• Duplicate the layer and adjust the formula:\ ( count() - count()*\)

Finally, we provide 4 generic formulas that are ready to use to spice up your dashboards:

| | | | ----------------------- | :-----------------------------------------------------: | | 1. Only the highest | | | Layer 1 | count()*floor(count()/overall_max(count())) | | Layer 2 | count() - count()*floor(count()/overall_max(count())) |

| | | | --------------------------------------------------------------------- | :-----------------------------------------------------: | | 2.1. Above the surface : high values (above 50% of the max value) | | | Layer 1 | count()*floor(count()/overall_max(count())) | | Layer 2 | count() - count()*floor(count()/overall_max(count())) |

| | | | ----------------------------------------------------------------- | :--------------------------------------------------------------------: | | 2.2. Above the surface : all values above the overall average | | | Layer 1 | count()*clamp(floor(count()/overall_average(count())),0,1) | | Layer 2 | count() - count()*clamp(floor(count()/overall_average(count())),0,1) |

| | | | --------------------------------------------------------------------------- | :------------------------------------------------------------------------: | | 2.2. Above the surface : all the values greater than 60% of the highest | | | Layer 1 | count()*clamp(floor(count()/(0.6*overall_max(count()) ) ),0,1) | | Layer 2 | count() - count()*clamp(floor(count()/(0.6*overall_max(count()) ) ),0,1) |

Try these examples out for yourself by signing up for a free trial of Elastic Cloud or download the self-managed version of the Elastic Stack for free. If you have additional questions about getting started, head on over to the Kibana forum or check out the Kibana documentation guide.\

Note: Starting from version 8.6, the formulas using clamp() can take advantage of the  function ifelse(), this will make it easier to write. An example can be found in this discussion.

References:

• Designing intuitive Kibana dashboards as a non-designer

• Kibana: Lens editor - use formula to perform math

• Discovering the clamp() function in this discussion (Thanks Marco!)

Kubernetes audit logs are essential for ensuring the security, compliance, and transparency of Kubernetes clusters. However, with managed Kubernetes infrastructure, traditional audit file-based log shipping is often not supported, and audit logs are only available via the control plane API or the Cloud Provider logging facility. In this blog, we will show you how to ingest the audit logs from these other sources and still take advantage of the Elastic Kubernetes Audit Log Integration.

In this blog we will be focusing on AWS as our cloud provider and when ingesting logs from AWS you have several options:

• AWS Custom Logs integration (which we will utilize in this blog)
• AWS Firehose to send logs from Cloudwatch to Elastic
• AWS General integration which supports many AWS sources

Kubernetes auditing documentation describes the need for auditing in order to get answers to the questions below:

• What happened?
• When did it happen?
• Who initiated it?
• What resource did it occur on?
• Where was it observed?
• From where was it initiated (Source IP)?
• Where was it going (Destination IP)?

Answers to the above questions become important when an incident occurs and an investigation follows. Alternatively, it could just be a log retention use case for a regulated company trying to fulfill compliance requirements. 

We are giving special importance to audit logs in Kubernetes because audit logs are not enabled by default. Audit logs can take up a large amount of memory and storage. So, usually, it’s a balance between retaining/investigating audit logs against giving up resources budgeted otherwise for workloads to be hosted on the Kubernetes cluster. Another reason we’re talking about audit logs in Kubernetes is that, unlike usual container logs, after being turned on, these logs are orchestrated to write to the cloud provider’s logging service. This is true for most cloud providers because the Kubernetes control plane is managed by the cloud providers. It makes sense for cloud providers to use their built-in orchestration workflows involving the control plane for a managed service backed by their implementation of a logging framework.

Kubernetes audit logs can be quite verbose by default. Hence, it becomes important to selectively choose how much logging needs to be done so that all the audit requirements are met for the organization. This is done in the audit policy file. The audit policy file is submitted against thekube-apiserver. It is not necessary that all flavors of cloud-provider-hosted Kubernetes clusters allow you to play with the kube-apiserver directly. For example, AWS EKS allows for this logging to be done only by the control plane.

In this blog we will be using Elastic Kubernetes Service (Amazon EKS) on AWS with the Kubernetes Audit Logs that are automatically shipped to AWS CloudWatch.

A sample audit log for a secret by the name “empty-secret” created by an admin user on EKS  is logged on AWS CloudWatch in the following format: 

Once the audit logs show up on CloudWatch, it is time to consider how to transfer them to Elasticsearch. Elasticsearch is a great platform for creating dashboards that visualize different audit events recorded in a Kubernetes cluster. It is also a powerful tool for analyzing various audit events. For example, how many secret object creation attempts were made in an hour? 

Now that we established the Kubernetes audit logs are being logged in CloudWatch, let’s discuss how to get the logs ingested into Elasticsearch. Elasticsearch has an integration to consume logs written on CloudWatch. Just using this integration by default is going to get the JSON from CloudWatch as is i.e. the real audit log JSON is nested inside the wrapper CloudWatch JSON. When bringing logs to Elasticsearch, it is important that we use the Elastic Common Schema(ECS) to get the best search and analytics performance. This means that there needs to  be an ingest pipeline that parses a standard Kubernetes audit JSON message and creates an ECS Compliant document in Elasticsearch. Let’s dive into how to achieve this.

Elasticsearch has a Kubernetes integration using Elastic Agent to consume Kubernetes container logs from the console and audit logs written to a file path. For a cloud-provider use case, as described above, it may not be feasible to write audit logs to a path on the Kubernetes cluster. So, how do we leverage the ECS designed for parsing the Kubernetes audit logs already implemented in the Kubernetes integration to work on the CloudWatch audit logs? That is the most exciting plumbing piece! Let’s see how to do it.

What we’re going to do is:

• Read the Kubernetes audit logs from the cloud provider’s logging module, in our case, AWS CloudWatch since this is where logs reside. We will use Elastic Agent and Elasticsearch AWS Custom Logs integration to read from logs from CloudWatch. Note: please be aware, there are several Elastic AWS integration, we are specifically using the AWS Custom Logs integration.

• Create two simple ingest pipelines (we do this for best practices of isolation and composability) 

• The first pipeline looks for Kubernetes audit JSON messages and then redirects them to the second pipeline

• The second custom pipeline will associate the JSON message field with the correct field expected by the Elasticsearch Kubernetes Audit managed pipeline (aka the Integration) and then reroute the message to the correct data stream, kubernetes.audit_logs-default, which in turn applies all the proper mapping and ingest pipelines for the incoming message

• The overall flow will be

1. Create an AWS CloudWatch integration:

a.  Populate the AWS access key and secret pair values

b. In the logs section, populate the log ARN, Tags and Preserve the original event if you want to, and then Save this integration and exit from the page

2. Next, we will configure the custom ingest pipeline

We are doing this because we want to override what the generic managed pipeline does. We will retrieve the custom component name by searching for managed pipeline created as an asset when we install the AWS CloudWatch integration. In this case we will be adding the custom ingest pipeline logs-aws_logs.generic@custom

From the Dev tools console, run below. Here, we are extracting the message field from the CloudWatch JSON and putting the value in a field called kubernetes.audit. Then, we are rerouting this message to the default Kubernetes audit dataset or ECS that comes with Kubernetes integration

PUT _ingest/pipeline/logs-aws_logs.generic@custom
{
    "processors": [
      {
        "pipeline": {
          "if": "ctx.message.contains('audit.k8s.io')",
          "name": "logs-aws-process-k8s-audit"
        }
      }
    ]
}

PUT _ingest/pipeline/logs-aws-process-k8s-audit
{
  "processors": [
    {
      "json": {
        "field": "message",
        "target_field": "kubernetes.audit"
      }
    },
    {
      "remove": {
        "field": "message"
      }
    },
    {
      "reroute": {
        "dataset": "kubernetes.audit_logs",
        "namespace": "default"
      }
    }
  ]
}

Let’s understand this further:

• When we create a Kubernetes integration, we get a managed index template called logs-kubernetes.audit_logs that writes to the pipeline called logs-kubernetes.audit_logs-1.62.2 by default

• If we look into the pipelinelogs-kubernetes.audit_logs-1.62.2, we see that all the processor logic is working against the field kubernetes.audit. This is the reason why our json processor in the above code snippet is creating a field called kubernetes.auditbefore dropping the original message field and rerouting. Rerouting is directed to the kubernetes.audit_logs dataset that backs the logs-kubernetes.audit_logs-1.62.2 pipeline (dataset name is derived from the pipeline name convention that’s in the format logs-<datasetname>-version)

3. Now let’s verify that the logs are actually flowing through and the audit message is being parsed

a. We will use Elastic Agent and enroll using Fleet and the integration policy we created in the Step 1. There are a number of ways to deploy Elastic Agent and for this exercise we will deploy using docker which is quick and easy.

% docker run --env FLEET_ENROLL=1 --env FLEET_URL=<<fleet_URL>> --env FLEET_ENROLLMENT_TOKEN=<<fleet_enrollment_token>>&nbsp; --rm docker.elastic.co/beats/elastic-agent:8.19.13

b. Check the messages in Discover. In 8.15 there is also a new feature called Logs Explorer which provides an ability to see Kubernetes Audit logs (and container logs) with a few clicks (see image below). Voila! We can see the Kubernetes audit messages parsed!

4. Let's do a quick recap of what we did

We configured CloudWatch integration in Elasticsearch to read Kubernetes audit logs from CloudWatch. Then, we created custom ingest pipelines to reroute the audit messages to the correct data stream and all the OOTB mappings and parsing that come with the Kubernetes Audit Logs integration.

In managing NGINX Elastic provides several capabilities:

1. Easy ingest, parsing, and out-of-the-box dashboards. Check out the simple how-to in our docs. Based on logs, these dashboards show several items over time, response codes, errors, top pages, data volume, browsers used, active connections, drop rates, and much more.

2. Out-of-the-box ML-based anomaly detection jobs for your NGINX logs. These jobs help pinpoint anomalies against request rates, IP address request rates, URL access, status codes, and visitor rate anomalies.

3. ES|QL which helps work through logs and build out charts during analysis.

4. Elastic’s GenAI Assistant provides a simple natural language interface that helps analyze all the logs and can pull out issues from ML jobs and even create dashboards. The Elastic AI Assistant also automatically uses ES|QL.

5. NGINX SLOs - Finally Elastic provides the ability to define and monitor SLOs for your NGINX logs. While most SLOs are metrics-based, Elastic allows you to create logs-based SLOs. We detailed this in a previous blog.

NGINX logs are another example of why logs are great.  Logging is an important part of Observability, for which we generally think of metrics and tracing. However, the amount of logs an application and the underlying infrastructure output can be significantly daunting and NGINX is usually the starting point for most analyses. 

In today’s blog, we’ll cover how the out-of-the-box ML-based anomaly detection jobs can help RCA, and how Elastic’s GenAI Assistant helps easily work through logs to pinpoint issues in minutes. 

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).

• Bring up an NGINX server on a host. OR run an application with NGINX as a front end and drive traffic.

• Install the NGINX integration and assets and review the dashboards as noted in the docs.

• Ensure you have an ML node configured in your Elastic stack

• To use the AI Assistant you will need a trial or upgrade to Platinum.

In our scenario, we use data from 3 months from our Elastic environment to help highlight the features. Hence you might need to run your application with traffic for a specific time frame to follow along.

Analyzing the issues with AI Assistant

As detailed in a previous blog, you can get alerted on issues via SLO monitoring against NGINX logs. Let’s assume you have an SLO based on status codes as we outlined in the previous blog. You can immediately analyze the issue via the AI Assistant. Because it's a chat interface we simply open the AI Assistant and work through some simple analysis: (See Animated GIF for a demo)

AI Assistant analysis:

• Using lens graph all http response status codes < 400 and > =400 from filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer - We wanted to simply understand the amount of requests resulting in status code >= 400 and graph the results. We see that 15% of the requests were not successful, hence an SLO alert being triggered.

• Which ip address (field source.adress) has the highest number of http.response.status.code >= 400 from filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer  - We were curious is there was a specific IP address not having successful requests. 72.57.0.53, with a count of 25,227 occurrences is daily high but not the ensure 2 failed requests.

• What country (source.geo.country_iso_code) is source.address=72.57.0.53 coming from. Use filebeat-nginx-elasticco-anon-2017. - Again we were curious if this came from a specific country. And the IP address 72.57.0.53 is coming from the country with the ISO code IN, which corresponds to India. Nothing out of the ordinary.

• Did source.address=72.57.0.53 have any (http.response.status.code < 400) from filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer -  Oddly the IP address in question only had 4000+ successful responses. Meaning its not malicious, and points to something else.

• What are the different status codes (http.response.status.code>=400), from source.address=72.57.0.53. Use filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer. Provide counts for each status code - We are curious whether or not we see any 502, which there were none, but most of the failures were 404. 

• What are the different status codes (http.response.status.code>=400). Use filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer. Provide counts for each status code - Regardless of a specific address, what is the largest number of status code occurrences > 400. This also points to 404. 

• What does a high 404 count from a specific IP address mean from NGINX logs? - Asking this question, we need to understand the potential causes of this from our application. From the answers, we can rule out security probing and web scraping, as we validated that a specific address 72.57.0.53 has a low non-success request status code. It also rules out User error. Hence this points potentially to Broken Links or Missing Resources.

Watch the flow:

Potential issue:

It seems that we potentially have an issue with the backend serving specific answers or having issues with resources (database, or broken links). This is cursing the higher-than-normal non-successful status codes>=400.

Key highlights from AI Assistant:

As you watched this video you will notice a few things:

1. We analyzed millions of logs in a matter of minutes using a set of simple natural language queries. 

2. We didn’t need to know any special query language. The AI Assistant used Elastic’s ES|QL but can similarly use KQL also. 

3. The AI Assistant easily builds out graphs

4. The AI Assistant is accessing and using internal information stored in Elastic’s indices. Vs a simple “google foo” based AI Assistant. This is enabled through RAG, and the AI Assistant can also bring up known issues in github, runbooks, and other useful internal information.

Check out the following blog on how the AI Assistant uses RAG to retrieve internal information. Specifically using github and runbooks.

Locating anomalies with ML

While using the AI Assistant is great for analyzing information, another important aspect of NGINX log management is to ensure you can manage log spikes and anomalies. Elastic has a machine learning platform that allows you to develop jobs to analyze specific metrics or multiple metrics to look for anomalies.When using NGINX, there are several out-of-the-box anomaly detection jobs. These work specifically on NGINX access logs.

• Low_request_rate_nginx - Detect low request rates

• Source_ip_request_rate_nginx - Detect unusual source IPs - high request rates

• Source_ip_url_count_nginx - Detect unusual source IPs - high distinct count of URLs

• Status_code_rate_nginx - Detect unusual status code rates

• Visitor_rate_nginx - Detect unusual visitor rates

Being right out of the box, lets look at the job - Status_code_rate_nginx, which is related to our previous analysis.

With a few simple clicks we immediately get an analysis showing a specific IP address - 72.57.0.53, having higher than normal non-successful requests. Oddly we also found this is using the AI Assistant.

We can take this further with conversations with the AI Assistant, look at the logs, and/or even look at the other ML anomaly jobs.

Conclusion:

You’ve now seen how easily Elastic’s RAG-based AI Assistant can help analyze NGINX logs without even the need to know query syntax, understand where the data is, and understand even the fields. Additionally, you’ve also seen how we can alert you when a potential issue or degradation in service (SLO). 

Check out other resources on NGINX logs:

Out-of-the-box anomaly detection jobs for NGINX

Using the NGINX integration to ingest and analyze NGINX Logs

NGINX Logs based SLOs in Elastic

Using GitHub issues, runbooks, and other internal information for RCAs with Elastic’s RAG based AI Assistant

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on the cloud? Start a free trial.

All of this is also possible in your environment. Learn how to get started today.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Find the logs you’re looking for

Focus on logs from specific integrations or data views

We've added the Data Source selector, a handy new feature for viewing specific logs. Now, you can easily filter your logs based on your integrations, like System Logs, Nginx, or Elastic APM, or switch between different data views, like logs or metrics. This new selector is all about making your data easier to find and helping you focus on what matters most in your analysis.

Dive into your logs

Analyze logs with Smart Fields in Kibana

Logs in Kibana have undergone a significant transformation, particularly in the way log data is presented. The once-basic table view has evolved with the introduction of Smart Fields, providing users with a more insightful and dynamic log analysis experience.

Resource Smart Field - centralizing log source information

The resource column further elevates the Logs Explorer page by providing users with a single column for exploring the resource that created the log event. This column groups various resource-indicating fields together, streamlining the investigation process. Currently, the following ECS fields are grouped under this single column and we recommend including them in your logs:

• service.name
• container.name
• orchestrator.namespace
• host.name
• cloud.instance.id

We know this does not include all use cases and would like your feedback on other fields you use/are important for you to help us provide a tailored and user-centric log analysis experience.

Content Smart Field - a deeper dive into log data

The content column revolutionizes log analysis by seamlessly rendering log.level and message fields. Notably, it automatically handles fallbacks, ensuring a smooth transition when the actual message field is not available. This enhancement simplifies the log exploration process, offering users a more comprehensive understanding of their data.

Actions column - unleashing additional columns

As part of our commitment to empowering users, we are introducing the actions column, adding a layer of functionality to the document table. This column includes two powerful actions:

• Degraded document indicator: This indicator provides insights about the quality of your data by indicating fields were ignored when the document was indexed and ended up in the _ignored property of the document. To help analyze what caused the document to degrade, we suggest reading this blog - The antidote for index mapping exceptions: ignore_malformed.
• Stacktrace indicator: This indicator informs users of the presence of stack traces in the document. This makes it easy to navigate through logs documents and know if they have additional information.

Investigate individual logs by expanding log details

Now, when you click the expand icon in the actions column, it opens up the Log details flyout for any log entry. This new feature gives you a detailed overview of the entry right at your fingertips. Inside the flyout, the Overview tab is neatly organized into four sections—Content breakdown, Service & Infrastructure, Cloud, and Others—each offering a snapshot of the most crucial information. Plus, you'll find the same handy controls you're used to in the main table, like filtering in or out, adding or removing columns, and copying data, making it easier than ever to manage your logs directly from the flyout.

The Observability AI Assistant is fully integrated into this view providing contextual insights about the log event and helping to find similar messages.

Experience a streamlined approach to log exploration

These enhancements simplify the process of finding and focusing on specific logs and offer more intuitive and insightful data presentation. Dive into your logs with these I tools and streamline your workflow, uncovering deeper insights with ease. Try it now and transform your log analysis!

1. Where traffic is coming in from and going out to from the deployment, and within the deployment. This helps identify unusual or unauthorized communications

2. Traffic volumes detecting spikes or drops which could indicate service issues in production or an increase in customer traffic

3. Latency and Performance bottlenecks - with VPC Flow logs, you can look at latency for a flow (in and outflows), and understand patterns

4. Accepted and rejected traffic helps determine where potential security threats and misconfigurations lie. 

AWS VPC Logs is a great example of how logs are great. Logging is an important part of Observability, for which we generally think of metrics and tracing. However, the amount of logs an application and the underlying infrastructure output can be significantly daunting with VPC Logs. However, it also provides a significant amount of insight.

Before we proceed, it is important to understand what Elastic provides in managing AWS and VPC Flow logs:

1. A full set of integrations to manage VPC Flows and the entire end-to-end deployment on AWS. 

2. Elastic has a simple-to-use AWS Firehose integration. 

3. Elastic’s tools such as Discover, spike analysis,  and anomaly detection help provide you with better insights and analysis.

4. And a set of simple Out-of-the-box dashboards

In today’s blog, we’ll cover how Elastics’ other features can support analyzing and RCA for potential VPC flow logs even more easily. Specifically, we will focus on managing the number of rejects, as this helps ensure there weren’t any unauthorized or unusual activities:

1. Set up an easy-to-use SLO (newly released) to detect when things are potentially degrading

2. Create an ML job to analyze different fields of the VPC Flow log

3. Using our newly released RAG-based AI Assistant to help analyze the logs without needing to know Elastic’s query language nor how to even graph on Elastic

4. ES|QL will help understand and analyze add latency for patterns.

In subsequent blogs, we will use AI Assistant and ESQL to show how to get other insights beyond just REJECT/ACCEPT from VPC Flow log.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).

• Follow the steps in the following blog to get AWS’s three-tier app installed instructed in git, and bring in the AWS VPC Flow logs.

• Ensure you have an ML node configured in your Elastic stack

• To use the AI Assistant you will need a trial or upgrade to Platinum.

SLO with VPC Flow Logs

Elastic’s SLO capability is based directly on the Google SRE Handbook. All the definitions and semantics are utilized as described in Google’s SRE handbook. Hence users can perform the following on SLOs in Elastic:

• Define an SLO on Logs not just metrics - Users can use KQL (log-based query), service availability, service latency, custom metric, histogram metric, or a timeslice metric.
• Define SLO, SLI, Error budget and burn rates. Users can also use occurrence versus time slice-based budgeting. 
• Manage, with dashboards, all the SLOs in a singular location.
• Trigger alerts from the defined SLO, whether the SLI is off, the burn rate is used up, or the error rate is X.

Setting up an SLO for VPC is easy. You simply create a query you want to trigger off. In our case, we look for all the good events where aws.vpcflow.action=ACCEPT and we define the target at 85%. 

As the following example shows, over the last 7 days, we have exceeded our budget by 43%. Additionally, we have not complied for the last 7 days.

Analyzing the SLO with AI Assistant

Now that we see that there is an issue with the VPC Flows, we immediately work with the AI Assistant to start analyzing the SLO. Because it's a chat interface we simply open the AI Assistant and work through some simple analysis: (See Animated GIF for a demo below)

AI Assistant analysis:

• what were the top 3 source.address that had aws.vpcflow.action=REJECT over the last 7 days, which is causing this SLO issue? - We wanted to simply see what could be causing the loss in error budget. Were there any particular source.addresses causing a heavy reject rate.

• The answer: A table with the highest count = 42670 and source.address = 79.110.62.185

• There is one singular source.address that is causing the loss in SLO. 

• What is the largest number of  aws.vpcflow.action=REJECT in a 30 min time frame for the last 3 days where the source.address=79.110.62.185? - After understanding that a specific source.address is causing the loss in SLO, we want to understand the averages. 

• **The answer: ** "The largest number of aws.vpcflow.action=REJECT in a 30-minute time frame for the last 3 days where the source.address is 79.110.62.185 is 229. This occurred on 2024-06-01T04:00:00.000Z.”

• It means there must be a low REJECT rate but fairly consistent vs spiky over the last 7 days.

• for the logs with source.address="79.110.62.185" was there any country code of source.geo.country_iso_code field present. If yes what is the value - Given the last question showed a low REJECT rate, it only means that this was fairly consistent vs spiky over the last 7 days.

• The answer: Yes, there is a country code present in the source.geo.country_iso_code field for logs with source.address="79.110.62.185". The value is BG (Bulgaria).

• Is there a specific destination.address where source.address=79.110.62.185 is getting a aws.vpcflow.action=REJECT. Give me both the destination.address and the number of REJECTs for that destination.address?

• The answer: destination.address of 10.0.0.27 is giving a reject number of 53433 in this time frame.

• Graph the number of REJECT vs ACCEPT for source.address="79.110.62.185" over the last 7 days. The graph is on a daily basis in a singular graph - We asked this question to see what the comparison is between ACCEPT and REJECT. 

• The answer: See the animated GIF to see that the generated graph is fairly stable

• Were there any source.address that had a spike, high reject rate in. a 30min period over the 30 days? - We wanted to see if there was any other spike 

• The answer - Yes, there was a source.address that had a spike in high reject rates in a 30-minute period over the last 30 days. source.address: 185.244.212.67, Reject Count: 8975, Time Period: 2024-05-22T03:00:00.000Z

Watch the flow

Potential issue:

he server handling requests from source 79.110.62.185 is potentially having an issue.

Again using logs, we essentially asked the AI Assistant to give the eni ids where the internal ip address was 10.0.0.27

From our AWS console, we know that this is the webserver. Further analysis in Elastic, and with the developers we realized there is a new version that was installed recently causing a problem with connections.

Locating anomalies with ML

While using the AI Assistant is great for analyzing information, another important aspect of VPC flow management is to ensure you can manage log spikes and anomalies. Elastic has a machine learning platform that allows you to develop jobs to analyze specific metrics or multiple metrics to look for anomalies.

VPC Flow logs come with a large amount of information. The full set of fields is listed in AWS docs. We will use a specific subset to help detect anomalies.

We were setting up anomalies for aws.vpcflow.action=REJECT, which requires us to use multimetric anomaly detection in Elastic.

The config we used utilizes:

Detectors:

• destination.address

• destination.port

Influencers:

• source.address

• aws.vpcflow.action

• destination.geo.region_iso_code

The way we set this up will help us understand if there is a large spike in REJECT/ACCEPT against destination.address values from a specific source.address and/or destination.geo.region_iso_code location.

The job once run reveals something interesting:

Notice that source.address 185.244.212.67 has had a high REJECT rate in the last 30 days. 

Notice where we found this before? In the AI Assistant!!!!!

While we can run the AI Assistant and find this sort of anomaly, the ML job can be setup to run continuously and alert us on such spikes. This will help us understand if there are any issues with the webserver like we found above or even potential security attacks.

Conclusion:

You’ve now seen how easily Elastic’s RAG-based AI Assistant can help analyze VPC Flows without even the need to know query syntax, understand where the data is, and understand even the fields. Additionally, you’ve also seen how we can alert you when a potential issue or degradation in service (SLO). Check out our other blogs on AWS VPC Flow analysis in Elastic:

1. A full set of integrations to manage VPC Flows and the entire end-to-end deployment on AWS. 

2. Elastic has a simple-to-use AWS Firehose integration. 

3. Elastic’s tools such as Discover, spike analysis,  and anomaly detection help provide you with better insights and analysis.

4. And a set of simple Out-of-the-box dashboards

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on the cloud? Start a free trial.

All of this is also possible in your environment. Learn how to get started today.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Site reliability engineers (SREs) face the challenging task of sifting through vast quantities of data, not just from the applications themselves but also from the underlying infrastructure. In addition to data analysis, they are responsible for ensuring the effective use and development of operational tools. The growing volume of data, the daily resolution of issues, and the continuous evolution of tools and processes can detract from the focus on business performance.

Elastic Observability offers a solution to this challenge. It enables SREs to integrate and examine all telemetry data (logs, metrics, traces, and profiling) in conjunction with business metrics. This comprehensive approach to data analysis fosters operational excellence, boosts productivity, and yields critical insights, all of which are integral to maintaining high-performing applications in a demanding digital environment.

To help manage operations and business metrics, Elastic Observability's SLO (Service Level Objectives) feature was introduced in 8.12. This feature enables setting measurable performance targets for services, such as availability, latency, traffic, errors, and saturation or define your own. Key components include:

• Defining and monitoring SLIs (Service Level Indicators)

• Monitoring error budgets indicating permissible performance shortfalls

• Alerting on burn rates showing error budget consumption

Users can monitor SLOs in real-time with dashboards, track historical performance, and receive alerts for potential issues. Additionally, SLO dashboard panels offer customized visualizations.

Service Level Objectives (SLOs) are generally available for our Platinum and Enterprise subscription customers.

In this blog, we will outline the following:

• What are SLOs? A Google SRE perspective

• Several scenarios of defining and managing SLOs

Service Level Objective overview

Service Level Objectives (SLOs) are a crucial component for Site Reliability Engineering (SRE), as detailed in Google's SRE Handbook. They provide a framework for quantifying and managing the reliability of a service. The key elements of SLOs include:

• Service Level Indicators (SLIs): These are carefully selected metrics, such as uptime, latency, throughput, error rates, or other important metrics, that represent the aspects of the service and are important from an operations or business perspective. Hence, an SLI is a measure of the service level provided (latency, uptime, etc.), and it is defined as a ratio of good over total events, with a range between 0% and 100%.

• Service Level Objective (SLO): An SLO is the target value for a service level measured as a percentage by an SLI. Above the threshold, the service is compliant. As an example, if we want to use service availability as an SLI, with the number of successful responses at 99.9%, then any time the number of failed responses is > .1%, the SLO will be out of compliance.

• Error budget: This represents the threshold of acceptable errors, balancing the need for reliability with practical limits. It is defined as 100% minus the SLO quantity of errors that is tolerated.

• Burn rate: This concept relates to how quickly the service is consuming its error budget, which is the acceptable threshold for unreliability agreed upon by the service providers and its users.

Understanding these concepts and effectively implementing them is essential for maintaining a balance between innovation and reliability in service delivery. For more detailed information, you can refer to Google's SRE Handbook.

One main thing to remember is that SLO monitoring is not incident monitoring. SLO monitoring is a proactive, strategic approach designed to ensure that services meet established performance standards and user expectations. It involves tracking Service Level Objectives, error budgets, and the overall reliability of a service over time. This predictive method helps in preventing issues that could impact users and aligns service performance with business objectives.

In contrast, incident monitoring is a reactive process focused on detecting, responding to, and mitigating service incidents as they occur. It aims to address unexpected disruptions or failures in real time, minimizing downtime and impact on service. This includes monitoring system health, errors, and response times during incidents, with a focus on rapid response to minimize disruption and preserve the service's reputation.

Elastic®’s SLO capability is based directly off the Google SRE Handbook. All the definitions and semantics are utilized as described in Google’s SRE handbook. Hence users can perform the following on SLOs in Elastic:

• Define an SLO on an SLI such as KQL (log based query), service availability, service latency, custom metric, histogram metric, or a timeslice metric. Additionally, set the appropriate threshold.

• Utilize occurrence versus time slice based budgeting. Occurrences is the number of good events over the number of total events to compute the SLO. Timeslices break the overall time window into slammer slices of a defined duration and compute the number of good slices over the total slices to compute the SLO. Timeslice targets are more accurate and useful when calculating things like a service’s SLO when trying to meet agreed upon customer targets.

• Manage all the SLOs in a singular location.

• Trigger alerts from the defined SLO, whether the SLI is off, burn rate is used up, or the error rate is X.

• Create unique service level dashboards with SLO information for a more comprehensive view of the service.

SREs need to be able to manage business metrics.

SLOs based on logs: NGINX availability

Defining SLOs does not always mean metrics need to be used. Logs are a rich form of information, even when they have metrics embedded in them. Hence it’s useful to understand your business and operations status based on logs.

Elastic allows you to create an SLO based on specific fields in the log message, which don’t have to be metrics. A simple example is a simple multi-tier app that has a web server layer (nginx), a processing layer, and a database layer.

Let’s say that your processing layer is managing a significant number of requests. You want to ensure that the service is up properly. The best way is to ensure that all http.response.status_code are less than 500. Anything less ensures the service is up and any errors (like 404) are all user or client errors versus server errors.

If we use Discover in Elastic, we see that there are close to 2M log messages over a seven-day time frame.

Additionally, the number of messages with http.response.status_code > 500 is minimal, like 17K.

Rather than creating an alert, we can create an SLO with this query:

We chose to use occurrences as the budgeting method to keep things simple.

Once defined, we can see how well our SLO is performing over a seven-day time frame. We can see not only the SLO, but also the burn rate, the historical SLI, and error budget, and any specific alerts against the SLO.

Not only do we get information about the violation, but we also get:

• Historical SLI (7 days)

• Error budget burn down

• Good vs. bad events (24 hours)

We can see how we’ve easily burned through our error budget.

Hence something must be going on with nginx. To investigate, all we need to do is utilize the AI Assistant, and use its natural language interface to ask questions to help analyze the situation.

Let’s use Elastic’s AI Assistant to analyze the breakdown of http.response.status_code across all the logs from the past seven days. This helps us understand how many 50X errors we are getting.

As we can see, the number of 502s is minimal compared to the number of overall messages, but it is affecting our SLO.

However, it seems like Nginx is having an issue. In order to reduce the issue, we also ask the AI Assistant how to work on this error. Specifically, we ask if there is an internal runbook the SRE team has created.

AI Assistant gets a runbook the team has added to its knowledge base. I can now analyze and try to resolve or reduce the issue with nginx.

While this is a simple example, there are an endless number of possibilities that can be defined based on KQL. Some other simple examples:

• 99% of requests occur under 200ms

• 99% of log message are not errors

Application SLOs: OpenTelemetry demo cartservice

A common application developers and SREs use to learn about OpenTelemetry and test out Observability features is the OpenTelemetry demo.

This demo has feature flags to simulate issues. With Elastic’s alerting and SLO capability, you can also determine how well the entire application is performing and how well your customer experience is holding up when these feature flags are used.

Elastic supports OpenTelemetry by taking OTLP directly with no need for an Elastic specific agent. You can send in OpenTelemetry data directly from the application (through OTel libraries) and through the collector.

We’ve brought up the OpenTelemetry demo on a K8S cluster (AWS EKS) and turned on the cartservice feature flag. This inserts errors into the cartservice. We’ve also created two SLOs to monitor the cartservice’s availability and latency.

We can see that the cartservice’s availability is violated. As we drill down, we see that there aren’t as many successful transactions, which is affecting the SLO.

As we drill into the service, we can see in Elastic APM that there is a higher than normal failure rate of about 5.5% for the emptyCart service.

We can investigate this further in APM, but that is a discussion for another blog. Stay tuned to see how we can use Elastic’s machine learning, AIOps, and AI Assistant to understand the issue.

Conclusion

SLOs allow you to set clear, measurable targets for your service performance, based on factors like availability, response times, error rates, and other key metrics. Hopefully with the overview we’ve provided in this blog, you can see that:

• SLOs can be based on logs. In Elastic, you can use KQL to essentially find and filter on specific logs and log fields to monitor and trigger SLOs.

• AI Assistant is a valuable, easy-to-use capability to analyze, troubleshoot, and even potentially resolve SLO issues.

• APM Service based SLOs are easy to create and manage with integration to Elastic APM. We also use OTel telemetry to help monitor SLOs.

For more information on SLOs in Elastic, check out Elastic documentation and the following resources:

• What’s new in Elastic Observability 8.12

• Introducing the Elastic AI Assistant

• Elastic OpenTelemetry support

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I’ve outlined above to get the most value and visibility out of your SLOs.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

As a single platform, Elastic® helps SREs unify and correlate limitless telemetry data, including metrics, logs, traces, and profiling, into a single datastore — Elasticsearch®. By then applying the power of Elastic’s advanced machine learning (ML), AIOps, AI Assistant, and analytics, you can break down silos and turn data into insights. As a full-stack observability solution, everything from infrastructure monitoring to log monitoring and application performance monitoring (APM) can be found in a single, unified experience.

In Elastic 8.11, a technical preview is now available of Elastic’s new piped query language, ES|QL (Elasticsearch Query Language), which transforms, enriches, and simplifies data investigations. Powered by a new query engine, ES|QL delivers advanced search capabilities with concurrent processing, improving speed and efficiency, irrespective of data source and structure. Accelerate resolution by creating aggregations and visualizations from one screen, delivering an iterative, uninterrupted workflow.

Advantages of ES|QL for SREs

SREs using Elastic Observability can leverage ES|QL to analyze logs, metrics, traces, and profiling data, enabling them to pinpoint performance bottlenecks and system issues with a single query. SREs gain the following advantages when managing high dimensionality and high cardinality data with ES|QL in Elastic Observability:

• Improved operational efficiency: By using ES|QL, SREs can create more actionable notifications with aggregated values as thresholds from a single query, which can also be managed through the Elastic API and integrated into DevOps processes.
• Enhanced analysis with insights: ES|QL can process diverse observability data, including application, infrastructure, business data, and more, regardless of the source and structure. ES|QL can easily enrich the data with additional fields and context, allowing the creation of visualizations for dashboards or issue analysis with a single query.
• Reduced mean time to resolution: ES|QL, when combined with Elastic Observability's AIOps and AI Assistant, enhances detection accuracy by identifying trends, isolating incidents, and reducing false positives. This improvement in context facilitates troubleshooting and the quick pinpointing and resolution of issues.

ES|QL in Elastic Observability not only enhances an SRE's ability to manage the customer experience, an organization's revenue, and SLOs more effectively but also facilitates collaboration with developers and DevOps by providing contextualized aggregated data.

In this blog, we will cover some of the key use cases SREs can leverage with ES|QL:

• ES|QL integrated with the Elastic AI Assistant, which uses public LLM and private data, enhances the analysis experience anywhere in Elastic Observability.
• SREs can, in a single ES|QL query, break down, analyze, and visualize observability data from multiple sources and across any time frame.
• Actionable alerts can be easily created from a single ES|QL query, enhancing operations.

I will work through these use cases by showcasing how an SRE can solve a problem in an application instrumented with OpenTelemetry and running on Kubernetes. The OpenTelemetry (OTel) demo is on an Amazon EKS cluster, with Elastic Cloud 8.11 configured.

You can also check out our Elastic Observability ES|QL Demo, which walks through ES|QL functionality for Observability.

ES|QL with AI Assistant

As an SRE, you are monitoring your OTel instrumented application with Elastic Observability, and while in Elastic APM, you notice some issues highlighted in the service map.

Using Elastic AI Assistant, you can easily ask for analysis, and in particular, we check on what the overall latency is across the application services.

My APM data is in traces-apm*. What's the average latency per service over the last hour? Use ESQL, the data is mapped to ECS

The Elastic AI Assistant generates an ES|QL query, which we run in the AI Assistant to get a list of the average latencies across all the application services. We can easily see the top four are:

• load generator
• front-end proxy
• frontendservice
• checkoutservice

With a simple natural language query in the AI Assistant, it generated a single ES|QL query that helped list out the latencies across the services.

Noticing that there is an issue with several services, we decide to start with the frontend proxy. As we work through the details, we see significant failures, and through Elastic APM failure correlation , it becomes apparent that the frontend proxy is not properly completing its calls to downstream services.

ES|QL insightful and contextual analysis in Discover

Knowing that the application is running on Kubernetes, we investigate if there are issues in Kubernetes. In particular, we want to see if there are any services having issues.

We use the following query in ES|QL in Elastic Discover:

from metrics-* | where kubernetes.container.status.last_terminated_reason != "" and kubernetes.namespace == "default" | stats reason_count=count(kubernetes.container.status.last_terminated_reason) by kubernetes.container.name, kubernetes.container.status.last_terminated_reason | where reason_count > 0

ES|QL helps analyze 1,000s/10,000s of metric events from Kubernetes and highlights two services that are restarting due to OOMKilled.

The Elastic AI Assistant, when asked about OOMKilled, indicates that a container in a pod was killed due to an out-of-memory condition.

We run another ES|QL query to understand the memory usage for emailservice and productcatalogservice.

ES|QL easily found the average memory usage fairly high.

We can now further investigate both of these services’ logs, metrics, and Kubernetes-related data. However, before we continue, we create an alert to track heavy memory usage.

Actionable alerts with ES|QL

Suspecting a specific issue, that might recur, we simply create an alert that brings in the ES|QL query we just ran that will track for any service that exceeds 50% in memory utilization.

We modify the last query to find any service with high memory usage:

FROM metrics*
| WHERE @timestamp >= NOW() - 1 hours
| STATS avg_memory_usage = AVG(kubernetes.pod.memory.usage.limit.pct) BY kubernetes.deployment.name | where avg_memory_usage > .5

With that query, we create a simple alert. Notice how the ES|QL query is brought into the alert. We simply connect this to pager duty. But we can choose from multiple connectors like ServiceNow, Opsgenie, email, etc.

With this alert, we can now easily monitor for any services that exceed 50% memory utilization in their pods.

Make the most of your data with ES|QL

In this post, we demonstrated the power ES|QL brings to analysis, operations, and reducing MTTR. In summary, the three use cases with ES|QL in Elastic Observability are as follows:

• ES|QL integrated with the Elastic AI Assistant, which uses public LLM and private data, enhances the analysis experience anywhere in Elastic Observability.
• SREs can, in a single ES|QL query, break down, analyze, and visualize observability data from multiple sources and across any time frame.
• Actionable alerts can be easily created from a single ES|QL query, enhancing operations.

Elastic invites SREs and developers to experience this transformative language firsthand and unlock new horizons in their data tasks. Try it today at https://ela.st/free-trial now in technical preview.

• Elastic Observability Tour
• The power of effective log management
• Transforming Observability with the AI Assistant
• ES|QL announcement blog

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

OpenSearch is a fork of Elasticsearch 7.10 that has diverged quite a bit from itself lately, resulting in a different set of features and also different performance, as this benchmark shows (hint: it’s currently much slower than Elasticsearch).

Given the differences between the two solutions, restoring a snapshot from OpenSearch is not possible, nor is reindex-from-remote, so our only option is then using something in between that will read from OpenSearch and write to Elasticsearch.

This blog will show you how easy it is to migrate from OpenSearch to Elasticsearch for better performance and less disk usage!

1 billion log lines

We are going to use part of the data set we used for the benchmark, which takes about half a terabyte on disk, including replicas, and spans over a week ( January 1–7, 2023).

We have in total 1,009,165,775 documents that take 453.5GB of space in OpenSearch, including the replicas. That’s 241.2KB per document. This is going to be important later when we enable a couple optimizations in Elasticsearch that will bring this total size way down without sacrificing performance!

This billion log line data set is spread over nine indices that are part of a datastream we are calling logs-myapplication-prod. We have primary shards of about 25GB in size, according to the best practices for optimal shard sizing. A GET _cat/indices show us the indices we are dealing with:

index                              docs.count pri rep pri.store.size store.size
.ds-logs-myapplication-prod-000049  102519334   1   1         22.1gb     44.2gb
.ds-logs-myapplication-prod-000048  114273539   1   1         26.1gb     52.3gb
.ds-logs-myapplication-prod-000044  111093596   1   1         25.4gb     50.8gb
.ds-logs-myapplication-prod-000043  113821016   1   1         25.7gb     51.5gb
.ds-logs-myapplication-prod-000042  113859174   1   1         24.8gb     49.7gb
.ds-logs-myapplication-prod-000041  112400019   1   1         25.7gb     51.4gb
.ds-logs-myapplication-prod-000040  113362823   1   1         25.9gb     51.9gb
.ds-logs-myapplication-prod-000038  110994116   1   1         25.3gb     50.7gb
.ds-logs-myapplication-prod-000037  116842158   1   1         25.4gb     50.8gb

Both OpenSearch and Elasticsearch clusters have the same configuration: 3 nodes with 64GB RAM and 12 CPU cores. Just like in the benchmark, the clusters are running in Kubernetes.

Moving data from A to B

Typically, moving data from one Elasticsearch cluster to another is easy as a snapshot and restore if the clusters are compatible versions of each other or a reindex from remote if you need real-time synchronization and minimized downtime. These methods do not apply when migrating data from OpenSearch to Elasticsearch because the projects have significantly diverged from the 7.10 fork. However, there is one method that will work: scrolling.

Scrolling

Scrolling involves using an external tool, such as Logstash^®, to read data from the source cluster and write it to the destination cluster. This method provides a high degree of customization, allowing us to transform the data during the migration process if needed. Here are a couple advantages of using Logstash:

• Easy parallelization: It’s really easy to write concurrent jobs that can read from different “slices” of the indices, essentially maximizing our throughput.
• Queuing: Logstash automatically queues documents before sending.
• Automatic retries: In the event of a failure or an error during data transmission, Logstash will automatically attempt to resend the data; moreover, it will stop querying the source cluster as often, until the connection is re-established, all without manual intervention.

Scrolling allows us to do an initial search and to keep pulling batches of results from Elasticsearch until there are no more results left, similar to how a “cursor” works in relational databases.

A scrolled search takes a snapshot in time by freezing the segments that make the index up until the time the request is made, preventing those segments from merging. As a result, the scroll doesn’t see any changes that are made to the index after the initial search request has been made.

Migration strategies

Reading from A and writing in B in can be slow without optimization because it involves paginating through the results, transferring each batch over the network to Logstash, which will assemble the documents in another batch and then transfer those batches over the network again to Elasticsearch, where the documents will be indexed. So when it comes to such large data sets, we must be very efficient and extract every bit of performance where we can.

Let’s start with the facts — what do we know about the data we need to transfer? We have nine indices in the datastream, each with about 100 million documents. Let’s test with just one of the indices and measure the indexing rate to see how long it takes to migrate. The indexing rate can be seen by activating the monitoring functionality in Elastic^® and then navigating to the index you want to inspect.

Scrolling in the deep
The simplest approach for transferring the log lines over would be to make Elasticsearch scroll over the entire data set and check it later when it finishes. Here we will introduce our first two variables: PAGE_SIZE and BATCH_SIZE. The former is how many records we are going to bring from the source every time we query it, and the latter is how many documents are going to be assembled together by Logstash and written to the destination index.

With such a large data set, the scroll slows down as this deep pagination progresses. The indexing rate starts at 6,000 docs/second and steadily descends down to 700 docs/second because the pagination gets very deep. Without any optimization, it would take us 19 days (!) to migrate the 1 billion documents. We can do better than that!

Slice me nice
We can optimize scrolling by using an approach called Sliced scroll, where we split the index in different slices to consume them independently.

Here we will introduce our last two variables: SLICES and WORKERS. The amount of slices cannot be too small as the performance decreases drastically over time, and it can’t be too big as the overhead of maintaining the scrolls would counter the benefits of a smaller search.

Let’s start by migrating a single index (out of the nine we have) with different parameters to see what combination gives us the highest throughput.

| | | | | | | ------ | --------- | ------- | ---------- | --------------------- | | SLICES | PAGE_SIZE | WORKERS | BATCH_SIZE | Average Indexing Rate | | 3 | 500 | 3 | 500 | 13,319 docs/sec | | 3 | 1,000 | 3 | 1,000 | 13,048 docs/sec | | 4 | 250 | 4 | 250 | 10,199 docs/sec | | 4 | 500 | 4 | 500 | 12,692 docs/sec | | 4 | 1,000 | 4 | 1,000 | 10,900 docs/sec | | 5 | 500 | 5 | 500 | 12,647 docs/sec | | 5 | 1,000 | 5 | 1,000 | 10,334 docs/sec | | 5 | 2,000 | 5 | 2,000 | 10,405 docs/sec | | 10 | 250 | 10 | 250 | 14,083 docs/sec | | 10 | 250 | 4 | 1,000 | 12,014 docs/sec | | 10 | 500 | 4 | 1,000 | 10,956 docs/sec |

It looks like we have a good set of candidates for maximizing the throughput for a single index, in between 12K and 14K documents per second. That doesn't mean we have reached our ceiling. Even though search operations are single threaded and every slice will trigger sequential search operations to read data, that does not prevent us from reading several indices in parallel.

By default, the maximum number of open scrolls is 500 — this limit can be updated with the search.max_open_scroll_context cluster setting, but the default value is enough for this particular migration.

Let’s migrate

Preparing our destination indices

We are going to create a datastream called logs-myapplication-reindex to write the data to, but before indexing any data, let’s ensure our index template and index lifecycle management configurations are properly set up. An index template acts as a blueprint for creating new indices, allowing you to define various settings that should be applied consistently across your indices.

Index lifecycle management policy
Index lifecycle management (ILM) is equally vital, as it automates the management of indices throughout their lifecycle. With ILM, you can define policies that determine how long data should be retained, when it should be rolled over into new indices, and when old indices should be deleted or archived. Our policy is really straightforward:

PUT _ilm/policy/logs-myapplication-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "25gb"
          }
        }
      },
      "warm": {
        "min_age": "0d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      }
    }
  }
}

Index template (and saving 23% in disk space)
Since we are here, we’re going to go ahead and enable Synthetic Source, a clever feature that allows us to store and discard the original JSON document while still reconstructing it when needed from the stored fields.

For our example, enabling Synthetic Source resulted in a remarkable 23.4% improvement in storage efficiency , reducing the size required to store a single document from 241.2KB in OpenSearch to just 185KB in Elasticsearch.

Our full index template is therefore:

PUT _index_template/logs-myapplication-reindex
{
  "index_patterns": [
    "logs-myapplication-reindex"
  ],
  "priority": 500,
  "data_stream": {},
  "template": {
    "settings": {
      "index": {
        "lifecycle.name": "logs-myapplication-lifecycle-policy",
        "codec": "best_compression",
        "number_of_shards": "1",
        "number_of_replicas": "1",
        "query": {
          "default_field": [
            "message"
          ]
        }
      }
    },
    "mappings": {
      "_source": {
        "mode": "synthetic"
      },
      "_data_stream_timestamp": {
        "enabled": true
      },
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "agent": {
          "properties": {
            "ephemeral_id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "aws": {
          "properties": {
            "cloudwatch": {
              "properties": {
                "ingestion_time": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "log_group": {
                  "type": "keyword",
                  "ignore_above": 1024
                },
                "log_stream": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            }
          }
        },
        "cloud": {
          "properties": {
            "region": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "data_stream": {
          "properties": {
            "dataset": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "namespace": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "event": {
          "properties": {
            "dataset": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            },
            "ingested": {
              "type": "date"
            }
          }
        },
        "host": {
          "type": "object"
        },
        "input": {
          "properties": {
            "type": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "path": {
                  "type": "keyword",
                  "ignore_above": 1024
                }
              }
            }
          }
        },
        "message": {
          "type": "match_only_text"
        },
        "meta": {
          "properties": {
            "file": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "metrics": {
          "properties": {
            "size": {
              "type": "long"
            },
            "tmin": {
              "type": "long"
            }
          }
        },
        "process": {
          "properties": {
            "name": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "tags": {
          "type": "keyword",
          "ignore_above": 1024
        }
      }
    }
  }
}

Building a custom Logstash image

We are going to use a containerized Logstash for this migration because both clusters are sitting on a Kubernetes infrastructure, so it's easier to just spin up a Pod that will communicate to both clusters.

Since OpenSearch is not an official Logstash input, we must build a custom Logstash image that contains the logstash-input-opensearch plugin. Let’s use the base image from docker.elastic.co/logstash/logstash:9.3.2 and just install the plugin:

FROM docker.elastic.co/logstash/logstash:9.3.2

USER logstash
WORKDIR /usr/share/logstash
RUN bin/logstash-plugin install logstash-input-opensearch

Writing a Logstash pipeline

Now we have our Logstash Docker image, and we need to write a pipeline that will read from OpenSearch and write to Elasticsearch.

The input

input {
    opensearch {
        hosts => ["os-cluster:9200"]
        ssl => true
        ca_file => "/etc/logstash/certificates/opensearch-ca.crt"
        user => "${OPENSEARCH_USERNAME}"
        password => "${OPENSEARCH_PASSWORD}"
        index => "${SOURCE_INDEX_NAME}"
        slices => "${SOURCE_SLICES}"
        size => "${SOURCE_PAGE_SIZE}"
        scroll => "5m"
        docinfo => true
        docinfo_target => "[@metadata][doc]"
    }
}

Let’s break down the most important input parameters. The values are all represented as environment variables here:

• hosts: Specifies the host and port of the OpenSearch cluster. In this case, it’s connecting to “os-cluster” on port 9200.
• index: Specifies the index in the OpenSearch cluster from which to retrieve logs. In this case, it’s “logs-myapplication-prod” which is a datastream that contains the actual indices (e.g., .ds-logs-myapplication-prod-000049).
• size: Specifies the maximum number of logs to retrieve in each request.
• scroll: Defines how long a search context will be kept open on the OpenSearch server. In this case, it’s set to “5m,” which means each request must be answered and a new “page” asked within five minutes.
• docinfo and docinfo_target: These settings control whether document metadata should be included in the Logstash output and where it should be stored. In this case, document metadata is being stored in the [@metadata][doc] field — this is important because the document’s _id will be used as the destination id as well.

The ssl and ca_file are highly recommended if you are migrating from clusters that are in a different infrastructure (separate cloud providers). You don’t need to specify a ca_file if your TLS certificates are signed by a public authority, which is likely the case if you are using a SaaS and your endpoint is reachable over the internet. In this case, only ssl => true would suffice. In our case, all our TLS certificates are self-signed, so we must also provide the Certificate Authority (CA) certificate.

The (optional) filter
We could use this to drop or alter the documents to be written to Elasticsearch if we wanted, but we are not going to, as we want to migrate the documents as is. We are only removing extra metadata fields that Logstash includes in all documents, such as "@version" and "host". We are also removing the original "data_stream" as it contains the source data stream name, which might not be the same in the destination.

filter {
    mutate {
        remove_field => ["@version", "host", "data_stream"]
    }
}

The output
The output is really simple — we are going to name our datastream logs-myapplication-reindex and we are using the document id of the original documents in document_id, to ensure there are no duplicate documents. In Elasticsearch, datastream names follow a convention \<type>-\<dataset>-\<namespace> so our logs-myapplication-reindex datastream has “myapplication” as dataset and “prod” as namespace.

elasticsearch {
    hosts => "${ELASTICSEARCH_HOST}"

    user => "${ELASTICSEARCH_USERNAME}"
    password => "${ELASTICSEARCH_PASSWORD}"

    document_id => "%{[@metadata][doc][_id]}"

    data_stream => "true"
    data_stream_type => "logs"
    data_stream_dataset => "myapplication"
    data_stream_namespace => "prod"
}

Deploying Logstash

We have a few options to deploy Logstash: it can be deployed locally from the command line, as a systemd service, via docker, or on Kubernetes.

Since both of our clusters are deployed in a Kubernetes environment, we are going to deploy Logstash as a Pod referencing our Docker image created earlier. Let’s put our pipeline inside a ConfigMap along with some configuration files (pipelines.yml and config.yml).

In the below configuration, we have SOURCE_INDEX_NAME, SOURCE_SLICES, SOURCE_PAGE_SIZE, LOGSTASH_WORKERS, and LOGSTASH_BATCH_SIZE conveniently exposed as environment variables so you just need to fill them out.

apiVersion: v1
kind: Pod
metadata:
  name: logstash-1
spec:
  containers:
    - name: logstash
      image: ugosan/logstash-opensearch-input:8.10.0
      imagePullPolicy: Always
      env:
        - name: SOURCE_INDEX_NAME
          value: ".ds-logs-benchmark-dev-000037"
        - name: SOURCE_SLICES
          value: "10"
        - name: SOURCE_PAGE_SIZE
          value: "500"
        - name: LOGSTASH_WORKERS
          value: "4"
        - name: LOGSTASH_BATCH_SIZE
          value: "1000"
        - name: OPENSEARCH_USERNAME
          valueFrom:
            secretKeyRef:
              name: os-cluster-admin-password
              key: username
        - name: OPENSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: os-cluster-admin-password
              key: password
        - name: ELASTICSEARCH_USERNAME
          value: "elastic"
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: es-cluster-es-elastic-user
              key: elastic
      resources:
        limits:
          memory: "4Gi"
          cpu: "2500m"
        requests:
          memory: "1Gi"
          cpu: "300m"
      volumeMounts:
        - name: config-volume
          mountPath: /usr/share/logstash/config
        - name: etc
          mountPath: /etc/logstash
          readOnly: true
  volumes:
    - name: config-volume
      projected:
        sources:
          - configMap:
              name: logstash-configmap
              items:
                - key: pipelines.yml
                  path: pipelines.yml
                - key: logstash.yml
                  path: logstash.yml
    - name: etc
      projected:
        sources:
          - configMap:
              name: logstash-configmap
              items:
                - key: pipeline.conf
                  path: pipelines/pipeline.conf
          - secret:
              name: os-cluster-http-cert
              items:
                - key: ca.crt
                  path: certificates/opensearch-ca.crt
          - secret:
              name: es-cluster-es-http-ca-internal
              items:
                - key: tls.crt
                  path: certificates/elasticsearch-ca.crt
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-configmap
data:
  pipelines.yml: |
    - pipeline.id: reindex-os-es
      path.config: "/etc/logstash/pipelines/pipeline.conf"
      pipeline.batch.size: ${LOGSTASH_BATCH_SIZE}
      pipeline.workers: ${LOGSTASH_WORKERS}
  logstash.yml: |
    log.level: info
    pipeline.unsafe_shutdown: true
    pipeline.ordered: false
  pipeline.conf: |
    input {
        opensearch {
          hosts => ["os-cluster:9200"]
          ssl => true
          ca_file => "/etc/logstash/certificates/opensearch-ca.crt"
          user => "${OPENSEARCH_USERNAME}"
          password => "${OPENSEARCH_PASSWORD}"
          index => "${SOURCE_INDEX_NAME}"
          slices => "${SOURCE_SLICES}"
          size => "${SOURCE_PAGE_SIZE}"
          scroll => "5m"
          docinfo => true
          docinfo_target => "[@metadata][doc]"
        }
    }

    filter {
        mutate {
            remove_field => ["@version", "host", "data_stream"]
        }
    }

    output {
        elasticsearch {
            hosts => "https://es-cluster-es-http:9200"
            ssl => true
            ssl_certificate_authorities => ["/etc/logstash/certificates/elasticsearch-ca.crt"]
            ssl_verification_mode => "full"

            user => "${ELASTICSEARCH_USERNAME}"
            password => "${ELASTICSEARCH_PASSWORD}"

            document_id => "%{[@metadata][doc][_id]}"

            data_stream => "true"
            data_stream_type => "logs"
            data_stream_dataset => "myapplication"
            data_stream_namespace => "reindex"
        }
    }

That’s it.

After a couple hours, we successfully migrated 1 billion documents from OpenSearch to Elasticsearch and even saved 23% plus on disk storage! Now that we have the logs in Elasticsearch how about extracting actual business value from them? Logs contain so much valuable information - we can not only do all sorts of interesting things with AIOPS, like Automatically Categorize those logs, but also extract business metrics and detect anomalies on them, give it a try.

| | | | | | | | | ---------------------------------- | --------- | ----------- | --------------------------------- | --------- | ----------- | ------ | | OpenSearch | | | Elasticsearch | | | | | Index | docs | size | Index | docs | size | Diff. | | .ds-logs-myapplication-prod-000037 | 116842158 | 27285520870 | logs-myapplication-reindex-000037 | 116842158 | 21998435329 | 21.46% | | .ds-logs-myapplication-prod-000038 | 110994116 | 27263291740 | logs-myapplication-reindex-000038 | 110994116 | 21540011082 | 23.45% | | .ds-logs-myapplication-prod-000040 | 113362823 | 27872438186 | logs-myapplication-reindex-000040 | 113362823 | 22234641932 | 22.50% | | .ds-logs-myapplication-prod-000041 | 112400019 | 27618801653 | logs-myapplication-reindex-000041 | 112400019 | 22059453868 | 22.38% | | .ds-logs-myapplication-prod-000042 | 113859174 | 26686723701 | logs-myapplication-reindex-000042 | 113859174 | 21093766108 | 23.41% | | .ds-logs-myapplication-prod-000043 | 113821016 | 27657006598 | logs-myapplication-reindex-000043 | 113821016 | 22059454752 | 22.52% | | .ds-logs-myapplication-prod-000044 | 111093596 | 27281936915 | logs-myapplication-reindex-000044 | 111093596 | 21559513422 | 23.43% | | .ds-logs-myapplication-prod-000048 | 114273539 | 28111420495 | logs-myapplication-reindex-000048 | 114273539 | 22264398939 | 23.21% | | .ds-logs-myapplication-prod-000049 | 102519334 | 23731274338 | logs-myapplication-reindex-000049 | 102519334 | 19307250001 | 20.56% |

Interested in trying Elasticsearch? Start our 14-day free trial.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

This blog dives into how input packages provide an extremely generic and flexible solution to the advanced users for customizing their ingestion experience in Elastic.

What are input packages?

An Elastic Package is an artifact that contains a collection of assets that extend the Elastic Stack, providing new capabilities to accomplish a specific task like integration with an external data source. The first use of Elastic packages is integration packages, which provide an end-to-end experience — from configuring Elastic Agent, to collecting signals from the data source, to ingesting them correctly and using the data once ingested.

However, advanced users may need to customize data collection, either because an integration does not exist for a specific data source, or even if it does, they want to collect additional signals or in a different way. Input packages are another type of Elastic package that provides the capability to configure Elastic Agent to use the provided inputs in a custom way.

Let’s look at an example

Say hello to Julia, who works as an engineer at Ascio Innovation firm. She is currently working with Oracle Weblogic server and wants to get a set of metrics for monitoring it. She goes ahead and installs Elastic Oracle Weblogic Integration, which uses Jolokia in the backend to fetch the metrics.

Now, her team wants to advance in the monitoring and has the following requirements:

1. We should be able to extract metrics other than the default ones, which are not supported by the default Oracle Weblogic Integration.

2. We want to have our own bespoke pipelines, visualizations, and experience.

3. We should be able to identify the metrics coming in from two different instances of Weblogic Servers by having data mapped to separate indices.

All the above requirements can be met by using the Jolokia input package to get a customized experience. Let's see how.

Julia can add the configuration of Jolokia input package as below, fulfilling the first requirement.

hostname, JMX Mappings for the fields you want to fetch for the JVM application, and the data set name to which the response fields would get mapped.

Julia can customize her data by writing her own ingest pipelines and providing her customized mappings. Also, she can then build her own bespoke dashboards, hence meeting her second requirement.

Let’s say now Julia wants to use another instance of Oracle Weblogic and get a different set of metrics.

This can be achieved by adding another instance of Jolokia input package and specifying a new data set name as shown in the screenshot below. The resultant metrics will be mapped to a different index/data set hence fulfilling her third requirement. This will help Julia to differentiate metrics coming in from two different instances of Oracle Weblogic.

The resultant metrics of the query will be indexed to the new data set, jolokia_second_dataset in the below example.

As we can see above, the Jolokia input package provides the flexibility to get new metrics by specifying different JMX Mappings, which are not supported in the default Oracle Weblogic integration (the user gets metrics from a predetermined set of JMX Mappings).

The Jolokia Input package also can be used for monitoring any Java-based application, which pushes its metrics through JMX. So a single input package can be used to collect metrics from multiple Java applications/services.

Elastic input packages

Elastic has started supporting input packages from the 8.8.0 release. Some of the input packages are now available in beta and will mature gradually:

1. SQL input package: The SQL input package allows you to execute queries against any SQL database and store the results in Elasticsearch^®.

2. Prometheus input package: This input package can collect metrics from Prometheus Exporters (Collectors).It can be used by any service exporting its metrics to a Prometheus endpoint.

3. Jolokia input package: This input package collects metrics from Jolokia agents running on a target JMX server or dedicated proxy server. It can be used for monitoring any Java-based application, which pushes its metrics through JMX.

4. Statsd input package: The statsd input package spawns a UDP server and listens for metrics in StatsD compatible format. This input can be used to collect metrics from services that send data over the StatsD protocol.

5. GCP Metrics input package: The GCP Metrics input package can collect custom metrics for any GCP service.

Try it out!

Now that you know more about input packages, try building your own customized integration for your service through input packages, and get started with an Elastic Cloud free trial.

We would love to hear from you about your experience with input packages on the Elastic Discuss forum or in the Elastic Integrations repository.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Why “Generic SQL”?

Elastic already has metricbeat and integration packages targeted for specific databases. One example is metricbeat for MySQL — and the corresponding integration package. These beats modules and integrations are customized for a specific database, and the metrics are extracted using pre-defined queries from the specific database. The queries used in these integrations and the corresponding metrics are not available for modification.

Whereas the Generic SQL inputs (metricbeat or input package) can be used to scrape metrics from any supported database using the user's SQL queries. The queries are provided by the user depending on specific metrics to be extracted. This enables a much more powerful mechanism for metrics ingestion, where users can choose a specific driver and provide the relevant SQL queries and the results get mapped to one or more Elasticsearch documents, using a structured mapping process (table/variable format explained later).

Generic SQL inputs can be used in conjunction with the existing integration packages, which already extract specific database metrics, to extract additional custom metrics dynamically, making this input very powerful. In this blog, Generic SQL input and Generic SQL are used interchangeably.

Functionalities details

This section covers some of the features that would help with the metrics extraction. We provide a brief description of the response format configuration. Then we dive into the merge_results functionality, which is used to combine results from multiple SQL queries into a single document.

The next key functionality users may be interested in is to collect metrics from all the custom databases, which is now possible with the fetch_from_all_databases feature.

Now let's dive into the specific functionalities:

Different drivers supported

The generic SQL can fetch metrics from the different databases. The current version has the capability to fetch metrics from the following drivers: MySQL, PostgreSQL, Oracle, and Microsoft SQL Server(MSSQL).

Response format

The response format in generic SQL is used to manipulate the data in either table or in variable format. Here’s an overview of the formats and syntax for creating and using the table and variables.

Syntax: response_format: table {{or}} variables

Response format table
This mode generates a single event for each row. The table format has no restrictions on the number of columns in the response. This format can have any number of columns.

Example:

driver: "mssql"
sql_queries:
 - query: "SELECT counter_name, cntr_value FROM sys.dm_os_performance_counters WHERE counter_name= 'User Connections'"
   response_format: table

This query returns a response similar to this:

"sql":{
      "metrics":{
         "counter_name":"User Connections ",
         "cntr_value":7
      },
      "driver":"mssql"
}

The response generated above adds the counter_name as a key in the document.

Response format variables
The variable format supports key:value pairs. This format expects only two columns to fetch in a query.

Example:

driver: "mssql"
sql_queries:
 - query: "SELECT counter_name, cntr_value FROM sys.dm_os_performance_counters WHERE counter_name= 'User Connections'"
   response_format: variables

The variable format takes the first variable in the query above as the key:

"sql":{
      "metrics":{
         "user connections ":7
      },
      "driver":"mssql"
}

In the above response, you can see the value of counter_name is used to generate the key in variable format.

Response optimization: merge_results

We are now supporting merging multiple query responses into a single event. By enabling merge_results , users can significantly optimize the storage space of the metrics ingested to Elasticsearch. This mode enables an efficient compaction of the document generated, where instead of generating multiple documents, a single merged document is generated wherever applicable. The metrics of a similar kind, generated from multiple queries, are combined into a single event.

Syntax: merge_results: true {{or}} false

In the below example, you can see how the data is loaded into Elasticsearch for the below query when the merge_results is disabled.

Example:

In this example, we are using two different queries to fetch metrics from the performance counter.

merge_results: false
driver: "mssql"
sql_queries:
  - query: "SELECT cntr_value As 'user_connections' FROM sys.dm_os_performance_counters WHERE counter_name= 'User Connections'"
    response_format: table
  - query: "SELECT cntr_value As 'buffer_cache_hit_ratio' FROM sys.dm_os_performance_counters WHERE counter_name = 'Buffer cache hit ratio' AND object_name like '%Buffer Manager%'"
    response_format: table

As you can see, the response for the above example generates a single document for each query.

The resulting document from the first query:

"sql":{
      "metrics":{
         "user_connections":7
      },
      "driver":"mssql"
}

And resulting document from the second query:

"sql":{
      "metrics":{
         "buffer_cache_hit_ratio":87
      },
      "driver":"mssql"
}

When we enable the merge_results flag in the query, both the above metrics are combined together and the data gets loaded in a single document.

You can see the merged document in the below example:

"sql":{
      "metrics":{
         "user connections ":7,
         “buffer_cache_hit_ratio”:87
      },
      "driver":"mssql"
}

However, such a merge is possible only if the table queries are merged, and each produces a single row. There is no restriction on variable queries being merged.

Introducing a new capability: fetch_from_all_databases

This is a new functionality to fetch all the database metrics automatically from the system and user databases of the Microsoft SQL Server, by enabling the fetch_from_all_databases flag.

Keep an eye out for the 8.10 release version where you can start using the fetch all database feature. Prior to the 8.10 version, users had to provide the database names manually to fetch metrics from custom/user databases.

Syntax: fetch_from_all_databases: true {{or}} false

Below is the sample query with fetch all databases flag as disabled:

fetch_from_all_databases: false
driver: "mssql"
sql_queries:
  - query: "SELECT @@servername AS server_name, @@servicename AS instance_name, name As 'database_name', database_id FROM sys.databases WHERE name='master';"

The above query fetches metrics only for the provided database name. Here the input database is master, so the metrics are fetched only for the master.

Below is the sample query with the fetch all databases flag as enabled:

fetch_from_all_databases: true
driver: "mssql"
sql_queries:
  - query: SELECT @@servername AS server_name, @@servicename AS instance_name, DB_NAME() AS 'database_name', DB_ID() AS database_id;
    response_format: table

The above query fetches metrics from all available databases. This is useful when the user wants to get data from all the databases.

Please note: currently this feature is supported only for Microsoft SQL Server and will be used by MS SQL integration internally, to support extracting metrics for all user DBs by default.

Using generic SQL: Metricbeat

The generic SQL metricbeat module provides flexibility to execute queries against different database drivers. The metricbeat input is available as GA for any production usage. Here, you can find more information on configuring the generic SQL for different drivers with various examples.

Using generic SQL: Input package

The input package provides a flexible solution to advanced users for customizing their ingestion experience in Elastic. Generic SQL is now also available as an SQLinput package. The input package is currently available for early users as a beta release. Let's take a walk through how users can use generic SQL via the input package.

Configurations of generic SQL input package:

The configuration options for the generic SQL input package are as below:

• Driver** :** This is the SQL database for which you want to use the package. In this case, we will take mysql as an example.
• Hosts: Here the user enters the connection string to connect to the database. It would vary depending on which database/driver is being used. Refer here for examples.
• SQL Queries: Here the user writes the SQL queries they want to fire and the response_format is specified.
• Data set: The user specifies a data set name to which the response fields get mapped.
• Merge results** :** This is an advanced setting, used to merge queries into a single event.

Metrics extensibility with customized SQL queries

Let's say a user is using MYSQL Integration, which provides a fixed set of metrics. Their requirement now extends to retrieving more metrics from the MYSQL database by firing new customized SQL queries.

This can be achieved by adding an instance of SQL input package, writing the customized queries and specifying a new data set name as shown in the screenshot below.

This way users can get any metrics by executing corresponding queries. The resultant metrics of the query will be indexed to the new data set, sql_second_dataset.

When there are multiple queries, users can club them into a single event by enabling the Merge Results toggle.

Customizing user experience

Users can customize their data by writing their own ingest pipelines and providing their customized mappings. Users can also build their own bespoke dashboards.

As we can see above, the SQL input package provides the flexibility to get new metrics by running new queries, which are not supported in the default MYSQL integration (the user gets metrics from a predetermined set of queries).

The SQL input package also supports multiple drivers: mssql, postgresql and oracle. So a single input package can be used to cater to all these databases.

Note: The fetch_from_all_databases feature is not supported in the SQL input package yet.

Try it out!

Now that you know about various use cases and features of generic SQL, get started with Elastic Cloud and try using the SQL input package for your SQL database and get customized experience and metrics. If you are looking for newer metrics for some of our existing SQL based integrations — like Microsoft SQL Server, Oracle, and more — go ahead and give the SQL input package a swirl.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

1. Common metadata across tracing, metrics, and logging to facilitate contextual correlation, including metadata passed between services as part of REST or RPC APIs; this is a critical element of service observability in the age of distributed, horizontally scaled systems

2. An optional unified data path for tracing, metrics, and logging to facilitate common tooling and signal routing to your observability backend

Adoption of metrics and tracing among developers to date has been relatively small. Further, the number of proprietary vendors and APIs (compared to adoption rate) is relatively large. As such, OpenTelemetry took a greenfield approach to developing new, vendor-agnostic APIs for tracing and metrics. In contrast, most developers have nearly 100% log coverage across their services. Moreover, logging is largely supported by a small number of vendor-agnostic, open-source logging libraries and associated APIs (e.g., Logback and ILogger). As such, OpenTelemetry’s approach to logging meets developers where they already are using hooks into existing, popular logging frameworks. In this way, developers can add OpenTelemetry as a log signal output without otherwise altering their code and investment in logging as an observability signal.

Notably, logging is the least mature of OTel supported observability signals. Depending on your service’s language, and your appetite for adventure, there exist several options for exporting logs from your services and applications and marrying them together in your observability backend.

The intent of this article is to explore the current state of the art of OpenTelemetry logging and to provide guidance on the available approaches with the following tenants in mind:

• Correlation of service logs with OTel-generated tracing where applicable
• Proper capture of exceptions
• Common context across tracing, metrics, and logging
• Support for slf4j key-value pairs (“structured logging”)
• Automatic attachment of metadata carried between services via OTel baggage
• Use of an Elastic^® Observability backend
• Consistent data fidelity in Elastic regardless of the approach taken

OpenTelemetry logging models

Three models currently exist for getting your application or service logs to Elastic with correlation to OTel tracing and baggage:

1. Output logs from your service (alongside traces and metrics) using an embedded OpenTelemetry Instrumentation library to Elastic via the OTLP protocol

2. Write logs from your service to a file scraped by the OpenTelemetry Collector, which then forwards to Elastic via the OTLP protocol

3. Write logs from your service to a file scraped by Elastic Agent (or Filebeat), which then forwards to Elastic via an Elastic-defined protocol

Note that (1), in contrast to (2) and (3), does not involve writing service logs to a file prior to ingestion into Elastic.

Logging vs. span events

It is worth noting that most APM systems, including OpenTelemetry, include provisions for span events. Like log statements, span events contain arbitrary, textual data. Additionally, span events automatically carry any custom attributes (e.g., a “user ID”) applied to the parent span, which can help with correlation and context. In this regard, it may be advantageous to translate some existing log statements (inside spans) to span events. As the name implies, of course, span events can only be emitted from within a span and thus are not intended to be a general purpose replacement for logging.

Unlike logging, span events do not pass through existing logging frameworks and therefore cannot (practically) be written to a log file. Further, span events are technically emitted as part of trace data and follow the same data path and signal routing as other trace data.

Polyfill appender

Some of the demos make use of a custom Logback “Polyfill appender” (inspired by OTel’s Logback MDC), which provides support for attaching slf4j key-value pairs to log messages for models (2) and (3).

Elastic Common Schema

For log messages to exhibit full fidelity within Elastic, they eventually need to be formatted in accordance with the Elastic Common Schema (ECS). In models (1) and (2), log messages remain formatted in OTel log semantics until ingested by the Elastic APM Server. The Elastic APM Server then translates OTel log semantics to ECS. In model (3), ECS is applied at the source.

Notably, OpenTelemetry recently adopted the Elastic Common Schema as its standard for semantic conventions going forward! As such, it is anticipated that current OTel log semantics will be updated to align with ECS.

Getting started

The included demos center around a “POJO” (no assumed framework) Java project. Java is arguably the most mature of OTel-supported languages, particularly with respect to logging options. Notably, this singular Java project was designed to support the three models of logging discussed here. In practice, you would only implement one of these models (and corresponding project dependencies).

The demos assume you have a working Docker environment and an Elastic Cloud instance.

1. git clone https://github.com/ty-elastic/otel-logging

2. Create an .env file at the root of otel-logging with the following (appropriately filled-in) environment variables:

# the service name
OTEL_SERVICE_NAME=app4

# Filebeat vars
ELASTIC_CLOUD_ID=(see https://www.elastic.co/guide/en/beats/metricbeat/current/configure-cloud-id.html)
ELASTIC_CLOUD_AUTH=(see https://www.elastic.co/guide/en/beats/metricbeat/current/configure-cloud-id.html)

# apm vars
ELASTIC_APM_SERVER_ENDPOINT=(address of your Elastic Cloud APM server... i.e., https://xyz123.apm.us-central1.gcp.cloud.es.io:443)
ELASTIC_APM_SERVER_SECRET=(see https://www.elastic.co/guide/en/apm/guide/current/secret-token.html)

1. Start up the demo with the desired model:

• If you want to demo logging via OTel APM Agent, run MODE=apm docker-compose up
• If you want to demo logging via OTel filelogreceiver, run MODE=filelogreceiver docker-compose up
• If you want to demo logging via Elastic filebeat, run MODE=filebeat docker-compose up

1. Validate incoming span and correlated log data in your Elastic Cloud instance

Model 1: Logging via OpenTelemetry instrumentation

This model aligns with the long-term goals of OpenTelemetry: integrated tracing, metrics, and logging (with common attributes) from your services via the OpenTelemetry Instrumentation libraries, without dependency on log files and scrappers.

In this model, your service generates log statements as it always has, using popular logging libraries (e.g., Logback for Java). OTel provides a “Southbound hook” to Logback via the OTel Logback Appender, which injects ServiceName, SpanID, TraceID, slf4j key-value pairs, and OTel baggage into log records and passes the composed records to the co-resident OpenTelemetry Instrumentation library. We further employ a custom LogRecordProcessor to add baggage to the log record as attributes.

The OTel instrumentation library then formats the log statements per the OTel logging spec and ships them via OTLP to either an OTel Collector for further routing and enrichment or directly to Elastic.

Notably, as language support improves, this model can and will be supported by runtime agent binding with auto-instrumentation where available (e.g., no code changes required for runtime languages).

One distinguishing advantage of this model, beyond the simplicity it affords, is the ability to more easily tie together attributes and tracing metadata directly with log statements. This inherently makes logging more useful in the context of other OTel-supported observability signals.

Architecture

Although not explicitly pictured, an OpenTelemetry Collector can be inserted in between the service and Elastic to facilitate additional enrichment and/or signal routing or duplication across observability backends.

Pros

• Simplified signal architecture and fewer “moving parts” (no files, disk utilization, or file rotation concerns)
• Aligns with long-term OTel vision
• Log statements can be (easily) decorated with OTel metadata
• No polyfill adapter required to support structured logging with slf4j
• No additional collectors/agents required
• Conversion to ECS happens within Elastic keeping log data vendor-agnostic until ingestion
• Common wireline protocol (OTLP) across tracing, metrics, and logs

Cons

• Not available (yet) in many OTel-supported languages
• No intermediate log file for ad-hoc, on-node debugging
• Immature (alpha/experimental) Unknown “glare” conditions, which could result in loss of log data if service exits prematurely or if the backend is unable to accept log data for an extended period of time

Demo

MODE=apm docker-compose up

Model 2: Logging via the OpenTelemetry Collector

Given the cons of Model 1, it may be advantageous to consider a model that continues to leverage an actual log file intermediary between your services and your observability backend. Such a model is possible using an OpenTelemetry Collector collocated with your services (e.g., on the same host), running the filelogreceiver to scrape service log files.

In this model, your service generates log statements as it always has, using popular logging libraries (e.g., Logback for Java). OTel provides a MDC Appender for Logback (Logback MDC), which adds SpanID, TraceID, and Baggage to the Logback MDC context.

Notably, no log record structure is assumed by the OTel filelogreceiver. In the example provided, we employ the logstash-logback-encoder to JSON-encode log messages. The logstash-logback-encoder will read the OTel SpanID, TraceID, and Baggage off the MDC context and encode it into the JSON structure. Notably, logstash-logback-encoder doesn’t explicitly support slf4j key-value pairs. It does, however, support Logback structured arguments, and thus I use the Polyfill Appender to convert slf4j key-value pairs to Logback structured arguments.

From there, we write the log lines to a log file. If you are using Kubernetes or other container orchestration in your environment, you would more typically write to stdout (console) and let the orchestration log driver write to and manage log files.

We then configure the OTel Collector to scrape this log file (using the filelogreceiver). Because no assumptions are made about the format of the log lines, you need to explicitly map fields from your log schema to the OTel log schema.

From there, the OTel Collector batches and ships the formatted log lines via OTLP to Elastic.

Architecture

Pros

• Easy to debug (you can manually read the intermediate log file)
• Inherent file-based FIFO buffer
• Less susceptible to “glare” conditions when service prematurely exits
• Conversion to ECS happens within Elastic keeping log data vendor-agnostic until ingestion
• Common wireline protocol (OTLP) across tracing, metrics, and logs

Cons

• All the headaches of file-based logging (rotation, disk overflow)
• Beta quality and not yet proven in the field
• No support for slf4j key-value pairs

Demo

MODE=filelogreceiver docker-compose up

Model 3: Logging via Elastic Agent (or Filebeat)

Although the second model described affords some resilience as a function of the backing file, the OTel Collector filelogreceiver module is still decidedly “beta” in quality. Because of the importance of logs as a debugging tool, today I generally recommend that customers continue to import logs into Elastic using the field-proven Elastic Agent or Filebeat scrappers. Elastic Agent and Filebeat have many years of field maturity under their collective belt. Further, it is often advantageous to deploy Elastic Agent anyway to capture the multitude of signals outside the purview of OpenTelemetry (e.g., deep Kubernetes and host metrics, security, etc.).

In this model, your service generates log statements as it always has, using popular logging libraries (e.g., Logback for Java). As with model 2, we employ OTel’s Logback MDC to add SpanID, TraceID, and Baggage to the Logback MDC context.

From there, we employ the Elastic ECS Encoder to encode log statements compliant to the Elastic Common Schema. The Elastic ECS Encoder will read the OTel SpanID, TraceID, and Baggage off the MDC context and encode it into the JSON structure. Similar to model 2, the Elastic ECS Encoder doesn’t support sl4f key-vair arguments. Curiously, the Elastic ECS encoder also doesn’t appear to support Logback structured arguments. Thus, within the Polyfill Appender, I add slf4j key-value pairs as MDC context. This is less than ideal, however, since MDC forces all values to be strings.

From there, we write the log lines to a log file. If you are using Kubernetes or other container orchestration in your environment, you would more typically write to stdout (console) and let the orchestration log driver write to and manage log files.We then configure Elastic Agent or Filebeat to scrape the log file. Notably, the Elastic ECS Encoder does not currently translate incoming OTel SpanID and TraceID variables on the MDC. Thus, we need to perform manual translation of these variables in the Filebeat (or Elastic Agent) configuration to map them to their ECS equivalent.

Architecture

Pros

• Robust and field-proven
• Easy to debug (you can manually read the intermediate log file)
• Inherent file-based FIFO buffer
• Less susceptible to “glare” conditions when service prematurely exits
• Native ECS format for easy manipulation in Elastic
• Fleet-managed via Elastic Agent

Cons

• All the headaches of file-based logging (rotation, disk overflow)
• No support for slf4j key-value pairs or Logback structured arguments
• Requires translation of OTel SpanID and TraceID in Filebeat config
• Disparate data paths for logs versus tracing and metrics
• Vendor-specific logging format

Demo

MODE=filebeat docker-compose up

Recommendations

For most customers, I currently recommend Model 3 — namely, write to logs in ECS format (with OTel SpanID, TraceID, and Baggage metadata) and collect them with an Elastic Agent installed on the node hosting the application or service. Elastic Agent (or Filebeat) today provides the most field-proven and robust means of capturing log files from applications and services with OpenTelemetry context.

Further, you can leverage this same Elastic Agent instance (ideally running in your Kubernetes daemonset) to collect rich and robust metrics and logs from Kubernetes and many other supported services via Elastic Integrations. Finally, Elastic Agent facilitates remote management via Fleet, avoiding bespoke configuration files.

Alternatively, for customers who either wish to keep their nodes vendor-neutral or use a consolidated signal routing system, I recommend Model 2, wherein an OpenTelemetry collector is used to scrape service log files. While workable and practiced by some early adopters in the field today, this model inherently carries some risk given the current beta nature of the OpenTelemetry filelogreceiver.

I generally do not recommend Model 1 given its limited language support, experimental/alpha status (the API could change), and current potential for data loss. That said, in time, with more language support and more thought to resilient designs, it has clear advantages both with regard to simplicity and richness of metadata.

Extracting more value from your logs

In contrast to tracing and metrics, most organizations have nearly 100% log coverage over their applications and services. This is an ideal beachhead upon which to build an application observability system. On the other hand, logs are notoriously noisy and unstructured; this is only amplified with the scale enabled by the hyperscalers and Kubernetes. Collecting log lines reliably is the easy part; making them useful at today’s scale is hard.

Given that logs are arguably the most challenging observability signal from which to extract value at scale, one should ideally give thoughtful consideration to a vendor’s support for logging in the context of other observability signals. Can they handle surges in log rates because of unexpected scale or an error or test scenario? Do they have the machine learning tool set to automatically recognize patterns in log lines, sort them into categories, and identify true anomalies? Can they provide cost-effective online searchability of logs over months or years without manual rehydration? Do they provide the tools to extract and analyze business KPIs buried in logs?

As an ardent and early supporter of OpenTelemetry, Elastic, of course, natively ingests OTel traces, metrics, and logs. And just like all logs coming into our system, logs coming from OTel-equipped sources avail themselves of our mature tooling and next-gen AI Ops technologies to enable you to extract their full value.Interested? Reach out to our pre-sales team to get started building with Elastic!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

filebeat.inputs:
  - type: filestream
    id: my-logging-app
    paths:
      - /var/tmp/other.log
      - /var/log/*.log
processors:
  - drop_event:
      when:
        and:
          - equals:
            url.scheme: http
          - equals:
            url.path: /profile

filebeat.inputs:
  - type: filestream
    id: my-logging-app
    paths:
      - /var/tmp/other.log
      - /var/log/*.log
processors:
  - drop_fields:
      when:
        and:
          - equals:
            url.scheme: http
          - equals:
            http.response.status_code: 200
        fields: ["event.message"]
        ignore_missing: false

input {
  file {
    id => "my-logging-app"
    path => [ "/var/tmp/other.log", "/var/log/*.log" ]
  }
}
filter {
  if [url.scheme] == "http" && [url.path] == "/profile" {
    drop {
      percentage => 80
    }
  }
}
output {
  elasticsearch {
        hosts => "https://my-elasticsearch:9200"
        data_stream => "true"
    }
}

# Input configuration omitted
filter {
  if [url.scheme] == "http" && [http.response.status_code] == 200 {
    drop {
      percentage => 80
    }
    mutate {
      remove_field: [ "event.message" ]
    }
  }
}
# Output configuration omitted

PUT _ingest/pipeline/my-logging-app-pipeline
{
  "description": "Event and field dropping for my-logging-app",
  "processors": [
    {
      "drop": {
        "description" : "Drop event",
        "if": "ctx?.url?.scheme == 'http' && ctx?.url?.path == '/profile'",
        "ignore_failure": true
      }
    },
    {
      "remove": {
        "description" : "Drop field",
        "field" : "event.message",
        "if": "ctx?.url?.scheme == 'http' && ctx?.http?.response?.status_code == 200",
        "ignore_failure": false
      }
    }
  ]
}

PUT _ingest/pipeline/my-logging-app-pipeline
{
  "description": "Event and field dropping for my-logging-app with failures",
  "processors": [
    {
      "drop": {
        "description" : "Drop event",
        "if": "ctx?.url?.scheme == 'http' && ctx?.url?.path == '/profile'",
        "ignore_failure": true
      }
    },
    {
      "remove": {
        "description" : "Drop field",
        "field" : "event.message",
        "if": "ctx?.url?.scheme == 'http' && ctx?.http?.response?.status_code == 200",
        "ignore_failure": false
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "description": "Set 'ingest.failure.message'",
        "field": "ingest.failure.message",
        "value": "Ingestion issue"
        }
      }
  ]
}

receivers:
  filelog:
    include: [/var/tmp/other.log, /var/log/*.log]
processors:
  filter/denylist:
    error_mode: ignore
    logs:
      log_record:
        - 'url.scheme == "info"'
        - 'url.path == "/profile"'
        - "http.response.status_code == 200"
  attributes/errors:
    actions:
      - key: error.message
        action: delete
  memory_limiter:
    check_interval: 1s
    limit_mib: 2000
  batch:
exporters:
  # Exporters configuration omitted
service:
  pipelines:
    # Pipelines configuration omitted

Elasticsearch, with its long experience in machine learning, provides various options to bring in custom models, such as large language models (LLMs), and provides its own models. These models will help implement PII redaction.

If you would like to learn more about natural language processing, machine learning, and Elastic, please be sure to check out these related articles:

• Introduction to modern natural language processing with PyTorch in Elasticsearch
• How to deploy natural language processing (NLP): Getting started
• Elastic Redact Processor Documentation
• Introducing Elastic Learned Sparse Encoder: Elastic’s AI model for semantic search
• Accessing machine learning models in Elastic

In this blog, we will show you how to set up PII redaction through the use of Elasticsearch’s ability to load a trained model within machine learning and the flexibility of Elastic’s ingest pipelines.

Specifically, we’ll walk through setting up a named entity recognition (NER) model for person and location identification, as well as deploying the redact processor for custom data identification and removal. All of this will then be combined with an ingest pipeline where we can use Elastic machine learning and data transformations capabilities to remove sensitive information from your data.

Loading the trained model

Before we begin, we must load our NER model into our Elasticsearch cluster. This may be easily accomplished with Docker and the Elastic Eland client. From a command line, let’s install the Eland client via git:

git clone https://github.com/elastic/eland.git

Navigate into the recently downloaded client:

cd eland/

Now let’s build the client:

docker build -t elastic/eland .

From here, you’re ready to deploy the trained model to an Elastic machine learning node! Be sure to replace your username, password, es-cluster-hostname, and esport.

If you’re using the Elastic Cloud or have signed certificates, simply run this command:

docker run -it --rm --network host elastic/eland eland_import_hub_model --url https://<username>:<password>@<es-cluster-hostname>:<esport>/ --hub-model-id dslim/bert-base-NER --task-type ner --start

If you’re using self-signed certificates, run this command:

docker run -it --rm --network host elastic/eland eland_import_hub_model --url https://<username>:<password>@<es-cluster-hostname>:<esport>/ --insecure --hub-model-id dslim/bert-base-NER --task-type ner --start

From here you’ll witness the Eland client in action downloading the trained model from HuggingFace and automatically deploying it into your cluster!

Synchronize your newly loaded trained model by clicking on the blue hyperlink via your Machine Learning Overview UI “Synchronize your jobs and trained models.”

Now click the Synchronize button.

That’s it! Congratulations, you just loaded your first trained model into Elastic!

Create the redact processor and ingest pipeline

From DevTools, let’s configure the redact processor along with our inference processor to take advantage of Elastic’s trained model we just loaded. This will create an ingest pipeline named “redact” that we can then use to remove sensitive data from any field we wish. In this example, I’ll be focusing on the “message” field. Note: at the time of this writing, the redact processor is experimental and must be created via DevTools.

PUT _ingest/pipeline/redact
{
  "processors": [
    {
      "set": {
        "field": "redacted",
        "value": "{{{message}}}"
      }
    },
    {
      "inference": {
        "model_id": "dslim__bert-base-ner",
        "field_map": {
          "message": "text_field"
        }
      }
    },
    {
      "script": {
        "lang": "painless",
        "source": "String msg = ctx['message'];\r\n                for (item in ctx['ml']['inference']['entities']) {\r\n                msg = msg.replace(item['entity'], '<' + item['class_name'] + '>')\r\n                }\r\n                ctx['redacted']=msg"
      }
    },
    {
      "redact": {
        "field": "redacted",
        "patterns": [
          "%{EMAILADDRESS:EMAIL}",
          "%{IP:IP_ADDRESS}",
          "%{CREDIT_CARD:CREDIT_CARD}",
          "%{SSN:SSN}",
          "%{PHONE:PHONE}"
        ],
        "pattern_definitions": {
          "CREDIT_CARD": "\d{4}[ -]\d{4}[ -]\d{4}[ -]\d{4}",
          "SSN": "\d{3}-\d{2}-\d{4}",
          "PHONE": "\d{3}-\d{3}-\d{4}"
        }
      }
    },
    {
      "remove": {
        "field": [
          "ml"
        ],
        "ignore_missing": true,
        "ignore_failure": true
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "failure",
        "value": "pii_script-redact"
      }
    }
  ]
}

OK, but what does each processor really do? Let’s walk through each processor in detail here:

1. The SET processor creates the field “redacted,” which is copied over from the message field and used later on in the pipeline.

2. The INFERENCE processor calls the NER model we loaded to be used on the message field for identifying names, locations, and organizations.

3. The SCRIPT processor then replaced the detected entities within the redacted field from the message field.

4. Our REDACT processor uses Grok patterns to identify any custom set of data we wish to remove from the redacted field (which was copied over from the message field).

5. The REMOVE processor deletes the extraneous ml.* fields from being indexed; note we’ll add “message” to this processor once we validate data is being redacted properly.

6. The ON_FAILURE / SET processor captures any errors just in case we have them.

Slice your PII

Now that your ingest pipeline with all the necessary steps has been configured, let’s start testing how well we can remove sensitive data from documents. Navigate over to Stack Management, select Ingest Pipelines and search for “redact”, and then click on the result.

Click on the Manage button, and then click Edit.

Here we are going to test our pipeline by adding some documents. Below is a sample you can copy and paste to make sure everything is working correctly.

{
  "_source":
    {
      "message": "John Smith lives at 123 Main St. Highland Park, CO. His email address is jsmith123@email.com and his phone number is 412-189-9043.  I found his social security number, it is 942-00-1243. Oh btw, his credit card is 1324-8374-0978-2819 and his gateway IP is 192.168.1.2",
    },
}

Simply press the Run the pipeline button, and you will then see the following output:

What’s next?

After you’ve added this ingest pipeline to a data set you’re indexing and validated that it is meeting expectations, you can add the message field to be removed so that no PII data is indexed. Simply update your REMOVE processor to include the message field and simulate again to only see the redacted field.

Conclusion

With this step-by-step approach, you are now ready and able to detect and redact any sensitive data throughout your indices.

Here’s a quick recap of what we covered:

• Loading a pre-trained named entity recognition model into an Elastic cluster
• Configuring the Redact processor, along with the inference processor, to use the trained model during data ingestion
• Testing sample data and modifying the ingest pipeline to safely remove personally identifiable information

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I’ve outlined above to get the most value and visibility out of your OpenTelemetry data.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Elastic Observability offers a wide range of integrations that help you to monitor your applications and infrastructure. These integrations are added as policies to Elastic agents, which help ingest telemetry into Elastic Observability. Several examples of these integrations include the ability to ingest logs from systems that send a stream of logs from different applications, such as Amazon Kinesis Data Firehose, Kubernetes container logs, and syslog. One challenge is that these multiplexed log streams are sending data to the same Elasticsearch data stream, such as logs-syslog-default. This makes it difficult to create parsing rules in ingest pipelines and dashboards for specific technologies, such as the ones from the Nginx and Apache integrations. That’s because in Elasticsearch, in combination with the data stream naming scheme, the processing and the schema are both encapsulated in a data stream.

The reroute processor helps you tease apart data from a generic data stream and send it to a more specific one. You may use that mechanism to send logs to a data stream that is set up by the Nginx integration, for example, so that the logs are parsed with that integration and you can use the integration’s prebuilt dashboards or create custom ones with the fields, such as the url, the status code, and the response time that the Nginx pipeline has parsed out of the Nginx log message. You can also split out/separate regular Nginx logs and errors with the reroute processor, providing further separation ability and categorization of logs.

Example use case

To use the reroute processor, first:

1. Ensure you are on Elasticsearch 8.8

2. Ensure you have permissions to manage indices and data streams

3. If you don’t already have an account on Elastic Cloud, sign up for one

Next, you’ll need to set up a data stream and create a custom Elasticsearch ingest pipeline that is called as the default pipeline. Below we go through this step by step for the “mydata” data set that we’ll simulate ingesting container logs into. We start with a basic example and extend it from there.

The following steps should be utilized in the Elastic console, which is found at Management -> Dev tools -> Console. First, we need an an ingest pipeline and a template for the data stream:

PUT _ingest/pipeline/logs-mydata
{
  "description": "Routing for mydata",
  "processors": [
    {
      "reroute": {
      }
    }
  ]
}

This creates an ingest pipeline with an empty reroute processor. To make use of it, we need an index template:

PUT _index_template/logs-mydata
{
  "index_patterns": [
    "logs-mydata-*"
  ],
  "data_stream": {},
  "priority": 200,
  "template": {
    "settings": {
      "index.default_pipeline": "logs-mydata"
    },
    "mappings": {
      "properties": {
        "container.name": {
          "type": "keyword"
        }
      }
    }
  }
}

The above template is applied to all data that is shipped to logs-mydata-*. We have mapped container.name as a keyword, as this is the field we will be using for routing later on. Now, we send a document to the data stream and it will be ingested into logs-mydata-default:

POST logs-mydata-default/_doc
{
  "@timestamp": "2023-05-25T12:26:23+00:00",
  "container": {
    "name": "foo"
  }
}

We can check that it was ingested with the command below, which will show 1 result.

GET logs-mydata-default/_search

Without modifying the routing processor, this already allows us to route documents. As soon as the reroute processor is specified, it will look for data_stream.dataset and data_stream.namespace fields by default and will send documents to the corresponding data stream, according to the data stream naming scheme logs-\<dataset>-\<namespace>. Let’s try this out:

POST logs-mydata-default/_doc
{
  "@timestamp": "2023-03-30T12:27:23+00:00",
  "container": {
"name": "foo"
  },
  "data_stream": {
    "dataset": "myotherdata"
  }
}

As can be seen with the GET logs-mydata-default/_search command, this document ended up in the logs-myotherdata-default data stream. But instead of using default rules, we want to create our own rules for the field container.name. If the field is container.name = foo, we want to send it to logs-foo-default. For this we modify our routing pipeline:

PUT _ingest/pipeline/logs-mydata
{
  "description": "Routing for mydata",
  "processors": [
    {
      "reroute": {
        "tag": "foo",
        "if" : "ctx.container?.name == 'foo'",
        "dataset": "foo"
      }
    }
  ]
}

Let's test this with a document:

POST logs-mydata-default/_doc
{
  "@timestamp": "2023-05-25T12:26:23+00:00",
  "container": {
    "name": "foo"
  }
}

While it would be possible to specify a routing rule for each container name, you can also route by the value of a field in the document:

PUT _ingest/pipeline/logs-mydata
{
  "description": "Routing for mydata",
  "processors": [
    {
      "reroute": {
        "tag": "mydata",
        "dataset": [
          "{{container.name}}",
          "mydata"
        ]
      }
    }
  ]
}

In this example, we are using a field reference as a routing rule. If the container.name field exists in the document, it will be routed — otherwise it falls back to mydata. This can be tested with:

POST logs-mydata-default/_doc
{
  "@timestamp": "2023-05-25T12:26:23+00:00",
  "container": {
    "name": "foo1"
  }
}

POST logs-mydata-default/_doc
{
  "@timestamp": "2023-05-25T12:26:23+00:00",
  "container": {
    "name": "foo2"
  }
}

This creates the data streams logs-foo1-default and logs-foo2-default.

NOTE: There is currently a limitation in the processor that requires the fields specified in a {{field.reference}} to be in a nested object notation. A dotted field name does not currently work. Also, you’ll get errors when the document contains dotted field names for any data_stream.* field. This limitation will be fixed in 8.8.2 and 8.9.0.

API keys

When using the reroute processor, it is important that the API keys specified have permissions for the source and target indices. For example, if a pattern is used for routing from logs-mydata-default, the API key must have write permissions for logs-*-* as data could end up in any of these indices (see example further down).

We’re currently working on extending the API key permissions for our integrations so that they allow for routing by default if you’re running a Fleet-managed Elastic Agent.

If you’re using a standalone Elastic Agent, or any other shipper, you can use this as a template to create your API key:

POST /_security/api_key
{
  "name": "ingest_logs",
  "role_descriptors": {
    "ingest_logs": {
      "cluster": [
        "monitor"
      ],
      "indices": [
        {
          "names": [
            "logs-*-*"
          ],
          "privileges": [
            "auto_configure",
            "create_doc"
          ]
        }
      ]
    }
  }
}

Future plans

In Elasticsearch 8.8, the reroute processor was released in technical preview. The plan is to adopt this in our data sink integrations like syslog, k8s, and others. Elastic will provide default routing rules that just work out of the box, but it will also be possible for users to add their own rules. If you are using our integrations, follow this guide on how to add a custom ingest pipeline.

Try it out!

This blog post has shown some sample use cases for document based routing. Try it out on your data by adjusting the commands for index templates and ingest pipelines to your own data, and get started with Elastic Cloud through a 7-day free trial. Let us know via this feedback form how you’re planning to use the reroute processor and whether you have suggestions for improvement.

Enterprise software development and operations has become an interesting space. We have some incredibly powerful tools at our disposal, yet as an industry, we have failed to adopt many of these tools that can make our lives easier. One such tool that is currently underutilized is application performance monitoring (APM) and tracing, despite the fact that OpenTelemetry has made it possible to adopt at low friction.

Logging, however, is ubiquitous. Every software application has logs of some kind, and the default workflow for troubleshooting (even today) is to go from exceptions experienced by customers and systems to the logs and start from there to find a solution.

There are various challenges with this, one of the main ones being that logs often do not give enough information to solve the problem. Many services today return ambiguous 500 errors with little or nothing to go on. What if there isn’t an error or log file at all or the problem is that the system is very slow? Logging alone cannot help solve these problems. This leaves users with half broken systems and poor user experiences. We’ve all been on the wrong side of this, and it can be incredibly frustrating.

The question I find myself asking is why does the customer experience often come second to errors? If the customer experience is a top priority, then a strategy should be in place to adopt tracing and APM and make this as important as logging. Users should stop going to logs by default and thinking primarily in logs, as many are doing today. This will also come with some required changes to mental models.

What’s the path to get there? That’s exactly what we will explore in this blog post. We will start by talking about supporting organizational changes, and then we’ll outline a recommended journey for moving from just logging to a fully integrated solution with logs, traces, and APM.

Cultivating a new monitoring mindset: How to drive APM and tracing adoption

To get teams to shift their troubleshooting mindset, what organizational changes need to be made?

Initially, businesses should consider strategic priorities and goals that need to be shared broadly among the teams. One thing that can help drive this in a very large organization is to consider an entire product team devoted to Observability or a CoE (Center of Excellence) with its own roadmap and priorities.

This team (either virtual or permanent) should start with the customer in mind and work backward, starting with key questions like: What do I need to collect? What do I need to observe? How do I act? Once team members understand the answers to these questions, they can start to think about the technology decisions needed to drive those outcomes.

From a tracing and APM perspective, the areas of greatest concern are the customer experience, service level objectives, and service level outcomes. From here, organizations can start to implement programs of work to continuously improve and share knowledge across teams. This will help to align teams around a common framework with shared goals.

In the next few sections, we will go through a four step journey to help you maximize your success with APM and tracing. This journey will take you through the following key steps on your journey to successful APM adoption:

1. Ingest: What choices do you have to make to get tracing activated and start ingesting trace data into your observability tools?
2. Integrate: How does tracing integrate with logs to enable full end-to-end observability, and what else beyond simple tracing can you utilize to get even better resolution on your data?
3. Analytics and AIOPs: Improve the customer experience and reduce the noise through machine learning.
4. Scale and total cost of ownership: Roll out enterprise-wide tracing and adopt strategies to deal with data volume.

1. Ingest

Ingesting data for APM purposes generally involves “instrumenting” the application. In this section, we will explore methods for instrumenting applications, talk a little bit about sampling, and finally wrap up with a note on using common schemas for data representation.

Getting started with instrumentation

What options do we have for ingesting APM and trace data? There are many, many options we will discuss to help guide you, but first let's take a step back. APM has a deep history — in very first implementations of APM, people were concerned mainly with timing methods, like this below:

Usually you had a configuration file to specify which methods you wanted to time, and the APM implementation would instrument the specified code with method timings.

From here things started to evolve, and one of the first additions to APM was to add in tracing.

For Java, it’s fairly trivial to implement a system to do this by using what's known as a Java agent. You just specify -javagent command line argument, and the agent code gets access to the dynamic compilation routines within Java so it can modify the code before it is compiled into machine code, allowing you to “wrap” specific methods with timing or tracing routines. So, auto instrumenting Java was one of the first things that the original APM vendors did.

OpenTelemetry has agents like this, and most observability vendors that offer APM solutions have their own proprietary ways of doing this, often with more advanced and differing features from the open source tooling.

Things have moved on since then, and Node.JS and Python are now popular.

As a result, ways of auto instrumenting these language runtimes have appeared, which mostly work by injecting the libraries into the code before starting them up. OpenTelemetry has a way of doing this on Kubernetes with an Operator and sidecar here, which supports Python, Node.JS, Java, and DotNet.

The other alternative is to start adding APM and tracing API calls into your own code, which is not dissimilar to adding logging functionality. You may even wish to create an abstraction in your code to deal with this cross-cutting concern, although this is less of a problem now that there are open standards with which you can implement this.

You can see an example of how to add OpenTelemetry spans and attributes to your code for manual instrumentation below and here.

from flask import Flask
import monitor  # Import the module
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
import urllib
import os

from opentelemetry import trace
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.instrumentation.requests import RequestsInstrumentor


# Service name is required for most backends
resource = Resource(attributes={
    SERVICE_NAME: "your-service-name"
})

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=os.getenv('OTEL_EXPORTER_OTLP_ENDPOINT'),
        headers="Authorization=Bearer%20"+os.getenv('OTEL_EXPORTER_OTLP_AUTH_HEADER')))

provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
RequestsInstrumentor().instrument()

# Initialize Flask app and instrument it
app = Flask(__name__)

@app.route("/completion")
@tracer.start_as_current_span("do_work")
def completion():
        span = trace.get_current_span()
        if span:
            span.set_attribute("completion_count",1)

By implementing APM in this way, you could even eliminate the need to do any logging by storing all your required logging information within span attributes, exceptions, and metrics. The downside is that you can only do this with code that you own, so you will not be able to remove all logs this way.

Sampling

Many people don’t realize that APM is an expensive process. It adds a lot of CPU cycles and memory to your applications, and although there is a lot of value to be had, there are certainly trade-offs to be made.

Should you sample everything 100% and eat the cost? Or should you think about an intelligent trade-off with fewer samples or even tail-based sampling, which many products commonly support? Here, we will talk about the two most common sampling techniques — head-based sampling and tail-based sampling — to help you decide.

Head-based sampling
In this approach, sampling decisions are made at the beginning of a trace, typically at the entry point of a service or application. A fixed rate of traces is sampled, and this decision propagates through all the services involved in a distributed trace.

With head-based sampling, you can control the rate using a configuration, allowing you to control the percentage of requests that are sampled and reported to the APM server. For instance, a sampling rate of 0.5 means that only 50% of requests are sampled and sent to the server. This is useful for reducing the amount of collected data while still maintaining a representative sample of your application's performance.

Tail-based sampling
Unlike head-based sampling, tail-based sampling makes sampling decisions after the entire trace has been completed. This allows for more intelligent sampling decisions based on the actual trace data, such as only reporting traces with errors or traces that exceed a certain latency threshold.

We recommend tail-based sampling because it has the highest likelihood of reducing the noise and helping you focus on the most important issues. It also helps keep costs down on the data store side. A downside of tail-based sampling, however, is that it results in more data being generated from APM agents. This could use more CPU and memory on your application.

OpenTelemetry Semantic Conventions and Elastic Common Schema

OpenTelemetry prescribes Semantic Conventions, or Semantic Attributes, to establish uniform names for various operations and data types. Adhering to these conventions fosters standardization across codebases, libraries, and platforms, ultimately streamlining the monitoring process.

Creating OpenTelemetry spans for tracing is flexible, allowing implementers to annotate them with operation-specific attributes. These spans represent particular operations within and between systems, often involving widely recognized protocols like HTTP or database calls. To effectively represent and analyze a span in monitoring systems, supplementary information is necessary, contingent upon the protocol and operation type.

Unifying attribution methods across different languages is essential for operators to easily correlate and cross-analyze telemetry from polyglot microservices without needing to grasp language-specific nuances.

Elastic's recent contribution of the Elastic Common Schema to OpenTelemetry enhances Semantic Conventions to encompass logs and security.

Abiding by a shared schema yields considerable benefits, enabling operators to rapidly identify intricate interactions and correlate logs, metrics, and traces, thereby expediting root cause analysis and reducing time spent searching for logs and pinpointing specific time frames.

We advocate for adhering to established schemas such as ECS when defining trace, metrics, and log data in your applications, particularly when developing new code. This practice will conserve time and effort when addressing issues.

2. Integrate

Integrations are very important for APM. How well your solution can integrate with other tools and technologies such as cloud, as well as its ability to integrate logs and metrics into your tracing data, is critical to fully understand the customer experience. In addition, most APM vendors have adjacent solutions for synthetic monitoring and profiling to gain deeper perspectives to supercharge your APM. We will explore these topics in the following section.

APM + logs = superpowers!

Because APM agents can instrument code, they can also instrument code that is being used for logging. This way, you can capture log lines directly within APM. This is normally simple to enable.

With this enabled, you will also get automated injection of useful fields like these:

• service.name, service.version, service.environment
• trace.id, transaction.id, error.id

This means log messages will be automatically correlated with transactions as shown below, making it far easier to reduce mean time to resolution (MTTR) and find the needle in the haystack:

If this is available to you, we highly recommend turning it on.

Deploying APM inside Kubernetes

It is common for people to want to deploy APM inside a Kubernetes environment, and tracing is critical for monitoring applications in cloud-native environments. There are three different ways you can tackle this.

1. Auto instrumentation using sidecars
With Kubernetes, it is possible to use an init container and something that will modify Kubernetes manifests on the fly to auto instrument your applications.

The init container will be used simply to copy the required library or jar file into the container at startup that you need to the main Kubernetes pod. Then, you can use Kustomize to add the required command line arguments to bootstrap your agents.

If you are not familiar with it, Kustomize adds, removes, or modifies Kubernetes manifests on the fly. It is even available as a flag to the Kubernetes CLI — simply execute kubectl -k.

OpenTelemetry has an operator that does all this for you automatically (without the need for Kustomize) for Java, DotNet, Python, and Node.JS, and many vendors also have their own operator or helm charts that can achieve the same result.

2. Baking APM into containers or code
A second option for deploying out APM in Kubernetes — and indeed any containerized environment — is using Docker to bake the APM agents and configuration into a dockerfile.

Have a look at an example here using the OpenTelemetry Java Agent:

# Use the official OpenJDK image as the base image
FROM openjdk:11-jre-slim

# Set up environment variables
ENV APP_HOME /app
ENV OTEL_VERSION 1.7.0-alpha
ENV OTEL_JAVAAGENT_URL https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v${OTEL_VERSION}/opentelemetry-javaagent-${OTEL_VERSION}-all.jar

# Create the application directory
RUN mkdir $APP_HOME
WORKDIR $APP_HOME

# Download the OpenTelemetry Java agent
ADD ${OTEL_JAVAAGENT_URL} /otel-javaagent.jar

# Add your Java application JAR file
COPY your-java-app.jar $APP_HOME/your-java-app.jar

# Expose the application port (e.g. 8080)
EXPOSE 8080

# Configure the OpenTelemetry Java agent and run the application
CMD java -javaagent:/otel-javaagent.jar \
      -Dotel.resource.attributes=service.name=your-service-name \
      -Dotel.exporter.otlp.endpoint=your-otlp-endpoint:4317 \
      -Dotel.exporter.otlp.insecure=true \
      -jar your-java-app.jar

3. Tracing using a service mesh (Envoy/Istio)
The final option you have here is if you are using a service mesh. A service mesh is a dedicated infrastructure layer for handling service-to-service communication in a microservices architecture. It provides a transparent, scalable, and efficient way to manage and control the communication between services, enabling developers to focus on building application features without worrying about inter-service communication complexities.

The great thing about this is that we can activate tracing within the proxy and therefore get visibility into requests between services. We don’t have to change any code or even run APM agents for this; we simply turn on the OpenTelemetry collector that exists within the proxy — therefore this is likely the lowest overhead solution. Learn more about this option.

Synthetics Universal Profiling

Most APM vendors have add ons to the primary APM use cases. Typically we see synthetics and continuous profiling being added to APM solutions. APM can integrate with both, and there is some good value in bringing these technologies together to give even more insights into issues.

Synthetics
Synthetic monitoring is a method used to measure the performance, availability, and reliability of web applications, websites, and APIs by simulating user interactions and traffic. It involves creating scripts or automated tests that mimic real user behavior, such as navigating through pages, filling out forms, or clicking buttons, and then running these tests periodically from different locations and devices.

This gives Development and Operations teams the ability to spot problems far earlier than they might otherwise, catching issues before real users do in many cases.

Synthetics can be integrated with APM — inject an APM agent into the website when the script runs, so even if you didn’t put end user monitoring into your website initially, it can be injected at run time. This usually happens without any input from the user. From there, a tracing id for each request can be passed down through the various layers of the system, allowing teams to follow the request all the way from the synthetics script to the lowest levels of the application stack such as the database.

Universal profiling
“Profiling” is a dynamic method of analyzing the complexity of a program, such as CPU utilization or the frequency and duration of function calls. With profiling, you can locate exactly which parts of your application are consuming the most resources. “Continuous profiling” is a more powerful version of profiling that adds the dimension of time. By understanding your system’s resources over time, you can then locate, debug, and fix issues related to performance.

Universal profiling is a further extension of this, which allows you to capture profile information about all of the code running in your system all the time. Using a technology like eBPF can allow you to see all the function calls in your systems, including into things like the Kubernetes runtime. Doing this gives you the ability to finally see unknown unknowns — things you didn’t know were problems. This is very different from APM, which is really about tracking individual traces and requests and the overall customer experience. Universal profiling is about overcoming those issues you didn’t even know existed and even answering the question “What is my most expensive line of code?”

Universal profiling can be linked into APM, showing you profiles that occurred during a specific customer issue, for example, or by linking profiles directly to traces by looking at the global state that exists at the thread level. These technologies can work wonders when used together.

Typically, profiles are viewed as “flame graphs” shown below. The boxes represent the amount of “on-cpu” time spent executing a particular function.

3. Analytics and AIOps

The interesting thing about APM is it opens up a whole new world of analytics versus just logs. All of a sudden, you have access to the information flows from inside applications.

This allows you to easily capture things like the amount of money a specific customer is currently spending on your most critical ecommerce store, or look at failed trades in a brokerage app to see how much lost revenue those failures are impacting. You can even then apply machine learning algorithms to project future spend or look at anomalies occurring in this data, giving you a new window into how your business runs.

In this section, we will look at ways to do this and how to get the most out of this new world, as well as how to apply AIOps practices to this new data. We will also discuss getting SLIs and SLOs setup for APM data.

Getting business data into your traces

There are generally two ways of getting business data into your traces. You can modify code and add in Span attributes, an example of which is available here and shown below. Or you can write an extension or a plugin, which has the benefit of avoiding code changes. OpenTelemetry supports adding extensions in its auto-instrumentation agents. Most other APM vendors usually have something similar.

def count_completion_requests_and_tokens(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        counters['completion_count'] += 1
        response = func(*args, **kwargs)

        token_count = response.usage.total_tokens
        prompt_tokens = response.usage.prompt_tokens
        completion_tokens = response.usage.completion_tokens
        cost = calculate_cost(response)
        strResponse = json.dumps(response)

        # Set OpenTelemetry attributes
        span = trace.get_current_span()
        if span:
            span.set_attribute("completion_count", counters['completion_count'])
            span.set_attribute("token_count", token_count)
            span.set_attribute("prompt_tokens", prompt_tokens)
            span.set_attribute("completion_tokens", completion_tokens)
            span.set_attribute("model", response.model)
            span.set_attribute("cost", cost)
            span.set_attribute("response", strResponse)
        return response
    return wrapper

Using business data for fun and profit

Once you have the business data in your traces, you can start to have some fun with it. Take a look at the example below for a financial services fraud team. Here we are tracking transactions — average transaction value for our larger business customers. Crucially, we can see if there are any unusual transactions.

A lot of this is powered by machine learning, which can classify transactions or do anomaly detection. Once you start capturing the data, it is possible to do a lot of useful things like this, and with a flexible platform, integrating machine learning models into this process becomes a breeze.

SLIs and SLOs

Service level indicators (SLIs) and service level objectives (SLOs) serve as critical components for maintaining and enhancing application performance. SLIs, which represent key performance metrics such as latency, error rate, and throughput, help quantify an application's performance, while SLOs establish target performance levels to meet user expectations.

By selecting relevant SLIs and setting achievable SLOs, organizations can better monitor their application's performance using APM tools. Continually evaluating and adjusting SLIs and SLOs in response to changes in application requirements, user expectations, or the competitive landscape ensures that the application remains competitive and delivers an exceptional user experience.

In order to define and track SLIs and SLOs, APM becomes a critical perspective that is needed for understanding the user experience. Once APM is implemented, we recommend that organizations perform the following steps.

• Define SLOs and SLIs required to track them.
• Define SLO budgets and how they are calculated. Reflect business’ perspective and set realistic targets.
• Define SLIs to be measured from a user experience perspective.
• Define different alerting and paging rules, page only on customer facing SLO degradations, record symptomatic alerts, notify on critical symptomatic alerts.

Synthetic monitoring and end user monitoring (EUM) can also help with getting even more data required to understand latency, throughput, and error rate from the user’s perspective, where it is critical to get good business focused metrics and data from.

4. Scale and total cost of ownership

With increased perspectives, customers often run into scalability and total cost of ownership issues. All this new data can be overwhelming. Luckily there are various techniques you can use to deal with this. Tracing itself can actually help with volume challenges because you can decompose unstructured logs and combine them with traces, which leads to additional efficiency. You can also use different sampling methods to deal with scale challenges (i.e., both techniques we previously mentioned).

In addition to this, for large enterprise scale, we can use streaming pipelines like Kafka or Pulsar to manage the data volumes. This has an additional benefit that you get for free: if you take down the systems consuming the data or they face outages, it is less likely you will lose data.

With this configuration in place, your “Observability pipeline” architecture would look like this:

This completely decouples your sources of data from your chosen observability solution, which will future proof your observability stack going forward, enable you to reach massive scale, and make you less reliant on specific vendor code for collection of data.

Another thing we recommend doing is being intelligent about instrumentation. This will serve two benefits: you will get some CPU cycles back in the instrumented application, and your backend data collection systems will have less data to process. If you know, for example, that you have no interest in tracking calls to a specific endpoint, you can exclude those classes and methods from instrumentation.

And finally, data tiering is a transformative approach for managing data storage that can significantly reduce the total cost of ownership (TCO) for businesses. Primarily, it allows organizations to store data across different types of storage mediums based on their accessibility needs and the value of the data. For instance, frequently accessed, high-value data can be stored in expensive, high-speed storage, while less frequently accessed, lower-value data can be stored in cheaper, slower storage.

This approach, often incorporated in cloud storage solutions, enables cost optimization by ensuring that businesses only pay for the storage they need at any given time. Furthermore, it provides the flexibility to scale up or down based on demand, eliminating the need for large capital expenditures on storage infrastructure. This scalability also reduces the need for costly over-provisioning to handle potential future demand.

Conclusion

In today's highly competitive and fast-paced software development landscape, simply relying on logging is no longer sufficient to ensure top-notch customer experiences. By adopting APM and distributed tracing, organizations can gain deeper insights into their systems, proactively detect and resolve issues, and maintain a robust user experience.

In this blog, we have explored the journey of moving from a logging-only approach to a comprehensive observability strategy that integrates logs, traces, and APM. We discussed the importance of cultivating a new monitoring mindset that prioritizes customer experience, and the necessary organizational changes required to drive APM and tracing adoption. We also delved into the various stages of the journey, including data ingestion, integration, analytics, and scaling.

By understanding and implementing these concepts, organizations can optimize their monitoring efforts, reduce MTTR, and keep their customers satisfied. Ultimately, prioritizing customer experience through APM and tracing can lead to a more successful and resilient enterprise in today's challenging environment.

Learn more about APM at Elastic.

To simplify this process and reduce management overhead, AWS users can now leverage the new Amazon Kinesis Firehose Delivery Stream to ingest logs into Elastic Cloud in AWS in real time and view them in the Elastic Stack alongside other logs for centralized analytics. This eliminates the necessity for time-consuming and expensive procedures such as VM provisioning or data shipper operations.

Elastic Observability unifies logs, metrics, and application performance monitoring (APM) traces for a full contextual view across your hybrid AWS environments alongside their on-premises data sets. Elastic Observability enables you to track and monitor performance across a broad range of AWS services, including AWS Lambda, Amazon Elastic Compute Cloud (EC2), Amazon Elastic Container Service (ECS), Amazon Elastic Kubernetes Service (EKS), Amazon Simple Storage Service (S3), Amazon Cloudtrail, Amazon Network Firewall, and more.

In this blog, we will walk you through how to use the Amazon Kinesis Data Firehose integration — Elastic is listed in the Amazon Kinesis Firehose drop-down list — to simplify your architecture and send logs to Elastic, so you can monitor and safeguard your multi-account AWS environments.

Announcing the Kinesis Firehose method

Elastic currently provides both agent-based and serverless mechanisms, and we are pleased to announce the addition of the Kinesis Firehose method. This new method enables customers to directly ingest logs from AWS into Elastic, supplementing our existing options.

• Elastic Agent pulls metrics and logs from CloudWatch and S3 where logs are generally pushed from a service (for example, EC2, ELB, WAF, Route53) and ingests them into Elastic Cloud.
• Elastic’s Serverless Forwarder (runs Lambda and available in AWS SAR) sends logs from Kinesis Data Stream, Amazon S3, and AWS Cloudwatch log groups into Elastic. To learn more about this topic, please see this blog post.
• Amazon Kinesis Firehose directly ingests logs from AWS into Elastic (specifically, if you are running the Elastic Cloud on AWS).

In this blog, we will cover the last option since we have recently released the Amazon Kinesis Data Firehose integration. Specifically, we'll review:

• A general overview of the Amazon Kinesis Data Firehose integration and how it works with AWS
• Step-by-step instructions to set up the Amazon Kinesis Data Firehose integration on AWS and on Elastic Cloud

By the end of this blog, you'll be equipped with the knowledge and tools to simplify your AWS log management with Elastic Observability and Amazon Kinesis Data Firehose.

Prerequisites and configurations

If you intend to follow the steps outlined in this blog post, there are a few prerequisites and configurations that you should have in place beforehand.

1. You will need an account on Elastic Cloud and a deployed stack on AWS. Instructions for deploying a stack on AWS can be found here. This is necessary for AWS Firehose Log ingestion.
2. You will also need an AWS account with the necessary permissions to pull data from AWS. Details on the required permissions can be found in our documentation.
3. Finally, be sure to turn on VPC Flow Logs for the VPC where your application is deployed and send them to AWS Firehose.

Elastic’s Amazon Kinesis Data Firehose integration

Elastic has collaborated with AWS to offer a seamless integration of Amazon Kinesis Data Firehose with Elastic, enabling direct ingestion of data from Amazon Kinesis Data Firehose into Elastic without the need for Agents or Beats. All you need to do is configure the Amazon Kinesis Data Firehose delivery stream to send its data to Elastic's endpoint. In this configuration, we will demonstrate how to ingest VPC Flow logs and Firewall logs into Elastic. You can follow a similar process to ingest other logs from your AWS environment into Elastic.

There are three distinct configurations available for ingesting VPC Flow and Network firewall logs into Elastic. One configuration involves sending logs through CloudWatch, and another uses S3 and Kinesis Firehose; each has its own unique setup. With Cloudwatch and S3 you can store and forward but with Kinesis Firehose you will have to ingest immediately. However, in this blog post, we will focus on this new configuration that involves sending VPC Flow logs and Network Firewall logs directly to Elastic.

We will guide you through the configuration of the easiest setup, which involves directly sending VPC Flow logs and Firewalls logs to Amazon Kinesis Data Firehose and then into Elastic Cloud.

Note: It's important to note that this setup is only compatible with Elastic Cloud on AWS and cannot be used with self-managed or on-premise or other cloud provider Elastic deployments.

Setting it all up

To begin setting up the integration between Amazon Kinesis Data Firehose and Elastic, let's go through the necessary steps.

Step 0: Get an account on Elastic Cloud

Create an account on Elastic Cloud by following the instructions provided to get started on Elastic Cloud.

Step 1: Deploy Elastic on AWS

You can deploy Elastic on AWS via two different approaches: through the UI or through Terraform. We’ll start first with the UI option.

After logging into Elastic Cloud, create a deployment on Elastic. It's crucial to make sure that the deployment is on Elastic Cloud on AWS since the Amazon Kinesis Data Firehose connects to a specific endpoint that must be on AWS.

After your deployment is created, it's essential to copy the Elasticsearch endpoint to ensure a seamless configuration process.

The Elasticsearch HTTP endpoint should be copied and used for Amazon Firehose destination configuration purposes, as it will be required. Here's an example of what the endpoint should look like:

https://elastic-O11y-log.es.us-east-1.aws.found.io

Alternative approach using Terraform

An alternative approach to deploying Elastic Cloud on AWS is by using Terraform. It's also an effective way to automate and streamline the deployment process.

To begin, simply create a Terraform configuration file that outlines the necessary infrastructure. This file should include resources for your Elastic Cloud deployment and any required IAM roles and policies. By using this approach, you can simplify the deployment process and ensure consistency across environments.

One easy way to create your Elastic Cloud deployment with Terraform is to use this Github repo. This resource lets you specify the region, version, and deployment template for your Elastic Cloud deployment, as well as any additional settings you require.

Step 2: To turn on Elastic's AWS integrations, navigate to the Elastic Integration section in your deployment

To install AWS assets in your deployment's Elastic Integration section, follow these steps:

1. Log in to your Elastic Cloud deployment and open Kibana.
2. To get started, go to the management section of Kibana and click on " Integrations."
3. Navigate to the AWS integration and click on the "Install AWS Assets" button in the settings.This step is important as it installs the necessary assets such as dashboards and ingest pipelines to enable data ingestion from AWS services into Elastic.

Step 3: Set up the Amazon Kinesis Data Firehose delivery stream on the AWS Console

You can set up the Kinesis Data Firehose delivery stream via two different approaches: through the AWS Management Console or through Terraform. We’ll start first with the console option.

To set up the Kinesis Data Firehose delivery stream on AWS, follow these steps:

1. Go to the AWS Management Console and select Amazon Kinesis Data Firehose.

2. Click on Create delivery stream.

3. Choose a delivery stream name and select Direct PUT or other sources as the source.

1. Choose Elastic as the destination.

2. In the Elastic destination section, enter the Elastic endpoint URL that you copied from your Elastic Cloud deployment.

1. Choose the content encoding and retry duration as shown above.

2. Enter the appropriate parameter values for your AWS log type. For example, for VPC Flow logs, you would need to specify the es_datastream_name and logs-aws.vpc flow-default _.

3. Configure the Amazon S3 bucket as the source backup for the Amazon Kinesis Data Firehose delivery stream failed data or all data, and configure any required tags for the delivery stream.

4. Review the settings and click on Create delivery stream.

In the example above, we are using the es_datastream_name parameter to pull in VPC Flow logs through the logs-aws.vpcflow-default datastream. Depending on your use case, this parameter can be configured with one of the following types of logs:

• logs-aws.cloudfront_logs-default (AWS CloudFront logs)
• logs-aws.ec2_logs-default (EC2 logs in AWS CloudWatch)
• logs-aws.elb_logs-default (Amazon Elastic Load Balancing logs)
• logs-aws.firewall_logs-default (AWS Network Firewall logs)
• logs-aws.route53_public_logs-default (Amazon Route 53 public DNS queries logs)
• logs-aws.route53_resolver_logs-default (Amazon Route 53 DNS queries & responses logs)
• logs-aws.s3access-default (Amazon S3 server access log)
• logs-aws.vpcflow-default (AWS VPC flow logs)
• logs-aws.waf-default (AWS WAF Logs)

Alternative approach using Terraform

Using the " aws_kinesis_firehose_delivery_stream" resource in Terraform is another way to create a Kinesis Firehose delivery stream, allowing you to specify the delivery stream name, data source, and destination - in this case, an Elasticsearch HTTP endpoint. To authenticate, you'll need to provide the endpoint URL and an API key. Leveraging this Terraform resource is a fantastic way to automate and streamline your deployment process, resulting in greater consistency and efficiency.

Here's an example code that shows you how to create a Kinesis Firehose delivery stream with Terraform that sends data to an Elasticsearch HTTP endpoint:

resource "aws_kinesis_firehose_delivery_stream" “Elasticcloud_stream" {
  name        = "terraform-kinesis-firehose-ElasticCloud-stream"
  destination = "http_endpoint”
  s3_configuration {
    role_arn           = aws_iam_role.firehose.arn
    bucket_arn         = aws_s3_bucket.bucket.arn
    buffer_size        = 5
    buffer_interval    = 300
    compression_format = "GZIP"
  }
  http_endpoint_configuration {
    url        = "https://cloud.elastic.co/"
    name       = “ElasticCloudEndpoint"
    access_key = “ElasticApi-key"
    buffering_hints {
      size_in_mb = 5
      interval_in_seconds = 300
    }

   role_arn       = "arn:Elastic_role"
   s3_backup_mode = "FailedDataOnly"
  }
}

Step 4: Configure VPC Flow Logs to send to Amazon Kinesis Data Firehose

To complete the setup, you'll need to configure VPC Flow logs in the VPC where your application is deployed and send them to the Amazon Kinesis Data Firehose delivery stream you set up in Step 3.

Enabling VPC flow logs in AWS is a straightforward process that involves several steps. Here's a step-by-step details to enable VPC flow logs in your AWS account:

1. Select the VPC for which you want to enable flow logs.

2. In the VPC dashboard, click on "Flow Logs" under the "Logs" section.

3. Click on the "Create Flow Log" button to create a new flow log.

4. In the "Create Flow Log" wizard, provide the following information:

Choose the target for your flow logs: In this case, Amazon Kinesis Data Firehose in the same AWS account.

• Provide a name for your flow log.
• Choose the VPC and the network interface(s) for which you want to enable flow logs.
• Choose the flow log format: either AWS default or Custom format.

1. Configure the IAM role for the flow logs. If you have an existing IAM role, select it. Otherwise, create a new IAM role that grants the necessary permissions for the flow logs.

2. Review the flow log configuration and click "Create."

Create the VPC Flow log.

Step 5: After a few minutes, check if flows are coming into Elastic

To confirm that the VPC Flow logs are ingesting into Elastic, you can check the logs in Kibana. You can do this by searching for the index in the Kibana Discover tab and filtering the results by the appropriate index and time range. If VPC Flow logs are flowing in, you should see a list of documents representing the VPC Flow logs.

Step 6: Navigate to Kibana to see your logs parsed and visualized in the [Logs AWS] VPC Flow Log Overview dashboard

Finally, there is an Elastic out-of-the-box (OOTB) VPC Flow logs dashboard that displays the top IP addresses that are hitting your VPC, their geographic location, time series of the flows, and a summary of VPC flow log rejects within the selected time frame. This dashboard can provide valuable insights into your network traffic and potential security threats.

Note: For additional VPC flow log analysis capabilities, please refer to this blog.

Step 7: Configure AWS Network Firewall Logs to send to Kinesis Firehose

To create a Kinesis Data Firehose delivery stream for AWS Network firewall logs, first log in to the AWS Management Console, navigate to the Kinesis service, select "Data Firehose", and follow the step-by-step instructions as shown in Step 3. Specify the Elasticsearch endpoint, API key, add a parameter (_ es_datastream_name=logs-aws.firewall_logs-default _), and create the delivery stream.

Second, to set up a Network Firewall rule group to send logs to the Kinesis Firehose, go to the Network Firewall section of the console, create a rule group, add a rule to allow traffic to the Kinesis endpoint, and attach the rule group to your Network Firewall configuration. Finally, test the configuration by sending traffic through the Network Firewall to the Kinesis Firehose endpoint and verify that logs are being delivered to your S3 bucket.

Kindly follow the instructions below to set up a firewall rule and logging.

1. Set up a Network Firewall rule group to send logs to Amazon Kinesis Data Firehose:

• Go to the AWS Management Console and select Network Firewall.
• Click on "Rule groups" in the left menu and then click "Create rule group."
• Choose "Stateless" or "Stateful" depending on your requirements, and give your rule group a name. Click "Create rule group."
• Add a rule to the rule group to allow traffic to the Kinesis Firehose endpoint. For example, if you are using the us-east-1 region, you would add a rule like this:json

{
  "RuleDefinition": {
    "Actions": [
      {
        "Type": "AWS::KinesisFirehose::DeliveryStream",
        "Options": {
          "DeliveryStreamArn": "arn:aws:firehose:us-east-1:12387389012:deliverystream/my-delivery-stream"
        }
      }
    ],
    "MatchAttributes": {
      "Destination": {
        "Addresses": ["api.firehose.us-east-1.amazonaws.com"]
      },
      "Protocol": {
        "Numeric": 6,
        "Type": "TCP"
      },
      "PortRanges": [
        {
          "From": 443,
          "To": 443
        }
      ]
    }
  },
  "RuleOptions": {
    "CustomTCPStarter": {
      "Enabled": true,
      "PortNumber": 443
    }
  }
}

• Save the rule group.

1. Attach the rule group to your Network Firewall configuration:

• Go to the AWS Management Console and select Network Firewall.
• Click on "Firewall configurations" in the left menu and select the configuration you want to attach the rule group to.
• Scroll down to "Associations" and click "Edit."
• Select the rule group you created in Step 2 and click "Save."

1. Test the configuration:

• Send traffic through the Network Firewall to the Kinesis Firehose endpoint and verify that logs are being delivered to your S3 bucket.

Step 8: Navigate to Kibana to see your logs parsed and visualized in the [Logs AWS] Firewall Log dashboard

Wrapping up

We’re excited to bring you this latest integration for AWS Cloud and Kinesis Data Firehose into production. The ability to consolidate logs and metrics to gain visibility across your cloud and on-premises environment is crucial for today’s distributed environments and applications.

From EC2, Cloudwatch, Lambda, ECS and SAR, Elastic Integrations allow you to quickly and easily get started with ingesting your telemetry data for monitoring, analytics, and observability. Elastic is constantly delivering frictionless customer experiences, allowing anytime, anywhere access to all of your telemetry data — this streamlined, native integration with AWS is the latest example of our commitment.

Start a free trial today

You can begin with a 7-day free trial of Elastic Cloud within the AWS Marketplace to start monitoring and improving your users' experience today!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we’ll cover some of Elastic’s artificial intelligence for IT operations (AIOps) and machine learning (ML) capabilities for root cause analysis.

Elastic’s machine learning will help you investigate performance issues by providing anomaly detection and pinpointing potential root causes through time series analysis and log outlier detection. These capabilities will help you reduce time in finding that “needle” in the haystack.

Elastic’s platform enables you to get started on machine learning quickly. You don’t need to have a data science team or design a system architecture. Additionally, there’s no need to move data to a third-party framework for model training.

Preconfigured machine learning models for observability and security are available. If those don't work well enough on your data, in-tool wizards guide you through the few steps needed to configure custom anomaly detection and train your model with supervised learning. To help get you started, there are several key features built into Elastic Observability to aid in analysis, helping bypass the need to run specific ML models. These features help minimize the time and analysis for logs.

Let’s review some of these built-in ML features:

Anomaly detection: Elastic Observability, when turned on (see documentation), automatically detects anomalies by continuously modeling the normal behavior of your time series data — learning trends, periodicity, and more — in real time to identify anomalies, streamline root cause analysis, and reduce false positives. Anomaly detection runs in and scales with Elasticsearch and includes an intuitive UI.

Log categorization: Using anomaly detection, Elastic also identifies patterns in your log events quickly. Instead of manually identifying similar logs, the logs categorization view lists log events that have been grouped, based on their messages and formats, so that you can take action quicker.

High-latency or erroneous transactions: Elastic Observability’s APM capability helps you discover which attributes are contributing to increased transaction latency and identifies which attributes are most influential in distinguishing between transaction failures and successes. An overview of this capability is published here: APM correlations in Elastic Observability: Automatically identifying probable causes of slow or failed transactions.

AIOps Labs: AIOps Labs provides two main capabilities using advanced statistical methods:

• Log spike detector helps identify reasons for increases in log rates. It makes it easy to find and investigate causes of unusual spikes by using the analysis workflow view. Examine the histogram chart of the log rates for a given data view, and find the reason behind a particular change possibly in millions of log events across multiple fields and values.
• Log pattern analysis helps you find patterns in unstructured log messages and makes it easier to examine your data. It performs categorization analysis on a selected field of a data view, creates categories based on the data, and displays them together with a chart that shows the distribution of each category and an example document that matches the category.

In this blog, we will cover anomaly detection and log categorization against the popular “Hipster Shop app” developed by Google, and modified recently by OpenTelemetry.

Overviews of high-latency capabilities can be found here, and an overview of AIOps labs can be found here.

In this blog, we will examine a scenario where we use anomaly detection and log categorization to help identify a root cause of an issue in Hipster Shop.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here) on AWS. Deploying this on AWS is required for Elastic Serverless Forwarder.
• Utilize a version of the ever so popular Hipster Shop demo application. It was originally written by Google to showcase Kubernetes across a multitude of variants available, such as the OpenTelemetry Demo App. The Elastic version is found here.
• Ensure you have configured the app for either Elastic APM agents or OpenTelemetry agents. For more details, please refer to these two blogs: Independence with OTel in Elastic and Observability and security with OTel in Elastic. Additionally, review the OTel documentation in Elastic.
• Look through an overview of Elastic Observability APM capabilities.
• Look through our Anomaly detection documentation for logs and log categorization documentation.

Once you’ve instrumented your application with APM (Elastic or OTel) agents and are ingesting metrics and logs into Elastic Observability, you should see a service map for the application as follows:

In our example, we’ve introduced issues to help walk you through the root cause analysis features: anomaly detection and log categorization. You might have a different set of anomalies and log categorization depending on how you load the application and/or introduce specific issues.

As part of the walk-through, we’ll assume we are a DevOps or SRE managing this application in production.

Root cause analysis

While the application has been running normally for some time, you get a notification that some of the services are unhealthy. This can occur from the notification setting you’ve set up in Elastic or other external notification platforms (including customer related issues). In this instance, we’re assuming that customer support has called in multiple customer complaints about the website.

How do you as a DevOps or SRE investigate this? We will walk through two avenues in Elastic to investigate the issue:

• Anomaly detection
• Log categorization

While we show these two paths separately, they can be used in conjunction and are complementary, as they are both tools Elastic Observability provides to help you troubleshoot and identify a root cause.

Machine learning for anomaly detection

Elastic will detect anomalies based on historical patterns and identify a probability of these issues.

Starting with the service map, you can see anomalies identified with red circles and as we select them, Elastic will provide a score for the anomaly.

In this example, we can see that there is a score of 96 for a specific anomaly for the productCatalogService in the Hipster Shop application. An anomaly score indicates the significance of the anomaly compared to previously seen anomalies. More information on anomaly detection results can be found here. We can also dive deeper into the anomaly and analyze the details.

What you will see for the productCatalogService is that there is a severe spike in average transaction latency time, which is the anomaly that was detected in the service map. Elastic’s machine learning has identified a specific metric anomaly (shown in the single metric view). It’s likely that customers are potentially responding to the slowness of the site and that the company is losing potential transactions.

One step to take next is to review all the other potential anomalies that we saw in the service map in a larger picture. Use an anomaly explorer to view all the anomalies that have been identified.

Elastic is identifying numerous services with anomalies. productCatalogService has the highest score and a good number or others: frontend, checkoutService, advertService, and others, also have high scores. However, this analysis is looking at just one metric.

Elastic can help detect anomalies across all types of data, such as kubernetes data, metrics, and traces. If we analyze across all these types (via individual jobs we’ve created in Elastic machine learning), we will see a more comprehensive view as to what is potentially causing this latency issue.

Once all the potential jobs are selected and we’ve sorted by service.name, we can see that productCatalogService is still showing a high anomaly influencer score.

In addition to the chart giving us a visual of the anomalies, we can review all the potential anomalies. As you will notice, Elastic has also categorized these anomalies (see category examples column). As we scroll through the results, we notice a potential postgreSQL issue from the categorization, which also has a high 94 score. Machine learning has identified a “rare mlcategory,” meaning that it has rarely occurred, hence pointing to a potential cause of the issue customers are seeing.

We also notice that this issue is potentially caused by pgbench , a popular postgreSQL tool to help benchmark the database. pgbench runs the same sequence of SQL commands over and over, possibly in multiple, concurrent database sessions. While pgbench is definitely a useful tool, it should not be used in a production environment as it causes heavy load on the database host, likely causing the higher latency issues on the site.

While this may or may not be the ultimate root cause, we have rather quickly identified a potentially issue that has a high probability of being the root cause. An engineer likely intended to run pgbench against a staging database to evaluate its performance, and not the production environment.

Machine learning for log categorization

Elastic Observability’s service map has detected an anomaly, and in this part of the walk-through, we take a different approach by investigating the service details from the service map versus initially exploring the anomaly. When we explore the service details for productCatalogService, we see the following:

The service details are identifying several things:

1. There is an abnormally high latency compared to expected bounds of the service. We see that recently there was a higher than normal (upward of 1s latency) compared to the average to 275ms on average.
2. There is also a high failure rate for the same time frame as the high latency (lower left chart “ Failed transaction rate ”).
3. Additionally, we can see the transactions and one in particular /ListProduct has an abnormally high latency, in addition to a high failure rate.
4. We see productCatalogService has a dependency on postgreSQL.
5. We also see errors all related to postgreSQL.

We have an option to dig through the logs and analyze in Elastic or we can use a capability to identify the logs more easily.

If we go to Categories under Logs in Elastic Observability and search for postgresql.logto help identify postgresql logs that could be causing this error, we see that Elastic’s machine learning has automatically categorized the postgresql logs.

We notice two additional items:

• There is a high count category (message count of 23,797 with a high anomaly of 70) related to pgbench (which is odd to see in production). Hence we search further for all pgbench related logs in Categories .
• We see an odd issue regarding terminating the connection (with a low count).

While investigating the second error, which is severe, we can see logs from Categories before and after the error.

This troubleshooting shows postgreSQL having a FATAL error, the database shutting down prior to the error, and all connections terminating. Given the two immediate issues we identified, we have an idea that someone was running pgbench and this potentially overloaded the database, causing the latency issue that customers are seeing.

The next steps here could be to investigate anomaly detection and/or work with the developers to review the code and identify pgbench as part of the deployed configuration.

Conclusion

I hope you’ve gotten an appreciation for how Elastic Observability can help you further identify and get closer to pinpointing root cause of issues without having to look for a “needle in a haystack.” Here’s a quick recap of lessons and what you learned:

• Elastic Observability has numerous capabilities to help you reduce your time to find root cause and improve your MTTR (even MTTD). In particular, we reviewed the following two main capabilities in this blog:

1. Anomaly detection: Elastic Observability, when turned on (see documentation), automatically detects anomalies by continuously modeling the normal behavior of your time series data — learning trends, periodicity, and more — in real time to identify anomalies, streamline root cause analysis, and reduce false positives. Anomaly detection runs in and scales with Elasticsearch and includes an intuitive UI.
2. Log categorization: Using anomaly detection, Elastic also identifies patterns in your log events quickly. Instead of manually identifying similar logs, the logs categorization view lists log events that have been grouped based on their messages and formats so that you can take action quicker.

• You learned how easy and simple it is to use Elastic Observability’s log categorization and anomaly detection capabilities without having to understand machine learning (which help drive these features), nor having to do any lengthy setups. Ready to get started? Register for Elastic Cloud and try out the features and capabilities I’ve outlined above.

Additional logging resources:

• Getting started with logging on Elastic (quickstart)
• Ingesting common known logs via integrations (compute node example)
• List of integrations
• Ingesting custom application logs into Elastic
• Enriching logs in Elastic
• Analyzing Logs with Anomaly Detection (ML) and AIOps

Common use case examples with logs:

• Nginx log management
• AWS VPC Flow log management
• Using OpenAI to analyze Kubernetes errors
• PostgreSQL issue analysis with AIOps

Logging is an important part of observability, for which we generally think of metrics and/or tracing. However, the amount of logs an application or the underlying infrastructure output can be significantly daunting.

With Elastic Observability, there are three main mechanisms to ingest logs:

• The new Elastic Agent pulls metrics and logs from CloudWatch and S3 where logs are generally pushed from a service (for example, EC2, ELB, WAF, Route53, etc ). We reviewed Elastic agent metrics configuration for EC2, RDS (Aurora), ELB, and NAT metrics in this blog.
• Using Elastic’s Serverless Forwarder (runs on Lambda and available in AWS SAR) to send logs from Firehose, S3, CloudWatch, and other AWS services into Elastic.
• Beta feature (contact your Elastic account team): Using AWS Firehose to directly insert logs from AWS into Elastic — specifically if you are running the Elastic stack on AWS infrastructure.

In this blog we will provide an overview of the second option, Elastic’s serverless forwarder collecting VPC Flow Logs from an application deployed on EC2 instances. Here’s what we'll cover:

• A walk-through on how to analyze VPC Flow Log info with Elastic’s Discover, dashboard, and ML analysis.
• A detailed step-by-step overview and setup of the Elastic serverless forwarder on AWS as a pipeline for VPC Flow Logs into Elastic Cloud.

Elastic’s serverless forwarder on AWS Lambda

AWS users can quickly ingest logs stored in Amazon S3, CloudWatch, or Kinesis with the Elastic serverless forwarder, an AWS Lambda application, and view them in the Elastic Stack alongside other logs and metrics for centralized analytics. Once the AWS serverless forwarder is configured and deployed from AWS, Serverless Application Registry (SAR) logs will be ingested and available in Elastic for analysis. See the following links for further configuration guidance:

• Elastic’s serverless forwarder (runs Lambda and available in AWS SAR)
• Serverless forwarder GitHub repo

In our configuration we will ingest VPC Flow Logs into Elastic for the three-tier app deployed in the previous blog.

There are three different configurations with the Elastic serverless forwarder:

Logs can be directly ingested from:

• Amazon CloudWatch: Elastic serverless forwarder can pull VPC Flow Logs directly from an Amazon CloudWatch log group, which is a commonly used endpoint to store VPC Flow Logs in AWS.
• Amazon Kinesis: Elastic serverless forwarder can pull VPC Flow Logs directly from Kinesis, which is another location to publish VPC Flow Logs.
• Amazon S3: Elastic serverless forwarder can pull VPC Flow Logs from Amazon S3 via SQS event notifications, which is a common endpoint to publish VPC Flow Logs in AWS.

We will review how to utilize a common configuration, which is to send VPC Flow Logs to Amazon S3 and into Elastic Cloud in the second half of this blog.

But first let's review how to analyze VPC Flow Logs on Elastic.

Analyzing VPC Flow Logs in Elastic

Now that you have VPC Flow Logs in Elastic Cloud, how can you analyze them?

There are several analyses you can perform on the VPC Flow Log data:

1. Use Elastic’s Analytics Discover capabilities to manually analyze the data.
2. Use Elastic Observability’s anomaly feature to identify anomalies in the logs.
3. Use an out-of-the-box (OOTB) dashboard to further analyze data.

Using Elastic Discover

In Elastic analytics, you can search and filter your data, get information about the structure of the fields, and display your findings in a visualization. You can also customize and save your searches and place them on a dashboard. With Discover, you can:

• View logs in bulk, within specific time frames
• Look at individual details of each entry (document)
• Filter for specific values
• Analyze fields
• Create and save searches
• Build visualizations

For a complete understanding of Discover and all of Elastic’s analytics capabilities, look at Elastic documentation.

For VPC Flow Logs, an important stat is to understand:

• How many logs were accepted/rejected
• Where potential security violations are occur (for example, source IPs from outside the VPC)
• What port is generally being queried

I’ve filtered the logs on the following:

• Amazon S3: bshettisartest
• VPC Flow Log action: REJECT
• VPC Network Interface: Webserver 1

We want to see what IP addresses are trying to hit our web servers.

From that, we want to understand which IP addresses we are getting the most REJECTS from, and we simply find the source.ip field. Then, we can quickly get a breakdown that shows 185.242.53.156 is the most rejected for the last 3+ hours we’ve turned on VPC Flow Logs.

Additionally, I can see a visualization by selecting the “Visualize” button. We get the following, which we can add to a dashboard:

In addition to IP addresses, we want to also see what port is being hit on our web servers.
We select the destination port field, and the quick pop-up shows us a list of ports being targeted. We can see that port 23 is being targeted (this port is generally used for telnet), port 445 is being targeted (used for Microsoft Active Directory), and port 433 (used for https ssl). We also see these are all REJECT.

Anomaly detection in Elastic Observability logs

Addition to Discover, Elastic Observability provides the ability to detect anomalies on logs. In Elastic Observability -> logs -> anomalies you can turn on machine learning for:

• Log rate: automatically detects anomalous log entry rates
• Categorization: automatically categorizes log messages