This feature is available in stateful deployments only and isn’t supported on Elastic Cloud Serverless.

What cross-cluster search for synthetic monitoring adds to Elasticsearch CCS

Before 9.5, Synthetics and CCS weren't integrated. Connecting clusters at the Elasticsearch layer gave you raw ping data in Discover but not in the Synthetics UI.Elastic's site reliability engineering (SRE) team wrote about this pattern, using CCS to build a single pane of glass across 150+ clusters, but it applied to raw data, not the Synthetics app.

Monitor definitions are Kibana saved objects, and those objects don't travel across clusters. Wiring CCS at the Elasticsearch layer didn't make remote monitors appear in the Synthetics app. You'd get raw data in Discover, but none of the Synthetics UI experience: no status cards, journey waterfalls, step screenshots, or error drilldowns.

That's still true for raw CCS on saved objects. What shipped in 9.5 is a native integration built on top of remote cluster search. The app searches remote synthetics-* pings. It marks what's remote and sends you back to the origin Kibana whenever you need to change something.

What remote monitors look like in the Synthetics UI

Once remote clusters are configured, their monitors appear alongside your local ones throughout the Synthetics UI.

Find remote monitors on the Overview page

The Overview page shows remote monitors inline with local ones, tagged with a Remote badge next to the monitor type badge. Two additions help with navigation: a Remote cluster filter alongside the existing Type / Location / Tags / Frequency filters, and a Group by → Remote cluster option. When a remote cluster filter is active, local monitors are hidden and status counts reflect only the filtered set, so you can focus on one deployment at a time.

Open a remote monitor detail page

Monitor detail pages work the same way for remote monitors as for local ones, including overview, history, errors, journey runs, step details, screenshots, network waterfalls, and duration charts.

A Remote monitor callout shows the cluster name and the originating Kibana URL, and a View on remote cluster button takes you directly to the monitor on its home instance.

See remote TLS certificates on the Certificates page

Transport Layer Security (TLS) certificates observed on remote clusters appear on the Certificates page, with a tooltip showing which cluster they came from.

Can you edit remote monitors from the central Kibana?

No, not from the central Kibana. Remote monitors are read-only where you view them because their configuration lives as a saved object in the Kibana where they were defined.

Action

Behavior on the central Kibana

View status, history, errors, journey runs, screenshots, network waterfalls

Available

Enable / Disable

Not available

Run test manually

Not available

Create SLO

Not available

Add to dashboard

Not available

Edit

Opens the origin Kibana, when that cluster’s public URL is configured

Clone

Opens the origin Kibana, when that cluster’s public URL is configured

View on remote cluster

Opens the monitor on the Kibana where it lives

 To manage a remote monitor, click the View on remote cluster button on its detail page. It takes you to the monitor on the Kibana where it lives.

Keeping configuration on the origin cluster is intentional. A single source of truth for monitor definitions avoids the mess of cross-cluster writes.

One current limitation to note: alerting doesn't cross cluster boundaries yet. A remote monitor that's down shows as down in the federated UI, but it won't fire alerts in the central Kibana. Keep those rules on the cluster that owns the monitor. Federated alerting is on the roadmap.

How to set up cross-cluster search for synthetic monitoring

There are a few things to put in place.

1. Connect the remote clusters in Elasticsearch

In Stack Management → Remote Clusters, add the Elasticsearch clusters that hold Synthetics data. Synthetics doesn't create these connections; it uses the ones already configured at the Elasticsearch level. The Search and replicate data between Elastic Cloud and on-prem post covers the Elasticsearch-level setup in detail.

2. Set server.publicBaseUrl on every participating Kibana

The View on remote cluster deep link comes from a kibanaUrl field on ping documents, stamped from server.publicBaseUrl. If that setting is missing on a remote Kibana, you still see the monitor but there's no URL to click. Since kibanaUrl is written at ping time, you need to wait for new check results after setting this config before the link appears.

3. Choose which remote clusters Synthetics includes

Go to Synthetics → Settings → Remote Clusters. You need the All privilege for Synthetics and Uptime. Users with Read see the form as read-only.

Toggle on Use all remote clusters, or pick specific ones. Each option shows connected versus disconnected status.

4. Grant access to Kibana spaces

This is one shared configuration per deployment, not a separate copy per space. Use the Spaces field to grant access to specific spaces, or select All spaces. On first save, settings are anchored to the space you're in.

Note that clearing the Spaces field on a subsequent save doesn’t remove access from existing spaces. You must explicitly select the spaces you want to keep.

Open the Monitors overview, and remote monitors appear alongside your local ones.

How the index expression resolves across clusters

Nothing is replicated into the central cluster. CCS works by expanding the index expression at query time, which means that the central cluster never stores remote data; it just knows where to ask.

On each request, the server figures out which index expressions to query.

• If you select Use all remote clusters, it builds synthetics-*,*:synthetics-* with no connectivity check needed.

• If you pick specific clusters, it calls Elasticsearch's cluster.remoteInfo() to check which are currently reachable and only includes those, so a disconnected remote doesn't hang your overview page.

• The resolved indices are cached per space.

The result is a combined index pattern like the following:

The Overview status page and the Certificates page pick up CCS automatically, since they fan out across all configured clusters. Detail views scope each query to the right cluster. Pings, journeys, screenshots, and network events all resolve against a single remote rather than spreading across everything.

What cross-cluster search for Synthetics does not do

CCS for Synthetics is a visibility layer, not a management layer. You get a unified read view across clusters; each cluster stays the owner of its own monitors. If you need to create, edit, or disable monitors centrally, you still do that on the Kibana where each monitor lives.

What

Where it lives

Monitor configuration (saved objects)

Origin cluster only

Private locations

Origin cluster only

Global parameters

Origin cluster only

API keys

Origin cluster only

Alerting rules

Origin cluster only

Ping results

Federated, visible in the central Kibana

Journey steps and screenshots

Federated, visible in the central Kibana

TLS certificate status

Federated, visible in the central Kibana

CCS at the Elasticsearch level is also a prerequisite rather than the whole story. Synthetics needs its own Remote Clusters configuration to show remote monitors properly. Dashboards and Discover can still query synthetics-* via CCS for custom visualizations.

Start centralized monitoring across your clusters

Federated Synthetics ships in 9.5 for stateful deployments. If you run monitors across multiple clusters, go to Synthetics → Settings → Remote Clusters and connect your remotes.

If a remote monitor appears but the deep link is missing, set server.publicBaseUrl on the origin Kibana and wait for new check results. The kibanaUrl field is stamped on documents written after that config is in place.

We welcome your feedback in the Elastic discuss forum.

Documentation:

• Configure Synthetics settings

• Analyze data from synthetic monitors

Related reading:

• Use Cross Cluster Search with Elastic Cloud to improve observability: Elastic's SRE team on building a single pane of glass across 150+ clusters

• Search and replicate data between Elastic Cloud and on-prem: CCS/cross-cluster replication (CCR) general availability (GA) announcement

Most users will notice nothing, which is the point, and anything you do notice is a bug. The difference is for maintainers and for anyone writing their own instrumentation. You get classloader isolation, and you can debug instrumentation in an IDE. Writing your own no longer means forking the agent or relying on its internals.

I did this migration once before, on the Elastic APM Java Agent, with a team of three. On paper, the plan the second time was simple:

1. Take the same approach we had proven in production with the Elastic APM Java Agent.

2. Convince OTel maintainers to adopt it.

3. Migrate most OTel instrumentation.

4. Declare victory and celebrate.

We wrote a blog post when we first started that work. This post is about everything between step two and step four.

What invokedynamic is and why it matters for Java agent instrumentation

If you’re interested in technical details, I recommend reading Embracing invokedynamic to tame class loaders in Java agents, the original blog post from Felix Barnsteiner and Rafael Winterhalter (Byte Buddy author).

For those who haven’t read it, here’s a short summary. Both the Elastic APM Java Agent and OTel Java instrumentation are built on Byte Buddy, a bytecode manipulation library that lets you modify application bytecode at runtime.

By default, Byte Buddy provides an inlining strategy; that is, the instrumentation injects bytecode directly into the target method. This is simple, but it trades off some flexibility. All classes referenced by the injected bytecode must be visible to the target class classloader, and we can't debug the injected code with common tools, like integrated development environments (IDEs). Plus, the initial driver was allowing end users to provide their own instrumentation without having to maintain a fork of the instrumentation agent or rely on its internal APIs.

What we internally call the "indy instrumentation" uses the invokedynamic Java Virtual Machine (JVM) bytecode instruction introduced in Java 7 to delegate instrumentation hooks without inlining the instrumentation code into the original method body.

Here is how the two instrumentation modes compare across the areas that matter in practice.

Inlined bytecode

invokedynamic delegation

Class visibility

Referenced classes must be visible to the target classloader

Instrumentation runs in an isolated classloader

Debugging

Not possible with standard IDE tooling

Standard breakpoints and stepping work

JPMS

Inherits the instrumented code’s access

Access must be granted explicitly

Pre-Java 7 bytecode

No constraint

Handled via a generated invokestatic helper

Classloader instrumentation

Supported

Not supported, these modules stay inline

One major downside of this approach is that it requires modifying and reworking the instrumentation code. Another added challenge with instrumentation agents is that they’re deployed in thousands of production environments, so there’s a lot of pressure to avoid regressions and bugs, since they could cause dramatic crashes or performance regression in the application.

Although extensive testing is done before any release, the nature of instrumentation is that the end result is highly dependent on the target application. At best, we can validate with sample applications, but those will never cover all the edge cases that a real application and its deployment can have, such as proprietary frameworks and application servers, esoteric JVMs, OSGi, and complex class loading.

How we migrated the Elastic APM Java Agent with a team of three

At the time, there were only three engineers working on the Elastic APM Java Agent. Although that made it easier to coordinate and make decisions quickly, it also meant that we had very limited capacity to deal with support cases.

Because it's impossible and too risky to do a "big-bang" migration, we had to perform a gradual transition, moving only a few plugins at a time. One of the first lessons I learned was that even if you have limited capacity in a risk-averse environment, you can still make progress on large and complex changes; you just need to figure out how to apply changes incrementally. Also, doing so provides a natural safety net because the risk is spread across multiple releases.

A practical tool for this kind of task is what I call a "shopping list" of issues, so you can see the overall progress at a glance. We tracked our first migration here: #1337.

Although a few users systematically upgrade to the latest version whenever a new release is available, most upgrade organically over time, and this provides a natural incremental rollout of any new version. There’s a similar adoption pattern for most software and libraries. In rare cases, early adopters reported issues, but we fixed them quickly before the bulk of users upgraded.

What Log4Shell told us about the migration

However, we were in for a surprise. The Log4Shell critical vulnerability was disclosed in December 2021 (less than three weeks after the publication of our initial blog post), and it triggered a wave of emergency upgrades across the Java ecosystem. Unfortunately, the Elastic APM Java Agent was impacted, since log4j is used for logging in the agent itself, exploitable under a non-default configuration, as detailed in the security advisory.

If you were trying to upgrade log4j in your application at that time, you probably remember the chaos and the stress it caused. To me, it seemed like almost everyone was upgrading to the latest version of log4j and publishing new releases of their own software and libraries. As a consequence, the Maven central repository was under extreme load and most builds and releases failed due to timeouts.

Economists have the concept of natural experiments, where an unexpected anomaly in the real world is used to try out a hypothesis that can't be tested in a controlled environment. We don't have a direct equivalent, but this unintentional massive migration to the newer version of the Elastic APM Java Agent provided us a strong signal that the implementation was stable and without major flaws.

Retrospectively, this was a very stressful moment for the team. But when we look at the release changelog, we can see that we didn’t have any major issues that would have required more patch releases. As shown in the chart below, except for the 1.28.x releases with log4j updates, the number of patch releases remains very low and the release cadence remains stable:

Migrating the OpenTelemetry Java agent: what changed at scale

As part of Elastic’s contributions to OTel, we identified invokedynamic as something that could benefit OTel Java instrumentation.

This project includes more contributors and maintainers than the Elastic APM Java Agent did, so it’s naturally more challenging to coordinate and to convince others that it's worth the effort to migrate instrumentation for more than 100 supported technologies. Also, the OTel Java instrumentation is distributed and used by all observability vendors using OTel, so the impact of any regression is much larger.

The discussion started in July 2023, and the #8999 issue was opened to track the plan and progress of the migration. In the following weeks, my colleague Jonas Kunz wrote the initial proof-of-concept implementation and did most of the low-level work to support invokedynamic instrumentation and classloader isolation in the OTel Java instrumentation.

We faced some key challenges during this migration, some of which were similar to those with Elastic APM:

Migrating more than 100 instrumentation modules

The first is to handle a migration over a broad surface. More than 100 instrumentation modules needed to be reworked, as previously it wasn’t possible without a gradual migration and testing path.

We used multiple strategies and techniques to make this possible, including:

• Adding a runtime toggle to opt in to using invokedynamic instrumentation. This allowed local testing as well as continuous testing in continuous integration (CI) for months without impacting existing users.

• Transforming instrumentation code with Byte Buddy; for example, by setting inline = false on the @Advice annotation or making it compatible with both instrumentation modes through bytecode modification, doing all of that at runtime.

• Using large language models (LLMs) to help with mechanical code transformation in the final phases of the project.

All of this allowed testing and modifying the code over months with maximum safety. The main bottleneck remains testing and reviewing the changes.

Why classloader instrumentation stays inline

On a more technical level, some instrumentation modules are incompatible with invokedynamic and need to be kept inline; for example, with instrumentation of classloaders. The process of resolving the invokedynamic call site triggers classloading and thus can lead to a recursive resolution, which isn’t supported.

JPMS access when instrumentation moves to another module

Another interesting challenge is the compatibility with the Java Platform Module System (JPMS), introduced in Java 9. With inlined instrumentation, the instrumentation code inherits the same access as the instrumented code. When the instrumentation is instead delegated to another class in a different module and classloader, we need to grant access. Fortunately, fixing this was easy since we had already solved this in the Elastic APM Java Agent.

Injecting proxy classes into the application classloader

With a few instrumentation modules, we have to inject classes directly into the target classloader, when we need to make a class visible to the application code; for example, with the Spring framework. When the instrumentation code is no longer in the same classloader as the application code, we can't easily share a class between the two, because there’s no common classloader to load it from. eWe solved this by introducing proxy classes that are injected into the application classloader but that delegate to the instrumentation classloader, similar to what we have with Java Development Kit (JDK) proxies.

Instrumenting libraries compiled before Java 7

Last, but not least, my favorite challenge is making it work with Java versions prior to 7. Some libraries and frameworks have been compiled with older Java versions, long before the invokedynamic instruction was introduced. This is quite common in libraries that haven't been released in a long time or aren't maintained anymore. When the JVM validates such bytecode before loading it, it rejects any instruction that doesn’t match the version, even if the JVM supports it. You can solve this by upgrading the version of the application's bytecode before instrumenting it; in other words, we upgrade the bytecode to make it look like it was compiled with Java 7. 

Fortunately, the only tricky and costly operation is to recompute the stack map frames, since newer JVMs expect to know the types of everything stored on the stack. This technique is still used in the Elastic APM Java Agent, but OTel Java instrumentation has since replaced it with a different approach; when a class version doesn't support invokedynamic, it generates a helper class containing the invokedynamic instruction and invokes it via invokestatic, avoiding the need to bump the class version or recompute stack map frames at all (PR #15258).

What’s next for invokedynamic in the OpenTelemetry Java agent

OTel Java instrumentation is getting closer to the 3.0 release, in which invokedynamic instrumentation will become enabled by default for all supported modules. There are always aspects that can be improved, but the bulk of the work is done and we can consider this migration complete.

For example, in the current implementation, there are still parts of the OTel API included in the agent that are being shaded, meaning that they’re renamed at build time to prevent conflicts with another copy of the same library in the application classpath. Thanks to classloader isolation, we should be able to remove this for invokedynamic instrumentation like we did for the Elastic APM Java Agent, but doing so while preserving compatibility with inline instrumentation isn’t trivial. As usual, contributions are welcome, and I’d be happy to help anyone trying to solve this.

On a personal level, few projects that I've worked on combined this much technical depth with that amount of coordination, including classloader isolation, JPMS access rules, bytecode version tricks, and dozens of project participants, spread across two separate migrations that were three years apart. I’d like to thank the OTel maintainers and contributors, along with my colleagues at Elastic. None of this work would have been possible without them.

Kibana Universal Alerting will be generally available and on by default in Elastic Cloud Serverless on October 20, 2026, and in Elastic Stack 9.6 for Elastic Cloud Hosted and self-managed deployments. Already have alerts in Kibana? The new system runs alongside your existing Classic Alerting system, so you can adopt what you like at your own pace.

Kibana Classic Alerting continues to handle common scenarios well. Pick a rule type, fill in its parameters, and attach actions as you always have. Classic Alerting is as simple and reliable as ever. Universal Alerting goes further, enabling you to express any ES|QL query as an alerting rule, control flapping with alert and recovery delay, use custom recovery queries, shape alert data to fit your needs, and schedule actions that respond and group alerts across your rules.

When Universal Alerting fits

It's a good fit when one or more of these is true:

1. Your alert scenarios go beyond what Classic Alerting can offer, e.g., custom severities, workflow-powered actions, alert data control, full control over recovery, and more.
2. You want the flexibility of building rules using simplified query builders and later "dropping down" to edit and expand the generated ES|QL.
3. Your alerts come from more than one system, and you want to triage and analyze them in one view.
4. Fine-grained notification control matters, e.g., who is notified and how often, cross-rule grouping and processing, and more.
5. Your team wants all the flexibility offered by Discover, ES|QL, and Workflows. Universal Alerting integrates perfectly with and depends on all of these foundational Elastic features.

If a Classic rule type already covers your scenario, keep using it. We compare the two systems side by side later in this post.

How Universal Alerting works: From ES|QL query rules to alert data

A Universal Alerting rule runs an ES|QL query on a schedule. Each result is written as a rule event to a dedicated data stream, .rule-events. Four core ideas define the system:

• Rules are ES|QL queries, so there are no rule types to pick from. Each row the query returns, identified by its group, becomes a rule event, and the columns it returns are stored with the event. The query defines the shape of the alert.
• Alerts are data. Rule events live in a regular data stream with a documented schema, whether they came from Kibana alerting rules or an external tool. They can be searched in Discover, charted on dashboards, and used as the input for other rules.
• An alert follows one ongoing problem through its lifecycle (pending, active, recovering, inactive) and groups all of its underlying rule events as the alert's history. A rule doesn't trigger a new alert on every run, nor does it update an existing alert document. Rule events can be captured without triggering alerts. These are simply collected as data.
• Action policies live outside rules. They match on alert data, group and throttle them, and send them to workflows. Changing who gets notified doesn't mean editing an action per rule.

Each of these is a separate component, and each one offers flexibility and controls on its own.

The SRE workflow: Author and shape one rule

Say you run production hosts across several availability zones. The goal is to alert when a host stays hot, with the severity level and availability zone the on-call engineer needs. A brief spike shouldn't count. The host examples use metrics from the Elastic Agent System integration (system.* fields), and the pod example later uses the Kubernetes integration (kubernetes.* fields). Swap in your own fields as needed.

Author: One ES|QL rule, written the way you already query

This rule computes average CPU per host, assigns a severity and returns only the hosts worth a look:

/*
  Average CPU per host, with a severity and a minimum-sample guard.
*/
FROM metrics-*
| WHERE system.cpu.total.norm.pct IS NOT NULL
| STATS
    avg_cpu_pct = ROUND(AVG(system.cpu.total.norm.pct) * 100, 0),
    samples = COUNT(*)
  BY host.name, cloud.availability_zone
    /* A column named severity sets the alert's built-in severity field
       (info, low, medium, high or critical). Hosts below every threshold get
       no severity, and the WHERE below drops them. The other returned
       columns, such as avg_cpu_pct and the zone, are stored on the alert and
       available to action policies and dashboards later. */
| EVAL severity = CASE(avg_cpu_pct > 95, "critical", avg_cpu_pct > 90, "high", avg_cpu_pct > 85, "medium")
| WHERE samples >= 10 AND severity IS NOT NULL
    /* A minimum-sample guard skips hosts that just came up and have only
       reported a few data points. */
| KEEP host.name, cloud.availability_zone, avg_cpu_pct, severity, samples

Shape: Decide how the alert behaves over time

The query decides what matches. The rule definition decides what happens as matches come and go. Here's the same rule in YAML, split into a base query, a breach condition, and a recovery condition:

kind: alert

metadata:
  name: host-cpu-saturation
  routing_tags: [infra]                 # links this rule to action policies (lever 5)

schedule:
  every: 1m
  lookback: 5m

query:
  base: |
    FROM metrics-*
    | WHERE system.cpu.total.norm.pct IS NOT NULL
    | STATS
        avg_cpu_pct = ROUND(AVG(system.cpu.total.norm.pct) * 100, 0),
        samples = COUNT(*)
      BY host.name, cloud.availability_zone
    | EVAL severity = CASE(avg_cpu_pct > 95, "critical", avg_cpu_pct > 90, "high", avg_cpu_pct > 85, "medium")
  breach:
    segment: WHERE samples >= 10 AND severity IS NOT NULL

recovery:
  strategy: condition
  segment: WHERE avg_cpu_pct < 70       # recover well below the breach threshold

no_data:
  strategy: keep_last                   # a host that is rebooting keeps its last known status

state_transition:
  pending:
    count: 2                            # alert delay: the alert goes active on the third consecutive breach
  recovering:
    count: 2                            # recovery delay: the alert goes inactive on the third consecutive recovery

grouping:
  fields: [host.name, cloud.availability_zone]   # one alert per host

Recovery, state transitions, grouping and no-data handling are each tunable, and levers 1 to 4 below cover them. Levers 5 and 6 pick up once the alert is active: who gets notified, and how responders triage it.

Six levers for fine-tuning your alerts

Universal Alerting has a control at each stage of the pipeline, listed here from earliest to latest. Use only the ones a scenario calls for.

1. The query: Decide exactly what counts

ES|QL has everything needed to define a condition precisely. WHERE sets the scope, a minimum-sample guard ignores hosts that just came up, CASE assigns severities, and SORT with LIMIT restricts alerts to the worst offenders.

/*
  Alert on only the three hottest pods, not every pod above the line.
*/
FROM metrics-*
| WHERE kubernetes.pod.cpu.usage.node.pct IS NOT NULL
| STATS avg_cpu_pct = ROUND(AVG(kubernetes.pod.cpu.usage.node.pct) * 100, 1) BY kubernetes.pod.name
| SORT avg_cpu_pct DESC
| KEEP kubernetes.pod.name, avg_cpu_pct
| LIMIT 3
    /* LIMIT caps the number of alerts a single run can produce.
       Use it when "the worst few" is the question you care about. */

LIMIT has one side effect to plan for. A pod that drops out of the top three disappears from the results. With no_breach recovery, that counts as recovered, and the alert can activate again if the pod climbs back. If that matters for your rule, pair LIMIT with a recovery delay (lever 3).

2. Grouping: Choose the unit of an alert

STATS ... BY defines what a single alert is. Group by host.name for one alert per host, or by service.name for one per service. Pick the unit someone can act on: a grouping that's too wide hides problems, and one that's too narrow multiplies them. If a grouping field is missing from some documents, those rows fall into a single null group and share one alert, while the other groups keep their own alerts. Handle missing values explicitly in the query if that matters.

Alerting rules return at most 1,000 rows unless you set a LIMIT. If a single run could produce more groups than that, add an explicit LIMIT.

3. Alert delay and recovery delay: Define how an alert starts and ends

Thirty seconds over a threshold isn't always an incident. An instance of recovery isn't always indicative of a real recovery. This kind of flapping is one of the biggest sources of alert noise, and two delay configurations handle it without touching the query.

The alert delay (state_transition.pending) is the number of evaluations an alert spends in the pending state before it goes active. With count: 2, the alert goes active on the third consecutive breach, so a short spike stays pending and never becomes an alert.

The recovery delay (state_transition.recovering) does the same for the recovering state before the alert goes inactive. A signal that bounces around the threshold stays one alert. Each phase accepts a count, a timeframe or both, plus an operator (and or or) that says whether both must be met or just one.

Recovery is a separate setting, and every alert rule must set one (no_breach by default):

• no_breach recovers an alert when its group stops appearing in the breach results.
• condition recovers it when a different recovery condition (as opposed to alert condition) on the same base query is met, as in the rule above.
• query recovers it when a separate ES|QL query returns the alert's group. Recovery is per group, so each alert recovers only when the query returns its group.
• manual never recovers on its own. The alert stays active until someone manually resolves it.

A common pattern is to recover well below the breach threshold; for example, breach at 90% CPU and recover under 67%. That way, an alert doesn't close the moment the metric dips under the line. And because query recovery can run its own ES|QL query, it can check different values than the breach does. The example below requires both CPU and load average to be back to normal before the alert closes:

# A recovery query that looks at more than the breach query does.
recovery:
  strategy: query
  query: |
    FROM metrics-*
    | STATS
        avg_cpu_pct = ROUND(AVG(system.cpu.total.norm.pct) * 100, 0),
        load_avg_5m = AVG(system.load.5)
      BY host.name, cloud.availability_zone
    | WHERE avg_cpu_pct < 67 AND load_avg_5m < 4
    | KEEP host.name, cloud.availability_zone

4. No-data handling: Decide what silence means

When a group with an active alert stops showing up in the breach results, there are two very different explanations. Either the problem cleared up or the data source stopped reporting, and the alert condition alone can't tell them apart. No-data handling lets you decide, per rule, what happens in that moment, so a broken metrics agent doesn't quietly recover a real alert.

Set it with no_data.strategy, or with the no-data setting in the rule builder:

| Option | no_data.strategy | What happens | | --- | --- | --- | | Keep last status | keep_last | Holds the last known state. An active alert stays active, and a recovering alert stays recovering. A status:no_data rule event is written to the data stream.| | Recover | resolve | Closes the alert the first time no data is found, moving it straight to inactive without waiting for the recovery delay. A status:no_data rule event is written to the data stream.| | Do nothing | ignore | Skips the no-data check. With no_breach recovery, a gap in data looks like a recovery. With condition or query recovery, no event is written and the alert stays active. |

The right choice depends on what silence means for the rule. For infrastructure CPU, use Keep last status: if the agent stops sending data, an active CPU breach stays active until data comes back. For a rule that triggers an alert when a job backlog grows too large, Recover makes sense. Once the queue is empty, the query has nothing to return and the alert closes.

5. Action policies: Route, group and throttle

Rules don't carry their own actions. Instead, an action policy selects alerts, bundles them into a payload, controls how often they repeat and dispatches them to a workflow. One action policy can cover alerts from many rules, so changing who gets paged for a hundred rules takes a single edit.

Policy scope selects the alerts an action policy handles, in two parts that can be combined. Routing tags limit the action policy to alerts from rules that carry at least one of those tags. Under Advanced matching, a KQL expression filters on alert fields such as severity, group_hash or data.*. When both are set, an alert has to pass both.

Routing tags connect an action policy to a rule. Add them in the Routing tags field on the Actions step of the rule form, and select the same tags in the action policy form. From then on, the action policy applies to every rule that carries one of those tags, including rules created later. The rule form's Applied policies panel lists every action policy that matches the rule, so the connection is easy to verify. To target a single rule, give it a routing tag that no other rule uses. Attaching or detaching a rule is then a matter of editing its routing tags, and the action policy itself doesn't change.

Routing tags aren't the same as a rule's regular tags. Regular tags group and filter rules in the rule list. They have no effect on notifications, so adding a regular tag to a rule never attaches an action policy to it.

Two more settings shape the notifications. Notify per controls bundling: one notification per alert, one per group of alerts across rules that share a field value such as the availability zone, or a single digest of everything the action policy matches. Frequency limits repeats, for example to at most once every 30 minutes.

Here are two action policies for the CPU rule, written as the settings chosen in the UI. Both are tied to the rule through its infra routing tag:

# Page the on-call engineer only for critical hosts, one page per availability zone.
Name:           infra-oncall-pager
Policy scope:   Routing tags: infra
                Advanced matching (KQL): severity : "critical"
Notify per:     Group, by data.cloud.availability_zone
Frequency:      At most once every 30 minutes
Destination:    pagerduty-infra workflow

# Send the lower severities to the infrastructure channel, batched per zone, at most hourly.
Name:           infra-team-digest
Policy scope:   Routing tags: infra
                Advanced matching (KQL): severity : "high" or severity : "medium"
Notify per:     Group, by data.cloud.availability_zone
Frequency:      At most once every 1 hour
Destination:    slack-infra-team workflow

An alert can match several action policies, and each one applies its own grouping and frequency. If a whole zone runs hot, the pager gets one grouped page rather than one per host, and the team channel gets an hourly summary of the high and medium hosts.

Action policies also work across rules. Give your CPU, memory and disk rules the same routing tag, select that tag in an action policy and set Notify per to Group by data.cloud.availability_zone. A zone where all three rules fire then sends one page instead of three. Switch Notify per to Digest, and the action policy sends a single notification for everything it matches. To alert on the pattern itself, see rules on top of your alerts. Expand commentComment on line R230Resolved Each destination in these examples is a workflow that calls your connector, such as Slack or PagerDuty. To learn how to build and enable one, see the Workflows documentation.

6. Triage controls and maintenance windows: Quiet things on purpose

Once an alert is active, a responder can acknowledge it, snooze the series for a set period or resolve it. Suppressed alerts stay out of notification channels but remain in the data. Nothing is deleted, and everything stays auditable. During a planned rollout, for example, snoozing the series keeps it out of notification channels until the snooze expires. Maintenance windows do the same for scheduled work and apply to Universal Alerting and Classic rules alike.

One place for all your alerts: Classic, Universal, and even external

Few teams have a single source of alerts. Some come from Classic Alerting rules, some from Universal Alerting rules, and some from other monitoring tools. Our new alerts views are built so responders can work from a single queue.

Alerts from other tools, as data

Alerts from external monitoring tools land in the same data stream as the alerts Elastic generates, and are searched, routed and triaged the same way. A workflow writes each one with the alerting.create_alert step, and the source field records where it came from. For the on-call engineer, that's one place to check instead of a tab per tool. External alert ingestion is an experimental feature, so expect it to change and improve as we learn how it's used.

Universal vs. Classic feature comparison

The two systems share a goal but make different tradeoffs. Classic Alerting favors guided setup for known scenarios, while Universal Alerting favors flexibility and control.

| Feature | Classic | Universal | | --- | :---: | :---: | | Prebuilt rule types with guided parameters | ✓ | Partial, more to come | | Rules written as ES|QL queries | ✓ | ✓ | | Basic threshold rules | ✓ | ✓ | | Anomaly detection and APM anomaly rules | ✓ | Coming soon | | Machine learning job health and transform health rules | ✓ | Coming soon | | SLO burn rate rules | ✓ | Coming soon | | Synthetics monitor status rules | ✓ | Coming soon | | Alert fields defined by the rule's query | | ✓ | | Alert event historical data, per run | | ✓ | | Alert delay (consecutive breaches before an alert goes active) | ✓ | ✓ | | Recovery delay (consecutive recoveries before an alert auto-recovers) | | ✓ | | Custom recovery condition or recovery query | | ✓ | | Actions attached to each rule, sent directly to connectors | ✓ | | | Reusable action policies shared across many rules, with grouping and throttling | | ✓ | | Import alerts from external tools (experimental) | | ✓ | | Triage and analysis in the combined Alerts page | ✓ | ✓ | | Maintenance windows | ✓ | ✓ |

Universal and Classic alerts appear together on the combined Alerts page, so responders triage everything in one view. Classic Alerting keeps running exactly as it does today. Maintenance windows, connectors, and workflows are shared across both systems.

New alerting capabilities are actively being built in Universal Alerting. Classic rules remain a good fit for the standard scenarios they were designed for, and you can adopt Universal Alerting at your own pace, one rule at a time.

What Universal Alerting doesn't do (yet)

Universal Alerting doesn't match Classic Alerting feature for feature yet. We'd rather you hear about the gaps now than discover them mid-incident, so here's what to know before moving a rule over.

• Not every Classic rule type has a Universal equivalent yet. Universal rules are ES|QL queries, written by hand or generated by the rule builder. Threshold-style rules, such as custom, metric, log and index threshold rules, map well to the threshold rule builder, and many other conditions can be written directly in ES|QL. Rule types that rely on results computed elsewhere in the stack, such as anomaly detection, don't have a Universal counterpart yet, so those rules stay on Classic for now. The comparison table above lists them, and the rule library on our roadmap will add ready-made starting points for common patterns.
• SLO and synthetics rules need extra work. Their conditions can be written in ES|QL against the SLO and synthetics data, but the multi-window burn rate logic and per-monitor settings of the Classic rules have to be rebuilt by hand. Keep those rules on Classic unless you need the extra control.
• Classic rules are not automatically migrated to the Universal system. They keep running unchanged, and moving one to Universal Alerting means rebuilding its condition as an ES|QL rule.
• Universal Alerting actions depend on workflows. A Classic rule sends its actions straight to connectors. A Universal rule has no actions of its own: action policies hand alerts to workflows, which are the only destination type, and the workflow then calls Slack, PagerDuty or any other connector. Before a Universal rule can notify anyone, at least one workflow must exist and be enabled.
• Workflows require an Enterprise license on Elastic Cloud Hosted and self-managed deployments. Without this license, Universal rules still run and alerts are still written, but action policies cannot deliver to any destination, so no notifications go out. Existing action policies can still be viewed, disabled, snoozed and deleted.
• Universal alerts don't provide flapping state configuration. Alert and recovery delays and custom recovery (lever 3) are the way to keep a noisy alert from firing too often.
• Field names may differ between the two alert types. A query on a Universal alert field such as group_hash won't match any Classic alerts. Our unified alerts table mostly hides this distinction, but use shared filters (status, severity, rule, tags) in your own combined queries, or query each system separately.

Two delivery details apply to every action policy. Notifications are delivered at least once, so in rare cases, such as a restart between dispatch and recording, the same notification can arrive twice. Make workflows idempotent. Notifications are also dispatched on a short polling interval rather than instantly, so don't depend on sub-second paging latency.

Understand your alerting with ES|QL

An alert and its history are documents in a data stream, so the language that defines a rule can also answer questions about your alerting. From an alert, jump into Discover and pivot to that host's metrics and logs without a separate tool. Both examples below query the .rule-events data stream.

/*
  The ten rules that activated the most alerts in the last 7 days.
  rule.id is the rule's saved object ID, not its name.
*/
FROM .rule-events
| STATS alerts = COUNT_DISTINCT(alert.id) BY rule.id
| SORT alerts DESC
| LIMIT 10
    /* Rules at the top are good candidates for tuning. Adjust a threshold,
       raise the alert delay or change the grouping, then re-run this. */

/*
  Alert volume per day for one rule, to confirm a change had the effect you wanted.
  rule.id is the rule's saved object ID, not its name.
*/
FROM .rule-events
| WHERE rule.id == "<your-rule-id>" AND type == "alert"
| STATS alerts = COUNT_DISTINCT(alert.id) BY day = BUCKET(@timestamp, 1 day)
| SORT day ASC

Rules on top of your alerts

Since alerts are data, rules can run over them too. A rule on alert data is an ordinary ES|QL rule that reads .rule-events instead of logs and metrics. That lets it catch a pattern no single alert shows, such as three or more rules firing in the same availability zone. Alerts from Universal rules and from external tools work the same way here, because both land in the same data stream with the same fields. The alerts such a rule produces get the same lifecycle, action policies and triage controls as any other alert.

Signal rules are the building blocks for this. A rule with kind: signal (labeled Events in the UI) records breach and recovery events as data, without triggering alerts or notifying anyone. Use signal rules for conditions that matter mostly in combination, like high CPU, low free disk and elevated I/O wait on the same host. Then write an alert rule that fires when several of those signals line up.

What we're working on next

Try upcoming features per space

New and upcoming Kibana Universal Alerting features will ship behind an experimental feature flag, so teams can try them before they're enabled for everyone. The flag is the Universal Alerting: Experimental Features setting (alerting:v2:experimentalFeatures). It's off by default and set per Kibana space, so you can turn it on in a test or staging space and leave production spaces as they are. Changing it requires a page reload.

The flag gates every upcoming Universal Alerting feature. Everything that ships at GA works without it.

What's coming

While we aren't committing to any specific dates, this is where we're focused after GA:

• Sequence rule builder GA: A guided way to build rules that detect events happening in a particular order, without writing the whole query by hand (already available via "experimental features").
• More rule builders: Provide more guided ways to build rule queries. (SLO, synthetics, anomaly detection, etc.)
• A rule library of ready-made rules to start from and adapt, so a new rule doesn't begin with a blank query.
• Alert deletion, for alerts that are no longer needed.
• External alert import, generally available, with simpler setup and management.
• Bi-directional alert syncing with external systems, so an action taken in one system is reflected in the other.
• Further "no data" features.

If one of these matters more to you than the rest, tell us. It helps us decide what to build first.

Get started

Universal Alerting is generally available on Elastic Cloud Serverless, and on Elastic Cloud Hosted and self-managed deployments starting with Elastic Stack 9.6. On Elastic Cloud Hosted and self-managed deployments, workflow-based action policies require an Enterprise license, which includes 10,000 workflow executions per month (see subscriptions). On Serverless, Elastic manages the alerting system.

Give your team access

Access is controlled with Kibana feature privileges, set per role in Stack Management > Roles. Universal Alerting adds four, listed under Universal Alerting in the role editor, each with its own All or Read level:

| Privilege | Read | All | | --- | --- | --- | | Rules | View rules | Create, edit and delete rules | | Alerts | View alerts | Triage alerts | | Action Policies | View action policies | Create and edit action policies | | Execution history | View execution history | Same as Read |

Most responders need Alerts: All and Rules: Read. Rule authors need Rules: All, plus Action Policies: All if they also manage notifications. Notifications run through workflows, so anyone who builds or edits them also needs All for Analytics > Workflows. See Set up Workflows for the full list of Workflows privileges.

Privileges control what a user can manage, not what data a rule can read. Each rule runs with the API key of the user who last saved it, and that key determines what data the rule can query. Save your rules as a user who can read the indices they query. For how rules, action policies and workflows are authorized, see the authorization documentation.

Create your first rule

Open the Universal rules tab, create your first rule and tell us what you find. If something comes up that we didn't cover here, we want to hear about it.

Don't have a deployment yet? Start an Elastic Cloud free trial.

Why OpenTelemetry needs declarative configuration

The OTel specification defines three ways to configure an SDK, and each makes a different tradeoff. They differ in their ability to express structure and the need for source access, along with whether changing them means a new build:

Mechanism

Expresses structure

Needs code access

Needs redeploy to change

Environment variables

No, flat key/value only

No

No

Programmatic

Yes

Yes

Yes

Declarative file

Yes

No*

No

* Needs a package reference and one time registration to opt-in to declarative config. Thereafter, the config file can change and is applied at application launch.

Environment variables are the mechanism that most people encounter first. They’re portable, available everywhere and they ask nothing of your application code. Setting OTEL_SERVICE_NAME or OTEL_EXPORTER_OTLP_ENDPOINT works nicely because those values are single strings. Where they fall down is structural configuration, such as defining a sampler and its related options, or two exporters with different batching behavior. The flat key/value model has no way to express that shape, so you either go without the capability or smuggle structure into a string, and neither option ages well.

Programmatic configuration sits at the other end. It’s completely expressive, but it asks a lot in return, including access to the source and the ability to change it, in addition to a rebuild and redeploy every time somebody wants to adjust sampling. In many environments, that isn’t a reasonable ask. Production systems and legacy applications share the same constraint, as does anything instrumented automatically without touching code. The people who need to tune the telemetry are frequently not those who can ship a new build.

Declarative configuration fills that gap. It introduces a standardized, file-based representation of a configuration data model, defined in the Configuration SDK specification. This excerpt from the spec’s getting-started example shows the shape. The sampler and the exporter sit inside the tracer provider that they belong to, and a second exporter with its own batching settings would be one more entry under processors:

A document like this offers the expressiveness of programmatic configuration without requiring code changes. And, because the format is language-agnostic, the same shape of document configures a .NET service and a Java one. The model also covers instrumentation alongside SDK components, so its reach isn’t limited to the parts that you would otherwise wire up yourself.

The specification was declared stable in March 2026, and the announcement listed .NET among the implementations still in development. The .NET auto-instrumentation project has its own implementation for zero-code setups, but the SDK itself doesn’t support declarative configuration yet. Bringing it into the SDK makes it available to any application or distribution. Issue #7658 tracks that work. In this post, we cover the early stages of those efforts.

Why YAML doesn’t fit .NET’s IConfiguration

Reading a YAML file is the easy part. The hard part is that .NET already has a configuration system, and it’s a very good one.

Microsoft.Extensions.Configuration gives every modern .NET application an IConfiguration; that is, a single view composed from an ordered list of sources, typically appsettings.json, an environment-specific overlay, environment variables, command-line arguments, and perhaps a secrets store. When two sources define the same key, the later source wins. The options pattern then binds that view onto typed classes. .NET developers rely on this layering, and they expect any new configuration mechanism to join in.

Before writing any code, we analyzed how the SDK consumes configuration today. The first finding was encouraging. No SDK component reads environment variables directly. Every OTEL_* value is read through IConfiguration, which is why you can already set OTEL_SERVICE_NAME in appsettings.json and it simply works. Most option types are built from IConfiguration through the SDK's options factory. So a declarative file could plausibly slot in as one more configuration source, and the existing SDK would read it unchanged.

Projecting a declarative file into IConfiguration would only get part of the way, for five reasons:

• YAML is hierarchical, with snake_case keys and nested components. The SDK's options are flat and keyed like environment variables (OTEL_BSP_SCHEDULE_DELAY). Every value in IConfiguration is a string, so value: 3 and value: "3", an integer and a string in the configuration schema, become indistinguishable. An absent property and one explicitly set to null mean different things in the schema, and ordinary configuration access collapses them.

• Today, a sampler, exporter, or propagator exists because someone wrote SetSampler(new TraceIdRatioBasedSampler(0.5)) or AddOtlpExporter(). Where the SDK does turn a string into a component, as with OTEL_TRACES_SAMPLER, it uses a handwritten switch statement that third parties cannot extend. Some spec-defined variables, such as OTEL_PROPAGATORS and the OTEL_*_EXPORTER family, aren’t implemented at all. A document that says "use a parent_based sampler wrapping trace_id_ratio_based" needs a registry that maps names to components, and the SDK has none.

• The SDK resolves options through IOptionsMonitor<T>, which looks like latent reload support but isn't. That choice was made to fix a dependency-injection lifetime problem. Nothing subscribes to change notifications, and many values are captured into long-lived state at construction. That matters for the longer-term goal of dynamic control, and it means that reload was explicitly out of scope for the first steps.

• The spec says that when OTEL_CONFIG_FILE is set, "all other environment variables besides those referenced in the configuration file for environment variable substitution MUST be ignored," because "there is no intuitive way to merge the flat environment variable scheme with the structured file configuration scheme in all cases." A .NET developer's instinct is the opposite: the file should be another layer. Because environment variables reach the SDK through IConfiguration in the same key space as appsettings.json, nothing when a value is read records where it came from.

• We also looked at prior art. The Java SDK is the reference implementation of the specification, and the .NET auto-instrumentation project already ships its own file-based configuration. There were practical questions, too, such as whether the work belonged in the SDK repository or started in contrib, and whether to take a dependency on a YAML library at all.

So the question was: How does a structured, typed, language-neutral document fit into a platform whose configuration is flat, string-typed, and layered, without breaking either model? The question was never How do we parse YAML?

A typed model behind IConfiguration

The analysis produced a broad plan: 24 proposed phases covering foundation work, new option types, a component registry, and reload support. That was too much to land in one go and too much to ask maintainers to agree to in the abstract. The next step was a deliberately minimal proof of concept (PoC) that the maintainers could react to.

The first cut took the obvious route. A custom IConfigurationProvider walked the YAML and emitted flat OTEL_* keys for the existing SDK to read. It worked, but emitting strings straight from the YAML collapsed absent, null, and present into two states, and the spec requires Parse to distinguish missing from null.

So the PoC was refined into two stages, and that shape is what PR #7413 shipped:

1. Parse into a typed model. The YAML is parsed into internal records that carry no YAML dependency, with every scalar wrapped in a three-state value. Only the parser touches the YAML library (YamlDotNet), so a future source, such as an Open Agent Management Protocol (OpAMP) push, could build the same model.

2. Project into IConfiguration. The model is projected into the flat OTEL_* keys that the SDK already understands, through a configuration provider. disabled: true becomes OTEL_SDK_DISABLED=true, and every existing read site picks it up with no changes to the core SDK.

The PR added a new package, OpenTelemetry.Configuration.Declarative, so applications never take on its YAML dependency without opting in. It supported a deliberately small slice of the schema: file_format, disabled, resource.attributes, and resource.attributes_list, plus the spec's environment-variable substitution. Wiring it up takes one call using UseDeclarativeConfiguration on the OpenTelemetryBuilder.

The following code adds the OTel SDK with declarative configuration and tracing enabled:

The following basic YAML configuration adds two resource attributes that apply to telemetry data exported from this service:

We called this overlay mode. The YAML source is appended after every source already registered, so the file takes precedence over environment variables and appsettings.json. Keys that the file doesn’t set fall through to whatever was there before, and sources added later still override the file, exactly as .NET developers expect.

Overlay mode was the right place to start. It needed no changes to the stable core SDK. The file behaved like any other configuration source, so its precedence didn’t need an explanation. Plus, the parser and the model could be reviewed on their own, without a component engine attached, as could  the substitution rules.

The most important decision in that PR, though, was making the typed model, rather than IConfiguration, the source of truth. Flat keys are one projection of that model, feeding the SDK's existing options pipeline. The design also anticipated a second projection: a schemaless view of each node, the spec's ConfigProperties, for building components. This was necessary because IConfiguration cannot represent absent versus null or polymorphic component selection, nor can it represent arbitrary nesting. Only the first existed in PR #7413, but the second was designed in from the start.

The PR was also candid about what overlay mode didn’t not do, such as:

• Resource attributes were string-only, because typed values cannot survive projection into the OTEL_RESOURCE_ATTRIBUTES string.

• Sections such as tracer_provider were logged and not applied.

• The file was read once at start-up.

• There was no strict mode. Standard IConfiguration ordering applied, so leftover OTEL_* variables still took effect for any setting that the file left out.

The design is now heading somewhere different with that last gap. Take a partially written file, for example, which is the normal case during a migration. It sets no sampler, but the deployment still carries OTEL_TRACES_SAMPLER=always_off from before. Under overlay mode, that leftover variable quietly applies, so two services with identical files can behave differently, which no other OTel SDK would do. The spec's answer is that anyone who needs to combine sources should customize the configuration model, not merge keys.

With the initial phases landed, the pragmatic choice is to align with the spec and make strict behavior the default: when a document is in use, it’s the source of OTel settings. That closes off the partial-file problem entirely. The .NET-style layering that developers value isn’t being ruled out. We’ll investigate later how .NET might support model customization, with explicit, opt-in merge rules. Either way, the typed model at the center of PR #7413 makes the change possible without starting again.

Getting the OTel configuration spec right

Configuration is the first thing that an SDK touches and the last thing that anyone suspects when telemetry goes missing, so even small deviations carry risk. Much of the first PR went on details that a casual reading of the spec skips over. Roughly two thirds of its 8,800 lines are tests, including fuzz tests for the parser and for substitution.

• Three states, everywhere. C# nullable annotations cannot distinguish "absent" from "present and null". Union types, which could model this directly, are coming in C# 15 with .NET 11, but they’re still in preview, and the package also has to build for netstandard2.0 and .NET Framework. So every model field is a three-state value, and every field is tested in all three states, not only the happy path.

• YAML 1.2, not YAML 1.1. The spec requires the YAML 1.2 core schema. YAML 1.1 habits include merge keys (<<: *defaults) and a looser set of Boolean spellings. The package ships its own YAML 1.2 scalar resolver rather than relying on library defaults, and it rejects merge keys outright. Quoting is significant: file_format: "1.1" is a string and is accepted, while file_format: 1.1 is a number and is rejected.

• Substitution happens at the right moment. Environment-variable substitution runs on the decoded scalar value, after YAML escapes are processed and before types are resolved. That matches the YAML processing model and the Java implementation, in addition to the official configuration validator. So value: ${PORT} may resolve to a number, while value: "${PORT}" is always a string, and substitution can never inject structure into a document.

• Silence is a bug. Anything that the package couldn’t honor, such as skipped resource attributes, unapplied sections, or repeated registrations, is reported through the SDK's self-diagnostics rather than quietly dropped. Someone debugging a missing attribute should be able to find out why without reading the source.

How this work is planned and reviewed upstream

This all started with a discussion in the original issue, as well as an analysis of the current configuration landscape within the SDK and the proposed issues. It didn’t start with code. Even now, the design continues to evolve. I track it in a local repository, assisted by AI agents. The repo holds a high-level plan and a decision record in which every significant choice is dated with its context and consequences. It also holds a specification for each phase, written before implementation starts. That level of detail is too much for general consumption, so the high-level phases are published in the public meta issue.

I've planned the work in small, deliverable units to reduce the review burden and to keep code merging. This approach depends on a close working relationship with Martin Costello, one of the OTel .NET maintainers, who has reviewed each phase as it has landed. His feedback has repeatedly improved the design. After each phase lands, I reevaluate the phases that follow, fold in what review has taught us, and adjust direction where needed, recording the change in the decision record so that the plan never quietly drifts.

What’s next for declarative configuration in OpenTelemetry .NET

The package hasn’t shipped a release yet, and PR #7413 is the foundation rather than the feature. The plan that came out of it sets out the next steps, some of which are already implemented, including:

• A library-neutral value model. The public ConfigProperties contract for reading any node of the document, distinguishing absent, null, present, and type-mismatched values.

• Access to the parsed document. Parse once, and keep the result. Let distributions read it before providers are built, including sections that the SDK doesn’t yet apply.

• Typed resource attributes. Integers, Booleans, and arrays reach the Resource with their declared types, rather than through a flat string.

• Strict mode by default. Honor the spec's requirement that other OTEL_* settings are ignored when a file is used, and investigate how model customization could bring back .NET-style layering.

• Components. A plugin provider abstraction and the built-in samplers, along with the wiring that lets the document itself build providers.

• Coverage, generation, and Parse/Create. Extend coverage to the rest of the schema, and generate the model from it. Publish the spec's Parse and Create operations.

Access to the parsed document is a large part of why Elastic is investing in this work. The Elastic Distribution of OpenTelemetry .NET (EDOT .NET) builds on the upstream SDK, and a distribution needs to read the user's configuration before the SDK builds its providers. Building that capability upstream, rather than inside EDOT, means that every distribution and application gets the same behavior.

Once the core phases are in place, the plan is to ship a public alpha, so users can start experimenting and give feedback on the path to a stable release.

If you want to follow along or share your feedback, the meta issue #7658 tracks the planned phases and current progress. The design is still open to change before the first release, and feedback from anyone who configures OTel in production, or builds a distribution on top of it, is exactly what shapes it.

Why small log files were skipped

Filestream needs a stable identifier for every file it reads so that it can track how far it has read from each file. In earlier versions, we used filesystem metadata such as inode numbers. This was not enough. For example, filesystems such as ext4 can reuse an inode number after a file is deleted, leading to a new file being mistaken for the old one. This is why we introduced the fingerprint identity and made it the default in 9.0.

Fingerprinting means files are identified by their content, not their metadata. The first 1024 bytes (by default) are hashed and used as the unique identity of the file. A registry keeps track of all IDs we encounter beyond the agent's process lifetime. Restart your machine, change the underlying device, host the files over the network, it doesn't matter; Elastic Agent will remember.

The major downside is that we can't hash a file's first 1024 bytes if it doesn't have this many bytes to begin with. Imagine a critical Kubernetes pod that panics on startup and its small log output is the reason the panic never reaches your Kibana dashboard.

The main way we've dealt with this until now is through configuration: instead of 1024 bytes, we could hash 512. Or 256. Or even less. However, each step towards reducing the threshold leaves less file content available to distinguish one file from another. Distinct files are therefore more likely to share the same fingerprint material, making the resulting file identity less stable. This is where the new growing fingerprint can shine instead.

How growing fingerprint splits file identity into two phases

With the introduction of the growing fingerprint mode, filestream splits a file's identity into two phases based on its size. While a file is smaller than the 1024-byte fingerprint threshold, it's in "growing" mode. Filestream identifies it with a hash of everything the file contains so far. When the file grows past the threshold, it gets promoted to a static fingerprint, the one it would have had pre-9.5.

What about file-identity collisions? Two files with identical contents do share a growing fingerprint, but only for as long as they stay identical: the moment their contents diverge, so do their fingerprints.

Let's take a closer look inside the internals. Let's start with a simple 13-byte text file:

mkdir -p /tmp/demo
echo 'Hello world!' > /tmp/demo/file.txt

and start Filebeat, the component inside Elastic Agent that collects log files, with a minimal configuration:

cat >/tmp/demo/filebeat.yml <<YAML
path.home: /tmp/demo

filebeat.inputs:
  - type: filestream
    id: growing-demo
    paths: [ "/tmp/demo/file.txt" ]

output.file:
  path: /tmp/demo/out
logging.level: debug
YAML

filebeat -c /tmp/demo/filebeat.yml

Phase 1: the growing fingerprint of a 13-byte file

After a few moments, filestream will pick up our new file. Digging through the debug logs (vastly reduced here for simplicity), we see:

{"log.level":"debug","log.logger":"input.filestream.prospector","message":"A new file /tmp/demo/file.txt has been found"}

We can also directly check the registry write-ahead log (WAL) filestream writes to data/registry/filebeat/log.json:

{
  "k": "filestream::growing-demo::fingerprint::f04f9d5ad8c21a115cd207488c6aaadeb4d7db581ea675f48c7f5d23e038c805",
  "v": {
    "cursor": {
      "eof": false,
      "offset": 13
    },
    "meta": {
      "fingerprint_len": 13,
      "identifier_name": "fingerprint",
      "source": "/tmp/demo/file.txt"
    }
  }
}

The k field is the registry key filestream uses to identify this file state. Its final segment is the fingerprint hash, which we can reproduce from the file contents:

$ xxd -p /tmp/demo/file.txt | tr -d '\n' | sha256sum
f04f9d5ad8c21a115cd207488c6aaadeb4d7db581ea675f48c7f5d23e038c805  -

Still in phase 1: the fingerprint changes as the file grows

Let's write more lines to the file:

for i in {1..70}; do echo "Hello $i" >> /tmp/demo/file.txt; done

Filestream detects the write while the file remains below the threshold:

{"log.logger":"input.filestream.prospector","message":"File /tmp/demo/file.txt has been updated","operation":"write"}

In the registry, the larger file now has a new fingerprint and a 634-byte fingerprint_len:

{
  "k": "filestream::growing-demo::fingerprint::cb8e361971e7ba91e343d1b5bd8e6560be0bdadbbed2b2f89a6c50099e972d63",
  "v": {
    "cursor": {
      "eof": false,
      "offset": 634
    },
    "meta": {
      "fingerprint_len": 634,
      "identifier_name": "fingerprint",
      "source": "/tmp/demo/file.txt"
    }
  }
}

The updated key ends in the SHA-256 hash of the hexadecimal fingerprint material:

$ xxd -p /tmp/demo/file.txt | tr -d '\n' | sha256sum
cb8e361971e7ba91e343d1b5bd8e6560be0bdadbbed2b2f89a6c50099e972d63  -

Phase 2: promotion to a static 1024-byte fingerprint

Let's now grow the file beyond the 1024-byte threshold:

for i in {1..500}; do echo "More messages $i" >> /tmp/demo/file.txt; done

Once the file crosses the threshold, filestream reports another write:

{"log.logger":"input.filestream.prospector","message":"File /tmp/demo/file.txt has been updated","operation":"write"}

The registry key changes one final time as the growing fingerprint is promoted to its static form:

{
  "k": "filestream::growing-demo::fingerprint::c8e252d76d621a02161f92e7a59d1617e3079db666cefd6cfcab106329082d1e",
  "v": {
    "cursor": {
      "eof": false,
      "offset": 9526
    },
    "meta": {
      "identifier_name": "fingerprint",
      "source": "/tmp/demo/file.txt"
    }
  }
}

The promoted fingerprint hashes the first 1024 bytes. It is exactly what a pre-9.5 Filebeat would have computed for this file:

$ head -c 1024 /tmp/demo/file.txt | sha256sum
c8e252d76d621a02161f92e7a59d1617e3079db666cefd6cfcab106329082d1e

How filestream matches a growing fingerprint to an existing file

During each scan, filestream resolves a file's identity in this order:

1. Look for an exact registry key matching the file's current fingerprint. This part has not changed.
2. If no key is found, search the growing entries for a shorter fingerprint, trying the same path first. A prefix match means filestream saw the same file earlier, when it contained fewer bytes.
3. Move the existing registry state to the new key, preserving the read offset.

You might have noticed from the registry entries above that we don't store the raw file contents. That was our initial approach, but it quickly exploded the size of the registry. Instead, we use the SHA-256 hash as the registry key and store fingerprint_len, which records the number of raw bytes the fingerprint represents. This reduces the storage required per entry by nearly 100×.

When filestream looks for a prefix match, it first groups the growing fingerprints by length. It then streams the scanned file's hexadecimal fingerprint material through SHA-256, snapshotting the digest at each group length. Filestream compares each snapshot only with stored hashes for the same length. A match means that the stored fingerprint is a prefix of the scanned file.

Growing fingerprint benchmarks: memory and CPU at 10,000 log files

So what does all this extra work cost? We benchmark Elastic Agent 9.5 ingesting thousands of above-threshold and sub-threshold files. To make the benchmark fair, we compare the growing fingerprint against the workaround of reducing the fingerprint size to 128 bytes. Otherwise, we would be comparing doing something against doing nothing and nothing would always win.

The benchmarks run on an r8id.2xlarge EC2 instance (8 vCPUs, Ubuntu 24.04) and report the median of five runs. Memory numbers are the Elastic Agent's anonymous memory as reported by the kernel via cgroups.

At 10,000 files, the two modes compare as follows.

| Metric | Condition | 128-byte static | Growing | Difference | |---|---|---|---|---| | Peak memory | Files above 1 KiB | 1,057 MiB | 1,099 MiB | +4% | | Peak memory | Files below 1 KiB | 941 MiB | 1,012 MiB | +8% | | Peak memory | Files crossing the threshold | 968 MiB | 1,016 MiB | +5% | | Steady-state memory | Files above 1 KiB | 291 MiB | 290 MiB | none | | Steady-state memory | Files below 1 KiB | 270 MiB | 322 MiB | +52 MiB | | CPU, one core | Files above 1 KiB, idle | 4.1% | 4.1% | none | | CPU, one core | Files below 1 KiB, idle | 3.9% | 4.9% | +1.0pp | | CPU, one core | Files crossing the threshold | 16.8% | 25.5% | +8.7pp | | Registry size at rest | Files above and below 1 KiB | 8.6 MB | 8.6 MB | none |

Memory use at 100, 1,000 and 10,000 log files

At 100 and 1,000 files, 128-byte static and growing fingerprints differ by only a few MiB. At 10,000 files the peaks separate by a few percent, but the only lasting difference is for sub-threshold files: above the threshold both modes stabilize within a MiB of each other.

The most demanding case is when the fingerprints are still changing. We created 10,000 sub-threshold files and appended 2,000 lines/s across them until every file crossed the threshold. That repeatedly exercises the prefix-lookup path and moves registry state to new keys. The extra memory this costs is released once the files are past the threshold and their harvesters close: both modes settle under 300 MiB, within a few MiB of each other:

CPU use with and without writes

The CPU impact is larger: every scan re-hashes a sub-threshold file's full contents instead of a fixed 128-byte prefix. That costs one extra percentage point of a core with no writes coming in, and almost nine under the changing-fingerprint workload above:

What changes in 9.6

We are already working on improving these numbers. On current main (future 9.6 release), the same benchmarks peak at less than half the memory, and the growing fingerprint's steady-state overhead, in both memory and CPU, is roughly halved.

Upgrading and rolling back

Growing mode ships enabled by default in 9.5 through file_identity.fingerprint.growing. Set it to false to restore the pre-9.5 behavior. There is no need for a migration on upgrade; existing registry entries keep working and files that were previously too small get picked up on the next scan.

Rolling back is also possible, but you go back to pre-9.5 behaviour: small files are ignored and when they cross the threshold, they get re-ingested.

What growing fingerprint means for your log files

Elastic Agent 9.5 ingests log files smaller than 1024 bytes by default, with no configuration and no waiting for the file to grow. Shared identities between identical small files resolve themselves as soon as their contents diverge, which a static fingerprint cannot do. The cost stays small: at realistic node scales the growing fingerprint is indistinguishable from the old workarounds, and at a 10,000-file workload it costs a few percent of memory and single-digit percent of one CPU core. Remember when upgrading that small files that were invisible before count as new, however long they have been sitting in your paths. If the trade doesn't work for you, file_identity.fingerprint.growing: false brings back the old behavior.

References

• How to choose file identity for filestream covers all identity options and their trade-offs.
• Growing fingerprint reference documents the setting and the rollback caveat.
• PR #50566 is the implementation and PR #51784 the registry optimization; PR #51863 and PR #51914 trimmed per-scan allocations and fingerprint retention in 9.5.2.
• Optimizations coming up in 9.6: PR #51278 parks idle harvesters instead of keeping one goroutine per open file, PR #51694 scans each file in a single pass, and PR #52476 normalizes published events in place. PR #52103, which replaces the multiline timeout goroutine with a read deadline, is not exercised by these benchmarks.

How to change OpenTelemetry Java agent sampling at runtime

Until now, with some proprietary exceptions, configuring that telemetry had to be done at startup of the telemetry-generating component (e.g., SDK, agent, or OpenTelemetry Collector). Telemetry Policy is the emerging standard that will allow you to reconfigure some telemetry configurations without needing to restart the telemetry-generating component.

For example, let's say you want to:

• Change the trace sampling percentage to 1% during some special peak periods (eg special sales days), but keep it at 5% at all other times.
• Change the log level (e.g., to debug level or back to info) temporarily.
• Stop some instrumentations that are active.

These are all straightforward requirements. But applying any of them until now has meant changing the configuration of the component and restarting it. For a Java application (and applications in most languages) that previously meant also restarting the application, as there was no way to change the agent configuration without restarting the JVM.

Why central management needs one policy format

Central management adds another challenge. A controlling system that requests a change such as "increase log level to debug" must know how to apply that change to every component it controls. The configuration change for a collector will not look like the change for a Java SDK, even though both are applying the same decision.

Telemetry Policy addresses both problems by separating:

1. what we want to happen
2. from how a particular SDK, agent, or collector makes it happen.

Using the Elastic APM UI to set the Java agent sampling rate 0.1 (10%)

Elastic APM Java agent log statements confirming that the sampling rate value of 0.1 was received and applied

What is a Telemetry Policy in OpenTelemetry?

A Telemetry Policy is an independent rule describing the behaviour a user wants.

For example:

Sample traces at 5%

A policy that says to sample traces at 5% does not say which sampler to install, where to add a processor, or how the telemetry pipeline is laid out. Those are implementation details specific to each component. The policy only states the intended result.

The draft specification defines a policy as a typed, standalone object. Each policy has an ID, a name, and one target signal such as traces, logs, or metrics. The target determines the matching fields and actions that can be used.

Here is an example of that 5% sampling policy:

{
  "id": "sample-spans-5-percent",
  "name": "Sample traces at 5%",
  "trace": {
    "match": [
      {
        "exists": true
      }
    ],
    "keep": {
      "percentage": 5.0
    }
  }
}

The policy says to match every trace and apply 5% sampling using the policy with ID sample-spans-5-percent. Each telemetry component that understands that ID will be able to apply the policy. Any telemetry component that doesn't understand the policy will simply ignore it.

Which settings can Telemetry Policy change at runtime?

Policies are not expected to cover every configuration, or even many configurations of each component. Telemetry Policies are not a general-purpose "reconfigure anything" tool. Policies are specific to the things that users would most like to change without restarting. For application agents, our experience at Elastic suggests there are only a dozen or so important policy types that most customers need to be able to change at runtime without restarting the application, including:

• Changing the trace sampling percentage.
• Turning individual instrumentations or all instrumentations off and on.
• Turning individual signal exporters (traces, metrics, logs, profiles) off and on.
• Changing logging level.
• Changing the configuration of a select few instrumentations.
• Such as changing which HTTP routes to ignore when creating traces (eg ignore /heartbeat).
• Or changing which methods to instrument.

How a Telemetry Policy reaches an SDK, agent or collector

The overall telemetry policy pipeline has certain components, and at minimum covers the sequence:

Message -> Provider -> Policy -> Policy aggregator -> Implementer

A concrete example of changing the trace sampling percentage directed by central management using the Open Agent Management Protocol (OpAMP) helps to understand this flow:

| Stage | What it does | In the 5% sampling example | | --- | --- | --- | | Message | Carries the requested change into the component | An OpAMP message specifying a trace sampling percentage of 5%, like the JSON above | | Provider | Reads policies from an input stream | The OpAMP provider reads the message from the OpAMP data flow | | Policy | The internal representation of the change | A trace-sampling-percentage-policy with a target value of 5%, which is how the ID sample-spans-5-percent is implemented in the component* | | Policy aggregator | Combines the policy with others already applied or pending, handling source priority and merges | A pass-through, because there is only one provider | | Implementer | Applies the change to the running component | Installs a new sampler set to 5% |

Source priority decides which policy wins when two sources disagree. A message from an OpAMP source has higher priority than one from an HTTP source.

The implementer installs a new sampler unless the sampler already running can have its sampling percentage changed at runtime, in which case it updates that one.

*Note that the policy type may become a required part of the message in future. There is currently no standard mapping from a policy ID to a policy type, so the type must either be included in the message or supplied through an external mapping.

Delivering policies over OpAMP, HTTP or a local file

Telemetry Policy does not define a new transport mechanism or protocol. Policies can be delivered through existing transports:

• OpAMP.
• HTTP.
• Local files.
• Custom sources.

The same trace sampling policy could be read from a local file while testing, supplied over HTTP, or sent from a central management system using OpAMP. OpAMP is not a requirement for Telemetry Policy; it is a supported provider and is likely to be a common transport for centrally managed systems.

Telemetry Policy status in OpenTelemetry

Telemetry Policy is currently an accepted developing project. The schema, policy types, and implementation details are still being developed before they will become accepted OpenTelemetry standards.

The first experimental Java SDK implementation

The first experimental Java SDK implementation starts with dynamically changing the trace sampling probability. This is deliberately narrow, providing a simple practical way to test the complete policy flow.

Telemetry Policy separates observability decisions from the internal configuration of the component applying them. This allows a platform team to define a policy once and apply it consistently in SDKs, agents, and collectors.

In the next blog, I'll walk through the first experimental implementation in the OpenTelemetry Java contrib repo, including how it receives policies through OpAMP and changes trace sampling in a running Java application without requiring a restart.

What is the Elastic Cloud Managed OTLP Endpoint?

Managed inputs are ingestion frontends that Elastic operates on your behalf. You point a shipper at an Elastic-owned endpoint, authenticate with an API key, and Elastic takes care of receiving, buffering, and indexing the data into your deployment. There is no gateway to size, patch, or monitor, and no backend credentials to distribute to edge agents.

The Managed OTLP Endpoint is the managed input for OpenTelemetry data. It accepts standard OTLP over HTTP and gRPC from any OpenTelemetry SDK or Collector distribution, buffers it durably, and indexes it into your Elasticsearch deployment using the OpenTelemetry data model. It is generally available on both Elastic Cloud Serverless and ECH. For an overview across logs, traces, and metrics, read Now GA: Managed OTLP Endpoint on Elastic Cloud Hosted.

For metrics, clients send OTLP/HTTP requests to the /v1/metrics signal path and authenticate with an Elasticsearch API key that has the event:write privilege for the apm application:

POST https://<managed-otlp-endpoint>/v1/metrics
Authorization: ApiKey <encoded-api-key>

The /v1/metrics client-facing path never changes. This post focuses on what happens after the request is accepted, during the last hop between the managed pipeline and Elasticsearch.

Two ways to ingest OpenTelemetry metrics: the bulk API and the native OTLP endpoint

Elasticsearch offers two ways to receive OpenTelemetry metrics.

The bulk API (/_bulk) is the general-purpose document ingestion API. To use it for OTLP metrics, something upstream must convert each OTLP data point into a JSON document that follows the OpenTelemetry mapping mode, group the documents by target data stream, serialize them as newline-delimited JSON, and send them in bulk requests. In the OpenTelemetry Collector ecosystem, this conversion is handled by the Elasticsearch exporter, and until recently, it was the only option.

The native OTLP endpoint (/_otlp/v1/metrics) accepts OTLP/HTTP Protobuf directly. Elasticsearch decodes the payload, builds the documents itself, and indexes them into time series data streams. Because Elasticsearch sees the original OTLP structure, it can perform these steps directly:

• It reads the aggregation temporality of every data point and stores it as a dimension
• It hashes the shared resource attributes once for all data points in a resource
• It maps exponential histograms to the exponential_histogram field type without an intermediate T-Digest conversion

The following diagram shows the two lanes.

The two paths differ in three ways that matter for users:

• Work distribution. On the bulk path, the managed pipeline converts and serializes documents, then Elasticsearch parses them again. On the native path, the pipeline forwards batched Protobuf and Elasticsearch builds documents once.
• Temporality. The Elasticsearch exporter used by the managed bulk path does not support cumulative histograms and drops those points. The native path preserves temporality and accepts both delta and cumulative histograms.
• Histogram fidelity. The bulk path converts every histogram to a T-Digest style representation. The native path stores exponential histograms natively and converts explicit-boundary histograms into the same exponential representation.

Do not confuse the two paths with the client-facing path. Clients always send to /v1/metrics on the Managed OTLP Endpoint. The service chooses between /_bulk and /_otlp/v1/metrics behind it.

Native OpenTelemetry metrics support by Elastic Stack version

Native OTLP metrics support in Elasticsearch arrived in several steps.

1. Elastic Stack 9.2 introduced the Elasticsearch OTLP/HTTP metrics endpoint as a technical preview.
2. Elastic Stack 9.3 added the exponential_histogram field type as a technical preview.
3. Elastic Stack 9.4 made exponential_histogram generally available and the default mapping for OTLP histograms received on the native endpoint.
4. Elastic Stack 9.5 added metric temporality support as a generally available feature. The native endpoint now stores the temporality of every data point, which is what makes cumulative histograms possible. The same release added OTLP/HTTP endpoints for logs and traces as a technical preview.
5. Elastic Stack 9.5.3 is the current minimum version for native OTLP metrics routing through the Managed OTLP Endpoint on ECH. The routing floor was raised from 9.2.0 because earlier versions can fail native OTLP requests when audit request-body logging processes Protobuf payloads. The 9.5.3 floor includes the required fixes.

The Elasticsearch Prometheus remote write endpoint arrived in Elastic Stack 9.4 and follows the same capability-gating model, but it is out of scope for this post.

Keep two things apart when reading version numbers in this post. The first four milestones are capabilities of Elasticsearch itself, available to anyone who calls /_otlp/v1/metrics directly. The fifth is a routing decision of the Managed OTLP Endpoint, which is stricter than "the endpoint exists" because the managed service has to work reliably for every tenant.

How Elastic Cloud Hosted routes OpenTelemetry metrics

On Serverless, Elastic operates and upgrades the backend, so the Managed OTLP Endpoint always uses the native path and users never see a version matrix.

ECH is different. Users pick and control their Elastic Stack version, and the Managed OTLP Endpoint supports deployments on any version from 9.0 onward. A single multi-tenant service therefore receives metrics for deployments that have no native endpoint at all, deployments that have one but predate the fixes the service relies on, and deployments that are fully ready. It cannot enable the native path unconditionally.

How the Managed OTLP Endpoint routes a metrics request

The following diagram shows what happens to a metrics request for an ECH deployment.

1. The endpoint authenticates the API key and resolves the target ECH deployment.
2. It looks up the deployment's Elasticsearch version from Elastic's own control plane, not from anything the client sends.
3. It derives a small set of capability flags for the target, such as "native OTLP metrics supported", and attaches them to the request as metadata.
4. The request is buffered durably together with its metadata.
5. At the indexing stage, the consumer reads the flags and hands the batch to one of two exporters: the OTLP/HTTP exporter, which forwards Protobuf to /_otlp/v1/metrics, or the Elasticsearch exporter, which converts the batch into documents and posts them to /_bulk.

Because the flags are derived inside Elastic's trust boundary, a client cannot opt a deployment into a path its version does not support. Elastic can also pin specific deployments to the bulk API for compatibility reasons, so the version rule below describes the normal case rather than a guarantee.

Which ingestion path does my deployment use?

| ECH Elastic Stack version | Path the Managed OTLP Endpoint normally uses | | ------------------------- | ------------------------------------------- | | 9.0 to 9.5.2 | Bulk API (/_bulk) | | 9.5.3 or later | Native OTLP endpoint (/_otlp/v1/metrics) |

No client-side change is required. If you already send OTLP metrics to the Managed OTLP Endpoint on ECH, your traffic moves to the native path when you upgrade the deployment to 9.5.3 or later.

What changes when OpenTelemetry metrics move to the native path

Client configuration and data stream routing remain unchanged. Histogram representation and temporality handling change: the native path uses exponential_histogram by default and records temporality as a dimension. Review dashboards, alerts, and queries that depend on the previous histogram representation or aggregate cumulative snapshots. Queries spanning historical histogram fields and newer exponential_histogram fields may need an explicit ::exponential_histogram cast, as described in querying historical data alongside new data.

The distribution of work across the ingestion chain also changes. On the bulk path, the managed pipeline converts every data point into a JSON document, serializes it, and Elasticsearch parses that JSON back into a document and hashes the dimensions of each one to compute its time-series identifier. On the native path, Elasticsearch decodes a compact Protobuf payload once, and because it sees the OTLP resource and scope structure, it can hash the resource attributes once and reuse that partial hash for every data point that shares the resource, instead of re-hashing the full dimension set per document. The pull request that introduced the endpoint names both effects, the binary encoding and the reused partial hashes, as the reasons the native endpoint is more efficient than bulk ingestion of the same data.

There are also capabilities you gain compared with the Elasticsearch exporter currently used by the managed bulk path. The native endpoint records the aggregation temporality of each data point, stores histograms in the exponential_histogram field type, and accepts cumulative histograms. These are differences between the two managed ingestion implementations, not inherent restrictions of the Bulk API. Custom bulk clients can index exponential histogram documents and configure a temporality dimension. The rest of this post is about those differences.

OpenTelemetry histogram types and temporality explained

The most visible difference between the two paths is what happens to histograms, so the rest of this post is a tour through every histogram case.

OpenTelemetry defines two histogram data types. Histogram uses explicit bucket boundaries chosen by the producer. ExponentialHistogram uses exponentially spaced buckets controlled by a scale parameter, so the SDK adapts resolution to the data.

Each type can use one of two temporalities. With delta temporality, every data point covers one collection interval. With cumulative temporality, every data point covers everything since a fixed start time.

That gives four combinations, and the two ingestion paths treat them differently.

| Path | Explicit, delta | Explicit, cumulative | Exponential, delta | Exponential, cumulative | | -------------------------------- | -------------------- | -------------------- | ------------------ | ----------------------- | | Bulk API (ECH 9.0 to 9.5.2) | Supported, converted | Dropped | Supported, converted | Dropped | | Native (ECH 9.5.3 or later) | Supported, converted | Supported, converted | Supported | Supported |

"Converted" means the metric is accepted but stored with a different bucket representation than the one on the wire. On the native path, that representation is the exponential_histogram field type, which is the default mapping for OTLP histograms since 9.4. Cumulative histograms on the native path require that default; if a cluster overrides xpack.otel_data.histogram_field_type to histogram, cumulative histograms are not supported. If Elastic has pinned a deployment to the bulk API for compatibility, the bulk row applies regardless of version.

Each of the four examples below is a complete curl request you can run against your own deployment. The JSON encoding of OTLP/HTTP is verbose but self-documenting, which makes it a good learning tool. Production shippers use Protobuf (application/x-protobuf) instead.

Prerequisites: your endpoint URL and API key

In the Elastic Cloud Console, find your deployment under Hosted deployments and select Manage. In Application endpoints, cluster and component IDs, select Managed OTLP and copy the public endpoint.

Then open the API keys management page in Kibana and create an API key with the event:write privilege for the apm application. Use the Encoded value of the key. It is already base64-encoded in the id:api_key format that the Authorization header expects.

export MANAGED_OTLP_URL="https://your-endpoint.elastic-cloud.com"
export ELASTIC_API_KEY="your-encoded-api-key"

# Nanosecond-precision timestamps: T0 = 2 min ago, T1 = 1 min ago, T2 = now
T0="$(python3 -c 'import time; print(time.time_ns() - 120_000_000_000)')"
T1="$(python3 -c 'import time; print(time.time_ns() -  60_000_000_000)')"
T2="$(python3 -c 'import time; print(time.time_ns())')"

All four examples report the same metric, http.server.request.duration in seconds, for the same service. Each has a distinct example.case resource attribute so the examples form separate time series and can be queried independently. The explicit and exponential examples describe different latency distributions; each cumulative example contains two snapshots whose difference matches its delta counterpart. Run the timestamp setup again before repeating the examples to avoid resending points with the same dimensions and timestamp.

Explicit histogram with delta temporality (case 1)

This is the case that works on every supported ECH version, so it is the right place to learn the shape of an OTLP histogram.

A histogram distributes individual measurements across buckets. For an explicit histogram, you choose the bucket boundaries in advance. For example: 5 ms, 10 ms, and 25 ms, and the SDK counts how many requests fell into each bucket. Each data point carries the bucket counts, the total count, and the sum, which is enough to estimate percentiles at query time.

Percentiles are what you want for latency. An average of 50 ms hides the fact that 1% of requests take 5 seconds, while a histogram shows p50, p95, and p99 side by side.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "explicit-delta"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "histogram": {
                  "aggregationTemporality": 1,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T1}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 100,
                      "sum": 12.5,
                      "explicitBounds": [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
                      "bucketCounts":   [2,     5,    10,    20,   30,  18,   10,  4,   1,  0],
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

aggregationTemporality: 1 means delta. The data point covers the one-minute window from startTimeUnixNano to timeUnixNano and nothing before it.

explicitBounds lists nine upper boundaries, so bucketCounts has ten entries. The last entry is the overflow bucket for values above 2.5 seconds.

A successful response is HTTP 200 with an empty or absent partialSuccess object, meaning the managed endpoint durably accepted the data for processing. It does not confirm that Elasticsearch indexed the points: unsupported cumulative histograms can still be dropped on the managed bulk path, and indexing can fail downstream. Use the queries below to verify ingestion. See indexing errors and the failure store for troubleshooting.

How Elastic stores it. Both paths accept this payload, and both convert it. On the bulk path, the Elasticsearch exporter turns the buckets into a T-Digest style histogram field. On the native path, Elasticsearch converts the explicit boundaries into an exponential_histogram. In both cases percentile precision is bounded by the boundaries you chose on the producer side, because the conversion cannot recover detail that the original buckets did not have.

Explicit histogram with cumulative temporality (case 2)

This case uses the same metric and bucket boundaries as case 1, with cumulative counts and a fixed start time. This is the case that separates the two paths most sharply.

Cumulative temporality is the default in most OpenTelemetry SDKs and the model Prometheus users know. Every data point reports the totals since the process started, so the counts only grow and a consumer computes the rate over a window by subtracting two points. It is resilient to lost data points, because the next point still carries the full state.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "explicit-cumulative"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "histogram": {
                  "aggregationTemporality": 2,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T1}"'",
                      "count": 150,
                      "sum": 18.5,
                      "explicitBounds": [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
                      "bucketCounts":   [3,     7,    15,    30,   45,  27,   15,  6,   2,  0],
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    },
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 250,
                      "sum": 31.0,
                      "explicitBounds": [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
                      "bucketCounts":   [5,     12,   25,    50,   75,  45,   25,  10,  3,  0],
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

aggregationTemporality: 2 means cumulative. Both points retain T0 as the fixed start of the series. The first reports 150 requests through T1; the second reports 250 through T2. Subtracting their counts, sums, and corresponding buckets gives the 100-request distribution for T1 to T2 shown in case 1.

How Elastic stores it. On the native path Elasticsearch stores the point as an exponential_histogram and records cumulative in the temporality dimension. Time series functions in ES|QL use that dimension to compute per-interval rates from consecutive cumulative points.

On the bulk path this data point is dropped. The managed bulk path's Elasticsearch exporter does not support cumulative histograms; treating those buckets as delta would produce incorrect interval distributions. If your ECH deployment is on 9.0 to 9.5.2, configure your SDK or Collector to export histograms with delta temporality, or put a cumulativetodelta processor in front of the export.

Exponential histogram with delta temporality (case 3)

Now switch the data type while keeping delta temporality.

An exponential histogram removes the need to pick boundaries. Its buckets are exponentially spaced and controlled by a single scale parameter, and the SDK lowers the scale automatically when the data range grows. That is why exponential histograms are the recommended default for latency SLOs where p99 and p99.9 accuracy matters: the resolution is relative to the value, so the tail is captured as precisely as the body.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "exponential-delta"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "exponentialHistogram": {
                  "aggregationTemporality": 1,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T1}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 100,
                      "sum": 62.2,
                      "scale": 3,
                      "zeroCount": 0,
                      "positive": {
                        "offset": -10,
                        "bucketCounts": [2, 5, 10, 20, 30, 18, 10, 4, 1]
                      },
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

Reading the bucket layout takes a moment the first time.

The base of the histogram is 2^(2^-scale). With scale: 3 that is about 1.09, so each bucket is 9% wider than the previous one.

Bucket i in bucketCounts covers the range (base^(offset+i), base^(offset+i+1)], lower bound excluded and upper bound included. With offset: -10 the first bucket starts at about 0.42 seconds and the ninth ends at about 0.92 seconds.

zeroCount counts measurements that are exactly zero or within the zero threshold, and a negative bucket range exists for negative values, which latency never uses.

How Elastic stores it. On the native path Elasticsearch stores the scale, offset, and counts as an exponential_histogram without conversion. This is the highest-fidelity case: what you measured is what you query.

On the bulk path the Elasticsearch exporter converts the exponential buckets into a T-Digest style histogram. The data point is accepted and percentiles remain usable, but the original scale and bucket layout are not preserved.

Exponential histogram with cumulative temporality (case 4)

The last case combines the adaptive bucket layout of case 3 with the cumulative semantics of case 2. An SDK configured for exponential histogram aggregation and cumulative export temporality produces this shape. The OpenTelemetry SDK default aggregation for histogram instruments is explicit buckets, so selecting exponential aggregation is a separate configuration choice.

curl --fail-with-body --silent --show-error \
  -H "Authorization: ApiKey ${ELASTIC_API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-service"}},
            {"key": "example.case", "value": {"stringValue": "exponential-cumulative"}}
          ]
        },
        "scopeMetrics": [
          {
            "scope": {"name": "http"},
            "metrics": [
              {
                "name": "http.server.request.duration",
                "unit": "s",
                "exponentialHistogram": {
                  "aggregationTemporality": 2,
                  "dataPoints": [
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T1}"'",
                      "count": 150,
                      "sum": 93.3,
                      "scale": 3,
                      "zeroCount": 0,
                      "positive": {
                        "offset": -10,
                        "bucketCounts": [3, 7, 15, 30, 45, 27, 15, 6, 2]
                      },
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    },
                    {
                      "startTimeUnixNano": "'"${T0}"'",
                      "timeUnixNano":      "'"${T2}"'",
                      "count": 250,
                      "sum": 155.5,
                      "scale": 3,
                      "zeroCount": 0,
                      "positive": {
                        "offset": -10,
                        "bucketCounts": [5, 12, 25, 50, 75, 45, 25, 10, 3]
                      },
                      "attributes": [
                        {"key": "http.request.method",       "value": {"stringValue": "GET"}},
                        {"key": "http.response.status_code", "value": {"intValue": 200}}
                      ]
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }' \
  "${MANAGED_OTLP_URL}/v1/metrics"

How Elastic stores it. On the native path this is stored exactly like case 3, as an exponential_histogram, plus the cumulative temporality dimension. Elasticsearch can then downsample and compute rates over the series correctly. The two points share the same start time and bucket layout; their difference is the delta distribution in case 3.

On the bulk path the data point is dropped, for the same reason as case 2.

Requires Elastic Stack 9.5.3 or later on ECH. On ECH 9.0 to 9.5.2, cases 2 and 4 are dropped. Emit delta histograms until you upgrade.

Query OpenTelemetry histogram percentiles with ES|QL

Sending is only half of the showcase. Open Discover in Kibana, switch to ES|QL mode, and first confirm that the documents arrived:

FROM metrics-*.otel-*
| EVAL duration = metrics.http.server.request.duration::exponential_histogram
| WHERE service.name == "my-service" AND duration IS NOT NULL
| KEEP @timestamp, resource.attributes.example.case, metrics.http.server.request.duration, attributes.http.request.method
| SORT @timestamp DESC
| LIMIT 10

On a native-path deployment the histogram field shows the scale, the bucket indices, and the counts you sent. On a bulk-path deployment you see the converted values and counts arrays instead, which is the visible trace of the conversion.

Then ask the question the histogram was collected for, the request latency percentiles:

TS metrics-*.otel-*
| WHERE service.name == "my-service" AND @timestamp >= NOW() - 5 minutes
| STATS p50 = PERCENTILE(metrics.http.server.request.duration::exponential_histogram, 50),
        p95 = PERCENTILE(metrics.http.server.request.duration::exponential_histogram, 95),
        p99 = PERCENTILE(metrics.http.server.request.duration::exponential_histogram, 99)
    BY resource.attributes.example.case, attributes.http.request.method, TBUCKET(5 minutes)

TS merges histograms per time series and respects their temporality before computing percentiles. For cumulative series, it accounts for the change between consecutive snapshots instead of counting their overlapping observations repeatedly. FROM is useful for inspecting individual documents, but it ignores temporality when aggregating them. See how temporality affects queries.

Grouping by resource.attributes.example.case keeps the four examples separate. The explicit examples place the median in the 0.05 to 0.1 second bucket; the exponential examples place it around 0.6 seconds. These are different input distributions, so their percentile results should differ. The cumulative examples describe the same final one-minute distribution as their corresponding delta examples. Refer to the ES|QL aggregation functions reference for the full list of functions that accept exponential histogram fields.

Summary: OpenTelemetry histogram support on Elastic Cloud Hosted

• ECH deployments on Elastic Stack 9.5.3 or later receive OTLP metrics through the native Elasticsearch endpoint, with no client change.
• Those deployments accept all four histogram cases, and exponential histograms are stored without conversion.
• ECH deployments on 9.0 to 9.5.2 continue through the bulk API and should emit delta histograms.
• Serverless deployments always use the native path.

Clients keep sending to the same /v1/metrics path on the Managed OTLP Endpoint in every case.

Learn more about OpenTelemetry metrics on Elastic

• Elastic Cloud Managed OTLP Endpoint documentation
• Elasticsearch OTLP/HTTP endpoint documentation
• Exponential histogram field type
• Now GA: Managed OTLP Endpoint on Elastic Cloud Hosted

Both features follow the OTel GenAI semantic conventions and work with any OTel-instrumented provider. If your framework already emits OTel GenAI span attributes, there is nothing to change.

How OTel GenAI spans are structured

A GenAI span stores everything as span attributes. A typical chat span includes:

• gen_ai.provider.name: the provider (openai, anthropic, aws.bedrock, etc.); gen_ai.system is supported as a fallback for older instrumentation.
• gen_ai.operation.name: the operation type (chat, embeddings, etc.).
• gen_ai.request.model: the model being called.
• gen_ai.usage.input_tokens: tokens consumed by the prompt.
• gen_ai.usage.output_tokens: tokens generated in the response.
• gen_ai.input.messages, gen_ai.output.messages: conversation messages.
• gen_ai.system_instructions: the system prompt.

Both features read from these attributes:

| Feature | What it shows | Where it appears | Attributes it reads | | --- | --- | --- | --- | | GenAI tab | Details (operation type, request model, provider, input and output token counts, response model, response ID) and Conversation (system prompt, user messages, model response) | Span flyout in the APM trace view, and the span flyout in Discover | Appears with any gen_ai.* attribute. Conversation needs gen_ai.system_instructions, gen_ai.input.messages, and gen_ai.output.messages | | Token count badges | Input and output token counts for each GenAI span | Every GenAI span row in the trace waterfall | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |

How to read LLM prompts and responses in the GenAI tab

When any gen_ai.* attribute is present on a span, the span flyout shows a dedicated GenAI tab next to Metadata.

The Details section shows model metadata from the span attributes: operation type, request model, provider, input and output token counts, response model, and response ID. The Conversation section shows the full exchange, populated from gen_ai.system_instructions (system prompt), gen_ai.input.messages (user messages), and gen_ai.output.messages (model response), each with a copy button so you can pull the exact prompt or response out of the trace without scraping text from a formatted table.

All raw span attributes remain accessible on the Metadata tab.

The GenAI tab is also available in the span flyout in Discover, so you can inspect LLM prompts and responses directly alongside your log and trace data without switching to the APM view.

What instrumentation does LLM tracing require?

No Kibana-side configuration is needed. The GenAI tab appears automatically when any gen_ai.* attribute is present on a span. Full Conversation support requires the OTel GenAI v1.37.0 span-attribute model: gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions.

Frameworks that emit the older span-events model (gen_ai.user.message, gen_ai.assistant.message, gen_ai.choice) will show the Details metadata section but will not populate the Conversation section. For a current list of compatible instrumentations, see the OTel GenAI semantic conventions.

To verify, open the span in Discover, check that gen_ai.input.messages and gen_ai.output.messages are present, and confirm the Conversation section renders.

If your application already sends APM data to Elastic from a GenAI workload, open any GenAI span in the trace view and check for the GenAI tab.

LLM token usage in the trace waterfall

Token count badges now appear on each GenAI span row in the waterfall, so you can scan the full trace without drilling in. In agentic traces with ten or twenty LLM calls, this lets you identify which span is driving token consumption before opening any span.

Each row shows input and output token count badges sourced from gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The row label is the span name, which instrumentation frameworks typically set to something like chat gpt-4o-mini.

Waterfall-level token counts are most useful in agentic traces where a chain of LLM calls uses different models or the same model with varying context sizes.

Which LLM providers does Elastic APM support for GenAI tracing?

Elastic APM's LLM tracing works with any OTel-instrumented provider: the GenAI tab and waterfall token counts use the same OTel attribute schema regardless of which provider your application uses. Provider is read from gen_ai.provider.name, falling back to gen_ai.system for older instrumentation.

The OTel GenAI semantic conventions that enable this provider detection are currently in a Development lifecycle. Check the release notes before upgrading instrumentation.

How to enable LLM tracing in Elastic APM

Availability: Both features are available as a Technical Preview on Elastic Serverless and will be available as a Technical Preview in Elastic Stack 9.6.

To try these features:

1. Instrument your GenAI application with an OTel SDK that follows the OTel GenAI semantic conventions (v1.37.0 or later for full Conversation support).
2. Send traces to Elastic Observability using OTLP, the Elastic APM agent, or an EDOT SDK.
3. Open the APM section in Kibana, navigate to a service that makes LLM calls, and open the trace waterfall for any transaction.

The GenAI tab appears on any span with at least one gen_ai.* attribute set; token count badges appear when gen_ai.usage.input_tokens or gen_ai.usage.output_tokens are present.

If you don't have a GenAI application to test with, the otel-genai-chat-app repository is a minimal OpenAI chat app pre-instrumented with EDOT. Set OPENAI_API_KEY and follow the EDOT commands in the README to send traces to Elastic and see both features in action.

What's next for LLM observability in Elastic APM

We're exploring cost estimation per span (estimated spend based on model pricing and token counts, surfaced in the waterfall) and tool call rendering (structured display of tool/function call inputs and outputs for agentic spans).

If you are building GenAI applications and want early access or to share feedback, reach out through the Elastic community forums or open an issue in the kibana repository.

Forking the agent would fix this, but then you own every upstream merge. You could also use manual instrumentation for complete control, but that requires code changes and ongoing upkeep. OpenTelemetry Java extensions give you a cleaner path: a separate JAR the agent loads at startup, giving you precise control over what gets captured and exported, without touching agent or application code.

For example, the following challenges are very common:

• Health check probes are flooding your trace view.
• Span names reflect generic framework patterns rather than your domain operations.
• Some span names or attributes have high cardinality creating noise in your traces.
• Spans are missing attributes relevant to your business logic.
• Baggage headers are propagating to downstream services when they shouldn't.
• Resource attributes that describe your deployment are not automatically captured because they rely on custom environment variables.

Some of those can be solved through configuration, or by using an intermediate OpenTelemetry Collector for processing. However, this also might add complexity to the telemetry pipeline, and you might prefer to solve this at the source, where the data is captured.

What are OpenTelemetry Java extensions

An extension is a JAR file the agent loads at startup. It hooks into the agent's extension points through Java's Service Provider Interface (SPI) mechanism, the same mechanism the agent uses internally.

The extension mechanism works identically with the upstream OpenTelemetry Java agent and with Elastic's OpenTelemetry distribution. You write the extension once and it works with either.

For reference, the upstream extension documentation provides an exhaustive overview of extension points and a few examples.

This post does not aim to provide a complete reference, but focuses on simple use cases you're likely to reach for in production: renaming spans, filtering noisy traces, or propagating context that the agent doesn't cover in your environment.

Extensions also let you modify and extend the agent instrumentation itself. That goes beyond what this post covers. Here are two starting points:

• Modify instrumentation using instrumenter customizers.
• Modify or add instrumentation using the instrumentation module.

Setting up an OpenTelemetry Java extension project

An extension is a standard Java Gradle project with two requirements: the output must be a shadow JAR (a fat JAR with all extension dependencies bundled), and OpenTelemetry dependencies must be declared compileOnly so you don't bundle the SDK itself.

The shadow JAR requirement exists because the agent loads the extension in its own classloader. If you declare a dependency as implementation, it gets bundled and may conflict with the version already in the agent. Using compileOnly keeps those JARs out of the extension JAR entirely.

Here is a minimal build.gradle.kts for a simple extension that does not customize instrumentation and thus relies only on the OpenTelemetry SDK/API.

plugins {
  id("java")
  id("com.gradleup.shadow")
}

repositories {
  mavenCentral()
}

java {
  toolchain {
    languageVersion.set(JavaLanguageVersion.of(8))
  }
}

dependencies {
  // Use BOM to manage OpenTelemetry dependency versions
  compileOnly(platform("io.opentelemetry:opentelemetry-bom:1.64.0"))
  // OpenTelemetry SDK autoconfiguration SPI (provided by agent)
  compileOnly("io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi")
  // OpenTelemetry SDK
  compileOnly("io.opentelemetry:opentelemetry-sdk")
  // Annotation processor for automatic SPI registration
  compileOnly("com.google.auto.service:auto-service:1.1.1")
  annotationProcessor("com.google.auto.service:auto-service:1.1.1")
}

tasks.assemble {
  dependsOn(tasks.shadowJar)
}

Check Maven Central for the latest version of the BOM before starting.

Extensions only depend at compile-time on the OpenTelemetry SDK and the autoconfiguration SPI. The agent provides the rest of the SDK and instrumentation implementation at runtime.

Loading OpenTelemetry Java extensions at runtime

To load an OpenTelemetry Java extension at runtime, you can use the otel.javaagent.extensions system property or OTEL_JAVAAGENT_EXTENSIONS environment variable. The value is a comma-separated list of paths to extension JARs:

java -Dotel.javaagent.extensions=/path/to/my-extension.jar -javaagent:/path/to/opentelemetry-javaagent.jar -jar myapp.jar

The upstream OpenTelemetry Java agent also lets you embed extensions directly into the agent JAR to simplify deployment.

Filtering and renaming spans with OpenTelemetry Java extensions

You can modify spans in two ways:

• Using a SpanProcessor that is called synchronously when the span starts or ends.
• Using a SpanExporter that is called asynchronously when the span is exported.

Rename spans with a SpanProcessor

SpanProcessor.onStart receives a ReadWriteSpan, which means you can call span.updateName() before the span is exported. This is the right hook for renaming based on attributes that are available at span start.

public class OperationRenamingSpanProcessor implements SpanProcessor {

  @Override
  public void onStart(Context parentContext, ReadWriteSpan span) {
    String operation = span.getAttribute(AttributeKey.stringKey("app.operation"));
    if (operation != null) {
      span.updateName(operation);
    }
  }

  @Override
  public boolean isStartRequired() { return true; }

  @Override
  public void onEnd(ReadableSpan span) {}

  @Override
  public boolean isEndRequired() { return false; }

  @Override
  public CompletableResultCode shutdown() { return CompletableResultCode.ofSuccess(); }

  @Override
  public CompletableResultCode forceFlush() { return CompletableResultCode.ofSuccess(); }
}

Register the SpanProcessor via AutoConfigurationCustomizerProvider, composing it with whatever processor you have already configured:

@AutoService(AutoConfigurationCustomizerProvider.class)
public class RenamingCustomizerProvider implements AutoConfigurationCustomizerProvider {

  @Override
  public void customize(AutoConfigurationCustomizer customizer) {
    customizer.addTracerProviderCustomizer(this::configureSdkTracerProvider);
  }

  private SdkTracerProviderBuilder configureSdkTracerProvider(
      SdkTracerProviderBuilder tracerProvider, ConfigProperties config) {
    return tracerProvider.addSpanProcessor(new OperationRenamingSpanProcessor());
  }

}

The modify-span EDOT Java extension example provides a complete implementation.

Filter spans with a SpanExporter

A SpanExporter wrapper lets you modify or drop spans before they leave the process. This works well for known noisy endpoints like health checks.

public class FilteringSpanExporter implements SpanExporter {

  private final SpanExporter delegate;

  public FilteringSpanExporter(SpanExporter delegate) {
    this.delegate = delegate;
  }

  @Override
  public CompletableResultCode export(Collection<SpanData> spans) {
    List<SpanData> filtered = new ArrayList<>();
    for (SpanData span : spans) {
      if (!"GET /health".equals(span.getName())) {
        filtered.add(span);
      }
    }
    return delegate.export(filtered);
  }

  @Override
  public CompletableResultCode flush() { return delegate.flush(); }

  @Override
  public CompletableResultCode shutdown() { return delegate.shutdown(); }
}

Register the FilteringSpanExporter via addSpanExporterCustomizer:

customizer.addSpanExporterCustomizer((existing, config) -> new FilteringSpanExporter(existing));

The modify-span EDOT Java extension example provides a complete implementation.

The approach has two limitations:

• This won't discard any child span that may have been created, for example, if the healthcheck calls the database.
• Spans filtered at the exporter have already passed through the full processor pipeline and occupied buffer space in the batch processor.

If you're dropping a large fraction of your traffic at this stage, a custom Sampler (shown below) is more efficient because it drops spans before any processing happens and also filters out child spans. Also, when using declarative configuration, the rule-based sampler lets you implement filtering on rules using only configuration.

Adding custom resource attributes with a ResourceProvider

Resource attributes describe what's running: the service name, its version, the host. A ResourceProvider lets you attach additional attributes that the agent doesn't know about, such as deployment metadata your platform injects through environment variables.

The example below uses environment variables, but it could also be a configuration file, a cloud metadata service, or any other source available to the agent at startup.

Because the SDK initialization is synchronous, when querying an external service like a metadata endpoint, this can make the agent (and thus the application) startup slower. If possible, prefer checking environment variables and local config first before calling an external service.

@AutoService(ResourceProvider.class)
public class DeploymentResourceProvider implements ResourceProvider {

  @Override
  public Resource createResource(ConfigProperties config) {
    AttributesBuilder attributes = Attributes.builder();

    String region = System.getenv("DEPLOY_REGION");
    if (region != null) {
      attributes.put(AttributeKey.stringKey("deployment.region"), region);
    }

    String buildVersion = System.getenv("BUILD_VERSION");
    if (buildVersion != null) {
      attributes.put(AttributeKey.stringKey("build.version"), buildVersion);
    }

    return Resource.create(attributes.build());
  }
}

Attributes from a ResourceProvider merge with the agent's own resource. When two providers supply the same key, the one with the higher order() value wins. The agent's built-in providers use order 0, so overriding order() to return a positive integer gives your provider priority.

The resource-attribute EDOT Java extension example provides a complete implementation.

Custom sampling in OpenTelemetry Java

When filtering at the exporter is too late or too expensive, implement a Sampler directly. The sampler runs before any span processing, so dropped spans never touch the batch buffer.

However, the sampling decision can only rely on attributes that are provided when the span starts. For example, the status code of an HTTP response can't be used as it is only available when the span ends.

The key detail: wrap the existing sampler rather than replacing it. That way, your logic composes with whatever you configured, and parent-based decisions from an upstream service are still respected.

public class HealthCheckSampler implements Sampler {

  private final Sampler delegate;

  public HealthCheckSampler(Sampler delegate) {
    this.delegate = delegate;
  }

  @Override
  public SamplingResult shouldSample(
      Context parentContext,
      String traceId,
      String name,
      SpanKind spanKind,
      Attributes attributes,
      List<LinkData> parentLinks) {
    if (spanKind == SpanKind.SERVER && name.contains("health")) {
      return SamplingResult.create(SamplingDecision.DROP);
    }
    return delegate.shouldSample(parentContext, traceId, name, spanKind, attributes, parentLinks);
  }

  @Override
  public String getDescription() {
    return "HealthCheckSampler{" + delegate.getDescription() + "}";
  }
}

Register the HealthCheckSampler via addSamplerCustomizer, which gives you both the existing sampler and the resolved config:

customizer.addSamplerCustomizer((existing, config) -> new HealthCheckSampler(existing));

Community extensions in opentelemetry-java-contrib

The opentelemetry-java-contrib repository contains several community-maintained extensions.

Some of them are already included in the OpenTelemetry Java agent (and inherited in the Elastic distribution), but are opt-in:

• azure-resources
• aws-resources
• gcp-resources
• cloudfoundry-resources
• baggage-processor

Most Elastic distribution features exist as extensions in the contrib repository, so you can use them with the upstream agent in a vendor-neutral way.

• inferred-spans
• span-stacktrace

Further reading and extension examples

The upstream extension examples cover additional extension points not shown here, including custom propagators, ID generators, and ignored-type configurers.

The Elastic baggage example shows the filtering propagator for baggage running end-to-end with a two-service application, it also demonstrates custom instrumentation to add baggage without modifying the application code.

This post covered the project setup and the patterns most likely to come up in production. Both links above go deeper: the upstream examples add extension points not covered here, and the baggage example shows a complete two-service implementation you can run locally.

What changed in Elastic Agent 9.3: A native OTel Collector under the hood

Previously, Elastic Agent acted as a supervisor process, spinning up Beats-based subprocesses, such as Filebeat or Metricbeat. From 9.3 onward, that architecture has been replaced. Elastic Agent itself is now built on the EDOT Collector, turning it into a first-class OTel Collector under the hood while preserving its original functionality.

Key benefits of this architectural shift include:

• Reduced footprint: Fewer subprocesses mean significantly less memory overhead and a simpler deployment model. In future releases, this footprint will be even further reduced.
• Unified telemetry pipeline: Logs and metrics flow through a single, standards-based OTel pipeline, as do traces.
• Ecosystem interoperability: Elastic Agent can now receive data from any OTel-compatible source out of the box. It can also be configured to emit to OTel-compatible destinations.
• Aligned with the OTel ecosystem: As the OTel ecosystem matures with new receivers, processors, and exporters, Elastic Agent deployments gain access to those capabilities automatically.

When you deploy or update Elastic Agent from version 9.3 onward, you're deploying an OpenTelemetry Collector. EDOT is the technology foundation; Elastic Agent is the product.

How existing Beats configurations run inside the OTel Collector pipeline

Elastic has introduced Beats Receivers, which are Beat inputs and processors that execute natively inside the new OTel Collector pipeline. For your teams and customers, this means:

• Existing elastic-agent.yml configurations require no modification.
• Fleet-managed agents automatically translate policy configurations into OTel format internally.
• All integrations, dashboards, ingest pipelines, and alerting rules continue to function exactly as before.
• Data written via Beats Receivers lands in the same data streams as always.

Upgrading to 9.3 is transparent because it uses the same inputs and produces the same outputs.

Running Beats and OTel Collector pipelines in one Elastic Agent

The new Elastic Agent is a collector capable of simultaneously running traditional Beats-based collections alongside native OTel pipelines, all in a single deployment. One agent policy can collect Elastic Common Schema–schematized (ECS-schematized) data via Beats Receivers and ingest native OpenTelemetry Protocol (OTLP) data from OTel-instrumented applications and infrastructure. This same agent policy can apply OTel processing stages across all telemetry before export.

OTel integrations from Elastic's catalog can be added to any agent policy. When native OTel data is ingested, Elastic automatically installs the relevant dashboards and alerts, in addition to necessary content packs, without any manual setup.

What's the relationship between Elastic Agent and EDOT?

You may be familiar with EDOT, the Elastic Distribution of OpenTelemetry Collector, as a stand-alone product. With this architectural change, EDOT is the technology foundation that now powers Elastic Agent, not a separate product that users need to track or deploy independently.

Going forward, Elastic Agent is the supported, Fleet-manageable, fully featured product. A stand-alone deployment remains available for specific niche scenarios (environments where the full version of Elastic Agent cannot be installed), but it isn't the recommended path for the vast majority of users.

Elastic Agent deployment options: Fleet-managed vs. stand-alone

| | Fleet-managed Elastic Agent | Stand-alone Elastic Agent | | :------------------------- | :-----------------------------: | :-----------------------------------------------------------------------------------------------------: | | Fleet lifecycle management | Yes | Can enroll into Fleet in-field without reinstallation | | Beats Receivers | Yes | Yes | | Elastic Defend | Yes | No | | Cloud Security | Yes | No | | Profiler support | Yes | No | | OTel-native pipeline | Yes | Yes | | Best for | Most deployments | Environments where full Elastic Agent cannot be installed or management is handled by other tools |

Do I need to change anything when upgrading to Elastic Agent 9.3?

For users running Elastic Agent today, upgrading to 9.3 requires no changes to configurations or integrations, and no changes to workflows. For customers evaluating OTel adoption, Elastic Agent now provides a fully supported, production-ready OTel Collector with Fleet management and rich integrations, along with Elastic's full support matrix, and none of this requires a separate OTel deployment.

With Elastic Agent 9.3, Elastic's data collection is fully OpenTelemetry-native. Elastic Agent is now an OpenTelemetry Collector. Everything you have today still works, and you also get all the capabilities of OTel.

The gain is in where the intermediate set lives. When you ask what the saturated hosts logged, the list of saturated hosts is computed and consumed inside Elasticsearch. Six host names or 500 trace IDs never land in a clipboard or in an AI agent's context window, and the set is recomputed from current data every time the query runs.

That changes the unit of investigation. You go from "one slow request stalled on a lock" to "most of the slowest requests did," and only the second answer tells you which team to page. It matters more the further apart your signals are: in many observability stacks, logs, metrics, and traces live in three separate systems, each with its own query language, its own time picker, and its own idea of what a host is, so the same question has to be asked two or three times and the answers joined by hand.

In this post we walk through four investigations, each of which is self-contained. For every one of them, we set out the scenario that started it, the query that answers it, the table it returns, and a note on the difficulties you run into when you try to get the same answer any other way.

| Pattern | Starts from | Question it answers | |---|---|---| | Metrics to logs | CPU saturation | Which error patterns show up only on the saturated hosts? | | Logs to metrics | Error logs | Do the erroring hosts look any different from the healthy ones? | | Traces to logs | Slow spans | What did every service log during those specific requests? | | All three signals | Pod memory pressure | Which log lines sit behind the requests that failed under that pressure? |

The data was collected with OpenTelemetry and lands in the logs-*.otel-*, traces-*.otel-*, and metrics-*.otel-* data streams, where fields keep their semantic convention names rather than being rewritten into another schema. The correlation pattern works just as well on Elastic Agent integrations, though the queries need translating rather than just renaming: ECS carries log severity as the text field log.level instead of a numeric severity_number, the System integration reports CPU as separate system.cpu.*.pct fields instead of one metric with a state dimension, and APM records durations in microseconds. All of it lands in the same cluster either way, which is the part the subquery depends on.

In Discover, the time picker already applies the range, so the examples below omit an explicit @timestamp filter. Outside Discover, add a filter by time yourself, either with literal timestamps in the query or with ?_tstart and ?_tend in the query and values in the params array of your _query request. Every result below comes from a one hour window over a synthetic fleet of 300 hosts.

Metrics to logs: what are the saturated hosts complaining about?

An infrastructure alert tells you a handful of hosts in a fleet of a few hundred sat above 90% CPU over the last hour. That tells you which hosts are hot and nothing about why. The question worth answering is whether those hosts share a failure mode, or whether they are busy for unrelated reasons and the alert is a coincidence.

FROM logs-*.otel-*
| WHERE severity_number >= 17
  AND resource.attributes.host.name IN (
      TS metrics-hostmetrics.otel-*
      | WHERE attributes.state == "idle"
      | STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
          BY resource.attributes.host.name
      | WHERE idle < 0.1
      | KEEP resource.attributes.host.name
    )
| STATS errors = COUNT(*), hosts = COUNT_DISTINCT(resource.attributes.host.name)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT errors DESC

The two halves of the query map onto the two halves of the question. The subquery works out which hosts were saturated, averaging CPU utilization per host and keeping the ones that averaged under 10% idle, which is another way of saying above 90% busy for the window. The outer query then works out what those hosts were complaining about, pulling their error logs and using CATEGORIZE to collapse thousands of individual lines into a handful of error classes.

Two choices in there are worth pausing on.

The subquery uses TS rather than FROM because a host does not report one CPU number. It reports a separate time series per CPU state, and per logical core as well if your collector is configured to break them out, so the reduction has to happen in two stages. AVG_OVER_TIME collapses each series to a single value first, and the outer AVG then combines those into one figure per host.

Naming that inner function matters more than it looks. Write AVG(metrics.system.cpu.utilization) on its own and TS supplies LAST_OVER_TIME for you, averaging each series' final sample rather than its average over the window. In this dataset that one substitution moves a host from 7% idle to 10% idle, which is the difference between appearing in the results and not. Querying metrics with the TS command goes into the two aggregation phases in more depth.

The log filter tests severity_number (17 is the ERROR floor on the OpenTelemetry scale) rather than the severity text, because the numeric scale is fixed by the spec while the text is whatever the emitting library decided to write. That is not a hypothetical distinction here: the error logs in this cluster carry four different labels, including SEVERE from a Java service. Matching on the text alone returns 2,754 of checkout's errors and misses the billing service entirely, while the numeric filter returns all 5,910 of them and keeps billing too.

When you run the query in Discover, the result is a short table of error classes, scoped to the saturated hosts:

The subquery returned six saturated hosts, and on all six the same service is timing out against an upstream dependency and draining its connection pool. The hosts column is what lets you set the other two rows aside without opening anything: the certificate errors reach only two of the six, and the gateway declines amount to five lines in an hour. Neither tracks the cohort the way the checkout patterns do.

Having all three signals in one store already removed the exports from this investigation. The subquery removes the step after that, and closing that last gap matters more than it sounds. By the time you have read six host names off a chart and typed them into a log search, the set has moved: a host that crossed the threshold a minute ago is missing from your list, and one that has since recovered is still in it. Here the host list is derived from current data on every run, so re-running the query during an incident gives you the current cohort.

The stale list is only half of it. A correlation done by hand exists only in the head of the person who did it, so nobody else can check it, save it, or run it again tomorrow.

Logs to metrics: do the erroring hosts look different from the healthy ones?

Filtering metrics by a log-derived host set answers the opposite question. The payments service is throwing errors on some hosts and not others, and you want to know whether resource pressure explains the split before you start reading deploy history.

That is a comparison, so the query needs both cohorts. FORK runs two branches over the same input, and IN and NOT IN against the same log-derived host set divide the fleet between them.

TS metrics-hostmetrics.otel-*
| WHERE attributes.state == "idle"
| STATS idle = AVG(AVG_OVER_TIME(metrics.system.cpu.utilization))
    BY host = resource.attributes.host.name
| EVAL busy = 1 - idle
| FORK
    ( WHERE host IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "logging errors" )
    ( WHERE host NOT IN (
        FROM logs-*.otel-*
        | WHERE severity_number >= 17
          AND resource.attributes.service.name == "payments"
          AND resource.attributes.host.name IS NOT NULL
        | STATS errors = COUNT(*) BY resource.attributes.host.name
        | KEEP resource.attributes.host.name )
      | EVAL cohort = "no errors" )
| STATS hosts = COUNT(*), mean_busy = AVG(busy), busiest_host = MAX(busy)
    BY cohort

The query reads top to bottom as three stages. The metrics query runs first and reduces the whole fleet to one busy figure per host. FORK then splits that fleet in two using the same log query in both branches, separating the hosts that appear in it from the hosts that do not. The final STATS summarizes each group, so both cohorts come back as two rows of one table, measured the same way over the same window.

This query is longer than the others, and three parts of it are less obvious than they look.

The natural way to label the two cohorts would be EVAL cohort = CASE(host IN (...), "erroring", "healthy"), and ES|QL rejects it. In 9.5 an IN subquery has to be a top-level predicate in a WHERE condition rather than an argument to a scalar function, which is why the split happens at the command level with FORK.

The STATS ... BY inside each subquery looks redundant, since KEEP alone would return the same host names. It is not: without it, the subquery returns one row per matching log document instead of one row per host, and those rows are all held in memory for the outer query to filter against. Aggregating first turns millions of rows into a few hundred host names.

The IS NOT NULL filter guards the sharpest edge here, and this dataset is a live example rather than a hypothetical. NOT IN follows SQL null semantics, so a single null in the subquery result makes the predicate match nothing at all. Five of the payments error logs in this cluster came through a sidecar that dropped the host name. Remove that one line from both branches and the query still succeeds, but it returns a single row: the nulls quietly delete the entire 286-host "no errors" cohort, and what is left looks like a perfectly plausible answer to a different question.

Run the query in Discover and you get two rows, one per cohort:

CPU does not explain the split, and the numbers say so twice. The 14 erroring hosts run slightly cooler on average than the 286 quiet ones, and the busiest machine among them averaged 50% over the hour while the quiet cohort contains a host that averaged 95%. Whatever is failing on those 14, they had headroom the entire time, and deploy history is a better place to spend the next ten minutes.

A negative result like this is worth as much as a positive one, and it is usually the one people skip. Getting it the long way means running the metrics query twice against two hand-built host lists and lining the numbers up afterwards, which is enough friction that the check often just does not happen. Both cohorts here come from the same log query in the same execution, over identical time windows, so there is nothing to reconcile and no reason not to check.

Traces to logs: what did every service log during the slow requests?

A service level objective (SLO) burn alert fires on checkout latency. Tracing gives you the slow requests and their spans, and the next question is what the services involved were writing to their logs while those specific requests were in flight.

The trace ID is the join key, and there are far too many of them to move by hand.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND resource.attributes.service.name == "checkout"
        AND name == "POST /api/orders"
        AND duration > 2000000000
      | SORT duration DESC
      | LIMIT 500
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text),
       service = resource.attributes.service.name,
       severity_text
| SORT traces DESC

The subquery answers "which requests were slow." It looks at the inbound request span for the checkout endpoint rather than the client and internal spans beneath it, then keeps the 500 slowest requests over two seconds. Durations are recorded in nanoseconds, which is why the threshold has so many zeros.

The outer query answers "what got logged while they were running," gathering every log line that shares one of those trace IDs and grouping them into patterns.

Counting distinct traces per pattern is what makes the output readable. A log pattern that appears 30,000 times across four traces is one chatty request, while a pattern that shows up in 470 of the 500 slowest traces is a property of being slow.

From the results above, the top row is there by construction and can be set aside: checkout writes one order submitted line per order, so it appears in all 500 traces and says nothing about why these particular 500 were slow.

From the results above, the third row is the answer, and it points at a service nobody was looking at. A lock wait timeout in inventory, two hops downstream from where the alert fired, shows up in 470 of the 500 slowest requests, and the 947 lines behind those 470 traces mean a good share of them retried more than once. The payment gateway declines are real failures, and at 19 traces out of 500 they are not what is burning the SLO.

Done by hand, this means opening slow traces one at a time and reading the correlated logs for each, which is tedious at ten traces and nobody's idea of a plan at 500. When the spans and the logs are held in different systems, every trace you check is a copied ID and a context switch, and the sample size you can afford drops to about three. Three traces is enough to form a theory and not enough to test one. Treating the slow requests as a population is what turns "this trace had a lock wait" into "470 of the 500 slowest requests had a lock wait," and that difference decides whether you page the inventory team.

All three signals: from pod memory pressure to the log lines behind the failures

IN subqueries nest, so the pattern extends to as many signal types as the question needs.

A node pool starts reporting memory pressure after a rollout. You want the log lines from the requests that actually failed on the pods under pressure, which means going from metrics to traces to logs without stopping in between.

FROM logs-*.otel-*
| WHERE trace_id IN (
      FROM traces-*.otel-*
      | WHERE kind == "Server"
        AND status.code == "Error"
        AND resource.attributes.k8s.pod.uid IN (
            TS metrics-kubeletstats.otel-*
            | STATS peak = MAX(MAX_OVER_TIME(metrics.k8s.pod.memory_limit_utilization))
                BY resource.attributes.k8s.pod.uid
            | WHERE peak > 0.95
            | KEEP resource.attributes.k8s.pod.uid
          )
      | STATS failures = COUNT(*) BY trace_id
      | SORT failures DESC
      | LIMIT 1000
      | KEEP trace_id
    )
| STATS lines = COUNT(*), traces = COUNT_DISTINCT(trace_id)
    BY pattern = CATEGORIZE(body.text), service = resource.attributes.service.name
| SORT traces DESC

Reading the query inside out, you can see each layer answering one part of the question. The innermost subquery identifies the pods whose memory peaked above 95% of their limit, using MAX_OVER_TIME to take each pod's peak rather than its average. The middle one narrows to the requests that actually failed on those pods and reduces them to at most 1,000 trace IDs. The outer query then collects the logs for those traces from every service that took part, including services running on pods that were entirely healthy, and that last part turns out to be where the answer is.

The pods are matched on their UID rather than their name, because names repeat across namespaces and restarts. The query assumes Kubernetes metadata reaches your spans, which is what the collector's k8sattributes processor is for; if it does not, the host or container ID works the same way.

From the results above, the first row sits at exactly 1,000 because that is the subquery's LIMIT, so it describes the size of the sample rather than the size of the incident. Every trace in that sample carries the cart deadline line, which is the symptom you already knew about when you started.

Look at the second row instead. product-catalog is rejecting oversized payloads across 946 of the same 1,000 traces, and it never appeared in the pod subquery at all: its pods peaked at 75% of their memory limit, well under the 95% threshold. The rollout started sending larger payloads, which would account for both cart's memory climb and the failures. Checkout's retry budget gives out in about half of them, which is how the failure became visible to users.

Filtering logs directly by the pressured pods would have shown you the cart line and hidden the product-catalog one, which is to say it would have confirmed the symptom and buried the cause. Doing it without subqueries means three queries and two hand-built lists, and the second list is a thousand trace IDs. That is usually the point at which people stop after the first hop and go with the cart theory. The reason the second hop is cheap here is that all three signals sit in the same store behind the same query language, so widening from pods to traces to every service in the trace is a clause, not a project.

Why do ES|QL subqueries matter for AI agents?

Keeping the intermediate set inside the cluster is convenient for a person and close to essential for an agent querying on your behalf.

Split across two tool calls, the intermediate result has to travel. A list of 500 trace IDs comes back in a tool response and occupies the model's context, and the agent then has to rewrite every one of them into the next query. That costs tokens on every hop, and it is where truncation and transcription errors come from. With a subquery, the intermediate set stays inside Elasticsearch and the agent only ever sees the final table.

The problem compounds when the signals are spread across systems. An agent then needs credentials, a client, and a working knowledge of the query language for each one, plus the judgment to join results that use different names for the same host. One store and one query language reduce that to a single skill the agent has to be good at.

One ES|QL string is also a complete description of the correlation, which makes the investigation reproducible: an agent can put the query in its summary, and a human can paste it into Discover and get the same logic evaluated against current data. A two-call sequence with a hardcoded host list in the middle gives you neither. The one thing to watch is that an agent calling the _query API has to filter @timestamp itself, since nothing is binding a time picker.

Four things to know before writing ES|QL IN subqueries

IN subqueries are in technical preview in Elastic Stack 9.5 and on Elastic Cloud Serverless, while TS and FORK have been generally available since 9.4. CATEGORIZE has been generally available since 9.1 and requires a Platinum license; every query above works without it if you group by an existing field instead.

Four things are worth knowing before you write your own:

• In 9.5 the subquery returns exactly one column, which is what the trailing KEEP does in each example.
• Aggregate the subquery down to distinct values with STATS ... BY before returning them. Its result is materialized for the outer query to filter against, so handing back a few hundred host names instead of a few million rows is both faster and safer.
• Filter nulls out of any NOT IN subquery, because SQL null semantics mean one null makes the predicate match nothing.
• Subqueries are non-correlated. They run independently and cannot reference columns from the outer query, so this is a set filter rather than a row-by-row join. Reach for LOOKUP JOIN when you need per-row enrichment, which we covered in ES|QL joins for richer observability.

Try ES|QL signal correlation on your own data

The pattern under all four examples is the same. You start with a set you can describe in one signal and a question you can only answer in another, and the subquery carries that set across the boundary for you.

The syntax is the smaller part of what makes that work. It works because logs, metrics, and traces sit in one store behind one query engine, under field names they kept on the way in, so crossing from one signal to another is a clause in a query rather than an integration to build and maintain. Where that is not true, the same four investigations turn into a sequence of exports, translations, and manual joins, and that costs more than slower answers. It quietly shrinks the number of questions anyone is willing to ask, and the negative results are the first to go.

Each pattern here replaces two or three queries with one, and no host list or trace ID list has to move between them. Fewer steps mean fewer places to be wrong, and a correlation you can save as a single string and hand to someone else.

To try it:

1. Open an Observability project on Elastic Cloud Serverless, or upgrade to Elastic Stack 9.5.
2. Send data with the Elastic Distributions of OpenTelemetry, or point an existing collector at Elasticsearch.
3. In Discover, switch to ES|QL and start from the metrics to logs query above, swapping in your own data streams and thresholds.
4. Read the IN subquery reference for the full set of commands you can use inside a subquery.

That team still wants a model. So the request lands on an SRE's desk, and it sounds deceptively small: "Can you stand up an open-weight model for the claims team? Sixty people. It has to run on our hardware." They aren't even allowed to use a neocloud. There is a cost associated with this, but we won't explore that part. Just the part that covers running the model and observing the configuration.

Standing it up is the easy half. Four manifests and an afternoon, and you have a model answering questions. The hard part arrives a week later, when someone says "it feels slow" and you realize you have no idea whether the deployment is configured well, badly, or catastrophically — and no obvious way to find out.

This guide shows you how Elastic Observability can help you analyze the metrics from the configuration. It walks through tuning a real vLLM deployment using the metrics vLLM already emits. vLLM exposes these on a /metrics endpoint in Prometheus exposition format — no instrumentation, no sidecar, no code change — which is why every query in this guide starts from a Prometheus scrape. The goal: turn "it feels slow" into a specific, defensible decision.

Test environment: vLLM on a Kubernetes cluster using NVIDIA A10G with dcgm-exporter and Prometheus metrics

Every figure in this guide was measured on the following stack — one replica, one GPU, no autoscaling.

• Workload — Kubernetes-native load generator, scaled from 8 to 32 concurrent requests.
• Model — Qwen/Qwen2.5-3B-Instruct, bf16, --max-model-len 4096
• Engine — vLLM v0.23.0, OpenAI-compatible server, Prometheus /metrics on :8000
• GPU — NVIDIA A10G, 24 GB — an AWS g5.xlarge
• Cluster — Amazon EKS 1.30, tainted GPU node pool with minSize: 0
• Telemetry — Prometheus scraping every 15s, plus dcgm-exporter on :9400, shipped via remote_write
• Analysis — Elastic Observability, queried with ES|QL and PromQL
• Measured — 2026-07-27

Why is it hard for an SRE to self-host and tune an open-weight LLM?

The difficulty is not the deployment, it's the tuning which has no feedback loop. vLLM starts, serves, and reports success whether it's configured brilliantly or wastefully. Nothing tells you which.

When loading up the model, your manifest would have this configuration:

      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.23.0  # pin an exact release — metric names shift between versions
          args:
            - "--model=Qwen/Qwen2.5-3B-Instruct"
            - "--max-model-len=4096"         # cap context → predictable KV-cache size
            # A10G has native bf16 — do NOT add --dtype=half (T4-only).
          ports:
            - name: http
              containerPort: 8000            # OpenAI API + /metrics

But you can run into specific issues, such as: Hugging Face downloads take minutes. If your cluster expects a server to start in 30 seconds, it will assume the app is dead and kill it mid-download, putting you in an infinite crash loop.

Or you could have a hardware mismatch, and potentially degrade your model’s speed or precision because hardware architectures vary

or a bevy of other issues.

Once the model is finally running, optimizing performance is complete guesswork because default metrics don't tell you if you're being efficient.

You could use nvidia-smi, but this only understands raw hardware state, not application software logic.

Now that you have it running, a few hours to maybe even a day in, the team says "it feels slow." You are, functionally, tuning blind.

How do you tune a self-hosted vLLM deployment?

You're not an inference engineer, you own forty other services besides this one, and you don't have a forward-deployed engineer from a model vendor on call. But tuning an LLM server turns out to need exactly one skill you already have: reading telemetry and reasoning about saturation. The only missing piece is telemetry that exists and means something.

It does. vLLM emits a rich Prometheus endpoint out of the box — latency decomposed by inference phase, cache hit rates, batch occupancy, token accounting, completion outcomes. Almost nobody looks at it. The rest of this guide is how to read it.

Defining the workload: sixty users, short prompts, streaming responses

With the slowness detected and reported, you gather the usage profile of the users. Their usage pattern is as follows:

• ~60 users, but not concurrent. Realistic peak is 8–12 simultaneous in-flight requests; sustained is lower.
• Short prompts, long answers. The user pastes a paragraph and asks for a structured summary. Prompts run ~50 tokens; useful answers run 500–1,000.
• Interactive, streaming UI. Perceived speed is dominated by time to first token (TTFT), not total time — the same psychology as a chat interface.
• Heavy prompt reuse. Every request carries the same system prompt and the same policy-language boilerplate.

From that, you write down actual service objectives — the step most self-hosted LLM projects skip:

• TTFT p95 < 300 ms
• inter-token latency < 50 ms (≥ 20 tokens/sec, faster than reading speed)
• zero queueing at 12 concurrent
• error + abort rate < 0.5%

Those four numbers are the point of everything that follows. Without them, "it feels slow" has no answer. With them, every metric below passes or fails a stated bar.

How do you get Prometheus metrics out of vLLM and the GPU?

There are two sources, and you need both.

• vLLM reports on itself — latency by phase, cache hit rates, batch occupancy, token counts — on /metrics at its serving port, with no adapter and no instrumentation work.
• The GPU reports separately, through NVIDIA's dcgm-exporter on :9400 (NVIDIA Data Center GPU Manager (DCGM) is a suite of tools and libraries designed to comprehensively manage, monitor, and diagnose enterprise-grade NVIDIA GPUs in clusters and data centers). vLLM tells you what the engine thinks is happening; DCGM tells you what the card is actually doing. Step 6 is built entirely on the gap between those two answers.

On this EKS cluster that means three things running side by side:

• vLLM as a plain Deployment on the tainted g5.xlarge GPU node pool. For one model on one card, a Deployment and a Service is the whole architecture.
• dcgm-exporter as a DaemonSet, pinned to the same GPU nodes.
• A Prometheus server on a CPU node, scraping both endpoints every 15 seconds.

Nothing here is AWS-specific. The production version is the same manifests on an on-prem cluster with L40S or H100 nodes — which is the point of doing this on Kubernetes rather than on a vendor's platform.

What about KServe and llm-d?

Neither was run for this guide, and neither changes where the metrics come from. KServe and llm-d sit on top of vLLM rather than replacing it — vLLM is still the engine, so /metrics is still the source of every number here. Each adds its own layer on top (KServe: autoscaler and revision metrics; llm-d: router and cache-routing metrics), but the inference telemetry underneath is identical.

What they change is when you need them — and each promotion is triggered by a metric you're already collecting:

How do you ship the vLLM metrics and DCGM metrics to Observability

A Prometheus scraping inside the cluster only holds hours of data — the metrics have to reach a store you can still query next week. There are two paths for that, and they are not equivalent.

Path A — OpenTelemetry Collector. Puts inference metrics into the same pipeline as your traces and logs. One collector, one auth path, one mental model. The cost is that Prometheus metrics sent through OTLP get normalized: the stored schema ends up neither purely Prometheus nor purely OTel, and metric names shift.

Path B — native Prometheus remote_write. Stands up a small Prometheus that scrapes both endpoints and pushes to a backend speaking the remote-write protocol. Names and labels land untouched, _sum / _count / _bucket histogram parts stay intact, and existing queries keep working.

For a tuning exercise, choose Path B. That's what produced every number in this guide. The reason is narrow but decisive: tuning means comparing against the vLLM documentation and the vLLM community, and both speak in exact metric names. When your chart says vllm:kv_cache_usage_perc, you can search for it.

The deployment is two YAML files — a Prometheus Deployment with two scrape jobs and a remote_write block, plus a Secret holding the backend credential. In this build the destination was an Elastic Serverless project, which exposes a Prometheus remote-write endpoint and lands data in a time-series data stream, metrics-vllm.prometheus-inference.

Two things cost me real time. If your backend has a separate ingest host for OTLP versus its main API, remote-write usually lives on the main API host, not the ingest one — pointing at the wrong one returns a 404 that looks like a path error. And the credential needs index-write privileges, not just ingest authentication; a key that works fine for OTLP can authenticate successfully and then 403 on every sample. Check prometheus_remote_storage_samples_failed_total on the Prometheus itself before looking anywhere else.

How do you confirm vLLM metrics landed in Elastic?

Once the pipeline is up, look at the field list. Roughly 127 metric series arrive from a single vLLM pod plus DCGM:

This screen is more useful than it looks. Scanning the field list is how you confirm your vLLM version's exact metric names — they do shift between major vLLM releases, and a dashboard built against the wrong names fails silently by returning nothing rather than erroring.

Two gotchas when querying vLLM metrics

Both produce results that look like "the metrics aren't working" when the pipeline is perfectly healthy.

vLLM metric names contain a colon (vllm:num_requests_running), so they need escaping in most query languages. More insidiously, if you filter by metric name across several metrics and then aggregate only one of them, you get rows back — full of nulls, with no error. Each Prometheus metric lands in its own field, so naming the field is the filter; you don't need the name predicate at all.

Counters need rate functions, gauges don't. vllm:generation_tokens_total is cumulative and monotonic — taking a max of it gives the pod's lifetime total, not its throughput. Gauges like vllm:num_requests_running, vllm:num_requests_waiting and vllm:kv_cache_usage_perc are instantaneous and want max or average. Mixing these up produces charts that are wrong but plausible, which is considerably worse than charts that are empty.

Which vLLM Prometheus metrics actually matter?

A reference for the metrics used in this guide, what each tells you, and the condition worth watching. Names are as vLLM emits them; the DCGM_FI_* series come from dcgm-exporter.

| Metric | Type | What it tells you | Watch for | |---|---|---|---| | vllm:time_to_first_token_seconds | Histogram | TTFT — how long before the first token streams | p95 above your interactive bar (300 ms here) | | vllm:inter_token_latency_seconds | Histogram | Streaming speed after the first token | Above ~50 ms is slower than reading speed | | vllm:e2e_request_latency_seconds | Histogram | Total request time | Rising while TTFT is flat = decode or workload change | | vllm:request_queue_time_seconds | Histogram | Time waiting for admission | Earliest saturation signal — any sustained rise | | vllm:request_prefill_time_seconds | Histogram | Time processing the prompt | Dominant share = prefill-bound workload | | vllm:request_decode_time_seconds | Histogram | Time generating tokens | Dominant share = memory-bandwidth-bound | | vllm:num_requests_running | Gauge | Requests currently being decoded | Batch occupancy | | vllm:num_requests_waiting | Gauge | Requests queued for admission | Sustained non-zero = add a replica | | vllm:kv_cache_usage_perc | Gauge | Occupancy of the KV block pool — not VRAM | Autoscaling trigger (~60%) | | vllm:prompt_tokens_total + vllm:prompt_tokens_cached_total | Counters | Prefix-cache hit rate | A drop means routing scattered your prefixes | | vllm:generation_tokens_total | Counter | Output throughput in tokens/sec | Headline throughput number | | vllm:request_prompt_tokens + vllm:request_generation_tokens | Histograms | Per-request token sizes; their ratio is the workload's shape | A moving ratio means the workload changed character | | vllm:iteration_tokens_total | Histogram | Tokens advanced per forward pass | Near 1.0 with concurrency = batching broken | | vllm:request_success_total{finished_reason} | Counter | Completion outcomes | error/abort = SLO; length share = truncation | | http_requests_total{status} | Counter | Server-level requests | Catches 4xx and malformed requests vllm:* never sees | | DCGM_FI_DEV_FB_USED / _FB_FREE | Gauge | Physical VRAM | Capacity planning only — never alert on it | | DCGM_FI_DEV_GPU_UTIL | Gauge | "A kernel is resident" | Not a measure of useful work | | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE | Gauge | Tensor-core activity | Low here + high DRAM = memory-bound | | DCGM_FI_PROF_DRAM_ACTIVE | Gauge | Memory-bandwidth activity | High = the bottleneck is bandwidth | | DCGM_FI_DEV_POWER_USAGE | Gauge | Watts drawn | Pairs with throughput for tokens-per-watt |

Step 1: Where does vLLM latency go? TTFT, prefill, and decode decomposed

Decompose total latency into queue, prefill, and decode before optimizing anything. vLLM reports all three separately, and they have completely different fixes. This is the single most valuable chart in the setup.

The query averages each phase's cumulative time by request count in the same window — in Prometheus terms, rate(vllm:request_prefill_time_seconds_sum) / rate(vllm:e2e_request_latency_seconds_count), and the same for queue and decode:

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS reqs = SUM(RATE(`metrics.vllm:e2e_request_latency_seconds_count`)),
        q_s  = SUM(RATE(`metrics.vllm:request_queue_time_seconds_sum`)),
        pf_s = SUM(RATE(`metrics.vllm:request_prefill_time_seconds_sum`)),
        dc_s = SUM(RATE(`metrics.vllm:request_decode_time_seconds_sum`))
    BY minute = BUCKET(@timestamp, 1 minute)
| EVAL queue_ms   = ROUND(q_s  / reqs * 1000, 2),
       prefill_ms = ROUND(pf_s / reqs * 1000, 1),
       decode_ms  = ROUND(dc_s / reqs * 1000, 1)
| KEEP minute, queue_ms, prefill_ms, decode_ms
| SORT minute ASC

At 8 concurrent requests on the A10G:

minute    | queue_ms | prefill_ms | decode_ms | e2e_ms  | ttft_ms | inter_token_ms
22:55:00  | 0.01     | 37.98      | 1664.69   | 1715.18 | 50.81   | 16.23
22:56:00  | 0.01     | 40.99      | 1670.44   | 1723.78 | 53.50   | 16.20

Read it against the objectives:

• TTFT is 51 ms against a 300 ms target. Passing, with almost 6× headroom. Perceived responsiveness is not the problem, whatever was said in the meeting.
• Inter-token latency is 16 ms — about 62 tokens/sec against a 50 ms / 20 tok-s bar. Text arrives roughly three times faster than a person reads it.
• Queue time is 0.01 ms. Nothing is waiting for admission; the engine has capacity to spare at this concurrency.
• Decode is 1,665 ms against 38 ms of prefill — 97% of the time is decode.

That last line is the finding. Every optimization aimed at prefill is worthless for this workload. Chunked prefill, prompt compression, a faster attention kernel for long contexts — all real techniques, all irrelevant when prefill is 2% of the time. Decode is memory-bandwidth-bound, so the levers that would actually move it are quantization, tensor parallelism across two cards, or a smaller model. A single chart eliminated the wrong shopping list.

Watch vllm:request_queue_time_seconds specifically. It is the earliest saturation signal in the entire vLLM metric set — queue time climbs before vllm:num_requests_waiting becomes visibly non-zero, because a request can wait milliseconds for admission without ever registering as queued at scrape time. If you alert on one thing from this section, alert on queue time crossing a small absolute threshold.

Step 2: Is vLLM using the GPU efficiently? KV cache, prefix caching, and batch occupancy

Four metrics answer this: prefix-cache hit rate, tokens per iteration, KV-cache occupancy, and running-vs-waiting requests. Latency tells you the experience is good; these tell you whether you're overpaying for it.

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS ptok    = SUM(RATE(`metrics.vllm:prompt_tokens_total`)),
        cached  = SUM(RATE(`metrics.vllm:prompt_tokens_cached_total`)),
        gen     = SUM(RATE(`metrics.vllm:generation_tokens_total`)),
        it_s    = SUM(RATE(`metrics.vllm:iteration_tokens_total_sum`)),
        it_c    = SUM(RATE(`metrics.vllm:iteration_tokens_total_count`)),
        running = MAX(`metrics.vllm:num_requests_running`),
        waiting = MAX(`metrics.vllm:num_requests_waiting`),
        kv      = MAX(`metrics.vllm:kv_cache_usage_perc`)
    BY minute = BUCKET(@timestamp, 1 minute)
| EVAL prefix_cache_hit_pct = ROUND(cached / ptok * 100, 1),
       tokens_per_iteration = ROUND(it_s / it_c, 2),
       gen_tokens_per_sec   = ROUND(gen, 1),
       kv_cache_pct         = ROUND(kv * 100, 3)
| KEEP minute, prefix_cache_hit_pct, tokens_per_iteration,
       gen_tokens_per_sec, running, waiting, kv_cache_pct
| SORT minute ASC

minute   | prefix_cache_hit_pct | tokens_per_iteration | gen_tok/s | running | waiting | kv_cache_pct
22:55:00 | 32.5                 | 10.36                | 479.1     | 8.0     | 0.0     | 0.255
22:59:00 | 32.3                 | 10.34                | 480.0     | 8.0     | 0.0     | 0.121

A 32% prefix-cache hit rate is a third of all prefill work simply not done. The claims team's requests share a system prompt and policy boilerplate, and vLLM's automatic prefix caching recognizes that. This is a direct argument for raising prompt standardization: the more the application puts shared context in a consistent leading position, the higher this climbs and the cheaper every request gets. It is also the number that will crater the day a naive round-robin load balancer sits in front of two replicas — precisely the condition that justifies llm-d's cache-aware routing.

tokens_per_iteration ≈ 10.3 with 8 concurrent requests is continuous batching working correctly. Each forward pass through the model advances about ten sequences at once. If this sat near 1.0 with multiple requests in flight, batching would be broken and you'd be paying full model-forward cost per token per user. This metric proves you're getting vLLM's core value.

What does vllm:kv_cache_usage_perc actually measure?

vllm:kv_cache_usage_perc reports occupancy of vLLM's pre-allocated KV block pool — not physical GPU memory. At startup, vLLM reserves a fraction of VRAM (governed by --gpu-memory-utilization, default 0.9) and carves a KV block pool out of that reservation. This gauge reports how full that pool is.

That's why it read 0.25% here. Eight concurrent requests holding ~150 tokens each barely touch an A10G's block budget. The pool is large, and correctly so. Push the same server to 32 concurrent requests with 512–1,024 token generations and it moves — to about 2.8%. Still small.

The instinct is to read a number that low as "the cache is broken" or "I've massively over-provisioned." Both are wrong. Treating this gauge as a VRAM proxy is the most common self-hosted vLLM configuration error I see. Step 6 shows exactly how far apart the two are.

Step 3: What shape is your vLLM inference workload?

Confirm the workload is what you think it is before tuning anything. This is the panel that explains a latency "regression" that isn't your fault.

minute   | requests_per_min | avg_prompt_tokens | avg_generated_tokens | avg_max_tokens | gen_to_prompt_ratio
22:55:00 | 282.2            | 49.6              | 103.6                | 103.6          | 2.09
23:01:00 | 276.0            | 50.6              | 103.0                | 103.0          | 2.04

The generation-to-prompt ratio is 2.04 — this workload writes twice as much as it reads. That single ratio is the explanation for Step 1's 97%-decode finding, and it holds for any summarize-and-draft use case. If the team later adds a long-document RAG feature, prompts jump to thousands of tokens, the ratio inverts, the workload becomes prefill-bound, and the correct tuning changes completely. Watching this ratio is how you learn your workload changed character before someone files a ticket.

Now the column that should bother you: avg_generated_tokens equals avg_max_tokens exactly. Every request is stopping because it hit its token ceiling, not because the model finished its thought. The screenshot shows the same pattern holding as generation lengths scale to ~685 tokens against a ~777 ceiling.

In a load test that's an artifact of the generator. In production, that number is users getting cut off mid-sentence — and it is invisible in every latency metric you have. Which brings us to the metric that catches it.

Step 4: Are vLLM requests actually succeeding? Checking finished_reason

Break vllm:request_success_total down by its finished_reason label. This is the closest thing self-hosted inference has to an application-level SLI, and it catches a failure mode no latency chart can.

TS metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS completions_per_min = ROUND(SUM(RATE(`metrics.vllm:request_success_total`)) * 60, 2)
    BY minute = BUCKET(@timestamp, 1 minute), finish_reason = labels.finished_reason
| SORT minute ASC, finish_reason

Five outcomes, each meaning something different operationally:

| finished_reason | What it means | What to do about it | |---|---|---| | stop | The model finished naturally | This is the number you want large | | length | Truncated at max_tokens | High share means users are cut off — raise the ceiling, or shorten the ask | | abort | The client disconnected first | Users giving up, or a proxy timeout shorter than your generations | | error | The engine failed | Your hard SLO signal. Should be flat zero | | repetition | Degenerate looping output | A sampling-parameter problem, not an infrastructure one |

Under the small load generator the split was 100% length — expected, since it requested a fixed ceiling. In the screenshot, at a mixed load, stop and length run side by side at roughly 51 and 32 completions/min. That mix is the healthy shape: most requests finishing on their own, a minority hitting the ceiling.

The lesson generalizes. error and abort are what you page on. But the stop-to-length ratio is what you review weekly, because drift toward length means answers are being truncated and no latency dashboard on earth will tell you.

One blind spot to close: vllm:* metrics only count requests the engine accepted. Malformed JSON, 4xx, auth failures and dropped connections never reach it. Those live in http_requests_total with status and handler labels — worth a panel beside this one, because "the model is broken" reports frequently turn out to be the gateway in front of it.

Step 5: What NVIDIA DCGM metrics say the GPU is actually doing

Use NVIDIA DCGM as an independent witness to vLLM's own account. Everything so far is the engine describing itself; DCGM describes the silicon.

FROM metrics-vllm.prometheus-inference
| WHERE @timestamp > NOW() - 30 minutes
| STATS gpu_util_pct    = MAX(`metrics.DCGM_FI_DEV_GPU_UTIL`),
        mem_bw_util_pct = MAX(`metrics.DCGM_FI_DEV_MEM_COPY_UTIL`),
        vram_used_mib   = MAX(`metrics.DCGM_FI_DEV_FB_USED`),
        vram_free_mib   = MIN(`metrics.DCGM_FI_DEV_FB_FREE`),
        power_w         = ROUND(MAX(`metrics.DCGM_FI_DEV_POWER_USAGE`), 1),
        temp_c          = MAX(`metrics.DCGM_FI_DEV_GPU_TEMP`)
    BY minute = BUCKET(@timestamp, 1 minute)
| SORT minute ASC

→ 100% GPU util · 21,483 MiB used / 1,352 MiB free · 240 W · 73 °C

100% utilization and 94% VRAM. Under the conventional reading, this card is maxed out and it's time to ask for more hardware. That reading is wrong.

Why is GPU utilization a misleading metric for LLM inference?

DCGM_FI_DEV_GPU_UTIL means "a kernel is resident on the device," not "the device is doing useful work." It reads 100% for a perfectly-tuned server and 100% for a badly-tuned one, so it cannot distinguish them. The DCGM profiling counters can:

gr_engine_active 99.8%  ·  tensor_active 16.6%  ·  dram_active 80.4%

Read those three together. The GPU's compute engine is busy essentially all the time — but its tensor cores, the units that do the actual matrix math, are active only 16.6%, while DRAM is active 80.4%. The card is not computing. It is waiting on memory.

This is independent, hardware-level confirmation of what Step 1 inferred purely from timings: decode is memory-bandwidth-bound. Two entirely different instruments, two different layers of the stack, one conclusion — the difference between a hypothesis and a finding.

It also permanently retires GPU utilization as a capacity metric for LLM inference. If your GPU capacity planning rests on DCGM_FI_DEV_GPU_UTIL — and most does — it rests on nothing.

Step 6: vLLM KV cache vs GPU VRAM, and why they disagree

Put vllm:kv_cache_usage_perc and physical VRAM usage on one 0–100% axis. They describe the same GPU memory, they sit at opposite ends of the chart, and both are correct.

minute   | running | gen_tok_s | kv_cache_pct | vram_used_pct | gpu_util | dram_active_pct | tensor_active_pct | tokens_per_watt
00:05:00 | 31      | 1627.5    | 2.82         | 94.1          | 100.0    | 80.4            | 16.6              | 6.80

KV cache at 2.8%. VRAM at 94.1%.

vLLM pre-allocates a large fraction of VRAM at startup — governed by --gpu-memory-utilization, default 0.9 — and carves its KV block pool out of that reservation. vllm:kv_cache_usage_perc reports occupancy of the pool. DCGM reports what the driver sees, which is the whole reservation, whether or not it's holding anything.

The operational consequences are precise, and they're the practical payoff of the entire exercise:

• Autoscale on vllm:kv_cache_usage_perc and vllm:num_requests_waiting. These describe admission capacity — whether the engine can take another request right now.
• Capacity-plan on VRAM. This describes physical space — whether a second model could ever fit on this card. (It can't. 1.3 GB free.)
• Never alert on VRAM. It will page you at 3 a.m. for a healthy, mostly-idle server, every single night, forever.

And tokens_per_watt — generated tokens divided by power draw, 6.8 here — is a genuine cost-efficiency metric. It's comparable across GPU models, batch settings and quantization levels in a way that neither latency nor utilization is. When you go back to Finance for card number two, this is the number that makes the argument: at 32 concurrent we sustain 1,627 tokens/sec at 240 watts, and here's what that becomes on an L40S.

vLLM tuning decisions: what the SRE does with these Prometheus metrics

Six steps, thirty minutes, one server. The verdict against the stated objectives:

| Objective | Measured | Verdict | |---|---|---| | TTFT p95 < 300 ms | 51 ms | Pass, 6× headroom | | Inter-token < 50 ms | 16 ms (≈62 tok/s) | Pass | | Zero queueing at 12 concurrent | queue_ms 0.01, waiting 0 at 8; still 0 at 32 | Pass, large margin | | Error + abort < 0.5% | 0% | Pass |

The configuration is correct for this department, and the department is over-provisioned rather than under-provisioned. That's a defensible, evidence-backed answer to "it feels slow" — and it redirects the investigation to the app, the gateway, or the prompt, which is where the problem actually is.

The concrete follow-ups, each tied to a metric rather than a hunch:

1. Stop optimizing prefill. — vllm:request_decode_time_seconds vs vllm:request_prefill_time_seconds. Decode is 97% of the time against prefill's 2%, confirmed twice. Chunked prefill and prompt compression are off the table for this workload.
2. If more throughput is needed, quantize before buying hardware. — DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (16.6%) vs DCGM_FI_PROF_DRAM_ACTIVE (80.4%). The bottleneck is memory bandwidth, not compute, so an FP8 or AWQ build of the same model is the highest-leverage single change: it moves fewer bytes per token, which is exactly the constrained resource.
3. Raise the client-side max_tokens ceiling. — vllm:request_success_total{finished_reason}. Every request finishing on length rather than stop is a user getting cut off mid-answer. This is the one finding that's a live user-experience defect. You need to increase the prompt max_token limit.
4. Standardize the prompt prefix. — vllm:prompt_tokens_cached_total over vllm:prompt_tokens_total, 32% today. But it should be better (more like 70%) More shared boilerplate in a consistent leading position raises it, and it's free.
5. Set the autoscaling trigger now, before it's needed. — vllm:kv_cache_usage_perc and vllm:num_requests_waiting. Scale when the first crosses ~60% or the second stays above zero. Do not scale on DCGM_FI_DEV_GPU_UTIL — it's pinned at 100% regardless.
6. Alert on queue time, not on VRAM. — vllm:request_queue_time_seconds is the earliest true saturation signal; DCGM_FI_DEV_FB_USED is a constant that looks like an emergency.
7. Revisit when the workload changes shape. — vllm:request_generation_tokens over vllm:request_prompt_tokens, 2.04 today. When the RAG feature ships that ratio inverts, the workload becomes prefill-bound, and half of this analysis needs redoing. The chart tells you the day it happens.

Notice that most of the metrics are looking at the vLLM metrics not the GPU metrics in helping optimize. These are still within the limit of a single service, but when you get vllm:num_requests_waiting to persistently non-zero, then you need to run KServe or you can use KEDA and HPA autoscaling. But you get the metrics to help you determine or allow KServe to scale. So you can see that understanding these metrics are crucial to tuning the inference service.

Elastic Observability can provide this to you.

Why self-hosted LLM tuning is an SRE problem, not an ML problem

Self-hosting an open-weight model is not primarily an ML problem. It is a capacity and saturation problem — something SREs have been extremely good at for twenty years. The blocker was never skill. It was that the telemetry sat unexamined on a /metrics endpoint nobody scraped, in a schema nobody had mapped to the questions they actually had.

Once it's collected, the reasoning is familiar work in unfamiliar clothes:

• Decompose latency by phase before optimizing anything (queue / prefill / decode).
• Distinguish the logical resource from the physical one (KV block pool ≠ VRAM), and know which each metric describes.
• Never trust a single-source utilization number — corroborate the engine's account with the hardware's.
• Tie every knob to a metric and every metric to a stated objective, so tuning converges instead of wandering.

For a team that isn't allowed to send its data anywhere, that difference — between running a model and operating one — is the whole ballgame. The department gets a capability it's otherwise locked out of, and the SRE gets to answer questions about it with numbers.

Frequently asked questions

What does vllm:kv_cache_usage_perc measure? It measures occupancy of vLLM's pre-allocated KV block pool, not physical GPU memory. vLLM reserves a fraction of VRAM at startup (--gpu-memory-utilization, default 0.9) and carves the KV pool from that reservation. In this deployment it read 2.8% while DCGM reported 94.1% VRAM used on the same card at the same moment. Use it as an autoscaling signal; use VRAM for capacity planning.

Why is my vLLM deployment decode-bound? Because the workload generates more tokens than it reads. Compare vllm:request_decode_time_seconds against vllm:request_prefill_time_seconds, and check the generation-to-prompt token ratio. In this deployment the ratio was 2.04 — twice as many output tokens as input — which produced 1,665 ms of decode against 38 ms of prefill. Decode is memory-bandwidth-bound, so quantization, tensor parallelism, or a smaller model help; prefill optimizations do not.

Should I autoscale vLLM on GPU utilization? No. DCGM_FI_DEV_GPU_UTIL means a kernel is resident on the device, not that the device is doing useful work — it reads 100% for both a well-tuned and a badly-tuned server. Autoscale on vllm:kv_cache_usage_perc (around 60%) or on vllm:num_requests_waiting staying above zero, since those describe whether the engine can admit another request.

Why does my GPU show 100% utilization when it isn't fully used? Because GPU utilization only reports kernel residency. Check the DCGM profiling counters instead: in this deployment DCGM_FI_PROF_GR_ENGINE_ACTIVE was 99.8% while DCGM_FI_PROF_PIPE_TENSOR_ACTIVE was only 16.6% and DCGM_FI_PROF_DRAM_ACTIVE was 80.4%. That combination means the GPU is waiting on memory bandwidth rather than computing.

How do I get vLLM metrics into Prometheus? vLLM already exposes Prometheus exposition format on /metrics at its serving port — no adapter or instrumentation needed. Point a Prometheus scrape job at the vLLM Service, add a second job for dcgm-exporter on :9400, and use remote_write to ship to long-term storage. Sending through an OpenTelemetry Collector also works but normalizes the metric names, which makes them harder to match against vLLM documentation.

What TTFT should I target for an interactive LLM application? For a streaming chat-style interface, a p95 time-to-first-token under 300 ms feels immediate, and inter-token latency under 50 ms (about 20 tokens/sec) outpaces reading speed. This deployment measured 51 ms TTFT and 16 ms inter-token latency on a 3B model on a single NVIDIA A10G, leaving roughly 6× headroom.

Why are all my vLLM requests finishing with length? Because they're hitting the max_tokens ceiling instead of the model choosing to stop. Break vllm:request_success_total down by its finished_reason label: a high length share means answers are being truncated mid-sentence. This is invisible in every latency metric, so review the stop-to-length ratio regularly and raise the client-side ceiling if it drifts.

When should I move from a plain vLLM Deployment to KServe or llm-d? Move to KServe when vllm:num_requests_waiting is persistently non-zero at peak and you need replicas to appear without human intervention. Move to llm-d when your prefix-cache hit rate collapses across replicas — a sign that load balancing scattered conversations that shared a prefix — or when prefill time starts stealing measurably from decode.

The challenge deepens when something goes wrong. A crash on the device, a slow screen, or an error response might originate in the Android app itself, in a backend service, or somewhere in the network path between them. Debugging these problems without a connected, E2E view from the mobile client to the backend is time-consuming and frustrating. And without a standard instrumentation format, mobile teams often end up maintaining separate tooling that doesn't integrate with what the backend and infrastructure teams already use.

OpenTelemetry offers a way out: a unified, open-standard instrumentation model that works across platforms and languages, backed by a large community. The Elastic Distribution of OpenTelemetry Android, or EDOT Android, is an APM agent for native Android applications built on top of OpenTelemetry. It gives Android teams a practical path to observe mobile app behavior in Elastic, providing them with distributed tracing, crash reporting, session tracking, disk buffering, and automatic instrumentation, with as little code as possible while staying grounded in open standards.

To see what it all looks like, we will instrument a demo Android weather application end to end. You will run Elasticsearch, Kibana, and the Elastic Agent locally. The Elastic Agent provides the OTLP endpoint that receives telemetry from the Android app and backend. You will then generate distributed traces, custom spans, logs, and Android crashes from the app, and explore the results in Kibana using the Android OpenTelemetry dashboards.

This article focuses on a hands-on experiment to explore the E2E experience of observing Android apps with Elastic, using the EDOT Android agent. For more specific details on the EDOT Android agent, such as a list of supported features and a setup guide for your own Android project, take a look at EDOT Android docs.

Setting up EDOT Android with Elasticsearch and Kibana

We will use the EDOT Android demo application. The demo is intentionally small but covers the main workflows you need when evaluating mobile observability with Elastic.

The demo has two main components: an Android app, and a Spring Boot backend. Additionally, you'll need an Elastic Stack environment up and running; we'll explain more about how to get one later in this guide.

Prerequisites

• Java 17 or higher.
• Docker.
• Android Studio.
• An Android emulator.
• On Windows, use Windows Subsystem for Linux (WSL) to run the demo scripts.

Step 1: Clone the demo app's repository

We'll start by cloning the EDOT Android demo application:

git clone git@github.com:elastic/android-agent-demo.git

Step 2: Start the Elastic Stack

The demo uses start-local to run Elasticsearch, Kibana, and the Elastic Agent with a single command. In this setup, the Elastic Agent provides the OTLP endpoint that receives telemetry from the application and backend. Run this from the directory where you want the local Elastic files to be created:

curl -fsSL https://elastic.co/start-local | sh -s -- --edot

For more information on this step, take a look at the demo app's instructions.

Step 3: Start the local backend

The demo backend is a Spring Boot service instrumented with the EDOT Java agent. It handles the app's weather requests and calls the Open-Meteo public API for weather data.

./backend-manager start

For more information on managing the backend service, take a look at the demo app's instructions.

Step 4: Launch the Android application

Use Android Studio to open up the EDOT Android demo application repo and run the application in your emulator. More info on how to run Android apps from Android Studio here.

Generating distributed traces, errors and crashes from an Android app

The Android app has two screens: a city selector and a weather display screen that shows the current weather for the selected city on the previous screen. It includes two intentional failure paths: the first one is reached by selecting New York, which causes the backend to reject the request (the demo backend only supports European cities), and tapping the floating crash button intentionally crashes the app so you can review crash reporting in Kibana after relaunch. We'll take a look at those use cases in more detail below.

Tracing a successful request end to end

In the EDOT Android demo app, selecting "Paris" as the city triggers a successful backend request on the second screen, for which a span will be automatically generated using EDOT Android's OkHttp auto-instrumentation, which supports all OkHttp-generated HTTP requests and tools using it, such as Retrofit. Aside from the Android HTTP span, the successful city request continues e2e and creates a backend HTTP client span to Open-Meteo.

How backend errors appear in the Android distributed trace

The demo backend only supports European cities, so selecting "New York" causes it to fail, which in turn automatically creates an error associated with our Android app's HTTP span. This is done automatically. We'll see later how to find and inspect these issues from Kibana.

How EDOT Android captures and reports app crashes

The crash button creates a crash event that appears in Kibana after the app is reopened. This event contains session information that will help us narrow down its root cause from Kibana, as we'll see later.

Note: EDOT Android automatically attaches Android session context to spans and logs. That means that any span or log created before the crash can be reviewed together with the crash event and nearby spans from the same session, giving you a complete picture of what the user was doing. This even applies to manually created spans and logs.

Visualizing Android application monitoring data in Kibana

To see the whole story from our Android app in a single place, we'll install Kibana's Android OpenTelemetry Assets package by following the steps below.

1. In Kibana, search for "Android OpenTelemetry Assets" in the global search field.
2. Open it and click Install to add the Android dashboards to your Kibana instance.

Exploring Android application monitoring dashboards in Kibana

Once the content package is installed, open the [Android OTel] Application Overview dashboard:

1. In Kibana, search for Dashboards in the global search field or in Kibana's menu.
2. In Dashboards, search for Android OTel and open the "[Android OTel] Application Overview" dashboard.
3. Select your application by clicking its name in the Applications table of the Overview section.

The dashboard is organized into four collapsible sections: Overview, Logs & Spans, Crashes, and Event timeline. Together they provide metric panels for an overview of your app's health, performance, and RUM, as well as span and crash tables that can be further explored either in Discover or in the Exception Details dashboard, as explained below.

How to inspect the distributed tracing waterfall in Kibana

From the Application Overview dashboard, expand the Logs & Spans section, go to one of the span tables (either All spans or Failed spans) and click its Explore in Discover button.

In Discover, click the expand icon on the left side of any span row to open its details panel. The trace waterfall UI appears inside, showing the full span hierarchy and timing for that trace. You can expand the waterfall to full screen and drill down from there.

Analyzing failed spans and backend errors in Kibana

While you can find all kinds of spans in the dashboard's All spans panel, you can narrow them down to failed ones only by exploring the Failed spans panel instead.

For the New York path, find a failed span and expand it. The trace waterfall highlights the backend error, and the exception details show the intentional backend rule that only supports European cities.

Reviewing crash details and stack traces in the exception dashboard

Crash reporting is provided by the crash automatic instrumentation plugin. When an unhandled exception crashes the app, EDOT Android stores the crash event on disk. The event is exported the next time the app starts. Disk buffering ensures the crash event is not lost even if the network was unavailable at the time of the crash.

In the Application Overview dashboard, expand the Crashes section. Its table lists crashes grouped by a group ID computed from their stack trace. Click a group ID and select View crash details to open the [Android OTel] Exception Details dashboard.

The Exception Details dashboard shows a set of metrics to better understand the impact of the selected crash, as well as its full stack trace in the Stack trace section. Crash events are grouped based on their stack trace, which helps ensure that the same crash is counted and aggregated in this dashboard to better understand a single crash's impact on your application.

For this demo, the stack trace points to the intentional crash in MainActivity. The nearby session events should also include the custom Crash button clicked log created just before the crash, which helps explain how the crash was triggered. We'll take a look at how to inspect a session to get an idea of the user's journey within your application that led them to a crash.

Using sessions to understand user flow in EDOT Android

Mobile troubleshooting often starts with a single bad outcome (a crash, an error, a slow UX), but the useful question is what happened before that outcome. EDOT Android helps answer that by attaching session.id to every span and log emitted by the application, even for manually created ones.

A session is meant to cover a single user interaction with your application. A new one is created when there is no previous active session or when the previous session has expired. Sessions expire after 30 minutes of inactivity. If the app stays active, a session can last up to 4 hours.

This lets you query all the telemetry from a single session and review it in order. After finding a crash group, drill into one affected session from the Top affected sessions panel and review the event timeline. You can see the custom logs, app startup spans, HTTP request spans, and crash data together in one investigation path.

The Event timeline section on the Application Overview dashboard is also useful here: select a session from the dashboard's top filters, and its First 100 events panel shows the sequence of spans and logs in that session chronologically.

What EDOT Android adds as a mobile APM for production apps

The demo uses a local stack and simple code, but the same agent features apply to production apps.

Disk buffering stores telemetry locally before export. This reduces data loss when the device has poor connectivity or the app is temporarily offline.

Automatic instrumentation creates telemetry for supported targets without adding code around every call. Today that includes OkHttp, crash reporting, and an adapter for OpenTelemetry Android instrumentation.

Manual instrumentation lets you add spans, logs, and metrics for app-specific workflows. This is useful for screen loading times, checkout flows, login steps, feature usage, or any area where framework-level telemetry alone is not enough.

Central configuration can remotely adjust selected EDOT Android behavior through Kibana when the OpAMP endpoint is configured. At the time of writing, central configuration for EDOT Android is in preview and supports settings such as recording and session sample rate.

Distributed tracing connects Android app requests to backend service spans so you can trace the full path of any user action, from the tap on the screen to the database query on the server. EDOT Android ensures that your application's telemetry timestamps are in sync with the coordinated universal time. This ensures a proper trace waterfall hierarchy later on in Kibana, where different components are properly coordinated in time.

Clean up the demo

When you are finished, stop the backend in case you're planning to restart it later, or uninstall it otherwise:

./backend-manager stop
# ./backend-manager uninstall

Then stop or uninstall the local Elastic Stack:

cd elastic-start-local
./stop.sh
# ./uninstall.sh

Getting started with EDOT Android in your own app

EDOT Android gives native Android teams an OpenTelemetry-based path for mobile APM in Elastic. With a small Gradle setup and one early initialization call, you get automatic HTTP spans, crash reporting, session tracking, and direct access to the OpenTelemetry SDK for custom telemetry, and you can see it all tied together in Kibana's Android dashboards.

Observability is a crucial part of modern mobile development. Crashes, slow screens, and backend errors all impact real users, and the sooner you can identify root causes across the full request path, from the device to the database, the better. The demo app is a good first step because it exercises the complete workflow without requiring a production deployment. After that, the same setup model applies to your own app with production endpoints, API key authentication, and custom spans and logs tailored to your use cases.

Developer resources:

• EDOT Android demo application
• EDOT Android documentation
• EDOT Android getting started guide
• EDOT Android automatic instrumentation
• EDOT Android manual instrumentation
• EDOT Android configuration
• EDOT Android troubleshooting
• Android OpenTelemetry Assets dashboard docs

Don't have an Elastic Cloud account yet? Sign up for Elastic Cloud and try out mobile observability with EDOT Android as described in this guide. We'd love to hear about your experience gaining visibility into your Android application stack with Elastic.

What you need

• An Observability serverless project, or an Elastic Cloud Hosted or self-managed stack at version 9.4 or later. PromQL is generally available in Elastic Cloud Serverless and Elastic Stack 9.5, and available as a technical preview in Elastic Stack 9.4.
• Kubernetes state and container memory metrics in Elasticsearch.

What is the alert telling us?

This is the alert that opened the investigation:

It comes from this waiting-reason query:

PROMQL
  max by (namespace, pod, container) (
    max_over_time(
      kube_pod_container_status_waiting_reason{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api",
        reason="CrashLoopBackOff"
      }[2m]
    )
  ) == 1

A result of 1 means Kubernetes is delaying another start because the container has failed repeatedly. That is the correct paging signal here because the checkout path has a single replica: when that replica restarts, requests fail.

The max_over_time(...[2m]) range keeps the alert tied to recent samples. Without it, the last observed value of 1 can outlive the pod, and the rule keeps matching after that pod is gone.

That PromQL query ran every minute over a two-minute window and created an alert after one matching run.

Why did the last container stop?

The alert shows what Kubernetes is doing now. It does not show how the previous container ended:

PROMQL
  max by (namespace, pod, container) (
    kube_pod_container_status_last_terminated_reason{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      reason="OOMKilled"
    }
  ) == 1

A result of 1 for the same namespace, pod, and container means the last recorded exit was out of memory. Kube-state-metrics keeps that last reason as a gauge, so the value can stay visible after recovery. It points the investigation at memory; it does not prove that every restart in the window was an OOM kill.

Is the failure repeating?

A single restart can still be transient. The restart counter shows whether the failure keeps happening:

PROMQL
  max by (namespace, pod, container) (
    increase(
      kube_pod_container_status_restarts_total{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api"
      }[10m]
    )
  )

increase() shows how much the restart counter rose over the selected range. Repeated increases during the incident window explain why Kubernetes entered backoff.

How close is memory to the limit?

We need to know how close the container is to its memory limit, and whether that gap collapses right before each restart. This deployment allows only 128MiB, so the next query divides working-set memory by that configured limit:

PROMQL
  max by (namespace, pod, container) (
    container_memory_working_set_bytes{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      image!=""
    }
  )
  /
  max by (namespace, pod, container) (
    container_spec_memory_limit_bytes{
      namespace="checkout",
      pod=~"checkout-api-.*",
      container="api",
      image!=""
    }
  )

The chart shows a repeating sawtooth: memory approaches 90% of the limit, drops when the process stops, and climbs again after each restart.

Working set is the better signal here than total usage. Total usage includes reclaimable file cache, so it can sit near the limit without a kill. Working set is closer to the memory that triggers OOMKilled for this workload.

Is the node under memory pressure?

OOMKilled can mean the container hit its own limit, or the node ran low on memory and Kubernetes started reclaiming. To separate those cases, first find which node runs the pod, then check whether that node (or any peer) reported MemoryPressure.

PROMQL
  max by (namespace, pod, node) (
    kube_pod_info{
      namespace="checkout",
      pod=~"checkout-api-.*"
    }
  ) == 1

The pod sits on ip-10-0-2-18.ec2.internal. That is the node whose MemoryPressure result matters most for this incident:

PROMQL
  max by (node) (
    max_over_time(
      kube_node_status_condition{
        condition="MemoryPressure",
        status="true"
      }[30m]
    )
  )

Every node returns 0, including ip-10-0-2-18.ec2.internal. So the host was not under node-wide memory pressure. The kill came from the container limit itself.

Does raising the limit clear the alert?

checkout-api was healthy, then began building an in-memory cache that grows to 200MiB in 10MiB steps. The container only allows 128MiB, so the process is killed with OOMKilled before that cache is fully allocated.

We will raise the memory limit to 512MiB so the 200MiB cache fits with room for the runtime, then check whether CrashLoopBackOff clears:

kubectl set resources deployment/checkout-api -n checkout --limits=memory=512Mi

The same waiting-reason query then stops matching. CrashLoopBackOff drops off:

PROMQL
  max by (namespace, pod, container) (
    max_over_time(
      kube_pod_container_status_waiting_reason{
        namespace="checkout",
        pod=~"checkout-api-.*",
        container="api",
        reason="CrashLoopBackOff"
      }[2m]
    )
  ) == 1

And the alert that started this investigation? Gone.

That is how you detect and investigate a Kubernetes CrashLoopBackOff with PromQL: from the firing alert, through OOMKilled and the limit mismatch, to a recovered alert. Elasticsearch holds the metrics; Kibana runs the same PromQL queries you already know from Prometheus.

Try it

1. Open an Observability project on Elastic Cloud Serverless, or use Elastic Stack 9.4 or later.
2. In the ES|QL editor in Kibana, run the waiting-reason query against a workload you care about.
3. Follow the same path from that alert to termination reason, restarts, memory versus the limit, and recovery.

For more PromQL in Elastic, see Elasticsearch supports PromQL, PromQL queries in Kibana, and Investigate Kubernetes infrastructure with PromQL.

If you've been an SRE for any length of time, you know what usually comes next. You acknowledge the page, open your observability tool, and start the process: pull up the cluster dashboard, find the namespace, find the pod, check restart counts, pivot to logs, check whether an upstream dependency is degraded, compare against last week, and slowly assemble a picture from a dozen tabs. It works, but it's a process, and the process is where the minutes go.

Elastic's new Kubernetes Experience changes the starting point. When the CrashLoopBackOff alert fires, an Investigation Workflow runs automatically alongside it. By the time you open the alert, the evidence has already been gathered and a root-cause hypothesis is waiting for you. Instead of a blank dashboard, you open the alert to an answer. Or at minimum, a strong starting point that tells you exactly where to look next.

This post walks through a typical CrashLoopBackOff scenario end to end. The sections that follow break down what Elastic's UI shows at each step and why it saves you time.

The manual CrashLoopBackOff investigation: six steps every SRE runs

Here's the shape of a normal CrashLoopBackOff investigation, minus Elastic's workflow:

1. You get paged. Something restarted too many times.
2. You orient. Which pod? Which namespace? Which deployment owns it?
3. You characterize. How many restarts? What was the last termination reason — OOMKilled, a failed liveness probe, a bad exit code?
4. You classify. Is this a memory problem, a config problem, a dependency problem, a scheduling problem?
5. You corroborate. Pull Kubernetes events, read the pod logs, check whether an upstream service started erroring first, compare current behavior against a healthy baseline.
6. You conclude. Only now can you form a hypothesis and act.

Every one of those steps is a query, a click, or a context switch. None of them is hard on its own. Together, on a bad night, they're twenty minutes you don't have and they're twenty minutes of the same steps you ran during the last incident, and the one before that.

The insight behind Elastic's Kubernetes Experience is simple: that sequence is deterministic enough to automate. So Elastic automated it.

Alert to Root cause in minutes

Elastic's Kubernetes integration ships pre-built alert rule templates for states that are wrong by definition; no baseline or warmup required. A pod in CrashLoopBackOff is always a problem, so the rule fires the moment the restart count crosses your configured threshold within a rolling window.

Here's the alert firing in this scenario, both the [Kubernetes OTel] Pod CrashLoopBackOff and OOMKilled containers rules light up for the recommendation group in the otel-demo namespace:

The alert itself is defined by an ES|QL query, so it's transparent and tunable; you can read exactly what triggers it and adjust the threshold to match your environment. And its action is what makes the rest of this post possible: the rule runs the K8s CrashLoopBackOff Investigation (OTel) workflow per alert, the instant a new alert fires. (For a refresher on the alert library and how these templates work, see Part 1.)

In this scenario the rule fires on the recommendation pod (recommendation-788dc88c6c-56w74) in the otel-demo namespace. When you open the alert, Elastic's AI Agent is already summarizing what happened; no rule-details spelunking required:

But the alert firing is only half the story. Attached to it is a Kubernetes Investigation Workflow (technical preview), a directed graph of steps that triggers the instant the alert does. While your phone is still buzzing, the workflow is already querying your cluster, branching on what it finds, and synthesizing an answer.

Inside the investigation: what the workflow actually does

The workflow mirrors the exact sequence an experienced SRE would run by hand, except it runs in seconds and writes nothing down that it can't back up with evidence. For our CrashLoopBackOff scenario, here's the path it takes.

Step 1: What does the workflow check first on the crashing pod?

The workflow queries Kubernetes metrics for the restart count, the last termination reason, and utilization against the pod's declared limits.

Result: last termination reason OOMKilled, restart count 7. Memory utilization data happened to be unavailable at query time, but the OOMKilled reason is definitive: the container is being killed by the kernel for exceeding its memory limit on each startup, then immediately restarting.

The OOMKilled termination reason determines everything that follows. The workflow branches down the memory-investigation path rather than the log-investigation path it would take for a non-memory crash.

Step 2: How does the workflow tell a memory leak from a load spike?

The ML anomaly check is the step that separates a good investigation from a fast-but-wrong one. OOMKilled does not automatically mean "memory leak." Rather than recompute memory trends from scratch, the workflow queries the ML anomaly index for an active k8s_pod_memory_growth anomaly on this pod.

Result: no anomaly. The memory spike is flagged as load-driven, not a suspected leak. The ML baseline — established over the preceding days — didn't see the slow, creeping growth trajectory that characterizes a leak. It saw a jump consistent with real traffic.

Distinguishing a load-driven memory spike from a genuine leak would take a human several minutes and a good deal of judgment. The workflow reaches it because Part 1's anomaly detection jobs were already learning the workload's baseline in the background.

Step 3: Is the failure spreading to other Kubernetes services?

A crashing pod is often a symptom, not a cause, and it can also cause problems downstream. So the workflow enumerates the pod's dependencies from APM service_destination aggregates and compares the current error rate and latency against baseline. An AI classification step decides whether the failure is spreading.

Result: the sole direct caller is the frontend service, which is absorbing the impact with just a 0.14% error rate against the recommendation service and no other service exceeds its degradation threshold relative to baseline. The blast radius is isolated to the recommendation service; there's no significant downstream cascade. The problem is local to this pod.

Step 4: Did a recent change in the namespace cause the crash loop?

Finally, the workflow scans the namespace event log. It finds a continuous Pulled → Created → Started → Killing → BackOff cycle running from roughly 18:51 to 18:54 UTC, the textbook signature of an active crash loop at the time the alert fired. Nothing changed operationally; this is a steady-state resource problem.

What the CrashLoopBackOff root-cause hypothesis looks like

When you open the alert, this is what greets you:

ROOT CAUSE HYPOTHESIS (confidence: high)

The recommendation service pod (recommendation-788dc88c6c-56w74) is in a
crash-loop caused by repeated OOMKilled terminations. The pod has restarted
7 times and Kubernetes events confirm a continuous BackOff/restart cycle
since at least 18:51 UTC. Memory utilization data was unavailable at query
time, but the OOMKilled termination reason is definitive: the container is
exceeding its configured memory limit on each startup, being killed by the
kernel, and immediately restarting. No memory leak was detected by ML
anomaly analysis, indicating the memory pressure is load-driven — the
container's memory limit is simply insufficient for the current request
volume. The frontend service (the sole direct caller) is absorbing the
impact with a 0.14% error rate on the recommendation service itself, but
no significant downstream cascade is observed.

EVIDENCE
- Pod restarted 7 times; last termination reason: OOMKilled — container is
  consistently exceeding its memory limit
- ML memory anomaly check: no anomaly found; memory spike assessed as
  load-driven, not a leak
- Blast radius is isolated to the recommendation service; no other service
  exceeds degradation thresholds relative to baseline
- Continuous BackOff events from 18:51–18:54 UTC confirm active crash-loop
  at alert time

PROBABLE CAUSE: The recommendation container's memory limit is too low for
current traffic load, causing repeated OOMKilled terminations and a
crash-loop backoff.

RECOMMENDED NEXT STEPS
1. Immediately increase the memory limit (and request) for the
   recommendation container in its Deployment spec to provide headroom
   above the observed peak usage, then redeploy to break the crash-loop.
2. Profile the recommendation service under representative load to
   determine the actual memory working set and set a right-sized limit
   with a safe buffer (e.g., 20–30% above peak observed).
3. Add a Kubernetes HorizontalPodAutoscaler or VPA policy for the
   recommendation service so memory resources scale with traffic rather
   than requiring manual intervention.

Read that again from the perspective of the on-call engineer. You were paged. You opened the alert. And the alert didn't hand you a pile of logs and a dashboard; it handed you a calibrated root-cause hypothesis with the evidence attached and the next actions spelled out. The six manual steps from the "old way" are done. Your job is now to decide, not to dig.

That's the time savings: not shaving a few seconds off each query, but removing the entire investigative scavenger hunt from the critical path.

How does Elastic Observability avoid misdiagnosing OOMKilled as a memory leak?

Speed is worthless if the answer is wrong, so it's worth noting how the workflow avoids the classic misdiagnoses. It encodes the reasoning an experienced SRE applies instinctively:

• OOMKilled is not automatically a leak. It compares against a 7-day baseline before ever claiming one. Here, that check is what turned "the app is leaking memory" into the correct "the limit is undersized for real load."
• Co-symptoms are not causes. It explicitly checks whether the upstream degraded first before blaming or clearing it.
• Absence of evidence is not evidence. If a query returns zero rows, it reports "no data available" rather than inventing a failure mode.
• It's honest about confidence. The hypothesis is labeled high, medium, or low. When two failure modes fit the evidence, the workflow names both and says which it believes is causal and why. Manufacturing false confidence is treated as a failure of the investigation itself.

This is the same diagnostic protocol encoded in Elastic's observability-k8s-investigation Skill, a failure-mode taxonomy covering 16 distinct Kubernetes failure patterns, from OOMKilled and CPU throttling through scheduling and networking issues. (More on that in Part 2.)

Still want to double-check? Dashboards, Discover, and APM

The workflow gives you the answer. But a good root-cause tool should also make it trivial to verify that answer because sometimes you want to see it with your own eyes, and sometimes the workflow returns medium confidence and you need to close the gap yourself. Everything the workflow reasoned over is available to you directly.

Dashboards — confirm the restart cascade visually. The Kubernetes dashboards are built for drill-down. Start at the cluster Overview, where "top namespaces by container restarts" surfaces the problem at a glance. Click into the flagged namespace, then the pod driving the restarts. The Pods view flags container restarts on the recommendation pod and plots memory against requests and limits over time — you'll see the working set pressing against the limit exactly as the workflow described. It's roughly four clicks from cluster to container.

Discover — read the raw evidence. The pod detail dashboard links directly to correlated pod logs and events in Discover. Here you can confirm the OOMKilled events and the restart cadence in the raw log and event stream, and run your own ES|QL queries if you want to slice the data differently.

APM — verify the blast radius is really contained. The workflow found the impact isolated to the recommendation service, with frontend (the sole caller) absorbing it at a 0.14% error rate. You can confirm that independently in the APM UI: open the service map, check the caller's latency and error rate over the incident window, and compare against the weekly baseline yourself.

The point isn't that you have to do any of this; it's that the workflow's conclusion is fully auditable. Fast when you trust it, transparent when you want to check.

The same investigation from your IDE: the MCP App

Not every investigation starts from a Kibana alert. Sometimes a developer just asks, "why is this service crashing?" from their editor. Elastic's Observability MCP App (technical preview) exposes the same telemetry (and the same investigation workflow) as AI-callable tools that render interactive views inline in your chat or IDE, no context switch to Kibana required.

For our CrashLoopBackOff scenario, the flow looks like this from an MCP-compatible client such as Claude Desktop or VS Code:

"What's broken?" → the cluster health rollup returns an overall health badge, degraded services, top memory consumers, and a Kubernetes breakdown (CPU, memory, restarts, nodes) in one inline view. Here it flags a critical cluster with payment, cart, frontend, and frontend-proxy degraded.

"Is anything anomalous in the recommendation pod?" → the memory analysis view confirms the spike is load-driven, not a leak — the crashing 788dc88c6c pods report null memory (they die too fast to emit a sample) while the healthy pods sit flat at 45MB, exactly the ML result the workflow used.

"Why is the recommendation service crashing?" → the agent returns the same structured root-cause reasoning you'd see on the alert, rendered inline: memory limit set below what the container needs to boot, the ReplicaSet has been intermittently OOMing for weeks, and a concrete mitigation (roll back, then fix the limit), all without leaving the editor.

Same evidence, same root cause, delivered wherever you happen to be working. (For the full set of MCP App views and architecture, see Part 2.)

From paged to decided: skipping the Kubernetes investigation entirely

The change here is not "a better dashboard." It's a shift in what you do when you get paged.

Before, the alert was the start of the investigation. You were notified, and then you went and found the answer. Now, the alert arrives with the investigation already run: evidence gathered, dead ends eliminated, a calibrated hypothesis, and next steps in hand. You go straight from "notified" to "deciding," and you keep the full trail of dashboards, Discover, and APM for whenever you want to verify or dig deeper.

For a single incident that's a few minutes saved. Across a quarter of on-call rotations, across every engineer who no longer re-runs the same six steps at 3 a.m., it's real time back and a lot less alert fatigue.

Try it yourself

You don't need a production incident to see this. The OpenTelemetry Astronomy Shop demo environment ships with a feature-flag service that lets you trigger failure scenarios on demand. Enable a cart/checkout failure, watch the restart cascade unfold, and the CrashLoopBackOff alert rule fires with the investigation workflow running right behind it.

To get set up:

1. Install the Kubernetes integration — dashboards are available immediately. (Part 1: Getting started)
2. Deploy data collection via the EDOT Collector (OpenTelemetry) or standalone Elastic Agent, both Helm-based.
3. Enable the alert rule templates in Observability > Alerts, including CrashLoopBackOff, and connect your notification channel.
4. Let the ML modules warm up for 24–48 hours so anomaly baselines are ready when you need them.
5. Enable the Investigation Workflow (technical preview) — import the Kubernetes Crashloop Investigation Workflow from the Workflows page and configure it to trigger on the alert. (Part 2: Getting started)
6. Install the MCP App (technical preview) on your favorite agentic client to bring investigations into your IDE.

Running Kubernetes on Elastic today? Tell us which investigation steps you still repeat by hand on every incident, and which remediations you'd trust a workflow to propose. Join the Elastic Community Discussion.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The closest option today is to hand-code the mapping in the transform processor:

# otel.yml
processors:
  transform:
    log_statements:
      - context: log
        statements:
          - set(attributes["user.name"], "Alice") where attributes["user.id"] == "user001"
          - set(attributes["user.name"], "Bob") where attributes["user.id"] == "user002"
          - set(attributes["user.name"], "Carol") where attributes["user.id"] == "user003"
          # ...and one more line for every user

That works for a handful of entries, but it does not scale beyond 10 to 20 items. Every new mapping means another statement, the lookup data lives in the same file as your pipeline config, and there is no way to point at reference data that already exists. It proves the need is real, but it only covers the simplest, smallest cases.

If you run the Collector and you have been reaching for the transform processor, a sidecar script, or a downstream ingest pipeline just to add reference data, this component is for you.

Why the OpenTelemetry Collector needed a lookup processor

The Collector was already good at a couple of enrichment patterns. The transform processor reshapes and derives data from what is already on a record. The k8sattributes and resourcedetection processors attach system and environment metadata, like Kubernetes pod details or cloud host information.

What it could not do was look up related data by a key you already have. Three patterns in particular had no home:

• File-based lookups from static reference data in JSON, YAML, or CSV
• HTTP and API-based enrichment from an external service
• DNS lookups, such as resolving an IP to a hostname

These are everyday tasks in other data collectors and transformation tools. Mapping an internal service ID to a friendly name, attaching business metadata by customer ID, or resolving an IP all fall into this category. Without a native component, teams built brittle workarounds or pushed the work downstream where it is harder to reuse.

That gap is what the new lookup processor closes. Elastic's Data Processing team proposed it, the community accepted it, and it was built in partnership with Grafana, thanks to Sam DeHaan (GH: dehaansa). The lookup processor takes a value from your telemetry, uses OTTL to build a lookup key, queries a source such as a YAML file or DNS, and writes the result back as new attributes.

How the OpenTelemetry lookup processor is designed

The design grew out of repeated requests from the community for richer enrichment. A few examples that fed into it:

• Enrich attributes based on key matching from YAML or CSV definition (#40526)
• Enrich telemetry with resource metadata from an inventory datasource (#40936)
• Generic resource detector (#29627)
• Alert Manager receiver and exporter (#18526)
• gRPC processor / connector (#20888)

Rather than build a one-off component for each request, the goal was a single processor flexible enough to cover them. Four ideas shaped it:

• Multiple lookups per processor, so one instance can enrich several attributes in a single pass.
• Caching, so external sources like DNS do not get queried for the same key over and over.
• OTTL for key extraction, so you get a full expression language for pulling the lookup key off a record, including converters.
• Extensible sources, so you are not limited to the built-in set. You can register a custom source for your own data.

Extensible sources matter most for the processor's long-term value. A source is a small, well-defined interface, so the processor is a foundation for enrichment rather than a fixed list of features. YAML and CSV cover static reference data today, DNS covers dynamic resolution, and the same interface leaves the door open for HTTP APIs, key-value stores, or anything specific to your environment.

How the lookup processor evaluates keys with OTTL

Whatever source you configure, the processor runs the same three steps for every record.

First, it evaluates an OTTL expression to produce a lookup key from the record. Second, it hands that key to the configured source and gets a value back. Third, it writes the value to the attributes you name, on the record or on its parent resource. When a key has no match, the processor writes a configurable default so downstream queries stay predictable.

The next two sections walk through the two sources available today.

File-based lookups with YAML in the OpenTelemetry Collector

The most common case is static reference data. You keep a mapping file next to the Collector and enrich records against it. Here the processor reads a YAML file and adds user.name to each log based on its user.id.

# otel.yml
processors:
  lookup:
    source:
      type: yaml
      path: /etc/otel/mappings.yaml
    lookups:
      - key: log.attributes["user.id"]
        attributes:
          - destination: user.name
            default: "Unknown User"

The mapping file is a plain set of key-value pairs:

# /etc/otel/mappings.yaml
user001: "Alice"
user002: "Bob"

The key field is an OTTL value expression, so log.attributes["user.id"] reads the user.id attribute off the log record. The destination is where the looked-up value lands, and default is what gets written when the key is missing from the file.

Given this incoming log record:

{
  "body": "User logged in",
  "attributes": {
    "user.id": "user001",
    "http.method": "POST"
  }
}

The processor looks up user001, finds Alice, and produces:

{
  "body": "User logged in",
  "attributes": {
    "user.id": "user001",
    "user.name": "Alice",
    "http.method": "POST"
  }
}

The original attributes stay intact and the enriched value is added alongside them. The CSV source works the same way for teams that keep reference data in spreadsheets or exports rather than YAML.

DNS lookup enrichment inside the OpenTelemetry Collector

Static files are one thing, but some enrichment requires live, changing data. Resolving an IP address to a hostname is the classic example, and it is the first dynamic source the processor supports. Instead of a file, you point it at a DNS server.

# otel.yml
processors:
  lookup:
    source:
      type: dns
    lookups:
      - key: log.attributes["client.ip"]
        attributes:
          - destination: client.hostname
            default: "Not found"

The shape of the config is identical to the YAML example. Only the source changed. The processor pulls client.ip off the record, asks a DNS resolver to reverse-resolve it, and writes the hostname back.

Given this incoming record:

{
  "body": "User logged in",
  "attributes": {
    "client.ip": "8.8.8.8",
    "http.method": "POST"
  }
}

The DNS source resolves 8.8.8.8 and produces:

{
  "body": "User logged in",
  "attributes": {
    "client.ip": "8.8.8.8",
    "client.hostname": "dns.google",
    "http.method": "POST"
  }
}

Because DNS queries hit an external system, this is where caching earns its place. The processor keeps results in an in-memory cache so a stream of records sharing the same IP does not turn into a stream of identical DNS queries. That keeps latency down and avoids hammering your resolver.

Writing a custom lookup source

The built-in sources cover common cases, but the real design goal was extensibility. A source is a small contract: you implement a lookup function that takes a string key and returns a value. The processor takes care of OTTL key evaluation, caching, defaults, and writing attributes, so a custom source only has to answer the question "what value goes with this key?"

That means if you already run an internal metadata API, a Redis cache, or a custom database, you can wire it in as a source and reuse everything else the processor provides. HTTP-based sources and key-value stores are natural fits, and they are on the roadmap precisely because the interface makes them straightforward to add.

How Elastic plans to use this lookup processor

Elastic builds its Collector distributions on OpenTelemetry Collector Contrib. The plan is to include lookup processor so the enrichment work teams do today with brittle workarounds can run inside the pipeline instead of downstream.

Two patterns stand out. The first is reference-data enrichment: turning an internal service ID into a friendly name, or attaching metadata such as a team or customer by ID, so telemetry arrives already labeled for search and correlation. The second is DNS resolution: turning a client.ip into a hostname before the data lands, which matters for network and security telemetry where you want the name rather than the raw address.

Doing this in the Collector keeps reference data close to where telemetry is processed and avoids duplicating the logic in separate ingest steps. As the source interface grows to cover HTTP APIs and key-value stores, the same processor can back richer enrichment without changing how pipelines are configured.

Lookup processor roadmap and how to contribute

The main implementation of the lookup processor has merged upstream into OpenTelemetry Collector Contrib. The core processor and YAML source landed first, and the DNS source followed as the first dynamic lookup.

There is a healthy backlog of work ahead, and contributions are welcome:

• An HTTP lookup source for enrichment from external APIs.
• More DNS capabilities, including A and AAAA queries and support for multiple DNS servers.
• Component telemetry, so you can observe cache hit and miss rates, lookup latency, and error rates.
• Performance improvements as real-world usage grows.

If you want to try it, add the processor to a Collector build that includes Contrib, point a YAML source at a mapping file, and enrich a real record. Then open an issue or pull request on the OpenTelemetry Collector Contrib repository. The component is community-owned, and the more sources and feedback it gets, the more useful it becomes.

To go deeper, read the lookup processor README and the enrichment tracking issue. For more on how Elastic builds on OpenTelemetry, browse the OpenTelemetry articles on Elastic Observability Labs.

Availability

This is available in serverless today and is coming to Elastic Cloud Hosted and self-managed deployments in 9.5.

What you need for RED metrics breakdown in Elastic Observability

You need trace data from a service instrumented with any method Elastic APM supports.

• Application instrumentation: one of the following:
• Elastic APM agents for Java, .NET, Node.js, Python, PHP, Ruby, Go, and other supported languages.
• OpenTelemetry SDKs sending OTLP via Elastic Agent or an upstream OpenTelemetry Collector with the elasticapm connector under connectors (not processors).
• Useful attributes: breakdown works best when spans include the dimensions you want to compare (k8s.pod.name, k8s.deployment.name, service.version, and others). You can also declare custom attributes on spans: add labels to transactions and spans with Elastic APM agents, or set OpenTelemetry attributes on spans and resources with OpenTelemetry SDKs. Those custom fields work as breakdown dimensions too.
• Backend: Observability serverless today, or Elastic Stack 9.5 on Elastic Cloud Hosted and self-managed when 9.5 releases.

How to go from a RED metrics alert to the root-cause error log

Step 1: Review RED metrics on the alert detail page

When you receive a notification for a RED metric threshold breach, if you open the alert detail page, you can review the symptoms for the impacted service on one page.

In our example, failed transactions have clearly increased for the cart service, so we want to understand what is driving that RED metric change:

Step 2: Break down RED metrics by span attributes in Discover

To investigate why a RED metric changed, open Traces in Discover and use the new breakdown feature to split RED metrics by any attribute on your spans. In our example, we're going to check Kubernetes attributes and service version, but you could break down by any span attribute you send (e.g. cloud.region, cloud.availability_zone, or container.id).

Each breakdown shows which attribute values moved the metric, so you can see whether the problem is isolated to one pod, deployment, version, or whatever dimension you split on.

In our example, error rate clusters on a single Kubernetes deployment, which points the investigation at a release. We will break down by service.version to validate our hypothesis:

Step 3: Open the trace waterfall and read the linked error log

Once trace breakdown has identified a specific service version as the likely cause, we can filter by that service.version and look at sample failing spans to see if they explain why the version is causing failures.

Open the trace waterfall for one failing span and follow through to the linked error log.

In our example, the error log points to bad configuration that could be causing the issue. Either way, we have narrowed the investigation to a solid hypothesis we can act on in just a few clicks:

From RED metrics alert to error log in Elastic Observability

From a RED metric alert, you can review the symptomatic service, break down Traces in Discover by any attribute on your spans, and open a failing span's trace waterfall to reach the error log in just a few clicks.

What is Tail Sampling?

Distributed tracing is useful for debugging, but at production scale it comes with processing overhead and storage costs, at which point sampling becomes a natural way to maintain the value of tracing while keeping costs under control. Tail-based sampling, or tail sampling, is a technique that makes a sampling decision conditionally at a later stage, so that high-value traces like errors or slow transactions are more likely to be sampled. The opposite is head sampling, which makes the decision at the start of a trace, before any such information is available.

How does the tail sampling processor work?

The tail sampling processor buffers 100% of incoming traces (or spans, used interchangeably), then forwards the sampled subset after applying the sampling policies. Buffering is a major source of memory usage, and it scales proportionally to the volume of spans, a well known pain point in the community.

Memory usage is bounded by configuration parameters like decision_wait and num_traces. Setting decision_wait to 1 minute means a sampling decision is made for a trace after 1 minute, during which all spans for that trace are expected to have arrived. If a trace is slower than 1 minute, the decision is made with some spans missing.

As a side note, scaling out the tail sampling setup involves using the loadbalancingexporter to satisfy the requirement that all spans for a trace must be routed to the same collector. This introduces some operational complexity and potentially data loss during collector restarts. But this post focuses on the memory usage of a single tail sampling processor instance, regardless of horizontal scaling.

Why does tail sampling cause memory pressure?

These parameters introduce a tradeoff between data loss and memory usage, and they require assumptions about the shape of traces: how slow they can be, how many spans they contain, how large each span is. These assumptions can become stale as instrumentation evolves.

How much data loss is acceptable to limit memory usage, and can the tradeoff be improved? The following two contributions aim to give operators more flexibility.

How span-ingest reduces tail sampling memory by releasing spans early

sampling_strategy is a new configuration option added to the tail sampling processor in v0.149.0.

sampling_strategy defaults to trace-complete, which matches the original behavior: sampling policies are only evaluated when decision_wait has elapsed, at which point the trace is considered complete. (There is a similar config, decision_wait_after_root_received, for optimization, but it is excluded from this discussion for simplicity.) This means all spans are buffered in memory for roughly decision_wait before being released, regardless of whether a decision could have been made earlier. For example, health check spans that should always be dropped are still held in memory until policy evaluation time.

Alternatively, sampling_strategy can be set to span-ingest, where spans are evaluated individually at ingest time. This allows terminal decisions, specifically drop or sampled, to be made earlier, freeing memory by dropping or exporting all spans buffered so far for that trace before decision_wait elapses. In the health check example, a policy can be configured to drop the entire trace as soon as the root span belongs to a health check. It is worth noting that an unsampled decision, unlike an explicit drop, is not terminal, as it can be overruled by a sampled or drop decision from another span in the same trace, so unsampled traces cannot be released early.

Switching from trace-complete to span-ingest will require policy adjustments, as policies can no longer assume all spans are available at evaluation time. Moreover, not all policy types are supported with the span-ingest strategy.

Disk-backed tail sampling storage with Pebble

Even with span-ingest, all spans are still buffered in memory. As decision_wait is increased to accommodate slow traces and num_traces is increased to limit data loss, the collector will eventually hit its memory limit and get OOM killed, resulting in further data loss.

What if traces were buffered on disk instead, where there is an order of magnitude more capacity? The main drawback is performance: disk throughput and latency, even with SSDs, are at least an order of magnitude slower than memory, so disk writes need to be efficient. For this reason, Pebble, an LSM database, was chosen as the storage backend for its fast write performance. Read performance is less of a concern, as reads only happen for the sampled subset of traces when sampling_strategy is set to span-ingest.

The implementation introduces a TailStorage interface for trace storage operations, and a new tail_storage option in v0.150.0 (behind feature gate processor.tailsamplingprocessor.tailstorageextension) to configure the storage backend. The default in-memory behavior is unchanged, but it is now possible to swap in a different storage backend, like the new pebbletailstorageextension contributed to Collector Contrib.

Tail sampling memory benchmarks: trace-complete vs span-ingest with Pebble

Benchmark setup: OpenTelemetry Demo with fan-out collectors

The following benchmarks were produced by running OpenTelemetry Demo with increased load against a pipe collector, which receives all spans and fans them out to two identical collectors under observation (CUO-A and CUO-B), differing only in their tail sampling configuration. Measurements include pipe collector throughput, spans received, spans sent (sampled), CPU usage, and memory usage.

Benchmark setup diagram

                      demo ns
     +----------------------------------------+
     |  opentelemetry-demo                     |
     |    loadgenerator (locust)               |
     |    services: frontend, cart, ...        |
     |    demo-collector                       |
     +----------------------------------------+
                          |  OTLP/gRPC
                          v
                     chamber ns
     +----------------------------------------+
     |             pipe-collector             |
     |         receive once, fan out          |
     |      exporters: [otlp/a, otlp/b]       |
     +----------------------------------------+
              | OTLP                  | OTLP
              v                       v
     +----------------+      +----------------+
     |     CUO-A      |      |     CUO-B      |
     | tail_sampling  |      | tail_sampling  |
     |   (config A)   |      |   (config B)   |
     +----------------+      +----------------+

Tail sampling processor configurations

CUO-A

config:
  processors:
    tail_sampling:
      sampling_strategy: trace-complete
      decision_wait: 5m
      num_traces: 5000000
      block_on_overflow: true
      decision_cache:
        sampled_cache_size: 10000
        non_sampled_cache_size: 200000
      policies:
        - name: root_1pct
          type: and
          and:
            and_sub_policy:
              - name: root_span_only
                type: ottl_condition
                ottl_condition:
                  error_mode: ignore
                  span:
                    - "IsRootSpan()"
              - name: root_probabilistic
                type: probabilistic
                probabilistic:
                  sampling_percentage: 1.0

CUO-B

CUO-B uses the same tail sampling processor configuration as CUO-A, except it sets sampling_strategy: span-ingest and tail_storage: pebble_tail_storage/main, along with its corresponding pebbletailstorageextension configuration.

extensions:
  pebble_tail_storage/main:
    directory: /var/lib/otelcol/pebble

Memory, CPU and throughput results

The following tables compare trace-complete (CUO-A) against span-ingest with Pebble disk storage (CUO-B) across memory, CPU and throughput. The process RSS, Go heap allocation, and per-process CPU measurements come from OpenTelemetry Collector internal process and runtime metrics, while container working set and container CPU come from Kubernetes cgroup metrics scraped by kubelet/cAdvisor.

Memory (peak over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | process RSS | 916.4 MiB | 442.7 MiB | -51.7% | | Go heap alloc | 699.3 MiB | 241.7 MiB | -65.4% | | container working set | 763.0 MiB | 282.9 MiB | -62.9% |

CPU (total over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | per-process CPU | 11.9 core-s | 22.7 core-s | +90.7% | | container CPU | 11.9 core-s | 22.7 core-s | +90.1% |

Throughput (total over the window)

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | spans received | 257,804 | 257,804 | 0.0% | | spans sent | 2,477 | 2,477 | 0.0% |

Tail sampling

| Metric | cuo-a | cuo-b | Δ (B vs A) | | --- | ---: | ---: | ---: | | traces in memory peak | 29,284 | 29,245 | -0.1% | | traces sampled by root_1pct policy | 496 | 496 | 0.0% |

• cuo-a = trace-complete, cuo-b = span-ingest-pebble
• Window: 15m 39s (t+0:00 start, t+10:06 drain start, t+15:39 drain end)

The results show a significant memory reduction when using pebbletailstorageextension with span-ingest, at the cost of increased CPU usage from event serialization and database overhead.

What's next for OpenTelemetry tail sampling

Both sampling_strategy and pebbletailstorageextension are still in their early stages at the time of writing. Feedback and contributions are welcome in the OpenTelemetry Collector Contrib repo. Stay tuned for more improvements.

Elasticsearch now stores Prometheus metrics in the same columnar backend as your logs and traces, at 3.75 bytes per datapoint, with no custom-metric penalty. The graph, the logs, and the traces answer to one query language.

Using an existing Kubernetes cluster (AWS EKS in this example): point Prometheus at Elastic with one config block, see every metric render in Discover with no dashboard to build, run most of your existing PromQL unchanged, find where ES|QL takes you past what PromQL can express, and finish by reading your logs with the same query language.

Nothing about your collection changes. Your scrape configs, relabeling rules, and service discovery carry over as-is. Most of your PromQL comes with you too — see the coverage note in Step 5 for the gaps.

Step 1: get a Prometheus endpoint and an API key from Elastic Observability

You need Elastic Cloud Serverless or Elastic Cloud Hosted. Both expose the Prometheus endpoints with no configuration.

Serverless is the fastest start. Sign in at cloud.elastic.co and create an Observability project. There is nothing to size and nothing to provision.

Elastic Cloud Hosted works the same way for everything in this post, and is the right choice when you need a specific stack version, a specific region topology, or the deployment-level controls that come with a managed cluster.

Find the Prometheus endpoint and create an API key in the UI

For Prometheus endpoint and the API key go to Kibana, click Add data in the left nav.

For Prometheus, scroll to Connect directly to the endpoint at the bottom and select the Prometheus tab.

Two things to copy:

The endpoint. It looks like https://my-observability-project-xxx.ingest.us-west-2.aws.elastic.cloud. Note the .ingest. host. This is not the same host as your Elasticsearch search endpoint or your Kibana URL.

The API key. Click Create key. Copy the value before you close the dialog, because it is not retrievable afterward.

If you want to scope it by hand instead, Open in API keys takes you to the full editor, and the minimum privilege for metrics ingest is:

{
  "ingest": {
    "indices": [
      {
        "names": ["metrics-*"],
        "privileges": ["auto_configure", "create_doc"]
      }
    ]
  }
}

Keep both values. Every remaining step uses them.

Step 2: make sure Prometheus is scraping your Kubernetes cluster

Prometheus should already be scraping your cluster.

If it is not, the shortest path is the kube-prometheus-stack Helm chart, which installs the Prometheus Operator, Prometheus itself, kube-state-metrics, and node-exporter in one command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

That gives you the metrics you see for the rest of this post:

• cAdvisor container metrics (container_cpu_usage_seconds_total, container_memory_working_set_bytes)
• kube-state-metrics cluster objects (kube_deployment_spec_replicas, kube_daemonset_status_number_ready).

Step 3: configure Prometheus remote write to the Prometheus endpoint in Step 1

Elasticsearch implements the Prometheus Remote Write protocol natively. There is no adapter, no sidecar, and no translation layer. You add one block and the data flows on the next scrape interval.

If you run the Prometheus Operator

The Operator does not read a prometheus.yml you write by hand. It generates one from the Prometheus custom resource, and authorization.credentials there is a reference to a Kubernetes Secret, not an inline value. Create the secret first:

kubectl create secret generic elastic-prometheus \
  --namespace monitoring \
  --from-literal=api_key='YOUR_API_KEY'

Then reference it from values.yaml:

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: "https://my-observability-project-xxxx.ingest.us-west-2.aws.elastic.cloud:443/api/v1/write"
        authorization:
          type: ApiKey
          credentials:
            name: elastic-prometheus
            key: YOUR_API_KEY

And apply it:

helm upgrade prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f values.yaml

Configure Prometheus remote write with prometheus.yml

Same thing, in prometheus.yml:

remote_write:
  - url: "https://YOUR_ES_ENDPOINT/_prometheus/metrics/node/eks/api/v1/write"
    authorization:
      type: ApiKey
      credentials: YOUR_API_KEY

If you run Grafana Alloy uss the following configuration

prometheus.remote_write "elasticsearch" {
  endpoint {
    url = "https://YOUR_ES_ENDPOINT/_prometheus/metrics/node/eks/api/v1/write"
    headers = {"Authorization" = "ApiKey YOUR_API_KEY"}
  }
}

How the remote write URL maps to Elasticsearch data streams

You do not name an index anywhere. There is no index field in the payload and no data stream in your remote_write config. Elasticsearch derives the target data stream from the write path and creates it on the first sample. The two path segments after /metrics/ are the dataset and the namespace:

| URL | Data stream | |---|---| | /_prometheus/api/v1/write | metrics-generic.prometheus-default | | /_prometheus/metrics/{dataset}/api/v1/write | metrics-{dataset}.prometheus-default | | /_prometheus/metrics/{dataset}/{namespace}/api/v1/write | metrics-{dataset}.prometheus-{namespace} |

The examples above use /metrics/node/eks/, which writes to metrics-node.prometheus-eks. That is the data stream you will see in Discover in the next step. Use dataset and namespace to separate production from staging, or to give each cluster and each team a data stream with its own retention and downsampling policy.

If you would rather keep a bare /api/v1/write URL, you can route per time series instead: attach data_stream_dataset and data_stream_namespace labels to the series, and they take precedence over the URL path. These two are control fields, so they route the document without being stored in its labels object.

Elasticsearch installs the index template for metrics-*.prometheus-* itself. You do not create templates or mappings.

What Prometheus metrics look like in Elastic Observability

Every Prometheus sample becomes a document. Labels become keyword fields that serve as time series dimensions. The value goes under metrics.<metric_name>:

{
  "@timestamp": "2026-07-02T10:30:00.000Z",
  "data_stream": {
    "type": "metrics",
    "dataset": "node.prometheus",
    "namespace": "eks"
  },
  "labels": {
    "__name__": "container_memory_working_set_bytes",
    "pod": "checkout-7d9f6c4b8-x2kqp",
    "namespace": "oteldemo",
    "container": "checkout",
    "node": "ip-10-0-3-14.us-west-2.compute.internal"
  },
  "metrics": {
    "container_memory_working_set_bytes": 36700160
  }
}

The one gotcha to know now. Elasticsearch infers whether a metric is a counter or a gauge from its name. Names ending in _total, _sum, _count, or _bucket are counters. Everything else is a gauge. That inference is correct for container_cpu_usage_seconds_total and correct for container_memory_working_set_bytes. It is wrong for any metric in your estate that does not follow Prometheus naming convention, and a misclassified metric gets rejected by the function that should accept it: RATE(my_metric::counter) works on counters only. Step 6 shows how to override the inference.

Current limits. Remote Write v1 only. Classic histograms and summaries are supported through their _bucket, _sum, and _count series, each mapped to the right metric type. Native (sparse) histograms and exemplars arrive with Remote Write v2, which is on the roadmap. Staleness markers are not stored or respected, and non-finite values (NaN, Infinity) are dropped silently. See the Prometheus remote write ingest docs for the full list.

Step 4: verify Prometheus metrics are flowing into Elasticsearch

Do not go build a dashboard. Confirm the pipeline first, and Elastic makes that a single command.

In Kibana, open Discover, switch the query editor to ES|QL, and type the name of the data stream you just wrote to:

TS metrics-node.prometheus-eks

That is the whole query. Discover reads the data stream, finds every metric in it, and renders each one as its own time series chart. Fifty metrics, fifty charts, no dashboard to build and no query to write per metric. It reads the metric type and charts each one correctly: gauges as averages, counters as rates, histograms as p95 distributions.

You can also get here without typing anything. Observability → Streams lists every data stream in the cluster. A Time series badge means it is a time series data stream. Click View in Discover and the TS query is filled in for you.

This is your ingest health check. Three things to look for.

• Data is flowing. Recent, continuous values. Not gaps, and not a line that stops an hour ago.
• Values are plausible. Memory in the tens of megabytes for a small container. CPU as a fraction of a core. Network bytes tracking real traffic.
• Coverage is what you expected. If kube_pod_container_status_restarts_total is missing, your kube-state-metrics scrape config is wrong, and you want to know that now rather than when you are building an alert on it.

Widen the time picker before you conclude anything is broken. A 15-minute window over a quiet period makes healthy data look flat.

To list what actually has data rather than what the mapping declares:

TS metrics-node.prometheus-eks | METRICS_INFO | SORT metric_name

The No dimensions selected control above the charts lets you break every chart out by a label: select pod and each chart splits into one series per pod.

Step 5: run PromQL queries on Prometheus metrics in Elastic Observability

If your team writes PromQL, keep writing PromQL. PROMQL is a source command in ES|QL, alongside FROM and TS, and it runs anywhere ES|QL runs: Discover, dashboard panels, and alert rules.

It does not run a separate engine. It parses the expression, resolves each function to its ES|QL equivalent, and builds a TS execution plan, so your PromQL gets the same vectorized, parallel execution as native ES|QL.

Current limits. PROMQL is generally available on Elastic Cloud Serverless and a tech preview on Elastic Stack 9.4, benchmarked at over 80% query coverage against popular Grafana OSS dashboards. The gaps worth knowing before you paste a dashboard in: histogram_quantile is not yet implemented, which matters most because it is how nearly every latency dashboard computes p95; group modifiers (on(...) group_left(...)) and the set operators or, and, and unless are unsupported; and predict_linear, label_join, and label_replace are not yet available. Time buckets also align to fixed calendar boundaries rather than the query start time, so short ranges or large steps can differ slightly from Prometheus. See the PROMQL command reference for the current list.

CPU: per-pod CPU rate with PromQL

The per-second CPU rate across containers, grouped by pod. This is the first thing you look at when something is hot:

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Broken out by namespace instead, to find which team is burning the cluster:

PROMQL sum by (namespace) (rate(container_cpu_usage_seconds_total[5m]))

Note what came back: a pod column, a step column, and the value column named after the expression itself. That is a normal ES|QL table, which is the whole point and the thing Step 6 builds on.

Memory: working set and RSS with PromQL

Working set is the number that matters for OOM risk. It is what the kernel counts against the limit, and it is not the same as total allocated memory:

PROMQL sum by (pod) (container_memory_working_set_bytes)

Resident set, for comparison, when you are trying to tell a real leak from page cache:

PROMQL sum by (pod) (container_memory_rss)

Network: receive rate by pod with PromQL

PROMQL sum by (pod) (rate(container_network_receive_bytes_total[5m]))

Cluster objects: replica counts with PromQL

Deployments that are not running the replica count they declare:

PROMQL kube_deployment_spec_replicas

One nicety worth calling out: in Prometheus every query needs an explicit start, end, and step. In Kibana you drop all three. The date picker supplies the range and Kibana derives the step, which is why every query above is a single line.

Build a Kibana dashboard from PromQL queries

Every query above is a dashboard panel. In Discover, click Save, or go to Dashboards → Create → Add panel → ES|QL and paste the query in. The date picker drives start, end, and step, so a panel written once works at every zoom level.

Four panels, four one-line PromQL queries, no translation step. If you are coming from Grafana, this is the same dashboard you already have, rebuilt in about five minutes. If you would rather not rebuild it at all, keep Grafana and point its existing Prometheus datasource at Elasticsearch. That is covered at the end of the post.

Step 6: query Prometheus metrics with ES|QL, beyond what PromQL can express

The PROMQL command in ES|QL compiles to TS. These two queries are equivalent:

PROMQL sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

TS metrics-node.prometheus-eks
| WHERE TRANGE(1h)
| STATS SUM(RATE(metrics.container_cpu_usage_seconds_total, 5m)) BY labels.pod, TBUCKET(1m)

The second form is where metrics stop being a separate world from logs and traces — and where the real joins happen.

Top-N and filtering

A TS query returns a normal ES|QL table, so SORT, LIMIT, and WHERE all work downstream. Top-N needs no special function, and no topk. The ten pods using the most memory are a SORT and a LIMIT:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL
| STATS mem = SUM(metrics.container_memory_working_set_bytes) BY `labels.pod`
| SORT mem DESC
| LIMIT 10

Filtering is the same move, a WHERE on the aggregated column. Only the pods over 50 MB:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL
| STATS mem = SUM(metrics.container_memory_working_set_bytes) BY `labels.pod`
| WHERE mem > 50000000
| SORT mem DESC

You actually can pipe the results of a PROMQL query and post-process with regular ES|QL.

Memory usage against the request

A useful question during a memory scare: which pods are using more than they requested, and by how much. That combines two metrics, and ES|QL expresses it in a single pass with filtered aggregations, one MAX per metric:

TS metrics-node.prometheus-eks
| WHERE `labels.pod` IS NOT NULL AND `labels.namespace` IS NOT NULL
| STATS
    used_memory      = MAX(metrics.container_memory_working_set_bytes),
    requested_memory = MAX(metrics.kube_pod_container_resource_requests)
                       WHERE `labels.resource` == "memory"
  BY `labels.pod`, `labels.namespace`, time_bucket = TBUCKET(5 minute)
| EVAL pct_of_request = 100 * used_memory / requested_memory
| WHERE pct_of_request > 80
| SORT pct_of_request DESC
| LIMIT 100

The WHERE attached to requested_memory is a filtered aggregation: kube_pod_container_resource_requests carries both CPU and memory under a labels.resource dimension, and the filter keeps only the memory rows, so the ratio is memory over memory. Each metric lives in its own documents; the shared BY key lands both aggregates on one row.

Step 7: query Prometheus metrics and logs together with ES|QL

Over the last few sections we used ES|QL and PromQL to explore metrics. ES|QL reads logs too. Here is a quick query against the OpenTelemetry demo running on this cluster:

FROM logs-*
| WHERE TRANGE(30m) AND kubernetes.pod.name == "checkout-9656cbd88-fsr9v"
| STATS events = COUNT(*) BY log.level, TBUCKET(1m)
| SORT events DESC

Same editor, same date picker you used for metrics. Only the source command changed, from TS to FROM, and now you are reading log volume by level per minute. One query language across metrics and logs, no tab switch and no second tool.

What's running in Elastic Observability now

Prometheus is writing to Elastic with one config block. Every metric renders in Discover with no dashboard built. Most of your existing PromQL runs unchanged, and ES|QL takes you further: top-N, filtering, and cross-metric ratios you can pipe into the rest of the language. Metrics and logs answer to the same query in the same window.

You did not drop a single label to get here, and you are not being billed for cardinality.

Next steps: Grafana, dashboard migration, and OpenTelemetry

1. Keep Grafana if you want it. Elasticsearch exposes a Prometheus-compatible read API at <endpoint>/_prometheus. Point Grafana's existing Prometheus datasource at it, set httpMethod: GET on the datasource, and your PromQL dashboards keep working. Keep Grafana, replace Mimir.
2. Migrate your Grafana dashboards. When you are ready to move off Grafana rather than point it at Elasticsearch, the Observability Migration Platform translates Grafana dashboards, panels, and alert rules into Kibana-native equivalents. It is a source-agnostic CLI (obs-migrate) that converts what it can and flags what needs a human, with a migration report showing what translated cleanly and where semantic gaps remain, so nothing is silently dropped. The walkthrough covers a Grafana and Datadog migration end to end.
3. Want to add OpenTelemetry? Read Part 2 if you are also running OpenTelemetry, or if you would rather collect with an OTel Collector than with Prometheus. Both land in the same store and the same queries read across both.

Start a free trial: cloud.elastic.co/registration Docs: elastic.co/docs/solutions/observability

Frequently asked questions

How do I send Prometheus metrics to Elasticsearch? Add a remote_write block to your Prometheus configuration pointing to https://<YOUR_ES_ENDPOINT>/_prometheus/metrics/{dataset}/{namespace}/api/v1/write with an Authorization: ApiKey <YOUR_KEY> header. Elasticsearch implements the Prometheus Remote Write v1 protocol natively — no adapter or sidecar required.

Can I run existing PromQL queries in Elasticsearch? Most of them, yes. ES|QL includes a PROMQL source command that accepts standard PromQL expressions, benchmarked at over 80% coverage against popular Grafana OSS dashboards. It is generally available on Elastic Cloud Serverless and a tech preview on Elastic Stack 9.4. histogram_quantile, predict_linear, label_join, and label_replace are not yet implemented, and group modifiers and the set operators or, and, and unless are unsupported.

Does Elasticsearch charge based on Prometheus metric cardinality? No. Elasticsearch stores Prometheus metrics at 3.75 bytes per datapoint with no cardinality-based billing and no custom-metric penalty, regardless of how many unique label combinations your metrics produce.

How do I route Prometheus metrics to different data streams in Elasticsearch? The two path segments after /metrics/ in the Remote Write URL set the dataset and namespace. /metrics/node/eks/api/v1/write writes to metrics-node.prometheus-eks. You can also route per time series using data_stream_dataset and data_stream_namespace labels, which take precedence over the URL path.

What Prometheus metric types does Elasticsearch support? Elasticsearch supports Remote Write v1, including counters, gauges, classic histograms, and summaries. Counter vs. gauge classification is inferred from the metric name: names ending in _total, _sum, _count, or _bucket are counters; everything else is a gauge. Native histograms and exemplars require Remote Write v2, which is on the roadmap.

Can I query Prometheus metrics and logs together in Elasticsearch? Yes. ES|QL reads both time series metrics and logs in the same query editor with the same date picker. Switch from TS metrics-node.prometheus-eks to FROM logs-* — same syntax, same window, no second tool.

Related reading

• Elasticsearch: best-in-class for logs, now best-in-class for metrics
• Ship Prometheus metrics to Elasticsearch with Remote Write
• Query Prometheus metrics in Elasticsearch with native PromQL support
• Don't leave metrics on the table: query them with the ES|QL TS command
• Elasticsearch as a backend for Grafana
• How we rebuilt Elasticsearch as a columnar metrics engine

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.