This feature is available in stateful deployments only and isn’t supported on Elastic Cloud Serverless.

What cross-cluster search for synthetic monitoring adds to Elasticsearch CCS

Before 9.5, Synthetics and CCS weren't integrated. Connecting clusters at the Elasticsearch layer gave you raw ping data in Discover but not in the Synthetics UI.Elastic's site reliability engineering (SRE) team wrote about this pattern, using CCS to build a single pane of glass across 150+ clusters, but it applied to raw data, not the Synthetics app.

Monitor definitions are Kibana saved objects, and those objects don't travel across clusters. Wiring CCS at the Elasticsearch layer didn't make remote monitors appear in the Synthetics app. You'd get raw data in Discover, but none of the Synthetics UI experience: no status cards, journey waterfalls, step screenshots, or error drilldowns.

That's still true for raw CCS on saved objects. What shipped in 9.5 is a native integration built on top of remote cluster search. The app searches remote synthetics-* pings. It marks what's remote and sends you back to the origin Kibana whenever you need to change something.

What remote monitors look like in the Synthetics UI

Once remote clusters are configured, their monitors appear alongside your local ones throughout the Synthetics UI.

Find remote monitors on the Overview page

The Overview page shows remote monitors inline with local ones, tagged with a Remote badge next to the monitor type badge. Two additions help with navigation: a Remote cluster filter alongside the existing Type / Location / Tags / Frequency filters, and a Group by → Remote cluster option. When a remote cluster filter is active, local monitors are hidden and status counts reflect only the filtered set, so you can focus on one deployment at a time.

Open a remote monitor detail page

Monitor detail pages work the same way for remote monitors as for local ones, including overview, history, errors, journey runs, step details, screenshots, network waterfalls, and duration charts.

A Remote monitor callout shows the cluster name and the originating Kibana URL, and a View on remote cluster button takes you directly to the monitor on its home instance.

See remote TLS certificates on the Certificates page

Transport Layer Security (TLS) certificates observed on remote clusters appear on the Certificates page, with a tooltip showing which cluster they came from.

Can you edit remote monitors from the central Kibana?

No, not from the central Kibana. Remote monitors are read-only where you view them because their configuration lives as a saved object in the Kibana where they were defined.

Action

Behavior on the central Kibana

View status, history, errors, journey runs, screenshots, network waterfalls

Available

Enable / Disable

Not available

Run test manually

Not available

Create SLO

Not available

Add to dashboard

Not available

Edit

Opens the origin Kibana, when that cluster’s public URL is configured

Clone

Opens the origin Kibana, when that cluster’s public URL is configured

View on remote cluster

Opens the monitor on the Kibana where it lives

 To manage a remote monitor, click the View on remote cluster button on its detail page. It takes you to the monitor on the Kibana where it lives.

Keeping configuration on the origin cluster is intentional. A single source of truth for monitor definitions avoids the mess of cross-cluster writes.

One current limitation to note: alerting doesn't cross cluster boundaries yet. A remote monitor that's down shows as down in the federated UI, but it won't fire alerts in the central Kibana. Keep those rules on the cluster that owns the monitor. Federated alerting is on the roadmap.

How to set up cross-cluster search for synthetic monitoring

There are a few things to put in place.

1. Connect the remote clusters in Elasticsearch

In Stack Management → Remote Clusters, add the Elasticsearch clusters that hold Synthetics data. Synthetics doesn't create these connections; it uses the ones already configured at the Elasticsearch level. The Search and replicate data between Elastic Cloud and on-prem post covers the Elasticsearch-level setup in detail.

2. Set server.publicBaseUrl on every participating Kibana

The View on remote cluster deep link comes from a kibanaUrl field on ping documents, stamped from server.publicBaseUrl. If that setting is missing on a remote Kibana, you still see the monitor but there's no URL to click. Since kibanaUrl is written at ping time, you need to wait for new check results after setting this config before the link appears.

3. Choose which remote clusters Synthetics includes

Go to Synthetics → Settings → Remote Clusters. You need the All privilege for Synthetics and Uptime. Users with Read see the form as read-only.

Toggle on Use all remote clusters, or pick specific ones. Each option shows connected versus disconnected status.

4. Grant access to Kibana spaces

This is one shared configuration per deployment, not a separate copy per space. Use the Spaces field to grant access to specific spaces, or select All spaces. On first save, settings are anchored to the space you're in.

Note that clearing the Spaces field on a subsequent save doesn’t remove access from existing spaces. You must explicitly select the spaces you want to keep.

Open the Monitors overview, and remote monitors appear alongside your local ones.

How the index expression resolves across clusters

Nothing is replicated into the central cluster. CCS works by expanding the index expression at query time, which means that the central cluster never stores remote data; it just knows where to ask.

On each request, the server figures out which index expressions to query.

• If you select Use all remote clusters, it builds synthetics-*,*:synthetics-* with no connectivity check needed.

• If you pick specific clusters, it calls Elasticsearch's cluster.remoteInfo() to check which are currently reachable and only includes those, so a disconnected remote doesn't hang your overview page.

• The resolved indices are cached per space.

The result is a combined index pattern like the following:

The Overview status page and the Certificates page pick up CCS automatically, since they fan out across all configured clusters. Detail views scope each query to the right cluster. Pings, journeys, screenshots, and network events all resolve against a single remote rather than spreading across everything.

What cross-cluster search for Synthetics does not do

CCS for Synthetics is a visibility layer, not a management layer. You get a unified read view across clusters; each cluster stays the owner of its own monitors. If you need to create, edit, or disable monitors centrally, you still do that on the Kibana where each monitor lives.

What

Where it lives

Monitor configuration (saved objects)

Origin cluster only

Private locations

Origin cluster only

Global parameters

Origin cluster only

API keys

Origin cluster only

Alerting rules

Origin cluster only

Ping results

Federated, visible in the central Kibana

Journey steps and screenshots

Federated, visible in the central Kibana

TLS certificate status

Federated, visible in the central Kibana

CCS at the Elasticsearch level is also a prerequisite rather than the whole story. Synthetics needs its own Remote Clusters configuration to show remote monitors properly. Dashboards and Discover can still query synthetics-* via CCS for custom visualizations.

Start centralized monitoring across your clusters

Federated Synthetics ships in 9.5 for stateful deployments. If you run monitors across multiple clusters, go to Synthetics → Settings → Remote Clusters and connect your remotes.

If a remote monitor appears but the deep link is missing, set server.publicBaseUrl on the origin Kibana and wait for new check results. The kibanaUrl field is stamped on documents written after that config is in place.

We welcome your feedback in the Elastic discuss forum.

Documentation:

• Configure Synthetics settings

• Analyze data from synthetic monitors

Related reading:

• Use Cross Cluster Search with Elastic Cloud to improve observability: Elastic's SRE team on building a single pane of glass across 150+ clusters

• Search and replicate data between Elastic Cloud and on-prem: CCS/cross-cluster replication (CCR) general availability (GA) announcement

In this article, we follow six audit events from a company’s admin console through the Elastic Distribution of OpenTelemetry (EDOT) Collector into a dedicated data stream. One of them is a user named alex, who fails two login tries but gets in on the third and then attempts a role change. We reconstruct that timeline with one ES|QL query and read back the retention policy to confirm that it applied. Then we rerun the same query on a snapshot-backed index. The evidence comes back in a form that you can hand to an auditor and explain.

Prerequisites

• Elasticsearch and Kibana 9.5+

• Docker, to run the elastic-otel-collector:9.5 container

• An API key created by a superuser

You can find the companion notebook here. It reproduces the whole lab from this article, including the collector configuration and the searchable snapshot, and the steps in between, and it cleans up at the end.

What audit logs tell you that application logs don’t

Application logs usually explain how software behaved, such as a timeout or an exception, or even a failed connection. Audit logs establish accountability by recording an actor, action, target, outcome, and time. They can help debugging, too, but their value is in reconstructing activity and demonstrating how controls operated.

A Kubernetes RoleBinding change captured in full

In Kubernetes, a Role defines permissions and a RoleBinding grants them to an identity within a namespace. Changes to both are auditable when the audit policy includes those resources. Here’s a real RoleBinding creation that was captured in an isolated Kubernetes 1.35.0 lab. It was collected through OpenTelemetry (OTel) and queried in Elasticsearch

The successful request grants the deploy-bot service account the existing pod-reader role in payments. The RoleBinding's subjects and role reference come from its request body, captured by this lab's targeted Request audit level. Metadata logging alone would omit that body.

Where audit logs come from

Common sources cover different parts of the story:

Source

Questions it can answer

Kubernetes API audit logs

Who requested a resource change, and how did the API server respond?

Cloud provider trails

Which principal performed an API operation on which resource?

Authentication systems

Who signed in or failed authentication? Who changed privileges?

Database audit logs

Who accessed a sensitive object or changed database permissions?

Collection starts with enabling the right events at the source. Cloud data-access events may need separate configuration; database slow-query logs aren’t a complete access audit. For Kubernetes setup, use the existing managed Kubernetes audit-log tutorial.

Four requirements for an audit logging pipeline

Four requirements shape an audit pipeline:

Requirement

Design decision

Long retention

Keep audit data in streams with an explicit retention policy, separate from short-lived operational logs.

Immutability

Define who shouldn’t be able to alter or delete records, and for how long. A read-only index doesn’t prevent privileged administration.

Tamper-evidence

Preserve originals and independently protected verification material so changes or missing records can be detected.

Access control

Separate collection, investigation, and retention administration; record changes to those controls.

For System and Organization Controls 2 (SOC 2) or ISO/IEC 27001, logs support the evidence behind your controls; storing them doesn’t establish compliance. Agree which activities to record, how long to retain each copy, who can retrieve it, and how exceptions or legal holds work. Neither framework gives this lab a universal 365-day retention requirement.

Search and preservation need complementary controls. For example, CloudTrail integrity validation uses signed digest files that must actually be validated. Where write-once retention is required, design a source archive using appropriate controls, such as S3 Object Lock. An indexed JSON string isn’t a substitute for that archive or its validation material.

How to collect audit logs with the EDOT Collector

The retention walk-through uses six fictitious application-audit JSON events: four for the user alex and two for a second user, priya, so the actor filter has something to exclude. The EDOT Collector runs as a container and ships to Elasticsearch and Kibana 9.5.3 on Elastic Cloud Hosted (ECH). The configuration is based on Elastic's documented file log collection and JSON parsing procedure: a filelog receiver with a json_parser operator and a resource processor that sets data_stream.dataset and data_stream.namespace, along with the elasticsearch exporter, in otel mapping mode. This article focuses on the decisions and results; the complete configuration and commands are in the companion notebook. 

Why the json_parser uses parse_to: attributes

The collector parses the source timestamp and fields and then sends records to the dedicated logs-audit_demo.otel-lab data stream. The exporter builds that name from the dataset and namespace and appends .otel to the dataset.

Unlike the documented example, which uses parse_to: body, the lab's json_parser uses parse_to: attributes, so the parsed values go under attributes and the original JSON string stays in body.text. With parse_to: body, the fields would end up in body.structured and the original string would be lost. All six stored strings matched the fixture lines in this test. Elastic Common Schema–based (ECS-based) integrations can use root fields, such as user.name, so check the schema before reusing a query.

Reconstruct an actor’s timeline with one ES|QL query

In Kibana Discover, select ES|QL, Elastic's piped query language. The time picker uses the time zone of the dateFormat:tz advanced setting, which defaults to the browser's, so set it to UTC in Stack Management > Advanced Settings before entering the absolute window 2026-09-07T04:59:00Z through 2026-09-07T05:06:00Z. Then run the following:

Notice that the last event is a failed role change. So far, we only know that alex tried to change a role and was denied, which isn’t the same as a permission change or a compromised account. To find out what happened, the next step is to look at identity-provider and resource activity within the same tenant and time window.

When comparing the screenshot with the query, keep in mind that the fixture uses UTC and Kibana displays timestamps in the browser's time zone (in this case, UTC−05:00). That’s why the rows show 00:00, while the time window is set in UTC.

Discover's time picker constrains this query; API clients need an explicit timestamp filter, and the notebook adds one. When handing evidence to a reviewer, record:

• The query.

• The UTC window.

• The source.

• The retrieval time.

• The result limit.

LIMIT 100 is a bounded investigation view, not an exhaustive export.

Confirm that every audit event arrived

The data stream held six records with six distinct event IDs before and after a normal collector restart, because the filelog receiver keeps its file offsets and the exporter keeps its sending queue in the file_storage extension. At scale, also monitor source coverage, parse failures, queue saturation, and delivery lag. This six-event check is a functional test, not an ingestion benchmark or an exactly once guarantee.

Set a log retention policy that you can verify

A retention setting should be something that you can verify. On ECH, the built-in logs-*-* templates hand new data streams to the logs index lifecycle management (ILM) policy, so the lab gave its stream a dedicated index template, configured as follows: 

• Priority 500, higher than the built-in logs templates that also match the stream name, so Elasticsearch applies this template instead of them.

• Copies the built-in logs-otel@template component list.

• Restricted to logs-audit_demo.otel-*.

• Sets index.lifecycle.prefer_ilm to false.

• Defines a 365-day data stream lifecycle.

The data stream API then reported the backing index as managed by data stream lifecycle, and the lifecycle read-back reported effective_retention: 365d:

Why your audit data stream ignores the retention policy you set

Data stream lifecycle retention is a minimum retention mechanism based on backing-index lifecycle timing, not an exact per-event deletion deadline. A successful read-back proves configuration, not a year of availability. Rehearse retrieving the oldest required records, and review source coverage regularly. For the retention interface and ILM comparison, see retention management with Streams.

Capacity planning for long log retention

Long retention also changes capacity planning. Daily ingest volume multiplied by retention days gives a starting raw volume; mappings, compression, replicas, query concurrency, and tier placement affect the actual resources needed. Size and test against representative investigations across realistic data ages.

Query older audit logs with searchable snapshots

Searchable snapshots keep snapshot-backed indices queryable through Elasticsearch. Fully mounted indices can serve older data from the cold tier; partially mounted indices use a shared local cache in the frozen tier. This gives recent and older evidence a common query surface, with storage and retrieval tradeoffs to test for your workload:

The audit lab took a snapshot of its six-event backing index and mounted it. It then reran the same actor query:

The same four events were returned. This used full_copy on available hot nodes with an Enterprise license; it didn’t benchmark cold/frozen costs or latency. The backing snapshot must remain available while its mounted index uses it, and local cache isn’t a backup. The separate lab mount also needs its own retention and cleanup plan.

Who can read and change your audit logs

Use role-based access to give investigators read and view_index_metadata on the required audit data, plus Discover read access in the intended Kibana space. Keep retention and snapshot administration in separate roles. Roles are additive, so a reader role doesn’t remove permissions granted elsewhere.

The collector's API key only carries create_doc and auto_configure on its stream, and the has privileges API confirmed that it has no index, write, delete, or manage privilege there. That limits the collector's authority; it doesn’t make the records immutable against administrators. Also plan Elasticsearch and Kibana audit logging for the evidence platform itself, subject to deployment and subscription requirements.

Connecting Kubernetes audit logs to container runtime data

For the Security Labs crossover, correlating Kubernetes audit logs with container runtime data shows how API activity connects to what runs inside a workload. Stable identities and resource context make that connection possible. Our small application fixture demonstrates timeline reconstruction; it doesn’t validate those Kubernetes detection rules.

Where to start with audit log management

Start with one audit source and one question that you must answer months later. Verify delivery, preserve the required originals, rehearse searches across recent and retained data, and review who can change the evidence path. Elastic brings collection, structured investigation, searchable retention, and scoped access together; your policy defines the evidence that those capabilities must preserve.

This is an alert noise reduction exercise done with queries. Kibana writes every alert into a hidden index and keeps it there after recovery, so we put three questions to that alert history with ES|QL. Was this one incident or many, where did it start, and which rule produced more noise than signal? Then we change the grouping key on the noisy rule and replay the same failure, and the same query comes back with one episode that lasts seven minutes.

Kibana stamps every alert document with a rule revision, so the before and after of that tuning change sit in one index. At the end, an Agent Builder agent answers the same questions over the same alert history.

Everything below was tested on Elasticsearch and Kibana 9.5.2 on Elastic Cloud, with OpenTelemetry Collector Contrib 0.157.0.

Prerequisites

• Elasticsearch and Kibana 9.5 or later, on Elastic Cloud or self-managed.
• Kibana privileges to create and edit Elasticsearch query rules and Observability custom threshold rules, plus read access to the .alerts-* indices.
• OpenTelemetry Collector Contrib 0.157.0 or later, and an Elasticsearch API key with ingest privileges for the OTLP endpoint.
• Agent Builder enabled, for the final section only. Everything before it is plain ES|QL and works without it.

Why alert history beats alert notifications

A notification answers one short-lived question: who should look at this right now?

Most of the questions that make alerting better arrive later, and they are questions about data:

• Did 18 alert episodes represent 18 problems, or one incident evaluated from 18 angles?
• Which service showed the first symptom, and did the failure follow the dependency chain?
• Which rules produce short, repetitive episodes that nobody can act on?
• Did last month's threshold change actually reduce noise?

Inside Elastic, this framing has a name: alerts as data.

The idea is that every alert and every state transition is a fact in time, stored with explicit semantics, so correlation becomes a query across streams rather than a bespoke feature.

It also matches what shows up in real operations channels: a single infrastructure failure can open dozens of alerts within seconds, and flapping alerts with miscalibrated thresholds are a familiar source of page fatigue.

Both problems are hard to solve from notifications and straightforward to study from preserved alert history.

What Kibana alerting stores in every alert document

Kibana alerting rules write alerts-as-data documents into hidden indices that you can query like any other index.

The two aliases used in this article are .alerts-stack.alerts-default for Elasticsearch query rules and .alerts-observability.threshold.alerts-default for the Observability custom threshold rule.

The unit of storage is what we will call an episode: one document that covers one alert instance from the moment it becomes active until it recovers.

The fields that matter for investigation:

| Field | What it tells you | | :---- | :---- | | kibana.alert.instance.id | The grouping-key value this episode belongs to | | kibana.alert.start, kibana.alert.end | When the episode opened and recovered | | kibana.alert.duration.us | How long it stayed active | | kibana.alert.status | active or recovered | | kibana.alert.flapping, kibana.alert.flapping_history | Whether Kibana marked it as rapidly changing state | | kibana.alert.rule.name, kibana.alert.rule.revision | Which rule, and which version of it | | kibana.alert.grouping | The structured grouping values, for example service.name | | kibana.alert.reason | The human-readable condition at trigger time | | kibana.alert.url | A deep link back to the rule's data view |

Two properties make the rest of the article possible.

First, a new episode for the same instance creates a new document, so counting documents counts episodes.

Alert documents and notifications are decoupled: whether an episode also pages anyone depends on the actions attached to the rule and on their frequency, which can be every check, only on status changes, or a summary on a custom interval.

Our three rules have no actions attached, so nobody was paged during the lab; everything we count from here on is an alert document.

Second, grouping fields with ECS names such as service.name are copied to the top level of the alert document, which means alert history can join against other indices.

One caveat before you query: alert indices are wide.

The alerts index on this deployment maps roughly 1,400 fields, so every ES|QL query in this article projects a narrow set of columns with KEEP, and you should do the same, especially in tools that feed an LLM.

Because these are hidden system indices, the Discover editor cannot always introspect their fields, so it may flag the query with a warning or even an error badge while still running it and returning rows. Trust the result count in the toolbar, not the badge.

One more orientation note: Kibana screenshots below render timestamps in the browser's local time (UTC-5 in this lab), while the query result tables in the text use UTC.

The lab: how one bad deploy produces an alert storm

To generate an honest storm, we need services that fail in a chain, not a single threshold breach.

The scenario: fraud-scorer 2.3.0 ships with slow model loading, the payments queue backs up, payment-gateway requests start timing out, and customers see failed checkouts on payments-web.

Every service writes structured JSON logs, and an OpenTelemetry Collector tails them with a filelog receiver and ships them over OTLP/HTTP to the deployment's Elastic OpenTelemetry endpoint using an API key.

receivers:
  filelog/payment_gateway:
    include:
      - ${env:LAB_DIR}/events/payment-gateway.jsonl
    start_at: end
    resource:
      service.name: payment-gateway
      service.namespace: payments
      service.version: 5.1.2
      cloud.region: us-central1
    operators:
      - id: parse_json
        type: json_parser
        parse_from: body
      - id: parse_severity
        type: severity_parser
        parse_from: attributes.severity_text
      - id: use_message_as_body
        type: move
        from: attributes.message
        to: body

exporters:
  otlphttp/elasticsearch:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      Authorization: ApiKey ${env:ES_API_KEY}

A transform processor routes everything into one data stream, logs-payments.stormlab-default, and the records land with service.name, log.level, custom string attributes under labels.*, and numeric attributes under numeric_labels.*.

The full collector configuration for all three services, and a notebook that reproduces the whole lab, are in the companion repository.

We also load one small index that will do a lot of work later: a service catalog.

PUT payments_service_catalog
{
  "settings": { "index.mode": "lookup" },
  "mappings": {
    "properties": {
      "service.name": { "type": "keyword" },
      "team":         { "type": "keyword" },
      "tier":         { "type": "keyword" },
      "depends_on":   { "type": "keyword" }
    }
  }
}

Three documents map each service to its owning team, tier, and upstream dependency: fraud-scorer depends on payments-queue, payment-gateway on fraud-scorer, and payments-web on payment-gateway.

POST payments_service_catalog/_bulk
{ "index": {} }
{ "service.name": "fraud-scorer",    "team": "risk-ml",       "tier": "backend",  "depends_on": "payments-queue" }
{ "index": {} }
{ "service.name": "payment-gateway", "team": "payments-core", "tier": "edge",     "depends_on": "fraud-scorer" }
{ "index": {} }
{ "service.name": "payments-web",    "team": "storefront",    "tier": "frontend", "depends_on": "payment-gateway" }

index.mode: lookup is what later allows LOOKUP JOIN from ES|QL.

How grouping keys decide how many alerts you get

The storm is observed by three alerting rules, and their grouping keys are the real subject of this article.

| Rule | Rule type | Grouping key | Window | Threshold | | :---- | :---- | :---- | :---- | :---- | | gateway-5xx-per-endpoint-status | Elasticsearch query rule (ES|QL mode) | labels.endpoint, labels.status_code, cloud.region | 1 minute | 3 or more errors | | fraud-scorer-queue-lag | Elasticsearch query rule (ES|QL mode) | service.name, cloud.region | 1 minute | Max lag of 30 seconds or more | | payments-error-rate-per-service | Observability custom threshold rule | service.name | 2 minutes | More than 5 ERROR documents |

The first rule is deliberately miscalibrated in a way we have all shipped at some point: it groups gateway errors by endpoint and status code with a one-minute window and a low threshold.

FROM logs-payments.stormlab*
| WHERE service.name == "payment-gateway" AND log.level == "ERROR"
| STATS error_count = COUNT(*) BY labels.endpoint, labels.status_code, cloud.region
| WHERE error_count >= 3

It runs every minute as an Elasticsearch query rule in ES|QL mode, and each returned row becomes one alert instance, so /api/payments,503,us-central1 and /api/payments/confirm,504,us-central1 alert separately.

The second rule watches the first domino directly: consumer lag on the fraud queue, reported by the service in numeric_labels.queue_lag_seconds.

FROM logs-payments.stormlab*
| WHERE service.name == "fraud-scorer" AND numeric_labels.queue_lag_seconds IS NOT NULL
| STATS max_lag_seconds = MAX(numeric_labels.queue_lag_seconds) BY service.name, cloud.region
| WHERE max_lag_seconds >= 30

The third rule is an Observability custom threshold rule: more than five ERROR documents in two minutes, grouped by service.name.

It represents the sane default: one stable alert instance per affected service. It also demonstrates that different rule types feed the same investigation because all of them write alerts-as-data documents.

All three rules carry the tag alerts-as-data-v2, which is how every query below isolates them.

Tags on rules are not cosmetic: kibana.alert.rule.tags is stored on every alert document, so a tag is effectively a queryable dataset label for your alert history.

Replaying the incident: 18 alert episodes in seven minutes

We replay the incident: three minutes of healthy baseline, the 2.3.0 deploy, queue lag climbing past the threshold, six minutes of rotating gateway 5xx bursts, customer-visible errors, then a rollback and recovery.

The Alerts page during the storm looks like the pager felt.

Notice the table, not just the count.

The same instance IDs appear both as Active and as Recovered within minutes of each other, which is churn you can see but not yet measure.

When the dust settles, the preserved history can be queried in Discover's ES|QL mode.

The raw material of the storm, the error logs themselves, confirms the cascade shape:

FROM logs-payments.stormlab*
| WHERE labels.incident_id == "payments-brownout-20260722" AND log.level == "ERROR"
| STATS errors = COUNT(*) BY minute = DATE_TRUNC(1 minutes, @timestamp), service.name
| SORT minute

![Discover ES|QL view of error counts per minute and service: fraud-scorer errors first, then payment-gateway at seven per minute, then payments-web]]3

Now we leave the logs behind and interrogate only the alert history.

How to tell if an alert storm was one incident or many

FROM .alerts-*
| WHERE kibana.alert.rule.tags == "alerts-as-data-v2"
| STATS episodes = COUNT(*),
        alert_instances = COUNT_DISTINCT(kibana.alert.instance.id),
        first_episode = MIN(kibana.alert.start),
        last_episode = MAX(kibana.alert.start)
  BY rule = kibana.alert.rule.name
| SORT episodes DESC

| rule | episodes | alert_instances | first_episode | last_episode | | :---- | ----: | ----: | :---- | :---- | | gateway-5xx-per-endpoint-status | 14 | 7 | 16:36:38 | 16:42:38 | | payments-error-rate-per-service | 3 | 3 | 16:36:41 | 16:37:41 | | fraud-scorer-queue-lag | 1 | 1 | 16:35:35 | 16:35:35 |

Eighteen episodes, one incident, and every one of them opened inside a seven-minute window.

The gateway rule alone produced 14 of the 18 episodes while tracking only 7 distinct instances, which means half of its episodes were re-opens of something that had just recovered.

That single row is the quantified version of a complaint that usually stays anecdotal: this rule fires more than it helps.

How to find which service failed first with ES|QL

Because the grouping fields were ECS names, each episode carries service.name at the top level, and we can join the alert history against the service catalog.

FROM .alerts-*
| WHERE kibana.alert.rule.tags == "alerts-as-data-v2" AND service.name IS NOT NULL
| STATS first_symptom = MIN(kibana.alert.start) BY service.name
| LOOKUP JOIN payments_service_catalog ON service.name
| KEEP first_symptom, service.name, team, tier, depends_on
| SORT first_symptom

| first_symptom | service.name | team | tier | depends_on | | :---- | :---- | :---- | :---- | :---- | | 16:35:35 | fraud-scorer | risk-ml | backend | payments-queue | | 16:36:41 | payment-gateway | payments-core | edge | fraud-scorer | | 16:37:41 | payments-web | storefront | frontend | payment-gateway |

The alert order matches the dependency chain exactly: the first symptom belongs to team risk-ml, the edge degraded 66 seconds later, and customers noticed a minute after that.

There is a subtler deduction available in this table.

The earliest alerting service depends on payments-queue, which never alerted at all, so the most likely true origin sits one hop upstream of the first visible symptom, and that is where a post-incident review should start reading logs.

Also note which rule is absent from this analysis: the noisy gateway rule grouped by endpoint and status code, so its episodes carry no service.name and cannot participate in the topology join.

A grouping key is the data contract for your future investigations.

How to score which alerting rule is noisiest

A noise scorecard query condenses rule quality into one table.

FROM .alerts-*
| WHERE kibana.alert.rule.tags == "alerts-as-data-v2"
| EVAL minutes_active = COALESCE(kibana.alert.duration.us, 0) / 60000000.0
| STATS episodes = COUNT(*),
        alert_instances = COUNT_DISTINCT(kibana.alert.instance.id),
        flapping_episodes = SUM(CASE(kibana.alert.flapping == true, 1, 0)),
        median_minutes_active = MEDIAN(minutes_active)
  BY rule = kibana.alert.rule.name, revision = kibana.alert.rule.revision
| SORT episodes DESC

| rule | revision | episodes | instances | flapping | median minutes active | | :---- | ----: | ----: | ----: | ----: | ----: | | gateway-5xx-per-endpoint-status | 0 | 14 | 7 | 0 | 1.0 | | payments-error-rate-per-service | 0 | 3 | 3 | 0 | 7.0 | | fraud-scorer-queue-lag | 0 | 1 | 1 | 0 | 9.0 |

Read the last column first.

A median active time of one minute means the typical episode from the gateway rule was already resolving itself before the on-call could even sign in, while the two well-grouped rules produced episodes long enough to represent the actual incident.

The flapping column is Kibana's own attempt at the same problem, and it is damage control.

The real fix is upstream, in the rule definition, and the alert history has just told us exactly what to change: the grouping key manufactures instances, and the one-minute window manufactures re-opens.

Alert noise reduction: fixing the rule and proving it worked

The rule's own detail page already summarizes the problem: 14 alerts in 24 hours, every one of them from a single incident, produced by the definition on the right.

We update the rule rather than replace it, changing the grouping to service.name, widening the window to five minutes, and raising the threshold.

FROM logs-payments.stormlab*
| WHERE service.name == "payment-gateway" AND log.level == "ERROR"
| STATS error_count = COUNT(*) BY service.name, cloud.region
| WHERE error_count >= 10

Updating a rule in place matters for one specific reason: Kibana stamps every alert document with kibana.alert.rule.revision, so the before and after of a tuning change are separable in the same index with no bookkeeping on our side.

Then we replay a compressed version of the same failure: same deploy, same lag ramp, same rotating gateway bursts, same customer impact, rollback after seven minutes.

The scorecard query runs unchanged, and now the revision column earns its place:

| rule | revision | episodes | instances | flapping | median minutes active | | :---- | ----: | ----: | ----: | ----: | ----: | | gateway-5xx-per-endpoint-status | 0 | 14 | 7 | 0 | 1.0 | | payments-error-rate-per-service | 0 | 6 | 3 | 0 | 6.0 | | fraud-scorer-queue-lag | 0 | 2 | 1 | 0 | 8.0 | | gateway-5xx-per-endpoint-status | 1 | 1 | 1 | 0 | 7.0 |

The unchanged rules now show both incidents in their rows, six and two episodes, which is exactly what stable rules should do.

The gateway rule splits into two rows because the revision changed, and the contrast is the entire argument of this article in four numbers.

Same failure shape, one episode instead of fourteen, and the median episode now lasts seven minutes instead of one, so it spans the incident instead of a fraction of it.

This is the part of alerting work that usually relies on faith, and here it is a query result: the tuning change is verified against the same incident pattern, in the same index, by the same ES|QL.

Nothing about this loop is specific to our lab rule.

Any rule that tags its alerts can be scored, tuned, and re-scored this way, which turns alert tuning from an opinion into a measurement.

Querying alert history with an Agent Builder agent

Every query above is deterministic, which makes them ideal tools for an AI agent.

We register three ES|QL tools in Agent Builder, each one a parameterized version of a query from this article with an explicit KEEP projection, and wire them into an agent called Alert Historian.

POST kbn:/api/agent_builder/tools
{
  "id": "alert_noise_scorecard",
  "type": "esql",
  "description": "Noise scorecard per alerting rule and revision: episodes, distinct instances, flapping episodes, median minutes active.",
  "configuration": {
    "query": "FROM .alerts-* | WHERE kibana.alert.rule.tags == \"alerts-as-data-v2\" AND DATE_DIFF(\"hours\", kibana.alert.start, NOW()) <= ?lookback_hours | EVAL minutes_active = COALESCE(kibana.alert.duration.us, 0) / 60000000.0 | STATS episodes = COUNT(*), alert_instances = COUNT_DISTINCT(kibana.alert.instance.id), flapping_episodes = SUM(CASE(kibana.alert.flapping == true, 1, 0)), median_minutes_active = MEDIAN(minutes_active) BY rule = kibana.alert.rule.name, revision = kibana.alert.rule.revision | KEEP rule, revision, episodes, alert_instances, flapping_episodes, median_minutes_active | SORT episodes DESC",
    "params": {
      "lookback_hours": {
        "type": "integer",
        "description": "How many hours of alert history to analyze, for example 6"
      }
    }
  }
}

The agent's instructions define the vocabulary (an episode is one alert document), the method (timeline first, then propagation, then scorecard), and the honesty rules (report numbers and timestamps from tool output, never invent alerts).

We ask it one question: what happened in the last three hours, was it one incident or many, which team owns the first symptom, and did the gateway rule tuning reduce noise?

The agent called all three tools and got the structure right: it separated the history into two storms with a quiet gap, one per failure wave, rather than blending them into one event.

It attributed the first symptom to team risk-ml, reproduced the upstream deduction about the queue that never alerted, and then answered the tuning question with the revision table.

Its verdict on the fix reads like a review comment backed by data: fourteen episodes at revision 0 against one stable episode at revision 1, with the median episode going from one minute to seven.

The agent is not doing anything we could not do by hand.

That is precisely the point: because alerts are durable, structured data with explicit semantics, a language model can reason over patterns and histories instead of reacting to a single notification, and every number it states is traceable to a tool call you can rerun.

Running alert history queries in production

A few things to decide before you rely on alert history in production.

Retention is a learning-window decision

Alert indices follow an ILM policy, and the default keeps far more than the few hours this lab needed, but if you want quarter-over-quarter rule quality trends, align the policy with that ambition deliberately.

Scope alert index read access narrowly

Alert documents embed rule parameters, reasons, and deep links, so treat .alerts-* read privileges like any other operational dataset and grant the narrowest index patterns that work, for example only the observability alert aliases.

Grouping keys are a data contract

Use ECS field names in BY clauses when they exist, keep cardinality bounded, and never put secrets or personal data into grouping values, because they become kibana.alert.instance.id and live for the retention window.

Project columns aggressively with KEEP

With around 1,400 mapped fields in the alert indices, an unprojected query is unpleasant for humans and actively harmful inside LLM tools, so end every reusable query with KEEP.

Alert storms also stress your automation

If a rule triggers a workflow per alert, fourteen episodes mean fourteen executions doing the same work, which is another argument for pointing automation at the aggregated alert dataset rather than at each notification.

Where to start with your own noisiest rule

The lab in this article is intentionally small, and every piece of it generalizes.

Score your noisiest real rule with the scorecard query, fix its grouping or window, and let kibana.alert.rule.revision tell you whether the fix held.

Join your alert history against whatever catalog you already have; a CMDB export in a lookup index is enough, and the propagation order becomes a query.

If you use Agent Builder, wrap your two or three most-used investigation queries as tools, and the next storm review starts with a conversation instead of a wall of notifications.

The companion notebook builds everything used here: the catalog index, the log data stream, the three rules, both incident replays, the queries, and the Agent Builder tools and agent. It runs for about 25 minutes because the rules evaluate once a minute and the episode durations are the data.

A notification tells you something broke.

The alert history tells you how your detection is performing, and it is already sitting in an index waiting to be queried.

If you run Observability on Elastic Cloud Hosted (ECH), this will feel familiar. Cross-cluster search (CCS) has long enabled teams to query across remote clusters, and it's a mainstay in large scale ECH environments. CPS takes the same idea to Serverless, and makes linking even simpler: no topology mapping, certificate configuration or per connection credentials. You link projects in the Cloud console and Elastic manages the rest.

What is cross-project search?

Serverless organizes your data into projects: Elasticsearch, Observability, or Security, each with its own scaling, permissions, and lifecycle. That isolation is a feature: it preserves data locality where needed, simplifies cost attribution, and lets teams own their own data. The tradeoff, historically, has been visibility. If your logs live in one project and your APM data lives in another, or if every business unit runs its own Observability project, getting a unified view has typically meant duplicating data into a shared store.

CPS removes that tradeoff. You designate an origin project (the project you search from) and link it to other projects you want included in that search. Once linked, an unqualified query like FROM logs* or a Discover session automatically spans the origin plus every linked project; no special syntax, no remote index prefixes. Kibana's project picker lets anyone narrow a query to just the current project or expand it to everything in scope, and administrators can set a default scope per Kibana space so a local incident-response space stays local by default while a global operations space starts broad. Access is enforced per project using your existing Elastic Cloud roles, so linking a project for search doesn't quietly grant broader permissions than a user already has.

What's the difference between cross-project search and cross-cluster search?

Cross-cluster search (ECH, self-managed)

Cross-project search (Serverless)

Where you link

Remote cluster connections, configured per cluster

Link projects in the Elastic Cloud console

Credentials

Certificates or API keys you issue and rotate

Managed by Elastic

Network setup

Network topology you map and maintain

None to configure

Query syntax

Remote cluster name referenced in the query

Unqualified query, no remote index prefix

Scope control

Set per query

Project picker, per-space defaults, and tag-based routing

Where CCS asks administrators to configure remote cluster connections, exchange certificates or API keys, and reference remote cluster names in every query, CPS asks you to link projects in the UI and query the way you already do. It's the same underlying idea: remote search without moving data, adapted to a control plane where infrastructure is meant to disappear.

Why cross-project search matters for Observability teams

Cross-project search matters most the moment your observability data stops living in one place. Observability organizations typically split their data by team, environment, region, or customer, usually for good reasons: isolating noisy neighbors, controlling blast radius, meeting data residency requirements, or giving each team ownership over its own ingest and retention. The problem shows up during an incident or a cross-cutting investigation, when someone needs to see across those boundaries without waiting for a data migration or standing up a consolidated cluster.

A few examples of where CPS changes the equation:

Keep observability data in its region and still search globally

Teams operating in multiple regions or clouds often need to keep observability data in the region it was generated, whether for regulatory reasons or to limit cross-region egress. CPS lets each region keep its own Observability project while a central project links to all of them, so an SRE team can run one query to check whether an anomaly in one region is also showing up elsewhere, without pulling all that data into one place first.

Isolate environments without losing the incident view

Splitting production, staging, and per-service or per-team projects makes scaling and cost predictable, and it limits how far a bad deploy or a runaway job can spread. It also means an on-call engineer used to have to check several places during an incident. With CPS, on-call can work from a single origin project and pull in whichever linked environments matter to the investigation, then narrow back down once things are resolved.

Central teams monitoring many business units or customers

Platform or Observability Center of Excellence (CoE) teams that operate on behalf of many business units, or Managed Service Providers (MSPs) monitoring several customer environments, can give each unit or customer its own project, with its own retention, access, and cost, while the central team links across all of them for cross-cutting reporting and anomaly detection. Each project owner keeps control of their own data; the center gets the aggregate view it needs.

Avoid ingesting the same data twice

When two teams need the same data for different purposes, the old options were duplicate ingestion pipelines or a shared, less isolated deployment. With linked projects, data is ingested once and queried from wherever it's needed, which keeps both storage costs and pipeline maintenance down.

This applies across solution types too: CPS links Elasticsearch, Observability, and Security projects interchangeably, so a Security team and an Observability team can each keep their own project, with their own retention and access controls, while querying the same underlying logs. A security analyst investigating an alert can query infrastructure or service context from a linked Observability project in Discover, dashboards, or ES|QL, and an SRE debugging an incident can query related signals from a linked Security project the same way, all without either team re-ingesting the other's data.

What does cross-project search support in Elastic Observability?

At GA, CPS is available across Elasticsearch, Observability, and Security project types (Observability and Security projects require the Complete tier). For Elastic Observability specifically, application and infrastructure monitoring, alerting rules, SLOs, machine learning anomaly detection (including APM anomaly detection), and Synthetics work across linked projects, alongside the platform-level capabilities every project gets: Discover, dashboards, and ES|QL. Project routing lets you target a query to a subset of linked projects using predefined tags (alias, cloud provider, region) or your own custom tags, combined with boolean logic, so "all EU projects" or "every environment except staging" can be expressed once and reused.

This release also raises the default number of projects you can link to a single origin, well beyond the tech preview limit, with room to go higher for larger deployments. Any existing project can now act as an origin rather than only newly created ones. For programmatic access, you only need a Cloud API key to call CPS from your own tooling. A new dedicated cloud role can now grant those keys for programmatic access.

A few nuances are worth knowing before you design around CPS:

• Alerting: Alerting rules defined in the origin project can evaluate data across linked projects. The alerts they generate stay local to the project where the alerting rule lives.
• SLOs: SLOs don't follow the project picker in the Kibana header. Instead, you set scope in the SLO create and edit form, choosing this project, all linked projects, or a specific subset, since changing scope rebuilds the SLO's underlying computation. You can't view SLOs created in linked projects from the current project, but you can create an SLO here based on data that lives in those projects.
• Synthetics: Overview, monitor details, certificates, errors, and journeys honor the Kibana header project picker and show linked-project monitors as read-only. Creating, editing, and managing monitors stay in the origin project, and Synthetics status and TLS rules still evaluate origin monitors only.
• Streams: Not CPS-aware yet; Streams stay scoped to the project where they were created.

How do you set up cross-project search?

Setting up cross-project search takes a few minutes:

1. From your origin project's settings in the Elastic Cloud console, open Link projects.
2. Select the projects you want to link, individually or in bulk.
3. Save.
4. Open Discover, run an ES|QL query, or build a dashboard as you normally would: linked project data is included automatically, and the project picker lets you narrow or widen the scope at any time.

For the full configuration walkthrough, including linking topologies (hub-and-spoke, shared data project, or full mesh) and how to scope project routing with tags and expressions, see the cross-project search documentation and the Observability-specific guide.

Cross-project search pricing

CPS pricing has two components: a charge for data retained in each linked project you query, and a charge for data egress generated by cross-project query traffic, tracked as its own line item so you can see what CPS is adding to your transfer costs. There's no separate per-link fee. Because actual cost depends on your data volumes, retention settings, and how many projects you link, use the Elastic Cloud Serverless pricing estimator to model costs for your own environment against your own ingest and retention numbers.

Start using cross-project search

If you're already running Elastic Observability on Serverless, cross-project search is available in your project settings today: link your projects and run your first cross-project query in Discover. If you haven't tried Elastic Cloud Serverless yet, start a free trial and see how far a single query can reach.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic, Elasticsearch, and associated marks are trademarks, logos, or registered trademarks of Elasticsearch B.V. in the United States and other countries. All other company and product names are trademarks, logos, or registered trademarks of their respective owners.

The Datadog Kubernetes dashboard used in this migration

The walkthrough uses Kubernetes - Overview from infra/datadog/dashboards/integrations/kubernetes.json in the migration repository. It is a cluster-wide board with the signals operators check during an incident: pod counts, CPU and memory by host or pod, non-running pods, and containers stuck in CrashLoopBackOff.

Below are representative queries from the source dashboard:

# Pod CPU by host
sum:kubernetes.cpu.usage.total{$scope,$cluster,$label,$node} by {host}

# Pod memory by pod
sum:kubernetes.memory.usage{$scope,$deployment,$statefulset,$replicaset,$daemonset,$cluster,$namespace,!pod_name:no_pod,$label,$service,$node} by {pod_name}

# Pods not running (pressure / scheduling signal)
sum:kubernetes_state.pod.status_phase{$scope,$cluster,$namespace,$deployment,$statefulset,$replicaset,$daemonset,!pod_phase:running,!pod_phase:succeeded,$label,$node,$service} by {kube_cluster_name,kube_namespace,pod_phase}

# CrashLoopBackOff
sum:kubernetes_state.container.status_report.count.waiting{$cluster,$namespace,$deployment,$statefulset,$replicaset,$daemonset,reason:crashloopbackoff,$scope,$daemonset,$label,$node,$service} by {pod_name}

If this board translates cleanly, most production Datadog Kubernetes folders are worth testing with the same workflow.

Why Datadog-to-Elastic migration is faster now

The migration platform automates the query translation and panel rebuilds that used to dominate Datadog moves. Elasticsearch stores Kubernetes metrics efficiently and runs the ES|QL queries those panels use. See Elasticsearch as a metrics backend for benchmark context and storage comparisons.

The platform maps Datadog queries to Kibana panels, validates ES|QL against live data, and writes artifacts you can inspect before anything goes to production.

Prerequisites

You need an Elastic Observability Serverless project, a project API key, and the migration CLI installed from the Observability Migration Platform repository.

Export your endpoints and API key:

export ELASTICSEARCH_ENDPOINT="https://YOUR_ES_ENDPOINT"
export KIBANA_ENDPOINT="https://YOUR_KIBANA_ENDPOINT"
export KEY="YOUR_API_KEY"

Install the CLI and confirm the toolchain:

python3 -m venv .venv
.venv/bin/pip install ".[all]"
.venv/bin/obs-migrate doctor

The doctor command checks compile and lint dependencies. Resolve any errors before you migrate production dashboards. Pin a release tag if you plan to run this in CI.

To pull dashboards from the Datadog API instead of JSON files, copy datadog_creds.env.example to datadog_creds.env and set DD_API_KEY, DD_APP_KEY, and DD_SITE.

Ingest Kubernetes metrics first

Empty panels after upload usually mean Elasticsearch does not yet have the series the Datadog queries reference. Make sure to confirm ingest before you run the migration.

There are two common paths to do so:

1. OpenTelemetry into managed OTLP with Kubernetes receivers (kubeletstats, k8s_cluster), then explore in Discover
2. Existing Prometheus or agent pipelines that already write pod and node metrics to metrics-*

The migration CLI accepts --field-profile otel to map Datadog tags such as pod_name, kube_namespace, and kube_cluster_name to OpenTelemetry fields like kubernetes.pod.name and kubernetes.namespace. If panels are empty after migration, verify field mapping and the selected time range before you change translator settings.

Run the Datadog dashboard migration CLI

Export the Datadog dashboard JSON from the UI, or copy the sample kubernetes.json from infra/datadog/dashboards/integrations/ in the migration repo. Place files in a directory such as ./datadog_k8s_exports/.

Run the migration from that directory:

datadog-migrate \
  --source files \
  --input-dir ./datadog_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --logs-index "logs-*" \
  --upload \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --ensure-data-views \
  --create-alert-rules \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY"

These flags matter for Kubernetes boards:

• --field-profile otel maps Datadog Kubernetes fields to OpenTelemetry field names in Elasticsearch
• --assets all includes dashboards and Datadog monitor definitions when present
• --validate runs emitted ES|QL against your cluster before upload
• --create-alert-rules creates Kibana rules in a disabled state

The unified CLI performs the same work:

obs-migrate migrate \
  --source datadog \
  --input-mode files \
  --input-dir ./datadog_k8s_exports \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --logs-index "logs-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload \
  --create-alert-rules

To fetch a dashboard from Datadog directly:

datadog-migrate \
  --source api \
  --env-file datadog_creds.env \
  --dashboard-ids YOUR_DASHBOARD_ID \
  --output-dir ./migration_output \
  --assets all \
  --field-profile otel \
  --data-view "metrics-*" \
  --validate \
  --es-url "$ELASTICSEARCH_ENDPOINT" \
  --es-api-key "$KEY" \
  --kibana-url "$KIBANA_ENDPOINT" \
  --kibana-api-key "$KEY" \
  --upload

Validate the migrated Datadog dashboard in Kibana

Open Kibana → Dashboards and locate the migrated Kubernetes - Overview board. Confirm that cluster and namespace pod counts, CPU and memory series, pod phase panels, CrashLoopBackOff widgets, and deployment replica charts return data for your selected time range.

If you migrated monitors, open Observability → Rules. Imported rules remain disabled until you enable them after reviewing thresholds.

The CLI also writes local artifacts under ./migration_output/:

• dashboards/yaml/ contains the translated dashboard definition.
• dashboards/migration_report.json lists panels that translated automatically and panels flagged for manual review.
• alerts/ contains monitor translations when monitors were included in the export.

Handle manual-review panels

Some Datadog widget types do not translate on the first pass. Exotic formulas, log-only panels, and unsupported widgets appear as manual-review entries in the migration report rather than as silently broken charts.

| Result | Recommended action | | --- | --- | | Panel returns data | Accept the translation and continue | | Panel is empty | Confirm metric names and data_stream.dataset values in metrics-*, then widen or shift the time range | | Manual-review marker | Open the original Datadog query and simplify or redesign the panel | | Monitor never fires | Confirm the rule is enabled and thresholds match your environment |

Datadog coverage is narrower than Grafana in some areas. Read the migration report before you commit to full parity with leadership. The platform prefers conservative failures over uploading panels that look correct but query the wrong fields.

Related Datadog and Grafana migration guides

For the Grafana PromQL version of this workflow, see Migrate your Grafana Kubernetes dashboard to Elastic Observability. For platform-level context, see Migrating Datadog and Grafana dashboards and alerts to Kibana. Review known limitations before you migrate every production folder.

This post looks at what happens inside Elasticsearch when a Remote Write request arrives.

If you want to understand the implementation, evaluate how Elasticsearch compares to other Prometheus-compatible backends, or contribute, this is the post for you. A companion post, Ship Prometheus Metrics to Elasticsearch with Remote Write, covers the setup and configuration side.

Request lifecycle: from HTTP to indexed documents

A quick note on the Prometheus data model before we dive in: Prometheus stores all metric values as 64-bit floats and treats the metric name as just another label (__name__). The storage engine itself is agnostic of whether a value is a counter or a gauge. Keep this in mind as we walk through how Elasticsearch maps these concepts.

Here is the full path of a Remote Write request through Elasticsearch:

1. HTTP layer — The endpoint receives a compressed protobuf payload, checks indexing pressure, decompresses with Snappy, and parses the protobuf WriteRequest.
2. Document construction — Each sample in each time series becomes an Elasticsearch document with @timestamp, labels.*, and metrics.* fields.
3. Bulk indexing — All documents from a single request are written to the target data stream via a single bulk call.

The sections below walk through each stage in detail.

HTTP layer

The endpoint accepts application/x-protobuf POST requests. The incoming request body is tracked against the same indexing pressure limits that protect the bulk indexing API. If the cluster is already under heavy indexing load, the request gets rejected with a 429 before any parsing happens.

Prometheus compresses Remote Write payloads with Snappy. Elasticsearch decompresses the body in a streaming fashion without materializing it into a single contiguous allocation, and validates the declared uncompressed size against a configurable maximum to guard against decompression bombs.

The decompressed body is then deserialized as a protobuf WriteRequest. Each WriteRequest contains a list of TimeSeries entries, and each TimeSeries contains a set of labels (key-value pairs) and a list of samples (timestamp + float64 value).

Document construction

For each sample in each time series, Elasticsearch builds an index request. Here is what a single document looks like:

{
  "@timestamp": "2026-04-01T12:00:00.000Z",
  "data_stream": {
    "type": "metrics",
    "dataset": "generic.prometheus",
    "namespace": "default"
  },
  "labels": {
    "__name__": "http_requests_total",
    "job": "prometheus",
    "instance": "localhost:9090",
    "method": "GET",
    "status": "200"
  },
  "metrics": {
    "http_requests_total": 1027.0
  }
}

All labels from the Prometheus time series (including __name__) end up in the labels.* fields. The metric value goes into metrics.<metric_name>, where <metric_name> is the value of the __name__ label.

Time series without a __name__ label are dropped entirely, and the samples are counted as failures. Non-finite values (NaN, Infinity, negative Infinity) are silently skipped. This includes Prometheus staleness markers, which use a special NaN bit pattern (0x7ff0000000000002) to signal that a series has disappeared.

One sample, one document

You might wonder whether storing each individual sample as its own document creates significant storage overhead, especially for labels. A common pattern to reduce that overhead was to group all metrics sharing the same labels and timestamp into a single document.

With recent TSDB improvements, that optimization is no longer necessary. Elasticsearch has trimmed the per-document storage overhead to the point where there is negligible difference between packing many metrics in a single document and writing each sample separately. A dedicated post covering these TSDB storage improvements in detail is coming soon.

Bulk indexing

All documents from a single Remote Write request are sent to Elasticsearch via a single bulk request. Each document targets the data stream metrics-{dataset}.prometheus-{namespace} and is indexed as an append-only create operation.

Metric type inference

Remote Write v1 does not reliably transmit metric types alongside samples. Prometheus sends metadata (type, help text, unit) in separate requests roughly once per minute, and those requests may land on a different node than the samples. Buffering samples until metadata arrives is not practical in a distributed system, so Elasticsearch infers the type from naming conventions instead.

Metric names ending in _total, _sum, _count, or _bucket are mapped as counters. Everything else defaults to gauge. This is a well-established convention that other Prometheus-compatible backends use as well.

http_requests_total             → counter
request_duration_seconds_sum    → counter
request_duration_seconds_count  → counter
request_duration_seconds_bucket → counter
process_resident_memory_bytes   → gauge
go_goroutines                   → gauge

The heuristic can be wrong. A metric like temperature_total (if someone named a gauge that way) would be misclassified as a counter. The main consequence today is that some ES|QL functions like rate() require the metric type to be a counter and will reject a misclassified gauge. For PromQL, we plan to lift this restriction so that rate() works regardless of the declared type, which will make incorrect inference less consequential.

You can override the inference by creating a metrics-prometheus@custom component template with custom dynamic templates. For example, to treat all *_counter fields as counters:

PUT /_component_template/metrics-prometheus@custom
{
  "template": {
    "mappings": {
      "dynamic_templates": [
        {
          "counter": {
            "path_match": "metrics.*_counter",
            "mapping": {
              "type": "double",
              "time_series_metric": "counter"
            }
          }
        }
      ]
    }
  }
}

Custom dynamic templates are merged with the built-in ones, so the default naming-convention rules still apply for metrics you don't explicitly override.

The index template

Elasticsearch installs a built-in index template that matches metrics-*.prometheus-*. This template is what makes field type inference work without manual mapping configuration.

TSDS mode is enabled, which gives you time-based partitioning, optimized storage, deduplication, and the ability to downsample data as it ages.

Passthrough object fields are used for both the labels and metrics namespaces. This serves three purposes:

1. Namespace isolation: Labels and metrics live in separate object namespaces (labels.* and metrics.*), so a label named status and a metric named status cannot conflict with each other.

2. Dimension identification: The labels passthrough object is configured with time_series_dimension: true, which means every field under labels.* is automatically treated as a TSDS dimension. When Prometheus sends a time series with a label you have never seen before, it becomes a dimension without any explicit field mapping.

3. Transparent queries: You don't need to write the labels. or metrics. prefix in ES|QL or PromQL. A query can reference job instead of labels.job, or http_requests_total instead of metrics.http_requests_total. The passthrough mapping handles the resolution.

Dynamic inference for metrics applies the naming-convention heuristics described above. When a new metric name appears for the first time, its field mapping is created automatically under metrics.* with the correct time_series_metric annotation.

Failure store is enabled. Documents that fail indexing (for example, due to a mapping conflict where the same metric name appears with incompatible types) are routed to a separate failure store instead of being dropped silently.

Data stream routing

The three URL patterns map directly to data stream names:

| URL pattern | Data stream | |---|---| | /_prometheus/api/v1/write | metrics-generic.prometheus-default | | /_prometheus/metrics/{dataset}/api/v1/write | metrics-{dataset}.prometheus-default | | /_prometheus/metrics/{dataset}/{namespace}/api/v1/write | metrics-{dataset}.prometheus-{namespace} |

This lets you separate metrics from different Prometheus instances or environments into different data streams. That separation is useful for a few reasons.

Lifecycle isolation: you can apply different retention policies per data stream. Production metrics might be kept for 90 days, while dev metrics might expire after 7 days.

Access control: you can scope API keys to specific data streams. A team's Prometheus instance writes to metrics-teamA.prometheus-prod, and their API key only has access to that stream.

Query performance: PromQL queries and Grafana dashboards can be scoped to a specific index pattern, avoiding scans of unrelated data.

Error handling and the Remote Write spec

The Remote Write spec defines two response classes: retryable (5xx, 429) and non-retryable (4xx). Prometheus uses this distinction to decide whether to retry or drop a failed request.

Elasticsearch returns 429 (Too Many Requests) if any sample in the bulk request was rejected due to indexing pressure. This signals Prometheus to back off and retry with exponential backoff.

For partial failures (some samples indexed, others rejected), the response includes a summary. It reports how many samples failed, grouped by target index and status code, along with a sample error message from each group.

Time series without a __name__ label result in a 400 error for those samples. Non-finite values (NaN, Infinity) are silently dropped: Prometheus receives a success response and will not retry.

NaN appears most commonly for summary quantiles when no observations have been recorded (for example, a p99 latency metric before any requests arrive) and for staleness markers. The practical impact of dropping these is limited today: for most queries, a missing sample behaves similarly to a NaN one, since PromQL's lookback window fills the gap with the last known value either way. The more significant gap is staleness markers, which are covered below.

What's next: Remote Write v2 and beyond

Remote Write v2 is still experimental, which is why the current implementation starts with v1. But v2 addresses several of v1's shortcomings.

Metadata alongside samples: v2 sends metric type, unit, and description with each time series in the same request. This eliminates the need for naming-convention heuristics entirely.

Native histograms: v2 supports Prometheus native histograms, which map naturally to Elasticsearch's exponential_histogram field type. Classic histograms (one counter per bucket boundary) are verbose and lose precision at query time. Native histograms are more compact and more accurate.

Dictionary encoding: v2 replaces repeated label strings with integer references, reducing payload size significantly for high-cardinality label sets.

Created timestamps: counters in v2 include a "created" timestamp that marks when the counter was initialized. This allows backends to detect counter resets more accurately than the current heuristic (value decreased since last sample).

Beyond v2, there are two other items in consideration for future enhancements.

Staleness marker support: currently, staleness markers (the special NaN that Prometheus writes when a scrape target disappears) are dropped. Supporting them would allow correct PromQL lookback behavior and avoid the 5-minute "trailing data" artifact where a disappeared series still appears in query results.

Shared metric field: the current layout creates a separate field for each metric name (metrics.http_requests_total, metrics.go_goroutines, etc.). This works, but it means the number of field mappings grows with the number of distinct metric names, which is why the field limit is set to 10,000 for Prometheus data streams. A different approach we're considering is to store the metric name only in the __name__ label and write the metric value to a single shared field. This eliminates the field explosion problem entirely and more closely matches how Prometheus stores data internally. This direction is part of the broader effort to make Elasticsearch's metrics storage more efficient and more compatible with Prometheus conventions.

Availability

The Prometheus Remote Write endpoint is available now on Elasticsearch Serverless with no additional configuration.

For self-managed clusters, check out start-local to get up and running quickly.

If you run into issues or have feedback, open an issue on the Elasticsearch repository.

In benchmarks, LogsDB brought a dataset from 162.7 GB down to 39.4 GB — a 76% reduction in storage. You can explore the full nightly benchmark results at elasticsearch-benchmarks.elastic.co.

In this tutorial you'll reproduce the experiment yourself using Kibana Dev Tools and an Apache logs dataset. You'll create two identical indices, ingest the same documents into both, and measure the storage difference with the _stats API. By the end, you'll see a 44% reduction on your test data — and understand exactly why production numbers push even higher.

Already on Elasticsearch 9.2+? Any data stream with a logs- prefix already uses LogsDB by default. Jump to What about your existing logs? to verify your setup.

Want the full picture? For the engineering history behind these savings — how Lucene doc values, synthetic _source, index sorting, and ZSTD were developed and stacked over twelve years — see Elasticsearch over the years: how LogsDB cuts index size by up to 75%.

Prerequisites

• Elasticsearch 8.17+ cluster, Elastic Cloud deployment, or Serverless
• Kibana with Dev Tools access
• Some logs
• Basic familiarity with running API calls in Kibana Dev Tools

How LogsDB saves storage

LogsDB stacks three mechanisms to achieve its storage reduction:

• Index sorting — documents are sorted by host.name then @timestamp, grouping similar log lines so compression codecs find far more repeated patterns. Sorting alone accounts for roughly 30% of the savings.
• ZSTD compression with delta/GCD/run-length encoding — best_compression switches from LZ4 to Zstandard and applies numeric codecs to each doc values column. The standard index in this tutorial uses LZ4, so part of what you're measuring is the full package LogsDB delivers automatically.
• Synthetic _source — Elasticsearch skips storing the raw JSON blob entirely and reconstructs _source on demand from doc values, adding another 20–40% of savings on top.

Synthetic _source trade-offs: Field ordering in returned documents may differ from the original, and some edge cases around multi-value array fields behave differently. For most log analytics workloads these differences are invisible, but check the synthetic _source documentation before enabling it in latency-sensitive applications.

For a deep dive into the architecture behind each mechanism, see Elasticsearch over the years: how LogsDB cuts index size by up to 75%.

Let's now walk through the steps you can take to enable LogsDB and measure the storage savings.

Step 1: Collect logs with Elastic Agent

The recommended way to ingest Apache logs into Elasticsearch is through Elastic Agent with the Apache integration. It handles collection, parsing, ECS field mapping, and routing automatically.

Browse all available integrations in the Elastic integrations catalog.

Once the Agent is collecting logs and routing them to logs-apache.access-*, move to the next step.

Step 2: Create the two indices

All commands in this tutorial are run in Kibana Dev Tools.

Create one standard index and one LogsDB index with identical field mappings. The only difference is "index.mode": "logsdb".

Standard index:

PUT /apache-standard
{
  "mappings": {
    "properties": {
      "@timestamp":                  { "type": "date" },
      "host.name":                   { "type": "keyword" },
      "http.request.method":         { "type": "keyword" },
      "url.path":                    { "type": "keyword" },
      "http.version":                { "type": "keyword" },
      "http.response.status_code":   { "type": "integer" },
      "http.response.bytes":         { "type": "integer" },
      "http.request.referrer":       { "type": "keyword" },
      "user_agent.original":         { "type": "keyword" }
    }
  }
}

LogsDB index:

PUT /apache-logsdb
{
  "settings": {
    "index.mode": "logsdb"
  },
  "mappings": {
    "properties": {
      "@timestamp":                  { "type": "date" },
      "host.name":                   { "type": "keyword" },
      "url.path":                    { "type": "keyword" },
      "http.request.method":         { "type": "keyword" },
      "http.version":                { "type": "keyword" },
      "http.response.status_code":   { "type": "integer" },
      "http.response.bytes":         { "type": "integer" },
      "http.request.referrer":       { "type": "keyword" },
      "user_agent.original":         { "type": "keyword" }
    }
  }
}

That single "index.mode": "logsdb" line activates all three storage mechanisms. Elasticsearch enables these additional settings behind the scenes — you don't set any of them manually:

{
  "index.sort.field":              ["host.name", "@timestamp"],
  "index.sort.order":              ["asc", "desc"],
  "index.codec":                   "best_compression",
  "index.mapping.ignore_malformed": true,
  "index.mapping.ignore_above":    8191
}

Step 3: Reindex the logs

Use the _reindex API to copy the same documents into both test indices:

POST /_reindex
{
  "source": { "index": "logs-apache.access-*" },
  "dest":   { "index": "apache-standard" }
}

POST /_reindex
{
  "source": { "index": "logs-apache.access-*" },
  "dest":   { "index": "apache-logsdb" }
}

Both indices now hold identical documents, so the storage comparison in the next step reflects only the index mode difference.

Step 4: Force merge for a fair comparison

Before measuring, force merge both indices to a single segment:

POST /apache-standard/_forcemerge?max_num_segments=1

POST /apache-logsdb/_forcemerge?max_num_segments=1

These calls block until the merge finishes. Wait for both responses before continuing.

Why this matters: Elasticsearch writes data into multiple Lucene segments before merging them in the background. Measuring mid-merge gives artificially inflated numbers because each segment is compressed independently. Forcing a single segment shows the real steady-state storage footprint you'd see in a mature production index.

Only run _forcemerge on indices that are no longer being written to. Force merging an index that is still receiving writes is resource-intensive and can impact ingestion performance. In production, you can use Index Lifecycle Management (ILM) to automate force merges as part of the warm or cold phase, once an index is rolled over and no longer actively ingested into.

Step 5: Measure the difference

GET /apache-standard/_stats?filter_path=indices.*.primaries.store

GET /apache-logsdb/_stats?filter_path=indices.*.primaries.store

The filter_path parameter keeps the response focused. Look for primaries.store.size_in_bytes in each response.

In our test with Apache log records, the results were:

| Index | Documents | Size | |------------------|-----------|----------| | apache-standard | 111,818 | 15.37 MB | | apache-logsdb | 111,818 | 8.6 MB | | Reduction | | 44% |

To put this in perspective: at 1 TB of log data, LogsDB brings that down to around 560 GB. That's 450 GB saved without any changes to your queries. At production scale with billions of documents and synthetic _source enabled, savings push to 76% — taking 162.7 GB down to 39.4 GB in our benchmark.

Visualize in Kibana

To see the storage difference visually, open Kibana and go to Management → Stack Management → Index Management. You'll see both indices listed with their current sizes side by side.

Why Kibana shows larger numbers than _stats: Kibana Index Management displays the total index size including all replica shards. The _stats query above uses primaries to report primary shards only. The ratio between the two indices remains the same either way.

What about your existing logs?

Elasticsearch 9.2+ (already enabled by default)

Since 9.2, any data stream matching the logs-* naming pattern automatically uses LogsDB. You're likely already saving storage without any configuration change.

Verify your existing data streams:

GET /.ds-logs-*/_settings?filter_path=*.settings.index.mode

If you see "index.mode": "logsdb" in the responses, you're already getting the savings.

Elasticsearch 8.x or 9.0–9.1 (enable per data stream via index template)

For earlier versions, enable LogsDB on a data stream by updating its index template. This affects all new indices created from that template — existing indices are not changed, so the transition is safe and gradual.

Option A — Update an existing template:

PUT _index_template/logs-myapp-template
{
  "index_patterns": ["logs-myapp-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "index.mode": "logsdb"
    }
  },
  "priority": 200
}

Option B — Check and patch an existing integration template:

First, find the template managing your data stream:

GET _index_template/logs-apache*

Then add the index.mode setting to the template.settings block using a PUT _index_template/<name> call with the full template body including your addition.

After updating the template, the next index rollover will use LogsDB. Trigger a rollover immediately if you don't want to wait:

POST /logs-myapp-default/_rollover

Upgrading from 8.x to 9.0+: Existing data streams are not changed automatically. Only new rollovers will use LogsDB. There is no data loss and no reindexing required — the savings accumulate as new indices roll over.

What about query performance?

LogsDB does not significantly impact query performance for typical log analytics workloads. The index sorting by host.name and @timestamp can actually improve range query and aggregation performance on those fields, since matching documents are stored adjacently. Queries that don't filter on those fields perform comparably to a standard index.

For indexing throughput data across releases, see the performance section of the companion article.

Conclusion

LogsDB activates with a single "index.mode": "logsdb" setting and delivers measurable storage savings immediately: 44% in our hands-on test, and 76% (162.7 GB → 39.4 GB) in production benchmarks with synthetic _source. On Elasticsearch 9.2+, logs-* data streams already use LogsDB by default. For 8.x or earlier 9.x clusters, a one-line index template change enables it on your next rollover with no data loss and no reindexing required.

Next steps

• LogsDB index mode documentation
• Configuring a logs data stream
• LogsDB GA announcement
• LogsDB and TSDS performance benchmarks

The write overhead

Lucene, the library underneath, keeps multiple structures for every indexed document:

• The inverted index maps terms to documents. This is what makes text search fast.
• _source stores the original JSON blob, returned when you fetch a document.
• Doc values store field values in columns for sorting and aggregation.
• Points / BKD trees index numeric and date fields for range queries.

The inverted index earns its keep: it's what lets you search a billion log lines by keyword in milliseconds, and there's no cheaper way to build that capability. _source exists to give you back exactly what you indexed: search results and GET requests return this blob directly. The problem is that it stores the full event even though the same field values are already available through doc values and the other structures.

Take a log event with fields like host.name, @timestamp, http.response.status_code, and duration_ms. The entire event is serialized as JSON in _source. The same field values are also written into doc values columns, indexed into the inverted index, and stored in BKD trees for range queries. Same data, multiple structures, each with its own on-disk footprint.

For a search engine where you need fast retrieval across all dimensions, that overhead is a reasonable tradeoff. For logs, where you rarely need the raw JSON and almost never do relevance-ranked search, much of it is pure waste.

One write, four on-disk structures: _source (the raw JSON blob), the inverted index, doc values columns, and BKD / points trees for numeric range queries. The same field values end up in multiple places.

Why columnar storage matters for compression

Doc values are the key to everything LogsDB does. Unlike _source, which stores entire documents as blobs, doc values store each field as a separate column across all documents in a Lucene segment.

Picture a segment with a million log events. The _source representation is a million JSON blobs, one per event, each containing all fields jumbled together. The doc values representation is a set of columns: one column of a million timestamps, one column of a million host names, one column of a million status codes, and so on.

Row-oriented _source keeps all fields for each document in one blob — doc0 through doc5 each carry host.name, @timestamp, status, duration_ms, and more jumbled together. Column-oriented doc values restructure the same data so all host.name values sit in one column, all timestamps in another, all status codes in another. Compression codecs can then run on each contiguous column independently.

That columnar layout is what makes per-column compression possible. When all values of http.response.status_code sit in a contiguous column, Lucene can apply codecs that exploit patterns in the sequence.

Delta encoding stores differences between adjacent values instead of full values. GCD encoding finds a common factor and divides everything down. Run-length encoding collapses repeats. Lucene picks the codec per segment and re-evaluates when segments merge.

Four sorted @timestamps from the same host, compressed in four stages. RAW: four 32-bit integers, 128 bits total. DELTA: store differences instead of full values — base stays, deltas +100, +200, +300 take 59 bits. GCD: divide out the common factor of 100, leaving 1, 2, 3 at 39 bits. BIT-PACK: pack those three small integers into contiguous bit storage, 9 bits freed.

But here's the catch: these codecs only work well when adjacent documents have correlated values. Consider the @timestamp column.

If logs arrive from dozens of hosts interleaved randomly, the timestamps in the column jump around. The delta between adjacent values might be +3 seconds, then -47 seconds, then +120 seconds. Delta encoding can't do much with that.

Now consider what happens if you sort by host.name and @timestamp before writing to the segment. All logs from host-A land in a contiguous run, followed by all logs from host-B, and so on. Within each host's run, the timestamps are monotonically increasing and the deltas are predictable.

Four timestamps from the same host might look like 1706745600, +100s, +200s, +300s. Delta encoding shrinks those to a base value plus three small integers.

GCD encoding finds that 100, 200, 300 are all divisible by 100 and stores 1, 2, 3 instead. Bit-packing then fits those three values into a handful of bits. The same pattern applies to fields like host.name, service.name, or http.response.status_code: within a sorted run, long stretches of identical values collapse to near nothing under run-length encoding.

Five hosts — api-01, api-02, db-01, web-01, web-02 — scattered randomly in arrival order (left). Sorting by host.name groups them into five contiguous blocks of eight (center). Run-length encoding collapses each block to a single (value, count) pair — 5 pairs stored instead of 40, the remaining slots freed (right).

Elasticsearch never sorted by default. Documents landed in arrival order, compressed with DEFLATE. We left a lot on the table.

How we got here: 2012–2026

Not all of the individual techniques in LogsDB were designed for logs. They were built over twelve years to solve different problems, and LogsDB is what happens when you stack them.

The foundation (2012–2017). Lucene 4.0 introduced doc values in 2012. By Elasticsearch 5.0 in 2016, they were on by default for all keyword and numeric fields. Lucene 7.0 added sparse doc values, so fields that only appear in some documents don't waste space on every document in the segment. That fixed a significant force-merge bloat problem (up to 10× on sparse fields) and set up the storage model everything else depends on.

Dense encoding reserves an 8-byte slot per document regardless of presence. Sparse encoding stores only documents that have a value at 12 bytes each (value + doc ID). For error_code with 2 of 16 docs populated (12% fill), sparse is 81% smaller: 24 B vs 128 B. For request_path at 88% fill, sparse is larger: 168 B vs 128 B. Lucene picks per field; sparse wins below ~67% fill.

Incremental wins (2020–2021). Two smaller changes targeted observability workloads. Dictionary-based stored fields compression deduplicated repetitive string metadata for about a 10% win.

The match_only_text field type dropped term frequencies and positions from the inverted index. Term frequencies are what BM25 uses to score documents by relevance — how often a term appears in a document relative to the rest of the corpus. For log search that signal is meaningless: you don't care whether "timeout" appeared twice or seven times in a log line, you just want to find it. Positions are similar: they're stored so Elasticsearch can do exact phrase matching, but the position data is expensive and phrase queries on logs are rare enough that the tradeoff is worth it. When you do run a phrase query on a match_only_text field, it still works — it just falls back to a slower path that rescores candidates rather than using stored positions directly.

text stores each term with its frequency and every position it appears at. match_only_text keeps only the doc IDs — enough to find the document, nothing more. The timeout term appears twice in this message (positions 1 and 4), which is exactly the kind of data that gets dropped.

Dropping frequencies and positions cuts the inverted index for a text field by roughly 40%. The overall index impact in 2021 was only ~10%, which sounds like a poor return on a 40% field-level reduction. The reason is where storage was going at the time: _source was stored in full for every document as a raw JSON blob, doc values were uncompressed and unsorted, and nothing was using ZSTD. The message field's inverted index was a small slice of a much larger, poorly-compressed whole. As the next five years of work addressed those other structures, the same 40% field-level savings became a meaningful fraction of a much smaller total.

Neither change was decisive on its own, but they established that log-specific storage optimization was worth pursuing.

The TSDB turning point (April 2023). This is where the story really starts. We shipped synthetic _source and index sorting for time series metrics in Elasticsearch 8.7.

Synthetic source changes the write-and-read contract. At write time, we skip storing the raw JSON blob entirely. At read time, when a query needs to return the original document, we reconstruct it by reading each field's value out of doc values and stored fields and assembling them back into JSON. The result is functionally equivalent to the original _source (with minor differences like field ordering), but we never stored the blob.

Index sorting groups documents by dimension fields and timestamp before writing to disk. Together, synthetic source and index sorting cut metrics storage by up to 70%.

That result told us something important: the same architecture could work for logs.

Without LogsDB, Elasticsearch writes every log event twice: once as a raw _source blob on disk, once into doc values columns. LogsDB skips the blob entirely. At read time, a GET <index>/_doc/1 request gathers field values from doc values and assembles the document on the fly.

The TSDB codec (2024). In 8.13 and 8.14, we built a custom doc values codec with run-length encoding optimized for sorted consecutive values, PFOR-delta encoding, and cyclic ordinal encoding for multi-valued dimensions. The numbers were striking: kubernetes.pod.name doc values dropped from 110 MB to 7.25 MB in one benchmark. We extended coverage to all numeric and keyword types including ip, scaled_float, and unsigned_long.

LogsDB Tech Preview (August 2024). In 8.15, we combined everything into index.mode: logsdb: host-first sorting, synthetic _source, ZSTD compression, and the TSDB numeric codecs. One decision mattered more than expected: sort order. Sorting by host.name first, then @timestamp, delivers up to ~40% storage reduction. Sorting by timestamp first gives ≤10%. The host-first ordering co-locates documents that share field values, which is exactly what the numeric codecs need.

ZSTD and GA (November–December 2024). In 8.16, we switched best_compression from DEFLATE to ZSTD permanently (level 3, blocks up to 2,048 documents or 240 kB, native bindings via Panama FFI on JDK 21+). ZSTD gave us ~12% smaller stored fields and ~14% higher indexing throughput at the same time, which almost never happens. LogsDB went GA in 8.17.

At GA, we claimed up to 65% storage reduction.

Routing and recovery (April 2025). In 8.18, route_on_sort_fields started routing documents to shards by sort field values instead of _id. Without this optimization, Elasticsearch hashes the _id to pick a shard, so logs from the same host scatter across all shards. With routing on sort fields, logs with similar host.name values land on the same shard. This co-locates similar documents at the shard level, not just within segments, adding ~20% storage reduction at a 1–4% ingest penalty. Routing on sort fields requires auto-generated _id.

Data stream .ds-logs-nginx-default-00001 with six hosts across three shards. STANDARD (hashed by _id): all host colors scattered randomly. ROUTED (route_on_sort_fields): same-host logs land on the same shard, but remain in arrival order within it. ROUTED + SORTED (host-first sort): each shard contains contiguous blocks of a single host — the combination that lets numeric codecs and RLE reach their full potential.

We also switched peer recovery to synthetic source reconstruction, eliminating the duplicate _recovery_source blob. In 9.0, logs-*-* indices default to LogsDB.

Nightly synthetic source benchmark, December 2024. Index size written drops 39% — from ~279 GB to ~171 GB — the day peer recovery switches from copying the raw _recovery_source blob to reconstructing documents from doc values.

Merge and recovery overhaul: 9.1 (July 2025). We fully eliminated the recovery source. Peer recovery uses batched synthetic reconstruction, cutting write I/O by ~50% and boosting median indexing throughput ~19% over the 8.17 baseline. We replaced up to four separate doc values merge passes with a single pass, cutting background merge CPU by up to 40%. And we swapped _seq_no's BKD tree for Lucene doc value skippers, halving _seq_no storage.

pattern_text and Failure Store: 9.2–9.3 (October 2025–February 2026). In 9.2, we shipped pattern_text as a Tech Preview: a new field type that decomposes log messages into static templates and dynamic variable parts. A log line like Session opened for user alice from 10.0.1.42 via TLS gets split into the template Session opened for user {} from {} via TLS (stored once, as a template ID) and the variables alice, 10.0.1.42 (stored per document). For logs with high template repetition, this cuts message field storage by up to 50%. A companion template_id sub-field lets you sort by template, and the LogsDB setting index.logsdb.default_sort_on_message_template enables this automatically. pattern_text went GA in 9.3.

TEXT stores each log message as a full string per document — eight copies of near-identical blobs. PATTERN_TEXT decomposes them: the shared template Session opened for user {} from {} via TLS is stored once with ID T0, and only the variable columns (user, ip) are stored per document — alice/10.0.1.42, bob/10.0.1.87, carol/10.0.2.11, and so on.

pattern_text does come with an indexing CPU cost: decomposing each message into template and variables takes more work at write time than storing a raw string. Whether that tradeoff makes sense depends on your dataset and your priorities.

If your log messages follow highly repetitive patterns (structured application logs, Kubernetes events, access logs), the storage wins are large and the CPU overhead is bounded. If your messages are free-form or low-repetition, the compression gains shrink while the CPU cost stays roughly the same.

For data you keep for months or years, the cumulative storage reduction usually makes it worthwhile. For high-cardinality, rapidly changing messages where storage isn't the constraint, it may not be.

9.3 also brought compression for binary doc values, making wildcard field types significantly more storage-efficient. Internally, wildcard fields store an inverted index of trigrams in a binary doc values column; that column is now compressed with Zstandard instead of being stored raw. In one benchmark, a URL field dropped from 2.92 GB to 1.12 GB, more than 60% compression. If you use wildcard fields heavily, the gain is automatic with no mapping changes needed.

Also in 9.3, skip lists for @timestamp and host.name became available as an opt-in for LogsDB. Skip lists let Elasticsearch jump ahead in a doc values column without reading every entry, which speeds up time-range queries on large segments. Other index modes have skip lists disabled by default; in LogsDB you can enable them selectively for the fields you range-query most.

Also in 9.3, the Failure Store became enabled by default for logs-*-* data streams. Failed documents (mapping conflicts, ingest pipeline errors) now land in dedicated ::failures indices instead of being rejected, which means LogsDB's strict synthetic source requirements are less likely to cause silent data loss during migration.

Performance, not just storage

LogsDB started as a storage optimization, and the early releases came with a throughput cost — sorting, synthetic source reconstruction, and ZSTD all add work at write time. Over two years of releases, we clawed that back. Indexing throughput is now on par with what users had before enabling LogsDB. You get the storage reduction without giving up the ingest rate you were used to.

Throughput (teal) has climbed from ~25k to ~35k docs/s since the Tech Preview. Storage on disk (blue) has dropped from ~65 GB to ~36 GB on the same benchmark dataset. Both curves move in the right direction, driven by the same layered releases: ZSTD in 8.16, routing optimization in 8.18, the merge and recovery overhaul in 9.1. Live numbers at elasticsearch-benchmarks.elastic.co.

The two trends compound each other. Less storage means fewer segments to merge, which frees CPU for indexing. Synthetic source reconstruction is cheaper to compute than it is to store and replicate the raw blob. Each release that shrank the index also reduced background I/O, which fed back into throughput.

The practical result: if you were running standard Elasticsearch for log ingestion two years ago, the throughput you had then is roughly what LogsDB delivers now — with a 50–75% smaller index alongside it.

How to enable it

As of 9.0, logs-*-* data streams default to LogsDB automatically. If your data streams match that pattern, you're already using it.

Want a hands-on walkthrough? Cut Elasticsearch log storage costs by 76% with LogsDB walks through creating two indices, reindexing, and measuring the difference with the _stats API — including version-specific enable instructions for 8.x clusters.

For other index patterns, set it in your template:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "index.mode": "logsdb"
    }
  }
}

Synthetic _source turns on automatically with index.mode: logsdb.

For the routing optimization (8.18+), add one more setting:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "index.mode": "logsdb",
      "index.logsdb.route_on_sort_fields": true
    }
  }
}

This routes shards by sort field values instead of _id, adding ~20% storage reduction at a 1–4% ingestion penalty. It requires at least two sort fields beyond @timestamp and auto-generated _id.

Switching an existing index to LogsDB requires a reindex. So does rolling back. There's no in-place conversion, so try it on new data streams first.

Storage improves further as segments merge — freshly written data compresses well, but merged segments compress even better.

What's next

Elasticsearch still carries some structural overhead from its search engine roots. _id and _seq_no are two examples: both consume meaningful disk space (on small documents they can account for more than half the index size), but neither is essential for log analytics workloads.

We've already taken the first step for TSDB: PR #144026 eliminated stored _id bytes from TSDB indices by reconstructing the field on the fly from doc values, the same approach synthetic _source uses. We're exploring the same direction for LogsDB.

9.4 and beyond. The architecture still has room to improve, and we're on it.

For the full reference, see the logs data stream documentation.

Efficiency is important (again)

Before we jump into continuous profiling, let's start with the "Why should I care?" question. To do that, I'd like to talk a bit about efficiency and some large-scale trends happening in our industry that are making efficiency, specifically computational efficiency, important again. I say again because in the past, when memory and storage on a computer was very limited and you had to worry about every byte of code, efficiency was an important aspect of developing software.

The end of Moore’s Law

First, the Moore's Law era is drawing to a close. This was inevitable simply due to physical limits of how small you can make a transistor and the connections between them. For a long time, software developers had the luxury of not worrying about complexity and efficiency because the next generation of hardware would mitigate any negative cost or performance impact.

If you can't rely on an endless progression of ever faster hardware, you should be interested in computational efficiency.

The move to Software-as-a-Service

Another trend to consider is the shift from software vendors that sold customers software to run themselves to Software-as-a-Service businesses. A traditional software vendor didn't have to worry too much about the efficiency of their code. That issue largely fell to the customer to address; a new software version might dictate a hardware refresh to the latest and most performant. For a SaaS business, inefficient software usually degrades the customer’s experience and it certainly impacts the bottom line.

If you are a SaaS business in a competitive environment, you should be interested in computational efficiency.

Cloud migration

Next is the ongoing cloud migration to cloud computing. One of the benefits of cloud computing is the ease of scaling, both hardware and software. In the cloud, we are not constrained by the limits of our data centers or the next hardware purchase. Instead we simply spin up more cloud instances to mitigate performance problems. In addition to infrastructure scalability, microservices architectures, containerization, and the rise of Kubernetes and similar orchestration tools means that scaling services is simpler than ever. It's not uncommon to have thousands of instances of a service running in a cloud environment. This ease of scaling accounts for another trend, namely that many businesses are dealing with skyrocketing cloud computing costs.

If you are a business with ever increasing cloud costs, you should be interested in computational efficiency.

Our changing climate

Lastly, if none of those reasons pique your interest, let's consider a global problem that all of us should have in mind — namely, climate change. There are many things that need to be addressed to tackle climate change, but with our dependence on software in every part of our society, computational efficiency is certainly something we should be thinking about.

Thomas Dullien, distinguished engineer at Elastic and one of the founders of Optymize points out that if you can save 20% on 800 servers, and assume 300W power consumption for each server, that code change is worth 160 metric tons of CO₂ saved per year. That may seem like a drop in the bucket but if all businesses focus more on computational efficiency, it will make an impact. Also, let's not forget the financial benefits: those 160 metric tons of CO₂ savings also represent a significant annual cost savings.

If you live on planet Earth, you should be interested in computational efficiency.

Performance engineering

Who's job is it to worry about computational efficiency? Application developers usually pay at least some attention to efficiency as they develop their code. Profiling is a common approach for a developer to understand the performance of their code, and there is an entire portfolio of profiling tools available. Frequently, however, schedule pressures trump time spent on performance analysis and computational efficiency. In addition, performance problems may not become apparent until an application is running at scale in production and interacting (and competing) with everything else in that environment. Many profiling tools are not well suited to use in a production environment because they require code instrumentation and recompilation and add significant overhead.

When inefficient code makes it into production and begins to cause performance problems, the next line of defense is the Operations or SRE team. Their mission is to keep everything humming, and performance problems will certainly draw attention. Observability tools such as APM can shed light on these types of issues and lead the team to a specific application or service, but these tools have limits into the observability of the full system. Third-party libraries and operating system kernels functions remain hidden without a profiling solution in the production environment.

So, what can these teams do when there is a need to investigate a performance problem in production? That's where continuous profiling comes into the picture.

Continuous profiling

Continuous profiling is not a new idea. Google published a paper about it in 2010 and began implementing continuous profiling in its environments around that time. Facebook and Netflix followed suit not long afterward.

Typically, continuous profiling tools have been the domain of dedicated performance engineering or operating system engineering teams, which are usually only found at extremely large scale enterprises like the ones mentioned above. The key idea is to run profiling on every server, all of the time. That way, when your observability tools point you to a specific part of an application, but you need a more detailed view into exactly where that application is consuming CPU resources, the profiling data will be there, ready to use.

Another benefit of continuous profiling is that it provides a view of CPU intensive software across your entire environment — whether that is a very CPU intensive function or the aggregate of a relatively small function that is run thousands of times a second in your environment.

While profiling tools are not new, most of them have significant gaps. Let's look at a couple of the most significant ones.

• Limited visibility. Modern distributed applications are composed of a complex mix of building blocks, including custom software functions, third-party software libraries, networking software, operating system services, and more and more often, orchestration software such as Kubernetes. To fully understand what is happening in an application, you need visibility into each piece. However, even if a developer has the ability to profile their own code, everything else remains invisible. To make matters worse, most profiling tools require instrumenting the code, which adds overhead and therefore even your developers’ code is not profiled in production.
• Missing symbols in production. All of these pieces of code building blocks typically have descriptive names (some more intuitive than others) so that developers can understand and make sense of them. In a running program, these descriptive names are usually referred to as symbols. For a human being to make sense of the execution of a running application, these names are very important. Unfortunately, almost always, any software running in production has these human readable symbols stripped away for space efficiency since they are not needed by the CPU executing the software. Without all of the symbols, it makes it much more difficult to understand the full picture of what's happening in the application. To illustrate this, think of the last time you were in an SMS chat on your mobile device and you only had some of the people in the chat group in your address book while the rest simply appeared as phone numbers — this makes it very hard to tell who is saying what.

Elastic Universal Profiling: Continuous profiling for all

Our goal is to allow any business, large or small, to make computational efficiency a core consideration for all of the software that they run. Universal Profiling imposes very low overhead on your servers so it can be used in production and it provides visibility to everything running on every machine. It opens up the possibility of seeing the financial unit cost and CO₂ impact of every line of code running on every system in your business. How do we do that?

Whole-system visibility — SIMPLE

Universal Profiling is based on eBPF, which means that it imposes very low overhead (our goal is less than 1% CPU and less than 250MB of RAM) on your servers because it doesn't require code instrumentation. That low overhead means it can be run continuously, on every server, even in production.

eBPF also lets us deploy a single profiler agent on a host and peek inside the operating system to see every line of code executing on the CPU. That means we have visibility into all of those application building blocks described above — the operating system itself as well as containerization and orchestration frameworks without complex configuration.

All the symbols

A key part of Universal Profiling is our hosted symbolization service. This means that symbols are not required on your servers, which not only eliminates a need for recompiling software with symbols, but it also helps to reduce overhead by allowing the Universal Profiling agent to send very sparse data back to the Elasticsearch platform where it is enriched with all of the missing symbols. Since we maintain a repository of most popular third-party software libraries and Linux operating system symbols, the Universal Profiling UI can show you all the symbols.

Your favorite language, and then some

Universal Profiling is multilanguage. We support all of today’s popular programming languages, including Python, Go, Java (and any other JVM-based languages), Ruby, NodeJS, PHP, Perl, and of course, C and C++, which is critical since these languages still underly so many third-party libraries used by the other languages. In addition, we support profiling native code a.k.a. machine language.

Speaking of native code, all profiling tools are tied to a specific type of CPU. Most tools today only support the Intel x86 CPU architecture. Universal Profiling supports both x86 and ARM-based processors. With the expanding use of ARM-based servers, especially in cloud environments, Universal Profiling future-proofs your continuous profiling.

Many businesses today employ polyglot programming — that is, they use multiple languages to build an application — and Universal Profiling is the only profiler available that can build a holistic view across all of these languages. This will help you look for hotspots in the environment, leading you to "unknown unknowns" that warrant deeper performance analysis. That might be a simple interest rate calculation that should be efficient and lightweight but, surprisingly, isn't. Or perhaps it is a service that is reused much more frequently than originally expected, resulting in thousands of instances running across your environment every second, making it a prime target for efficiency improvement.

Visualize your impact

Elastic Universal Profiling has an intuitive UI that immediately shows you the impact of any given function, including the time it spends executing on the CPU and how much that costs both in dollars and in carbon emissions.

Finally, with the level of software complexity in most production environments, there's a good chance that making a code change will have unanticipated effects across the environment. That code change may be due to a new feature being rolled out or a change to improve efficiency. In either case, a differential view, before and after the change, will help you understand the impact.

Let's recap

Computational efficiency is an important topic, both from the perspective of the ultra-competitive business climate we all work in and from living through the challenges of our planet's changing climate. Improving efficiency can be a challenging endeavor, but we can't even begin to attempt to make improvements without knowing where to focus our efforts. Elastic Universal Profiling is here to provide every business with visibility into computational efficiency.

How will you use Elastic Universal Profiling in your business?

• If you are an application developer or part of the site reliability team, Universal Profiling will provide you with unprecedented visibility into your applications that will not only help you troubleshoot performance problems in production, but also understand the impact of new features and deliver an optimal user experience.
• If you are involved in cloud and infrastructure financial management and capacity planning, Universal Profiling will provide you with unprecedented visibility into the unit cost of every line of code that your business runs.
• If you are involved in your business’s ESG initiative, Universal Profiling will provide you with unprecedented visibility into your CO₂ emissions and open up new avenues for reducing your carbon footprint.

These are just a few examples. For more ideas, read how AppOmni benefits from Elastic Universal Profiling.

You can get started with Elastic Universal Profiling right now!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we are excited to share the significant strides we have made in improving the UI performance of Elastic APM. Maintaining a snappy user interface can be a challenge when interactively summarizing the massive amounts of data needed to provide an overview of the performance for an entire enterprise-scale service inventory. We want to assure our customers that we have listened, taken action, and made notable architectural changes to elevate the scalability and maturity of our solution.

Architectural enhancements

Our journey began back in the 7.x series where we noticed that doing ad-hoc aggregations on raw transaction data put Elasticsearch^® under a lot of pressure in large-scale environments. Since then, we’ve begun to pre-aggregate the transactions into transaction metrics during ingestion. This has helped to keep the performance of the UI relatively stable. Regardless of how busy the monitored application is and how many transaction events it is creating, we’re just querying pre-aggregated metrics that are stored at a constant rate. We’ve enabled the metrics-powered UI by default in 7.15.

However, when showing an inventory of a large number of services over large time ranges, the number of metric data points that need to be aggregated can still be large enough to cause performance issues. We also create a time series for each distinct set of dimensions. The dimensions include metadata, such as the transaction name and the host name. Our documentation includes a full list of all available dimensions. If there’s a very high number of unique transaction names, which could be a result of improper instrumentation (see docs for more details), this will create a lot of individual time series that will need to be aggregated when requesting a summary of the service’s overall performance. Global labels that are added to the APM Agent configuration are also added as dimensions to these metrics, and therefore they can also impact the number of time series. Refer to the FAQs section below for more details.

Within the 8.7 and 8.8 releases, we’ve addressed these challenges with the following architectural enhancements that aim to reduce the number of documents Elasticsearch needs to search and aggregate on-the-fly, resulting in faster response times:

• Pre-aggregation of transaction metrics into service metrics. Instead of aggregating all distinct time series that are created for each individual transaction name on-the-fly for every user request, we’re already pre-aggregating a summary time series for each service during data ingestion. Depending on how many unique transaction names the services have, this reduces the number of documents Elasticsearch needs to look up and aggregate by a factor of typically 10–100. This is particularly useful for the service inventory and the service overview pages.
• Pre-aggregation of all metrics into different levels of granularity. The APM UI chooses the most appropriate level of granularity, depending on the selected time range. In addition to the metrics that are stored at a 1-minute granularity, we’re also summarizing and storing metrics at a 10-minute and 60-minute granularity level. For example, when looking at a 7-day period, the 60-minute data stream is queried instead of the 1-minute one, resulting in 60x fewer documents for Elasticsearch to examine. This makes sure that all graphs are rendered quickly, even when looking at larger time ranges.
• Safeguards on the number of unique transactions per service for which we are aggregating metrics. Our agents are designed to keep the cardinality of the transaction name low. But in the wild, we’ve seen some services that have a huge amount of unique transaction names. This used to cause performance problems in the UI because APM Server would create many time series that the UI needed to aggregate at query time. In order to protect APM Server from running out of memory when aggregating a large number of time series for each unique transaction name, metrics were published without aggregating when limits for the number of time series were reached. This resulted in a lot of individual metric documents that needed to be aggregated at query time. To address the problem, we've introduced a system where we aggregate metrics in a dedicated overflow bucket for each service when limits are reached. Refer to our documentation for more details.

The exact factor of the document count reduction depends on various conditions. But to get a feeling for a typical scenario, if your services, on average, have 10 instances, no instance-specific global labels, 100 unique transaction names each, and you’re looking at time ranges that can leverage the 60m granularity, you’d see a reduction of documents that Elasticsearch needs to aggregate by a factor of 180,000 (10 instances x 100 transaction names x 60m x 3 because we’re also collapsing the event.outcome dimension). While the response times of Elasticsearch aggregations isn’t exactly scaling linearly with the number of documents, there is a strong correlation.

FAQs

When upgrading to the latest version, will my old data also load faster?

Updating to 8.8 doesn’t immediately make the UI faster. Because the improvements are powered by pre-aggregations that APM Server is doing during ingestion, only new data will benefit from it. For that reason, you should also make sure to update APM Server as well. The UI can still display data that was ingested using an older version of the stack.

If the UI is based on metrics, can I still slice and dice using custom labels?

High cardinality analysis is a big strength of Elastic Observability, and this focus on pre-aggregated metrics does not compromise that in any way.

The UI implements a sophisticated fallback mechanism that uses service metrics, transaction metrics, or raw transaction events, depending on which filters are applied. We’re not creating metrics for each user.id, for example. But you can still filter the data by user.id and the UI will then use raw transaction events. Chances are that you’re looking at a narrow slice of data when filtering by a dimension that is not available on the pre-aggregated metrics, therefore aggregations on the raw data are typically very fast.

Note that all global labels that are added to the APM agent configuration are part of the dimension of the pre-aggregated metrics, with the exception of RUM (see more details in this issue).

Can I use the pre-aggregated metrics in custom dashboards?

Yes! If you use Lens and select the "APM" data view, you can filter on either metricset.name:service_transaction or metricset.name:transaction, depending on the level of detail you need. Transaction latency is captured in transaction.duration.histogram, and successful outcomes and failed outcomes are stored in event.success_count. If you don't need a distribution of values, you can also select the transaction.duration.summary field for your metric aggregations, which should be faster. If you want to calculate the failure rate, here's a Lens formula: 1 - (sum(event.success_count) / count(event.success_count)). Note that the only granularity supported here is 1m.

Do the additional metrics have an impact on the storage?

While we’re storing more metrics than before, and we’re storing all metrics in different levels of granularity, we were able to offset that by enabling synthetic source for all metric data streams. We’ve even increased the default retention for the metrics in the coarse-grained granularity levels, so that the 60m rollup data streams are now stored for 390 days. Please consult our documentation for more information about the different metric data streams.

Are there limits on the amount of time series that APM Server can aggregate?

APM Server performs pre-aggregations in memory, which is fast, but consumes a considerable amount of memory. There are limits in place to protect APM Server from running out of memory, and from 8.7, most of them scale with available memory by default, meaning that allocating more memory to APM Server will allow it to handle more unique pre-aggregation groups like services and transactions. These limits are described in APM Server Data Model docs.

On the APM Server roadmap, we have plans to move to a LSM-based approach where pre-aggregations are performed with the help of disks in order to reduce memory usage. This will enable APM Server to scale better with the input size and cardinality.

A common pitfall when working with pre-aggregations is to add instance-specific global labels to APM agents. This may exhaust the aggregation limits and cause metrics to be aggregated under the overflow bucket instead of the corresponding service. Therefore, make sure to follow the best practice of only adding a limited set of global labels to a particular service.

Validation

To validate the effectiveness of the new architecture, and to ensure that the accuracy of the data is not negatively affected, we prepared a test environment where we generated 35K+ transactions per minute in a timespan of 14 days resulting in approximately 850 million documents.

We’ve tested the queries that power our service inventory, the service overview, and the transaction details using different time ranges (1d, 7d, 14d). Across the board, we’ve seen orders of magnitude improvements. Particularly, queries across larger time ranges that benefit from using the coarse-grained metrics in addition to the pre-aggregated service metrics saw incredible reductions of the response time.

We’ve also validated that there’s no loss in accuracy when using the more coarse-grained metrics for larger time ranges.

Every environment will behave a bit differently, but we’re confident that the impressive improvements in response time will translate well to setups of even bigger scale.

Planned improvements

As mentioned in the FAQs section, the number of time series for transaction metrics can grow quickly, as it is the product of multiple dimensions. For example, given a service that runs on 100 hosts and has 100 transaction names that each have 4 transaction results, APM Server needs to track 40,000 (100 x 100 x 4) different time series for that service. This would even exceed the maximum per-service limit of 32,000 for APM Servers with 64GB of main memory.

As a result, the UI will show an entry for “Remaining Transactions” in the Service overview page. This tracks the transaction metrics for a service once it hits the limit. As a result, you may not see all transaction names of your service. It may also be that all distinct transaction names are listed, but that the transaction metrics for some of the instances of that service are combined in the “Remaining Transactions” category.

We’re currently considering restructuring the dimensions for the metrics to avoid that the combination of the dimensions for transaction name and service instance-specific dimensions (such as the host name) lead to an explosion of time series. Stay tuned for more details.

Conclusion

The architectural improvements we’ve delivered in the past releases provide a step-function in terms of the scalability and responsiveness of our UI. Instead of having to aggregate massive amounts of data on-the-fly as users are navigating through the user interface, we pre-aggregate the results for the most common queries as data is coming in. This ensures we have the answers ready before users have even asked their most frequently asked questions, while still being able to answer ad-hoc questions.

We are excited to continue supporting our community members as they push boundaries on their growth journey, providing them with a powerful and mature platform that can effortlessly handle the demands of the largest workloads. Elastic is committed to its mission to enable everyone to find the answers that matter. From all data. In real time. At scale.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic Observability offers a wide range of integrations that help you to monitor your applications and infrastructure. These integrations are added as policies to Elastic agents, which help ingest telemetry into Elastic Observability. Several examples of these integrations include the ability to ingest logs from systems that send a stream of logs from different applications, such as Amazon Kinesis Data Firehose, Kubernetes container logs, and syslog. One challenge is that these multiplexed log streams are sending data to the same Elasticsearch data stream, such as logs-syslog-default. This makes it difficult to create parsing rules in ingest pipelines and dashboards for specific technologies, such as the ones from the Nginx and Apache integrations. That’s because in Elasticsearch, in combination with the data stream naming scheme, the processing and the schema are both encapsulated in a data stream.

The reroute processor helps you tease apart data from a generic data stream and send it to a more specific one. You may use that mechanism to send logs to a data stream that is set up by the Nginx integration, for example, so that the logs are parsed with that integration and you can use the integration’s prebuilt dashboards or create custom ones with the fields, such as the url, the status code, and the response time that the Nginx pipeline has parsed out of the Nginx log message. You can also split out/separate regular Nginx logs and errors with the reroute processor, providing further separation ability and categorization of logs.

Example use case

To use the reroute processor, first:

1. Ensure you are on Elasticsearch 8.8

2. Ensure you have permissions to manage indices and data streams

3. If you don’t already have an account on Elastic Cloud, sign up for one

Next, you’ll need to set up a data stream and create a custom Elasticsearch ingest pipeline that is called as the default pipeline. Below we go through this step by step for the “mydata” data set that we’ll simulate ingesting container logs into. We start with a basic example and extend it from there.

The following steps should be utilized in the Elastic console, which is found at Management -> Dev tools -> Console. First, we need an an ingest pipeline and a template for the data stream:

PUT _ingest/pipeline/logs-mydata
{
  "description": "Routing for mydata",
  "processors": [
    {
      "reroute": {
      }
    }
  ]
}

This creates an ingest pipeline with an empty reroute processor. To make use of it, we need an index template:

PUT _index_template/logs-mydata
{
  "index_patterns": [
    "logs-mydata-*"
  ],
  "data_stream": {},
  "priority": 200,
  "template": {
    "settings": {
      "index.default_pipeline": "logs-mydata"
    },
    "mappings": {
      "properties": {
        "container.name": {
          "type": "keyword"
        }
      }
    }
  }
}

The above template is applied to all data that is shipped to logs-mydata-*. We have mapped container.name as a keyword, as this is the field we will be using for routing later on. Now, we send a document to the data stream and it will be ingested into logs-mydata-default:

POST logs-mydata-default/_doc
{
  "@timestamp": "2023-05-25T12:26:23+00:00",
  "container": {
    "name": "foo"
  }
}

We can check that it was ingested with the command below, which will show 1 result.

GET logs-mydata-default/_search

Without modifying the routing processor, this already allows us to route documents. As soon as the reroute processor is specified, it will look for data_stream.dataset and data_stream.namespace fields by default and will send documents to the corresponding data stream, according to the data stream naming scheme logs-\<dataset>-\<namespace>. Let’s try this out:

POST logs-mydata-default/_doc
{
  "@timestamp": "2023-03-30T12:27:23+00:00",
  "container": {
"name": "foo"
  },
  "data_stream": {
    "dataset": "myotherdata"
  }
}

As can be seen with the GET logs-mydata-default/_search command, this document ended up in the logs-myotherdata-default data stream. But instead of using default rules, we want to create our own rules for the field container.name. If the field is container.name = foo, we want to send it to logs-foo-default. For this we modify our routing pipeline:

PUT _ingest/pipeline/logs-mydata
{
  "description": "Routing for mydata",
  "processors": [
    {
      "reroute": {
        "tag": "foo",
        "if" : "ctx.container?.name == 'foo'",
        "dataset": "foo"
      }
    }
  ]
}

Let's test this with a document:

POST logs-mydata-default/_doc
{
  "@timestamp": "2023-05-25T12:26:23+00:00",
  "container": {
    "name": "foo"
  }
}

While it would be possible to specify a routing rule for each container name, you can also route by the value of a field in the document:

PUT _ingest/pipeline/logs-mydata
{
  "description": "Routing for mydata",
  "processors": [
    {
      "reroute": {
        "tag": "mydata",
        "dataset": [
          "{{container.name}}",
          "mydata"
        ]
      }
    }
  ]
}

In this example, we are using a field reference as a routing rule. If the container.name field exists in the document, it will be routed — otherwise it falls back to mydata. This can be tested with:

POST logs-mydata-default/_doc
{
  "@timestamp": "2023-05-25T12:26:23+00:00",
  "container": {
    "name": "foo1"
  }
}

POST logs-mydata-default/_doc
{
  "@timestamp": "2023-05-25T12:26:23+00:00",
  "container": {
    "name": "foo2"
  }
}

This creates the data streams logs-foo1-default and logs-foo2-default.

NOTE: There is currently a limitation in the processor that requires the fields specified in a {{field.reference}} to be in a nested object notation. A dotted field name does not currently work. Also, you’ll get errors when the document contains dotted field names for any data_stream.* field. This limitation will be fixed in 8.8.2 and 8.9.0.

API keys

When using the reroute processor, it is important that the API keys specified have permissions for the source and target indices. For example, if a pattern is used for routing from logs-mydata-default, the API key must have write permissions for logs-*-* as data could end up in any of these indices (see example further down).

We’re currently working on extending the API key permissions for our integrations so that they allow for routing by default if you’re running a Fleet-managed Elastic Agent.

If you’re using a standalone Elastic Agent, or any other shipper, you can use this as a template to create your API key:

POST /_security/api_key
{
  "name": "ingest_logs",
  "role_descriptors": {
    "ingest_logs": {
      "cluster": [
        "monitor"
      ],
      "indices": [
        {
          "names": [
            "logs-*-*"
          ],
          "privileges": [
            "auto_configure",
            "create_doc"
          ]
        }
      ]
    }
  }
}

Future plans

In Elasticsearch 8.8, the reroute processor was released in technical preview. The plan is to adopt this in our data sink integrations like syslog, k8s, and others. Elastic will provide default routing rules that just work out of the box, but it will also be possible for users to add their own rules. If you are using our integrations, follow this guide on how to add a custom ingest pipeline.

Try it out!

This blog post has shown some sample use cases for document based routing. Try it out on your data by adjusting the commands for index templates and ingest pipelines to your own data, and get started with Elastic Cloud through a 7-day free trial. Let us know via this feedback form how you’re planning to use the reroute processor and whether you have suggestions for improvement.

Mid 2021, the Elasticsearch team embarked on making Elasticsearch a much better fit for metrics. The team created Time Series Data Streams (TSDS), which were released in 8.7 as generally available (GA).

This blog post dives into how TSDS works and how we use it in Elastic Observability, as well as how you can use it for your own metrics.

A quick introduction to TSDS

Time Series Data Streams (TSDS) are built on top of data streams in Elasticsearch that are optimized for time series. To create a data stream for metrics, an additional setting on the data stream is needed. As we are using data streams, first an Index Template has to be created:

PUT _index_template/metrics-laptop
{
  "index_patterns": [
    "metrics-laptop-*"
  ],
  "data_stream": {},
  "priority": 200,
  "template": {
    "settings": {
      "index.mode": "time_series"
    },
    "mappings": {
      "properties": {
        "host.name": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "packages.sent": {
          "type": "integer",
          "time_series_metric": "counter"
        },
        "memory.usage": {
          "type": "double",
          "time_series_metric": "gauge"
        }
      }
    }
  }
}

Let's have a closer look at this template. On the top part, we mark the index pattern with metrics-laptop-*. Any pattern can be selected, but it is recommended to use the data stream naming scheme for all your metrics. The next section sets the "index.mode": "time_series" in combination with making sure it is a data_stream: "data_stream": {}.

Dimensions

Each time series data stream needs at least one dimension. In the example above, host.name is set as a dimension field with "time_series_dimension": true. You can have up to 16 dimensions by default. Not every dimension must show up in each document. The dimensions define the time series. The general rule is to pick fields as dimensions that uniquely identify your time series. Often this is a unique description of the host/container, but for some metrics like disk metrics, the disk id is needed in addition. If you are curious about default recommended dimensions, have a look at this ECS contribution with dimension properties.

Reduced storage and increased query speed

At this point, you already have a functioning time series data stream. Setting the index mode to time series automatically turns on synthetic source. By default, Elasticsearch typically duplicates data three times:

• row-oriented storage (_source field)
• column-oriented storage (doc_values: true for aggregations)
• indices (index: true for filtering and search)

With synthetic source, the _source field is not persisted; instead, it is reconstructed from the doc values. Especially in the metrics use case, there are little benefits to keeping the source.

Not storing it means a significant reduction in storage. Time series data streams sort the data based on the dimensions and the time stamp. This means data that is usually queried together is stored together, which speeds up query times. It also means that the data points for a single time series are stored alongside each other on disk. This enables further compression of the data as the rate at which a counter increases is often relatively constant.

Metric types

But to benefit from all the advantages of TSDS, the field properties of the metrics fields must be extended with the time_series_metric: {type}. Several types are supported — as an example, gauge and counter were used above. Giving Elasticsearch knowledge about the metric type allows Elasticsearch to offer more optimized queries for the different types and reduce storage usage further.

When you create your own templates for data streams under the data stream naming scheme, it is important that you set "priority": 200 or higher, as otherwise the built-in default template will apply.

Ingest a document

Ingesting a document into a TSDS isn't in any way different from ingesting documents into Elasticsearch. You can use the following commands in Dev Tools to add a document, and then search for it and also check out the mappings. Note: You have to adjust the @timestamp field to be close to your current date and time.

# Add a document with `host.name` as the dimension
POST metrics-laptop-default/_doc
{
  # This timestamp neesd to be adjusted to be current
  "@timestamp": "2023-03-30T12:26:23+00:00",
  "host.name": "ruflin.com",
  "packages.sent": 1000,
  "memory.usage": 0.8
}

# Search for the added doc, _source will show up but is reconstructed
GET metrics-laptop-default/_search

# Check out the mappings
GET metrics-laptop-default

If you do search, it still shows _source but this is reconstructed from the doc values. The additional field added above is @timestamp. This is important as it is a required field for any data stream.

Why is this all important for Observability?

One of the advantages of the Elastic Observability solution is that in a single storage engine, all signals are brought together in a single place. Users can query logs, metrics, and traces together without having to jump from one system to another. Because of this, having a great storage and query engine not only for logs but also metrics is key for us.

Usage of TSDS in integrations

With integrations, we give our users an out of the box experience to integrate with their infrastructure and services. If you are using our integrations, eventually you will automatically get all the benefits of TSDS for your metrics assuming you are on version 8.7 or newer.

Currently we are working through the list of our integration packages, add the dimensions, metric type fields and then turn on TSDS for the metrics data streams. What this means is as soon as the package has all properties enabled, the only thing you have to do is upgrade the integration and everything else will happen automatically in the background.

To visualize your time series in Kibana, use Lens, which has native support built in for TSDS.

Learn more

If you switch over to TSDS, you will automatically benefit from all the future improvements Elasticsearch is making for metrics time series, be it more efficient storage, query performance, or new aggregation capabilities. If you want to learn more about how TSDS works under the hood and all available config options, check out the TSDS documentation. What Elasticsearch supports in 8.7 is only the first iteration of the metrics time series in Elasticsearch.

TSDS can be used since 8.7 and will be in more and more of our integrations automatically when integrations are upgraded. All you will notice is lower storage usage and faster queries. Enjoy!