CPU-based indexing can also degrade search performance when indexing and search happen simultaneously, especially in online systems, like ecommerce platforms and ad-serving pipelines. This happens because index builds consume significant CPU resources, leaving fewer cycles available for queries and driving up search latency and latency variance. Even after indexing completes, latency can remain high due to index fragmentation. Customers often run a forced merge to consolidate segments, but it can be slow on CPUs, further delaying recovery to the expected search latency by hours to days.

Elasticsearch has introduced GPU-accelerated vector indexing powered by NVIDIA cuVS. In this blog, we show how Elasticsearch achieves up to 7x faster vector indexing throughput by offloading hierarchical navigable small world (HNSW) index construction to GPUs, indexing 138 million vectors in under 10 minutes using eight NVIDIA RTX PRO 6000 GPUs.

Indexing over 100M Vectors in under 10 minutes

Figure 1 shows vector indexing throughput on 138 million 1,024-dimensional vectors from the MS MARCO dataset, representing roughly 4 TB of multimodal documents (Figure 1). The benchmark was run with Rally, Elasticsearch’s macro-benchmarking framework. The test ran on a server with 8 NVIDIA RTX Pro 6000 GPUs with a 2 socket AMD EPYC 9555. GPUs were turned on for the GPU test and turned off for the CPU test. The result was that Elasticsearch achieved 7x higher indexing throughput, lowering time to index 138M vectors from 1 hour on CPUs to under 10 minutes on GPUs.

Does GPU vector indexing change recall?

Index acceleration isn’t helpful if indexes built on GPU have lower search quality compared to CPUs. The throughput-versus-recall curves for GPU-built and CPU-built indexes overlap, showing that GPU indexing delivers the same recall profile as Elasticsearch’s CPU-based indexing path, with minimal accuracy or search-time performance penalty (Figure 2).

Search latency under concurrent indexing load

Offloading indexing to the GPU improved search latency on the CPU by 6x when running indexing and search simultaneously (Figure 3) because indexing on GPUs frees CPU resources for faster search latency. This allows a single Elasticsearch cluster to support real-time ingestion and low-latency retrieval at the same time, without forcing a tradeoff between freshness and search performance.

Force merge time for HNSW indexes on GPU versus CPU

Additionally, GPU-accelerated indexing in Elasticsearch reduced forced merge time from roughly four hours to five minutes, helping restore low-latency search in real-time (Figure 4). Note: Force merge was used to merge each shard into four segments.

Here are the full results for GPU versus CPU vector indexing in Elasticsearch:

Metric

CPU

GPU

Change

Index build time, 138 million vectors

~1 hour

Under 10 min

7x throughput

p90 search latency under indexing load

Baseline

6x lower

6x

Force merge, four segments per shard

~4 hours

~5 min

~48x

Recall at target

95%

95%

Unchanged

What GPU vector indexing means for Elasticsearch users

Elasticsearch’s integration of NVIDIA cuVS brings GPU-accelerated vector indexing to production AI search with over 84% reduction in search latency to deliver faster ingest with a 640% increase in indexing throughput with a lower CPU overhead and limited infrastructure tradeoffs. This unlocks a new class of applications, such as faster RAG pipelines, semantic search, multimodal retrieval, and fraud detection at scale.

Get started with GPU vector indexing in Elasticsearch

Install Elasticsearch with GPU-accelerated vector indexing by following these instructions. Replicate the above benchmarks by running rally msmarco track with parameters in the GitHub issue. Learn more about GPU-accelerated vector indexing, search, and preprocessing by visiting NVIDIA cuVS. 

Acknowledgments

The authors would like to thank Chris Hegarty, Gilad Gal, and Mayya Sharipova from Elastic, as well as Nathan Stephens from NVIDIA, for their contributions to this article.

Elasticsearch Workflows solves this with a persistent execution state. A workflow can pause at a human approval gate for days and resume exactly where it left off. Every decision is persisted in Elasticsearch as searchable data.

In this article, we’ll build this in practice with a real scenario: Documents that failed during ingestion get stuck in a data stream’s failure store. 

In this scenario, every step is reproducible. By the end, you’ll have an end-to-end workflow triggered by alerts, with pre-execution approval, automatic remediation, post-execution verification, and a rejection-and-revision path.

What you’ll learn

We’ll cover how to:

• Use the failure store to capture and remediate documents that failed ingestion.

• Build a long-running workflow with structured and binary approval gates using waitForInput and waitForApproval.

• Invoke Elastic Agent Builder AI agents automatically after human approval.

• Trigger workflows automatically from alerting rules.

• Handle approval and rejection paths (real human-in-the-loop, not just a checkbox).

What’s a long-running AI agent?

The difference between a session-bound agent and a long-running agent is the nature of the work rather than the speed of the large language model (LLM).

A session-bound agent lives within a session: It receives a question and processes it. Then it responds to the question. If the process dies, the context dies with it. 

Operational processes, like data remediation or infrastructure provisioning, or like change review, are different. The processing itself is fast. What takes time are human decisions at unpredictable moments. The person who needs to approve a reindex might be handling an incident or in a different time zone. Or they may simply be at lunch. Long-running agents handle these breaks in workflow by assembling separate infrastructure: a database to store execution state, a server for the orchestrator, a UI for approvals, and integrations for notifications. 

Elasticsearch Workflows runs where the data already lives. The workflow's long-term execution state is persisted in Elasticsearch, surviving restarts and waiting days between steps. AI agents come from Agent Builder and maintain a session only for the duration of each step, such as when the step ends or when the session ends. The workflow holds the context that spans across steps and days, and alerts come from the same rules already monitoring your data. The approval UI is in Kibana, and there’s no additional infrastructure to provision.

The workflow's internal execution state is managed by Kibana. The remediation-runs index is the audit trail that the workflow itself writes and that you can query like any other Elasticsearch index.

What’s the Elasticsearch failure store?

When Elasticsearch receives a document it cannot index, it has two options: Reject the document with an error, or store it somewhere safe for later analysis. The failure store is that second option.

Imagine a logs-demo-app data stream with the price field mapped as float. If the source application sends "price": "N/A" instead of a number, Elasticsearch cannot index the document. With the failure store enabled, it’s redirected to a dedicated index within the data stream itself rather than losing that document.

Documents in the failure store preserve the original content and include information about the error, including the exception type, the message, the pipeline and processor that failed, and even the stack trace. You can query the failure store using the data_stream::failures syntax. 

A typical document looks like this:

The document.source field contains the original document that failed, and the error field describes what went wrong. This is exactly the structure that the failure-analyst agent will read to diagnose the problem.

But the documents remain stuck there. Someone needs to diagnose the problem and fix the pipeline or mapping. They also need to reindex the documents back into the data stream. This is exactly the kind of work that combines AI automation with human approval, and it’s what we’re going to build.

How the AI agent orchestration works, from alert to resolution

The workflow connects four building blocks: an alerting rule, two AI agents, human approval gates, and an audit index. The diagram below shows how these components interact.

The alerting rule monitors logs-demo-app::failures every minute. When it finds documents written within the previous five minutes, it starts a workflow execution.

The workflow first runs read_failures to retrieve the failed documents. It then passes them to the failure-analyst agent through the diagnose step. The agent inspects the destination mapping and identifies the root cause. It then produces a structured remediation plan without making any changes.

The workflow records this plan in the remediation-runs index with the status awaiting_fix_approval and pauses at Gate 1.

Gate 1 uses waitForInput, a structured form that collects the reviewer's decision and optional notes. While the workflow waits, its state remains persisted in Elasticsearch. No agent session or polling loop needs to remain active, which means that the gate can wait for hours or days without continuously consuming compute resources.

If the reviewer approves the plan, the workflow invokes the remediation-executor agent through the execute_fix step. The agent calls the execute-failure-store-fix skill and creates the ingest pipeline. It also performs the bounded reindex and returns a detailed execution report.

Gate 2 appears only after the execution finishes. It uses waitForApproval to present the agent's report and asks the reviewer to choose between Yes, mark as resolved and No, escalate. The workflow then records the final outcome in remediation-runs and includes the complete report in agent_report.

If the reviewer rejects the initial plan at Gate 1, the workflow sends the diagnosis and the reviewer's feedback back to the failure-analyst agent, which then produces a revised plan that’s presented at Gate 1b. If approved, the revised plan follows the same automatic execution and verification path. If the plan is rejected again, the workflow records the case as fix_rejected and ends without applying any changes.

The remediation-runs index records the diagnosis, revision requests, revised plans, execution reports, and terminal outcomes as searchable audit documents. The shared workflow execution_id correlates records belonging to the same remediation run.

Production consideration: Exception-based verification

The diagram above shows the deliberately conservative implementation used in this tutorial: Every completed remediation pauses at Gate 2, including executions that appear fully successful. This makes the human-verification mechanism explicit and may be appropriate for low-volume or high-risk environments, but it can create review fatigue at scale. A production extension would add a deterministic verification step after execution. When the expected documents are present in the destination and the reindex reports no failures or policy-defined errors, the workflow can record resolved automatically. Only partial, failed, or ambiguous outcomes should pause at Gate 2 for site reliability engineering (SRE) review and possible escalation.

The following sections walk through each part: the data stream setup, the agents and skills, the alerting rule, and the workflow.

Prerequisites

To follow this tutorial, you’ll need:

• An Elastic Cloud Serverless project or an Elastic Stack 9.4+ deployment.

• Agent Builder (generally available [GA] on Elasticsearch projects, enabled by default).

• The following environment variables configured:

Setting up a data stream with the failure store enabled

All code in this tutorial, including the workflow YAML, setup scripts, and agent instructions, is available in this repository. 

Start by getting the code from the repository:

Then run the setup script:

The script prepares the test environment: It removes previous test resources, creates a failure-store-enabled index template, ingests three valid baseline documents, initializes the failure store, and creates the remediation-runs audit index. The price field is mapped as float with ignore_malformed: false ; nonnumeric values generate a parsing error and get redirected to the failure store instead of being silently dropped. The data stream itself is created automatically when the first document is indexed.

The failure store is enabled at the index template level, not on the data stream directly. The template includes a data_stream_options block that tells Elasticsearch to activate the failure store for any data stream created from that template:

The data stream is created automatically upon the first ingestion request, inheriting the failure store configuration from the index template. From that point on, any document that cannot be indexed is redirected to the failure store instead of being rejected with an indexing error.

The script also ingests three valid documents with numeric price values. These documents establish the healthy baseline: The data stream exists and contains valid data before any ingestion failures occur.

After the 01-setup-failure-store.sh script completes, the environment is ready:

The invalid documents that trigger the alerting rule and start the workflow are ingested later, when you run 02-trigger-test.sh in the testing section. This sequence shows a realistic failure scenario: The data stream is operating normally with valid data, and then an upstream change causes malformed documents to arrive.

Importing the agents and skills into Agent Builder

Run the restore script to import the two agents and two skills, along with the workflow into your Kibana instance:

Expected output:

Running the script again updates existing components without creating duplicates.

The AI agents that diagnose and execute the fix

After running the restore script, two agents will be available in Agent Builder: one to diagnose failures and another to apply approved fixes. In Kibana, go to Agent Builder and verify that both agents and their associated skills were created correctly.

failure-analyst, the read-only diagnosis agent

This agent reads failed documents from the failure store and inspects the destination mapping. It produces a structured remediation plan that includes an ingest pipeline definition and bounded replay instructions. The plan also includes a risk assessment. The agent doesn’t execute the proposed remediation.

The agent instructions establish its objective and read-only boundary. Key instructions include:

Its associated failure-store-remediation-planner skill provides the specialized rules used to construct a safe replay. For example:

The response contract is also constrained so that the workflow can consume the plan reliably:

Together, the agent instructions and skill allow failure-analyst to produce an actionable but bounded remediation plan while leaving execution under the control of the workflow and its human approval gate.

remediation-executor, the agent that applies the approved fix

The workflow invokes this agent after the remediation plan has been approved at Gate 1. Unlike the first agent, remediation-executor is allowed to perform the write operations required to apply the approved remediation.

Its agent instructions intentionally keep this role focused on immediate execution:

The associated execute-failure-store-fix skill defines how the approved plan is retrieved and executed. Key instructions include:

The skill searches by both execution_id and approval status. This ensures that the executor retrieves the plan approved for the current workflow execution rather than an unrelated remediation run. The status determines whether it uses the original diagnosis or the revised_diagnosis produced during the revision path.

This separation gives the two agents distinct responsibilities: failure-analyst investigates the failure and proposes a bounded plan without making changes, while remediation-executor applies the selected plan only after the workflow has completed the required human approval step.

The setup scripts import the complete agent instructions and skill definitions. The excerpts above highlight the instructions most relevant to understanding the responsibilities and behavior of each agent. The complete definitions are available in backup/save_agents.json and backup/save_skills.json in the repository.

After verifying that the agents and their skills have been imported correctly, the next step is to configure the alerting rule that detects new documents in the failure store and starts the remediation workflow.

How do I trigger a workflow from an Elasticsearch alert

The remediation process starts with an Elasticsearch alerting rule that monitors the failure store and starts the workflow when new failures are detected. Because the restore.py script has already imported and enabled failure_store_remediation, you can connect the workflow while creating the rule.

Creating the alerting rule that watches the failure store

The rule checks the failure store every minute and creates an alert when it finds documents written within the configured five-minute time window.

1. Go to Management → Rules → Create rule.

2. Select Elasticsearch query.

3. Configure:

• Query type: ES|QL.

• Query:

• Select time field: @timestamp.

• Select alert group: Create an alert if matches are found.

• Time window: 5 minutes.

• Check every: 1 minute.

4. In the Actions section, click Add action → Workflows.

5. Select the failure_store_remediation workflow.

6. Under Run workflow for, select New alerts.

7. Set Action frequency to Run per alert.

8. Save and enable the rule.

Whenever the rule creates a new alert, it starts one failure_store_remediation workflow execution for that alert. The failure store is still empty at this stage; the invalid documents are ingested later by 02-trigger-test.sh.

With the alert trigger configured, let’s examine the workflow that performs the diagnosis, approval, remediation, and verification steps.

Building the agentic workflow, step by step

At this point, the workflow has already been created in Kibana by the restore.py script executed earlier. To understand how the workflow operates, we’ll examine it incrementally, using relevant YAML excerpts to explain the role of each step in the remediation process. The complete workflow definition is available in the tutorial repository.

Declaring the alert trigger and the 7-day timeout

The workflow declares an alert trigger that allows it to be initiated by the alerting rule created earlier.

Two details are important here. First, the timeout: 7d defines that the workflow can remain active for up to seven days. We set this explicitly because the workflow will pause at human approval gates that may take hours or days to respond.

Second, declaring type: alert makes the workflow available to alerting. The workflow action configured in the rule establishes the connection between the alert and failure_store_remediation.

Reading failed documents from the failure store

The workflow begins by retrieving failed documents using the ::failures syntax:

The range query limits the search to failure-store documents created during the same five-minute window monitored by the alerting rule. Within that window, size: 50 bounds the batch to the 50 most recent matching documents. For larger volumes, use search_after or pagination.

How the AI agent diagnoses the root cause

The next step sends the failed documents to the failure-analyst agent:

The {{ steps.read_failures.output.hits.hits | json }} template injects the documents retrieved in the previous step into the agent's prompt. The agent analyzes the error patterns and identifies that the price field received string values where floating-point values were expected. It then proposes a remediation pipeline.

The agent's output arrives in steps.diagnose.output.message as a single JSON object. The exact diagnosis, proposed remediation, pipeline definition, and reindex request may vary between executions because they’re generated by the AI agent from the failures it receives. In our tests, the agent consistently identified the root cause and generated working remediation pipelines.

Despite this natural variability, the response contract remains fixed so that the workflow can persist the plan and the executor can reliably extract its executable components. The response always contains the six required top-level keys: root_cause, failure_type, affected_docs, proposed_fix, remediation_pipeline, and risk_assessment.

Recording the diagnosis for audit

Before requesting human approval, the workflow writes the diagnosis and proposed remediation plan to the remediation-runs index. This creates a durable audit record that can later be retrieved by the remediation-executor if the plan is approved:

The execution_id correlates the diagnosis with the current workflow execution. The record stores both the number of failure documents loaded into the bounded batch and the total number of failures matching the query. After human approval, the executor uses the execution_id together with the approval status to retrieve the correct remediation plan.

How do human approval gates work in Elasticsearch Workflows?

The workflow uses two kinds of pause steps for different kinds of human decisions. Gate 1 uses waitForInput to collect a structured approval decision and reviewer feedback. Gate 2 uses waitForApproval because verification requires a binary choice: Mark the case as resolved, or escalate it.

Gate 1

Gate 1b

Gate 2

Step name

gate_fix

gate_fix_revised

gate_verify (approved path) / gate_verify_v2 (revised path)

Step type

waitForInput

waitForInput

waitForApproval

Timeout

72h

72h

72h

Decision shape

Approve or reject, plus notes

Approve or reject, plus notes

Yes, mark as resolved / No, escalate

Presents

AI diagnosis and proposed plan

Revised plan

remediation-executor report

Appears

Always

Only after Gate 1 rejection

Only after execution completes

Gate 1 is defined like this:

When the workflow reaches this step, execution pauses, and its state is persisted in Elasticsearch. No agent session or polling loop needs to remain active while the gate waits. Every gate accepts a response for up to 72 hours, subject to the workflow's overall seven-day timeout.

The schema collects an approval decision and an optional notes field. When the proposal is rejected, the workflow passes the notes to the failure-analyst agent as context for revising the remediation plan. The testing section shows the approval and rejection payloads at the point where each is submitted.

What happens when the reviewer approves or rejects the plan

When Gate 1 is submitted, route_fix resumes the workflow without rerunning the completed steps. The simplified excerpt below shows the first transition in each branch. (The complete executable YAML is available in the tutorial repository.)

Approval sends the current workflow execution_id to the remediation-executor. The executor uses this identifier together with the approval status to retrieve the remediation plan associated with this specific execution before applying it and continuing to Gate 2.

If Gate 1 is rejected, the workflow stores the reviewer’s feedback and asks the failure-analyst agent to produce a revised proposal. Gate 1b (gate_fix_revised) then presents that proposal for another human decision. When the revised plan is approved, execute_fix_revised sends the same workflow execution ID to the executor:

The executor uses the approval status to determine which plan belongs to the current path: the original diagnosis associated with awaiting_fix_approval or the revised_diagnosis associated with awaiting_fix_approval_v2. Both paths then continue to Gate 2. If the revised proposal is rejected at Gate 1b, the workflow records fix_rejected and ends without applying any changes.

Verifying the result at Gate 2 before closing the case

Gate 2 is a verification checkpoint. By the time it appears, the remediation-executor agent has already created the ingest pipeline and run the reindex. It has also returned a detailed report. The gate embeds the executor's report, steps.execute_fix.output.message on the approved path, or steps.execute_fix_revised.output.message when the revised plan was executed, and asks the reviewer to select either Yes (mark as resolved) or No (escalate).

Gate 2 uses waitForApproval because the workflow only needs a binary verification decision. The operation details, including the pipeline ID, documents matched and reindexed, version conflicts, failures, and execution errors, come from the remediation-executor agent’s report rather than from fields entered manually by the reviewer.

In our validated execution, the agent created the logs-demo-app-price-remediation pipeline and matched and reindexed all five approved documents. It reported zero version conflicts and zero failures. The logs-demo-app data stream increased from three to eight documents, while the five original records remained in logs-demo-app::failures. This is expected because reindex copies documents; it doesn’t remove them from the failure store.

If the reviewer selects Yes, mark as resolved, route_verify writes a new document to remediation-runs with status resolved and stores the complete agent report in agent_report. If the reviewer selects No, escalate; the workflow writes an escalated record with the same report. The revised path mirrors this through route_verify_v2, producing the same statuses and the same agent_report field.

Gate 2 exists because a technically completed reindex can still produce partial or unexpected results. Escalating instead of automatically marking the remediation as resolved keeps the audit trail honest. Whether to keep this second gate depends on your environment and risk tolerance.

The complete workflow

After reviewing the workflow, remember that the complete code is available in the repository in the failure-store-remediation.yaml file.

The workflow can finish in five ways, depending on the decisions made at Gate 1, Gate 1b, and Gate 2. The diagram below shows how the original and revised remediation paths lead to resolved, escalated, or fix_rejected.

Every terminal outcome is appended to the remediation-runs audit history. Across the workflow paths, the audit records preserve the diagnosis, reviewer feedback when a revision is requested, the executor report, and the terminal outcome.

With the rule and its workflow action already configured, the end-to-end flow is ready to test.

Testing the workflow end to end

With the alerting rule enabled, run the trigger script to ingest five documents with invalid price values and current timestamps:

The rule searches the previous five minutes, so it detects the new failure-store documents during its next evaluation. Go to Workflows → Executions, and open the new execution. Confirm that read_failures, diagnose, and record_diagnosis have completed and that gate_fix is waiting for input.

Approving the fix at Gate 1

At Gate 1, approve the proposed remediation by submitting:

The workflow resumes at route_fix and invokes the remediation-executor agent through execute_fix. It pauses at Gate 2 with the execution report.

The following clip shows Gate 1 being approved and the workflow resuming automatically at route_fix and execute_fix.

Before completing Gate 2, verify that the five remediated documents were added to the data stream:

Expected result:

The data stream started with three valid documents, so a successful replay increases the total to eight. The original records remain in the failure store because reindex copies documents to the destination rather than removing the source records.

Complete Gate 2 by selecting Yes, mark as resolved or No, escalate. The workflow stores the selected outcome and the execution report in remediation-runs.

Rejecting the plan and reviewing a revised one

To test the rejection path, reset the environment and start another execution:

At Gate 1, reject the proposal with specific feedback:

Although notes is optional in the current schema, provide specific feedback whenever you reject a proposal. The workflow passes this value to the failure-analyst agent when requesting a revised plan.

The workflow records the feedback and asks the agent to revise the proposal. It pauses at Gate 1b. To approve the revised plan, submit:

To reject it definitively, submit:

Approving the revision sends it through the same execution and Gate 2 verification path described above. Rejecting it records fix_rejected and ends the workflow without applying the remediation.

Conclusion

The pattern built in this tutorial combines three elements: persistent workflow state, specialized AI agents, and human control at the points where judgment matters. The failure-analyst agent diagnoses the problem and proposes a bounded remediation. Gate 1 pauses before any change is made, giving the reviewer control over execution. After approval, the remediation-executor agent applies the fix automatically, and Gate 2 pauses again so the result can be verified before the case is marked as resolved or escalated.

Persistent execution state solves more than the problem of expiring agent sessions. The diagnosis, reviewer feedback, execution report, and final decision are stored as searchable data in Elasticsearch and correlated by the workflow execution_id. This correlation allows the executor to retrieve the plan approved for the current workflow run, even when multiple remediation cases exist in remediation-runs. It also makes it possible to analyze how many remediations were approved on the first attempt and which failure types are most frequently rejected, along with how long approval gates remain open and which cases require escalation.

This workflow pattern also applies to index promotions, mapping changes, enrichment pipeline validation, infrastructure operations, and other processes where automation and human judgment must coexist: Diagnose the problem, pause for approval, execute the approved action automatically, pause for verification, and record the final outcome.

Resources

• Elasticsearch Workflows documentation

• Agent Builder documentation

• Failure store documentation

• Tutorial repository

The changes are about storage and performance, along with the out-of-the-box (OOTB) experience. Up until 9.5.0, Elasticsearch operated as a document-based search engine by default. It could be set up to behave like a columnar system storage-wise, but that wasn’t the OOTB experience. Columnar index modes make a number of fundamental changes that allow Elasticsearch to optimize columnar analytic and search use cases:

• Fields are stored once as doc values only and are no longer indexed by default.

• New multi-value semantics. The original ordering of multiple values per field per document (for example, in arrays) is preserved by default.

• Mappings are always flat, and object and passthrough fields in mappings are always auto-flattened.

How columnar index modes fit into Elasticsearch

Many of the columnar index mode changes originate from time series data streams (TSDSs). As part of making TSDB a competitive metrics solution, we improved doc values format on disk and only store dimensions and metric fields once as doc values. We also improved query performance. TSDB is already columnar today. Essentially, this makes TSDB’s storage mode columnar. The lessons learned from TSDB are now being applied more broadly to Elasticsearch. 

Note that columnar index modes are opt-in and columnar, and document-based indices can coexist in the same cluster. An enterprise search use case can use a document-oriented index mode, while a logging use case can use logsdb_columnar index mode and be fully columnar, all in the same cluster. In fact, there are currently seven index modes, and indices can all use them in the same cluster.

How columnar storage stays fast without an inverted index

Indexed fields, either an inverted index for string-based fields or block k-dimensional (BKD) tree for numeric fields, allow Elasticsearch to query or filter by field very efficiently. However, the cost for this is an additional expensive data structure that uses a lot of disk space and is expensive to build at index time and at merge time. With the columnar index modes, fields are no longer indexed by default, so what did we do for query performance to be still acceptable on fields that were no longer indexed?

One major change was improving doc values scanning performance. This is key and is the cornerstone that any columnar system relies on. Previously, the scanning of doc values was essentially document by document. Lucene’s doc values API only allowed for looking up one value at a time. Historically, this fit the execution model of a search engine. In our own doc value format, we build the capability to allow bulk reading of values for Elasticsearch Query Language (ES|QL) queries. Also over recent minor Lucene releases, Lucene doc values API added support for bulk reading. Without this, fast columnar scanning wouldn’t have been possible.

Secondly, we fully adopted doc value skippers, a hierarchical skiplist over doc values. Contrary to an inverted index or a BKD tree, doc values skippers are lightweight data structures. At its core, a skipper allows queries to skip over a range of documents that don’t match a query. It can do this because it stores information, like min and max values. So, for example, when a range query is executed, an interval of documents can be skipped based on the intermediate result and a doc value skipper’s min and max values. The effectiveness of doc value skippers depends on the order in which documents are laid out on disk. This is why index sorting should be enabled or altered to match the use case.

By significantly improving our columnar scanning and doubling down on doc values skippers, we’re able to avoid indexing fields by default. Note that a field can still be indexed; the index mapping attribute just defaults to false in columnar mode. The exceptions to this rule are text-based fields, which are still indexed by default. This is because text fields provide free text search, which includes text analysis, along with phrase and wildcard matching. This is different from just filtering.

How Elasticsearch handles high and low cardinality fields

When setting up a schema with string fields, an important configuration parameter is often cardinality; that is, whether many unique values or a few unique string values are expected. Many systems have dedicated field or column types that target low and high cardinality string fields.

Fields that have low cardinality are typically stored with a dictionary, containing all unique values. Then, for each row offset, the offset into the dictionary containing the term the row has is stored. This is often called an ordinal. For low cardinality fields, this works well, as storing an ordinal per document takes up much less space. Encoding techniques, like delta encoding, offset encoding, and bitpacking, work well for ordinals to compact the per-document storage to just a few bits. 

However, for high cardinality fields, the dictionary and ordinal approach can work counterintuitively. If a larger percentage of the documents have a unique value, building the dictionary becomes expensive and storage savings diminish. The dictionary then becomes another level of indirection for reading values. This is why most systems in that case store values in a columnar fashion using block-based compression. For example, values of multiple rows are stored in 128KB blocks using a sliding-window dictionary-based compression algorithm (like zstandard or lz4). This, in general, is a simple and effective method to store higher cardinality fields and avoids building and maintaining a dictionary. 

With document-based Elasticsearch, there are two ways to map a string: using either the keyword field mapping or one of the text-based field mappings. The former stores an inverted index and dictionary-based doc values. The latter only stores an inverted index. This is why, typically, a text field mapper is often used in combination with a keyword mapper as a multi-field. Also, keyword field mapper (as the name suggests) is meant for keywords or fields that have a lower cardinality and uses dictionary-based doc values implementation. However, in practice, keyword field mappers are also used for high cardinality.

In columnar mode, every field is only stored once by default. For keyword fields, this means only doc values are stored with no inverted index. Text-based fields now also store doc values and an inverted index by default. Text-based field mappers are different from the keyword field mapper, as these are not automatically used via Elasticsearch’s dynamic mapping logic for columnar indices, and therefore text fields keep storing an inverted index by default.

For both keyword- and text-based fields, we didn’t choose to expose a cardinality mapping attribute. It’s not always possible to know ahead of time whether a field is low or high cardinality. When flushing and merging segments to disk, Elasticsearch sees all values and can determine the cardinality of a field. This is why we’re choosing to automatically determine whether the usage of a dictionary and ordinal-based encoding is beneficial over block-based compression using a simple cardinality threshold. If a field is below this threshold, dictionary and ordinal-based encoding is used; otherwise block-based compression is used. This simplifies configuration of the mappings and makes it possible to automatically optimize storage as data evolves, since some segments may use dictionaries while others may use blocks for the same field. However, this is currently not ready yet and so, as part of 9.5.0, in columnar mode, both keyword- and text-based field mappers store values in doc values in a block-based compressed layout on disk.

The two approaches compare as follows:

Dictionary and ordinal encoding

Block-based compression

Suits

Low cardinality fields

High cardinality fields

What’s stored

A dictionary of unique values, plus one ordinal per document

Values for many documents compressed together in blocks

Compression

Delta encoding, offset encoding, and bitpacking reduce each ordinal to a few bits

Sliding-window dictionary compression, such as zstandard or lz4, typically over 128KB blocks

Read path

Resolve the ordinal, and then look up the value in the dictionary

Decompress the block, and then read the value directly

Cost as cardinality rises

Dictionary grows large, savings shrink, and the extra indirection stays

Stable, with no dictionary to build or maintain

Used in columnar mode tech preview

Not yet

Yes, for both keyword and text fields

Columnar mapping attributes: multi_value, nullability, on_failure

The columnar index modes provide more control over how data is stored as doc values. By default, Elasticsearch is lenient and accepts all non-malformed values (for example, nulls and multiple values per field and document). If documents have fields with multiple values per document or no value, doc values store additional data structures to deal with them and therefore implicitly increase costs. 

With columnar, new mapping attributes provide additional control. Note that these new mapping attributes are currently only available with the columnar index modes but will eventually also be available for all index modes.

Three new mapping attributes are involved:

Attribute

Default

Enforces

On violation

Available

multi_value

true

One value per document per field

Document indexing fails

9.5.0

nullability

true

Field must have a value

Document indexing fails

9.5.0

on_failure

fail

How the above failures are handled

Sets fail or ignore behavior

Next minor release

multi_value: Enforcing single-valued fields

By default, Elasticsearch accepts multiple values per document. To understand the implications of this, we first should take a look at how Elasticsearch (using Lucene’s doc values) stores a dense numeric field where all documents have a single value:

With this layout, all values are stored in blocks. The number of values per block depends on the index mode but is typically 128 values and is always the same within an index. All values in a block are encoded using various encoding techniques, like delta encoding and bit packing, so each block can have a different size, depending on how well the encoding techniques compress the values. This is why a block index is required. 

Lucene has the notion of a docid (internal numbering for a document), which is essentially a row identifier.

1. Queries produce matching docids.

2. In case of a dense field, the block id can be resolved from the docid directly.

3. The offset of a block can be resolved from the block index.

4. Once that has been looked up, the target block gets decoded and all values are available.

5. Finally, from docid, the ordinal within the decoded values array can be resolved, which produces the final value.

Now let’s have a look at how the data layout changes when documents have multiple values per document:

To determine how many values belong to a single docid, an offset lookup is required.

In this case, a docid has one or more offsets. Each offset points to a block index. Values for a single document are adjacent but can stretch over blocks. In general, compaction of values works well in blocks because values are similar. However, multi-value fields can cause the compaction of values to be less efficient if the number of values per field and document is large and values aren’t similar. This and the additional storage of offsets result in multi-value fields typically having a higher storage footprint on disk.

If a field is truly single-valued, you may want to enforce this property. The multi_value mapping attribute makes that possible now. An example is a log level field. Logs typically have one log level (such as debug, info, or error). Enforcing that this field is single-valued in your mappings can help avoid accidentally using more storage than anticipated. A mapping snippet example that disallows the field log.level to have multiple values per document:

Note that even if a field allows multiple values, this doesn’t mean an offset lookup is stored. This only happens when a Lucene segment has at least one document with two or more values. The multi_value mapping attribute exists just for enforcement.

nullability: Requiring every document to have a value

By default, Elasticsearch accepts documents with fields that have no value or null value. Just as  multiple values per document require additional accounting, documents with no value require additional accounting to identify which of them have at least one value.

Doc values store a docid to offset lookup (known as IndexedDISI in Lucene) in case not all documents have a value in a segment. The offset either points directly to the block index for single-valued fields or the offset lookup in case of multi-valued fields. This lookup is compact compared to the value blocks being stored. However, if it were to be created for fields that should have at least one value per document, that would be a waste.

The nullability mapping attribute allows you to control whether documents are allowed to have no value. Just like the multi_value mapping attribute, the nullability attribute exists for enforcement.  Following is a mapping snippet example that requires the log.level field to have a value:

on_failure: What happens when validation fails

What happens if a document has multiple values for a field and if the multi_value mapping attribute is set to false or when a field is mapped with nullability set to false and a document doesn’t have that field? At the moment, indexing such documents will fail with a bad request error.

As part of the next minor release, the on_failure mapping attribute will be available. This allows you to indicate how to handle these validation failures, on a per-mapped field basis. This will support two values:

1. Fail: Fail indexing of the entire document with a client error. This is the current behavior in Elasticsearch 9.5.0.

2. Ignore: Ignore the validation error, mark the field as ignored, and store values for that field in a hidden field so that it can be introspected when requesting the source. 

Trying the columnar index modes

The columnar index modes are still under active development, but we encourage you to give them a test drive. As we prepare the columnar index modes for general availability (GA), we’ll add more performance and efficiency improvements. We believe that by adapting a columnar mindset, many use cases will benefit from being more cost effective or having better performance characteristics.

How historical metrics were loaded before backfill

Even if loading historical metrics isn’t a very common use case, it’s an important step when teams are adopting TSDB. Two scenarios have been the most prominent: bootstrapping a new time series data stream and migrating data from a different system or data stream to a time series one.

Bootstrapping a new time series data stream

You want to start a new time series data stream with a week of historical data so you have something meaningful to query from the start. With existing tooling, you had to set index.look_back_time to the seven-day maximum in the index template, and all historical data would land in a single backing index. For anything beyond seven days, you needed to create past backing indices manually.

Migrating metrics from another system

You have months worth of metrics stored on a different system and want to move your full dataset to TSDB. You need to load months of metrics history alongside live ingestion. The workaround was to manually create all the necessary past backing indices with the right time_series.start_time and time_series.end_time and to index into it directly using the index name. You then attached it to the data stream via the modify data stream API. It worked, but it required understanding the index time semantics and repeating the steps for each time window, along with coordinating that process around ongoing writes.

We wanted both scenarios to feel as close to normal bulk indexing as possible.

What time series data backfill changes

In 9.5, Elasticsearch can create backing indices covering past time ranges, which extends the eligible write window backward.

The eligible write windowis the range of @timestamp values that a time series data stream accepts for new documents. 

In the past, the eligible write window was determined only by the existing writable backing indices at the moment the request was received by Elasticsearch.

In 9.5, Elasticsearch can expand the eligible write window in the past by creating backing indices. This converts the eligible write window to a sliding window extending from the present back to the first read-only or destructive lifecycle action. Common examples of these actions, which are typically defined within your lifecycle configuration, are downsampling or searchable snapshots. Examples also include retention configurations. 

So, given that loading historical data is enabled in a cluster, the eligible write window of the data stream with the following lifecycle configuration is determined by the downsampling action, because it’s the first action that makes backing indices read-only. So, for this data stream Elasticsearch accepts documents whose @timestamp is no older than three months.

Why loading historical data into TSDB is hard

TSDB consists of data streams optimized for timestamped measurements. It uses a columnar storage layout and enforces immutable dimensions. It also organizes data into time-bound backing indices; each index covers a specific time range and accepts only documents whose @timestamp falls within it.

As time passes, rollover creates new backing indices to cover upcoming ranges. Until this release, there was no corresponding mechanism for the past. Creating indices in the past is tricky because historical data might span over a long period of time and can arrive at Elasticsearch out of order. Consequently, Elasticsearch cannot determine the write timeframe that its backing index should cover. Our solution to this is to use a preconfigured interval and lazily create past backing indices.

How Elasticsearch creates past backing indices

When a document is detected whose timestamp isn't covered by any existing backing index, Elasticsearch determines the time boundaries for the missing indices and creates them. It then adds them to the data stream in a single atomic operation. 

Lazily creating the indices ensures that a single request in the past won’t overwhelm the cluster by requiring the creation of 300 indices all at once. It also doesn’t create indices before there are docs to write into them.

Proactive vs. reactive: How we chose the index creation approach

We explored two ways to detect when a past backing index needs to be created.

The first is proactive. Inspect each incoming document's timestamp before routing, and create any missing past backing indices up front. This keeps the write path clean. By the time a document is routed, the index it needs already exists. It does require the data stream to already exist with at least one time series backing index, since that's what we inspect to determine the eligible write window and the time boundaries of the new index. The downside is that it adds work to every bulk request targeting a time series data stream, even requests that contain no past timestamps and need no backfill at all.

The second is reactive. Let the document fail the normal indexing, intercept that failure, create the missing index, and retry. This avoids any overhead on the common case, since the extra work only happens when a mismatch actually occurs. The tradeoff is more complexity in the failure handling path and a retry on every backfill document.

We ran performance tests on the proactive approach against bulk requests with no past timestamps and found no measurable regression. The overhead of inspecting timestamps turned out to be negligible. That settled it. Proactive creation is simpler and consistent with how index auto-creation already works in Elasticsearch. Plus, it adds no measurable cost to the workloads that don't use backfill.

How Elasticsearch determines past index boundaries

Each new past backing index has three properties to compute: its duration, its start time, and its end time.

Property

How it's set

Constraint

Duration

Defaults to one day, configurable via the cluster setting data_streams.past_tsdb_index_interval

Minimum one hour. If the triggering timestamp falls in a gap up to 1.3 times the configured duration, Elasticsearch collapses it into a single bridging index rather than creating many tiny ones.

Start time

Anchored to the start of the next existing backing index, working backward in multiples of the configured duration

Increased to match the end time of the previous neighboring index, where they would otherwise overlap.

End time

Start time plus the configured duration

Reduced to match the start time of the next index, where they would otherwise overlap.

Handling concurrent writes

In a distributed setup, multiple nodes can receive bulk requests with overlapping past timestamps at the same time. Each node collects the timestamps that aren’t matching any of the existing indices and sends a request to the master node. 

The master node executes a cluster update that sorts them and then, one by one, checks whether the timestamp is covered by an existing or newly created index. Otherwise, it issues a new create index request with the time boundaries calculated as described above. The cluster updates are always sequential and guaranteed to produce valid cluster states, so new indices are guaranteed to not overlap with existing indices.

How lifecycle age works for backfilled indices

Past backing indices hold old data but are new indices. Without an adjustment, lifecycle features would apply downsampling and retention based on when the index was created rather than when the data is from. We account for this by using the index.time_series.end_time as the index.lifecycle.origination_date. As a result, the age of the index as perceived by both data stream lifecycle and index lifecycle management (ILM) is based on the age of its data and not its creation time.

How to use time series data backfill

How to enable time series data backfill

Backfill support ships disabled by default. Enable it at the cluster level:

Bootstrapping with historical metrics

To load historical data into a new time series data stream:

1. Create your index template. 

2. Initialize your data stream. (This is an important step because an existing data stream is a requirement for creating past backing indices.)

3. Start indexing. 

Past backing indices are created automatically as documents with historical timestamps arrive, each covering one day of data by default. No additional configuration is needed.

Data migration into an existing data stream

Migrating data within the eligible write window

For data that falls within the eligible write window of your data stream, point your migration pipeline at the data stream and let Elasticsearch manage the rest.

Migrating data beyond a read-only action

For data older than the write window (for example, you're migrating 18 months of metrics but downsampling kicks in after seven days), you need a separate data stream without read-only lifecycle actions. Retention isn’t an issue since the data would be deleted anyway. The pattern is:

1. Create an index template for the historical data stream, using the same mappings as the original but without a lifecycle:

2. Create the historical data stream. If this step isn’t executed, the first indexing request might fail. During the first indexing request, Elasticsearch can create the data stream but it cannot yet create any past backing indices, so indexing a historical document might fail. Creating the data stream explicitly ensures that all indexing requests will be accepted:

3. Index historical data into the historical data stream while current data continues flowing into the original.

4. When the load is complete, add lifecycle. This is only supported by data stream lifecycle since this feature functions on a data stream level:

5. Query across both data streams with a wildcard pattern (my-metrics*) or a data stream alias.

6. If retention is configured, delete the historical data streams when their data expires. Data stream lifecycle will delete the data but it won't clean up the data stream itself.

As you see, the historical data needs to fit on the target tier as a whole because lifecycle will be enabled after the data is loaded. If you have a large historical import, you might choose to split it into batches. Make sure each batch can fit on the target tier as a whole at the time of indexing, to avoid running your cluster out of disk space. Lifecycle will start processing the batch's indices as soon as it's enabled, but it will need time to process the whole backlog.

Protecting the cluster during large migrations: Downsampling floodgate

When data stream lifecycle runs against a data stream with many indices that all qualify for downsampling, it queues them simultaneously. Downsampling is CPU and I/O intensive; it reads and rewrites all data in an index. Queuing dozens of operations at once can overwhelm the master node with persistent task updates while it coordinates them.

The downsampling floodgate scenario could occur before backfill support (for example, when adding a lifecycle policy to an existing data stream with months of accumulated data). Backfill makes it more likely by design.

In 9.5 and serverless, we added flood protection to data stream lifecycle. It now tracks how many indices per data stream are actively being downsampled. If that count reaches a threshold, data stream lifecycle pauses queuing further operations for that data stream until the count drops. The threshold is configurable via the cluster setting data_streams.lifecycle.downsampling.max_indices_in_progress. Other data streams aren't affected.

Limitations and prerequisites of time series data backfill

• Backfill doesn’t apply to read-only indices. If downsampling or a searchable snapshot transition has already run on a time period, documents for that period are still rejected.

• The feature requires a preexisting time series data stream with at least one time series backing index.

• System data streams are excluded.

• Replicated data streams rely on the leader data stream, so no direct backfilling is possible.

• Scaling remains your responsibility. Loading months of data can trigger significant storage usage, force merge operations, and lifecycle activity in parallel. Check that your cluster has the headroom to manage it before starting.

Conclusion

Prior to the Elasticsearch 9.5 release, loading historical data into TSDB was a manual process. By automating the generation and management of past backing indices, we aim to transform historical data migration to a native capability of your standard ingest pipelines. The inherent complexity of managing time-bound indices remains, but it has transitioned from a user responsibility into an internal Elasticsearch function. Whether you’re bootstrapping a fresh data stream or migrating extensive historical datasets, the platform now handles the heavy lifting, allowing you to focus on analyzing your metrics. We look forward to seeing how these improvements streamline your adoption of TSDB.

Observability workloads are storage-intensive by nature, and the composition of that storage determines both cost and query performance. ES95 introduces adaptive compression: rather than applying the same encoding to every numeric field, it automatically selects the encoding that best matches each field's structure. The result is a 33.6% reduction in total doc-values storage, 19% to 74% reduction on floating-point gauge metrics and a 92% reduction in @timestamp. No configuration or migration required.

Observability data is storage-intensive

Data processing systems are rarely limited by how fast they can compute. They’re limited by how fast they can move bytes: off disk, across the network, and through the memory hierarchy. Compression is how a storage engine trades CPU time for memory bandwidth, spending comparatively cheap CPU cycles so fewer bytes have to travel through the parts of the system that are usually constrained. In a read-heavy system like Elasticsearch, that trade-off pays back every time data is queried, often long after it was written.

Storage size and query performance move together; fewer bytes on disk means fewer bytes to read on every range query, every aggregation and every dashboard load. Compression is not just about saving storage. Every byte that is never written is also a byte that never has to be read.

The right encoding depends on the structure of the values themselves, and the largest wins come from exploiting the structure already present in the data rather than squeezing an opaque stream of bytes. Few workloads expose that structure more clearly than observability metrics.

A single host reports hundreds of metrics every few seconds, including CPU utilization, memory ratios, request latencies, and network throughput. Multiply that by thousands of hosts across weeks of retention, and the bytes accumulate fast. Most of that volume is structured but not uniform: timestamps arrive at near-constant intervals from thousands of concurrent series, counters increase monotonically, while gauges like 23.47 or1.15 are short decimal measurements.

A fixed compression approach cannot adapt to that variety. A timestamp column and a floating-point gauge column compress through fundamentally different techniques, but a codec that applies the same approach to both will necessarily handle one of them poorly. For most of Elasticsearch's time-series codec history, gauges were on the losing end of that trade-off.

The structure the old codec was not built to exploit

Elasticsearch stores time-series numeric values in doc values: a column-oriented structure where all values for the same field sit adjacent on disk. That adjacency makes compression possible: the codec compares consecutive values of the same field, finds patterns, and exploits them.

The time-series codec before ES95 applied the same fixed encoding to every numeric field: delta encoding followed by normalization, GCD (greatest common divisor) reduction, and bit-packing. Each encoding technique activated where it helped and skipped where it would not. For timestamps and integer counters, this approach was remarkably effective. For floating-point gauges, it could find almost nothing to work with.

The reason is how they’re stored. To support range queries, Elasticsearch stores floating-point values as integers that preserve numeric ordering. A change of 0.01 in a CPU percentage reading translates to a jump of trillions in that integer space. The codec sees those large jumps and has no strategy to further reduce their footprint. Storage stays near the original eight bytes per value.

The codec was doing the right thing with the representation it had, but the latter was chosen for querying, not compression, and the goals conflict at the bit level.

The cost of a fixed format

A compression stage for floating-point values was already on the roadmap, so the interesting part wasn’t the algorithm. The obstacle was architectural.

The previous codec baked its compression approach into the storage format. Adding a new encoding meant changing the meaning of existing bytes on disk, which forced a format migration, a rollout that can last weeks or months in large production clusters. Over time, that migration burden constrains codec development itself. The question stops being Is this a good compression idea? and becomes Is it worth another format migration? That rigidity limits the cadence of codec evolution and leads to missed compression improvements.

The right encoding without configuration

ES95 solves this at the architecture level for time-series indices. Each field's encoding is no longer baked into the format. It is selected automatically at write time, based on what the field mapping already declares: the field's name, its data type, and its metric role. Timestamps are encoded differently than counters. Counters are encoded differently than gauges. ES95 encodes all of them, and it chooses the right strategy for each.

Users already tell Elasticsearch everything the codec needs to know. The mapping describes the data; the codec chooses the compression strategy.

Compression strategy is a codec concern, not a user concern.

The alternative would have been to expose per-field encoding selection as a configuration parameter, letting users opt in to better compression for specific fields. That would shift the burden of knowing which encoding fits which data type onto those least equipped to make that call and would guarantee that most deployments never see the benefit. ES95 keeps that decision inside the codec, where it belongs. This matters most in managed and serverless deployments, where users expect the system to automatically make optimal storage decisions.

The timestamp result nobody planned for

With the adaptive architecture in place, the team set out to ship the planned float-compression algorithm. Before it arrived, the architecture proved itself by substantially improving compression for timestamps.

A time-series index is sorted first by its time-series identifier (_tsid constructed by the metric’s dimensions) and then by timestamp within each series. Timestamps on disk aren’t one smooth sequence; there are many smooth sequences laid end to end, one per series, with a large jump at every boundary where one series ends and the next begins.

The codec compresses data in fixed-size blocks without regard to those series boundaries. A block straddling a series boundary holds timestamps from two different series. The jump between them breaks monotonicity, reducing delta encoding effectiveness on blocks spanning different time series. A block that would otherwise compress to near-zero bits per value ended up needing nine or more, because bit-packing encodes every value in a block using the same fixed number of bits, so one large jump sets the cost for all of them.

In the ideal case, every block belongs to a single series: timestamps increase at near-constant intervals, delta encoding captures the regularity, and bit-packing compresses the result to near-zero bits per value. With few series, boundary blocks are rare and the overhead barely registers. On an observability cluster ingesting millions of documents across thousands of series, that changes. The cost scales along two dimensions: series count and data density. More series means more boundary events; sparser series means multiple jumps packed into single blocks. In high-churn environments, both compound, and boundary blocks accumulate into a standing tax on the most-read field in any time-series workload. It’s why @timestamp storage grew faster than the data that produced it.

The fix was to detect series boundaries and treat each run as its own independent sequence. Instead of trying to encode across the jump between two series, which forces every value in the block to pay the storage cost of that one large jump, each run is compressed on its own terms. The boundary simply becomes a seam: the jump is never seen by the encoder on either side.

That encoding is called SplitDelta. @timestamp and monotonic long counters now use it by default. No format change. No migration. Existing segments retain legacy encoding.

On the high-cardinality Elasticsearch Rally benchmark, that single unplanned encoding cut counters storage by 20%–30% and @timestamp storage by 92.4%, from 1.03 GB to 79 MB. Gigabytes to megabytes, and no, that isn’t a typo.

The pluggable pipeline had already paid for itself. SplitDelta, which wasn’t part of the original plan, slotted in without a format change or migration before ALP even shipped.

ALP: recovering the decimal that was always there

Most floating-point metrics can be expressed as short decimals with no loss of accuracy: CPU utilization at 23.47, load average at 1.15. ALP (Adaptive Lossless floating-Point compression) recovers that decimal structure from the floating-point representation, converting values into integers that the existing pipeline already handles well. ES95 feeds ALP's output into the same mature integer compression pipeline used for timestamps and counters, extracting additional savings rather than treating ALP as a standalone encoding. Values that don’t fit ALP's model (such as irregular high-precision floats or special values) fall back to direct bit-packing or the original representation without degrading the rest of the block.

ALP lets Elasticsearch treat floating-point metrics according to the structure they actually contain rather than the binary representation they happen to use. It’s applied automatically to double-valued gauge fields, selected by field type and metric role, through exactly the door the architecture had built for it.

What the numbers say

Here’s what the Elasticsearch Rally benchmark looks like on a high-cardinality workload containing 2.26 billion data points. Results are from an internal tsdb-metricsgen benchmark.

Field or metric

Storage reduction (%)

@timestamp

−92.4%

cpu.load_average.5m

−74.3%

system.cpu.utilization

−63%

memory.utilization

-19%

Total doc values

−33.6%

That overall 33.6% reduction deserves context.

A time-series index contains more than metrics. Every data point also carries the labels that identify the series: host names, IP addresses, regions, container IDs. Those dimension fields are stored as keywords. ES95 doesn’t target dimensions.

On this benchmark, two dimension fields, host.ip and host.mac, accounted for 44% of doc-values storage after ES95 ran. The 33.6% total reflects that mix. The per-field breakdown is the honest picture. Compression for dimension fields is an active area of work.

The per-field variation is the most convincing result. Some gauges shrank by nearly three quarters, while others moved by less than a fifth. That spread is direct evidence that ES95 matches compression to the structure actually present in each field. A fixed encoding treats every field identically and misses most of those wins.

Better compression without extra configuration

The storage reductions from SplitDelta and ALP are the most visible results of ES95. The more consequential result is the architecture that produced them.

Before ES95, every new compression technique required a format evolution. That reality shaped which ideas were practical to pursue. Today, new encodings become implementation decisions inside the codec rather than migration projects. Existing data never needs to move, and users gain better compression on newly written data simply by upgrading Elasticsearch. SplitDelta and ALP are the first encodings to benefit from this architecture. They will not be the last.

Asking users to choose compression algorithms would only duplicate information Elasticsearch already has. There are no per-field compression parameters to tune, and no expert knowledge is required to get good storage efficiency. Different fields get different strategies because ES95 understands what kind of data each field contains, not because a user configured it. As the codec evolves, those decisions evolve with it. The API does not.

In Elasticsearch Serverless, good defaults are part of the product. Users expect the system, not configuration, to make storage decisions. ES95 is designed to honor that expectation: encoding that starts right and gets better over time.

The compression was always there

ES95 establishes a new standard for how time-series codec evolution works. New encodings become implementation decisions, not migration projects. Users get better compression on newly written data with every Elasticsearch upgrade.

The compression was already in the data. ES95 just removed what was hiding it.

How to configure frozen_after in data stream lifecycle

frozen_after sits at the top level of the lifecycle, next to data_retention and downsampling.

That's the whole feature, at the API level. Indices in my-data-stream stay on hot for 30 days, then move to frozen for the remaining 60. After 90 days they're deleted, and the backing snapshot goes with them.

It composes with the rest of the lifecycle, including downsampling:

Same options in an index template:

The order of values is enforced: frozen_after has to be less than data_retention and greater than any downsampling.after. The API rejects configurations that don't make physical sense.

Where frozen tier data is stored: the default snapshot repository

Frozen tier data is held as partially-mounted searchable snapshots, which means DLM needs a snapshot repository to write into. Rather than make you choose a repository per lifecycle, 9.5 introduces a cluster-level default snapshot repository.

DLM uses this repository for every frozen tier index in the cluster. On Elastic Cloud Hosted (ECH), the default is pre-populated with found-snapshots so existing clusters work out of the box. You can change it to a repository you control if you'd rather keep your frozen data in a bucket you own (useful if you want object versioning, lifecycle backups to Glacier, or anything else that needs bucket-level access). Wherever you can set frozen_after in Kibana, the UI shows the current default repository inline and links to the place to change it, so you can see where frozen data will be written.

If the cluster doesn't have a default repository configured, you can still write a lifecycle with frozen_after. The API accepts it but returns a warning:

The data stays on hot until a default repository is configured and exists. The same logic applies if the cluster lacks a valid Enterprise license. Errors are visible in the data stream lifecycle status API for that stream.

Configuring frozen_after in Kibana

Kibana in 9.5 lets you set frozen_after and the default snapshot repository from the UI. In Streams, the Retention tab shows the frozen phase on the lifecycle timeline alongside hot and any downsampling steps. Click the timeline to open the data lifecycle flyout, set frozen_after, and see the timeline update before you save. Index Management's Data Streams page opens the same flyout.

How frozen tier conversion works in data stream lifecycle

When a backing index ages past frozen_after, DLM walks through five steps in order:

1. Clone. Mark the index read-only and clone it to a zero-replica copy so the original stays available while conversion runs.

2. Force merge. Merge the clone to a single segment. On completion a cluster state marker is written; duplicate force-merge requests (for instance after a master failover) are deduplicated, so a restart doesn't repeat the work.

3. Snapshot. Write the merged clone to the default repository, and record the snapshot name in cluster state on success. If a stalled snapshot from a previous attempt is detected, DLM deletes it and re-runs the step.

4. Mount. Create a partially-mounted searchable snapshot index from the snapshot.

5. Swap and delete.Once the mounted index's shards are fully allocated, atomically swap it in for the original in the data stream, then delete the original. The swap is atomic, so query results don't see a data volume dip during the transition.

On failure, DLM retries from the last successful step on the next run.

Each step is idempotent, and the cluster state markers make sure work already done isn't repeated after a master failover. Throttling caps concurrent conversions so a lifecycle change covering thousands of indices doesn't overwhelm the cluster. Errors surface in the data stream lifecycle status API and roll up to the lifecycle health indicator.

Scope and limitations of frozen_after

• Data indices only. frozen_after applies to a data stream's data indices. Failure store indices don't currently support the frozen tier and continue to be governed by their own data_retention setting.

• Enterprise license required. Frozen tier in DLM is implemented with searchable snapshots and requires an Enterprise license. You can write frozen_after on any license, but data won't move to frozen until the license is valid.

• Serverless ignores the field. In Elastic Cloud Serverless, frozen_after is accepted but ignored - Serverless manages tiering on your behalf. Built-in templates may include the field, so we don't reject it, but the step is skipped.

Getting started with frozen_after

1. In Kibana, open Streams or Index Management and choose a data stream backed by data stream lifecycle.

2. Open the Edit data lifecycle flyout, set a frozen-after value, and save. The lifecycle timeline shows the new phase.

3. On a self-managed cluster, set repositories.default_repository to a repository you control. On ECH, found-snapshots is already configured if you want zero setup.

4. For declarative workflows, write the same configuration into your index templates so new data streams pick up the lifecycle automatically.

Learn more

• Data stream lifecycle

• Frozen tier overview

• Searchable snapshots

• Manage data retention for Streams

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The palette takes shape: how multimodal search in Elasticsearch evolved from semantic_text

The semantic field is a convergence of several complementary features we've introduced over the past couple of years, bringing them together to create a cohesive multimodal search experience. Each solved an important piece of the semantic search puzzle on its own; together they enable native multimodal search.

The first brushstroke was semantic_text. Before it, running semantic search meant manually configuring mappings, wiring up ingest pipelines with an ML model, manually chunking content, and generating query-time embeddings yourself. The semantic_text field folds all of that away: it performs inference automatically at ingest time, chunks long documents for you, and simplifies the queries you write against it. Introduced in Elasticsearch 8.15 and released as GA in Elasticsearch 8.18, it has become the foundation for semantic search on the platform.

Next came the model to power multimodal search. jina-embeddings-v5-omni is our family of multimodal embedding models, capable of embedding text, images, video, audio, and PDFs into a shared vector space. Because those embeddings are semantically compatible across modalities, you can store diverse media in a single index and query across all of it at once, such as retrieving an image via a text description or matching audio against a written phrase, all without maintaining a separate pipeline for each content type. For more detailed information about how these embeddings are generated, see the model documentation.

We added the embedding query vector builder in Elasticsearch 9.4 to handle multimodal inputs at query time. Query vector builders are general-purpose tools you can use to convert input to a vector at query time as part of your request. For example, we have the text_embedding query vector builder for text-only models and input, and the lookup query vector builder for getting a vector from an existing document. The embedding query vector builder is a new type that works with multimodal models and accepts multimodal input, including text or base64-encoded binaries. This allows you to pose a query in whatever modality fits, and Elasticsearch generates the matching vector on the fly.

The final piece was multimodal ingest. The semantic_text field brought automatic embedding to text; the semantic field extends that same automatic experience to images, audio, video, and PDFs from ingest through query.

Painting the picture: creating an index with the semantic field

Let’s create an index with a semantic field. This is as simple as setting the field type to semantic and defining the inference endpoint you want to use:

In this example, we use the .jina-embeddings-v5-omni-small inference endpoint. This is our built-in jina-embeddings-v5-omni inference service, and it is available in all environments with access to the Elastic Inference Service (EIS). This includes:

• Serverless.

• Elastic Cloud Hosted (ECH).

• Self-managed with Cloud Connected Mode (CCM).

Indexing images, audio, video and PDFs

To index an image, provide an object with a type of image and a value containing the image as a base64-encoded data URL:

Arrays of objects are also accepted, allowing you to index multiple images in a single field value:

The semantic field also supports text values, just like semantic_text. You can provide such values standalone or intermix them with image values:

Text values are handled just like they are with semantic_text: long passages are chunked according to the chunking settings configured on either the inference service or field mapping. Multimodal values, such as images, are not chunked. Each multimodal value is represented as one chunk.

Other modalities are supported as well. Change the type value to match your content’s modality. Currently we support:

• image

• audio

• video

• pdf

For example, to index a video, the request would look like:

Image search and cross-modal retrieval with a text query

To find multimodal content using a text description, run a match query on the semantic field:

Just like with semantic_text, Elasticsearch automatically generates an embedding for the query text using the inference endpoint associated with the field. That query embedding is used to return semantically similar matches.

This query pattern enables easy text-to-image search. Just index an image and use a match query to retrieve it via text description! It also works for any other modality: index the multimodal input and search by description to retrieve it.

Querying with images, video, and other multimodal inputs

We can also search using a multimodal input by using the knn query with an embedding query vector builder. For example, we can search using an image:

The input object format is the same as when providing an image to index: set the type to image and value to a base64-encoded data URL.

Similar to when querying by text description, Elasticsearch automatically generates an embedding for the query image using the inference endpoint associated with the field. That query embedding is used to return semantically similar matches.

Just like with indexing, other modalities are supported, but are limited to those supported by your inference endpoint. For example, a search using a video clip would look like:

Extending the composition: highlighting, retrievers, and other semantic field features

The semantic field didn't start from a blank canvas. It's built on the same foundation as semantic_text, inheriting its behavior and its ergonomics, and extending them to multimodal content. In practice, that means nearly everything you already know about working with semantic_text carries over unchanged. If you've built with semantic_text before, the semantic field will feel immediately familiar.

Here’s a selection of the features that come along for the ride. See the documentation for a complete list.

Highlighting the best-matching chunks

If you index multiple values in a semantic field, you may want to know which value best matches the query. The semantic highlighter can be used to return the most relevant chunks as highlight fragments:

Setting order to score returns the fragments ranked by relevance, while number_of_fragments caps how many chunks come back. The response looks like:

Note how highlighted multimodal values are represented using their data URLs.

Controlling vector quantisation with index options

The semantic field stores its embeddings in an underlying vector field, and index_options lets you control how that vector field is indexed. For example, choosing a non-default quantization strategy:

Multi-field retrievers

The semantic field participates in the multi-field query format supported by the linear and rrf retrievers. Rather than hand-writing an inner retriever per field, you supply a single query and a list of fields, mixing lexical fields and semantic fields freely:

The retriever automatically separates lexical fields from semantic fields, queries each group, and normalizes the results so that each group contributes equally to the final ranking, preventing lexical matches from drowning out semantic ones.

Cross-cluster search

The semantic field supports cross-cluster search (CCS), enabling use of the field in large, multi-cluster deployments. Simply list the indices to query using the standard <cluster>:<index> format:

The fields queried across indices and clusters can use a mix of different inference endpoints that produce different query embeddings. The search request will automatically apply the proper query embedding to each individual field queried.

Off the easel, into the world: optimising multimodal embeddings for production

When you move multimodal search from experiment to production, the size of your multimodal inputs becomes a practical concern. Multimodal data is supplied as base64-encoded data URLs, and that data is stored in the index. Those strings can grow large in a hurry: a single high-resolution file can balloon into several megabytes of encoded text, which has several side effects:

• The index size on disk can increase significantly.

• Requests and responses containing multimodal data are larger, increasing transmission time and ingress/egress costs.

• Inference on larger multimodal inputs is slower.

The good news is that you don’t need that much fidelity. Multimodal embedding models reduce each input to a compact representation before generating a vector anyway, so a smaller, lower-fidelity version of a multimodal input (such as a downscaled image or a lower-bitrate audio clip) produces a very similar embedding, and similar search quality, to its full-size original. This also applies to PDF input. PDFs are generally processed visually by multimodal models, so the quality only needs to be good enough to perform operations like image embedding and OCR. Long PDFs should be broken up into chunks of smaller inputs, so the embeddings generated more accurately represent each chunk. Feeding the model small inputs keeps your documents lean, trims index and response sizes, and speeds up ingestion, all without meaningfully affecting relevance. 

Elasticsearch reinforces this practice with a guardrail: the indices.inference.max_binary_input_size cluster setting caps the size of each binary input, defaulting to 1 MB. Any individual value that exceeds the limit is rejected with a clear error, so oversized inputs surface as an actionable problem at index time rather than as silent bloat. This setting is adjustable in self-hosted and ECH through the cluster settings API. It is not adjustable in our serverless offering, where 1 MB is the hard limit for binary sizes.

When possible, it is also advised to use source filtering to exclude semantic fields from responses. For example:

This makes responses smaller, more performant, and easier to parse because multimodal data is not returned with each.

Try out the semantic field

The semantic field is available in Elasticsearch 9.5 and Serverless. Start a free trial and try it out today.

Elastic Connectors are a type of Elastic integrations that sync data from an original data source to an Elasticsearch index. Connectors enable you to create searchable, read-only replicas of your data sources.

There are a number of connectors that are supported for variety of 3rd-parties, such as:

• MongoDB

• Various SQL DBMS such as MySQL, PostgreSQL, MSSQL and OracleDB

• Sharepoint Online

• Amazon S3

• And many more. The full list is available here.

Connectors content synchronization jobs

Connectors support two types of content synchronization jobs: full syncs and incremental syncs.

1. Full syncs

Full sync is a sync that extracts all desired documents from a 3rd-party service and ingests them into Elasticsearch. So if you've set up your Network Drive connector to ingest all documents from a folder "\Documents/Reports\2022**.docx", during a full sync the connector will fetch all the documents that match this criteria and send all of them to Elasticsearch. Simplified pseudocode for this would look like:

This works well until the sync starts to take too long. This could happen because the connector fetches more data than needed. For instance, why fetch old files that have not changed and send them to Elasticsearch? One could argue that the metadata for files could be unreliable, so all files need to be fetched again and sent to Elasticsearch. Indeed, that could be the case, but if we can trust the metadata of the data fetched from 3rd-party, we can ingest less data. Incremental sync is the way to do so.

2. Incremental syncs

Most of the time, if written well, connector spends doing IO operations. Returning to the example code there are 3 places where IO happens:

Each of these places can become a bottleneck and take a significant amount of time during the sync.

Here's where incremental sync comes into play. Its purpose is to decrease the amount of IO on any of the stages, if possible.

Potential optimizations for incremental sync

Fetch fewer documents from 3rd-party systems

Modifying the example above, the code could look like this:

In cases where only a small number of documents change in our 3rd-party system, we can speed up the ingestion process significantly. However, for Network Drive it's not possible - its API does not support filtering documents by metadata. We won't be able to avoid scanning through the full content of Network Drive.

Skip download of content of files that haven't changed since previous sync

Downloading file content takes a significant amount of time in the syncs. If files are reasonably large, the connection is unstable or throughput is low, downloading the content of files would take most of the time when syncing the content from the 3rd-party. If we skip downloading some of them, it can already significantly speed up the connector.

Consider the following example pseudocode:

If no documents were updated, the sync will actually be magnitudes faster than fully syncing the content.

Skip ingestion of non-modified documents into Elasticsearch

While it may seem minor, ingestion of data into Elasticsearch takes a significant amount of time - albeit normally less than downloading the content from the 3rd-party system. We can start storing timestamps per each document and not send the documents into Elasticsearch if their timestamp did not change.

We can combine this approach with the previous approach to save the most time possible during the sync.

This approach helps save even more time when running a sync. Now let's take a look into performance considerations for such improvements.

Measuring incremental sync performance

Now since we've taken a look into simplified code that shows how incremental syncs can work, we can try to estimate potential performance improvements.

For some connectors, incremental sync is implemented in a certain manner that optimizes the way data is fetched from a 3rd-party. For example, the Sharepoint Online connector fetches some data via delta API - only collecting documents that changed after the last sync. This improves performance in an obvious manner - less data -> less time to sync the data to latest.

For other connectors (currently all connectors except Sharepoint Online connector) incremental sync is done by framework in a generic way which was described in one of previous sections "Skip ingestion of non-modified documents into Elasticsearch".

Connectors still collect all the data from 3rd-party data source (as it does not provide a way to fetch only the changed records). However if this data contains timestamps, the connector framework compares document IDs and timestamps of already ingested documents with incoming documents. If the document exists in Elasticsearch with the same timestamp that was received from the 3rd-party data source, then this document will not be sent to Elasticsearch.

We've described abstract approach for the performance improvements with incremental syncs, but we already have these implemented in connectors, so let's dive into real numbers!

Performance tests

We will estimate the rough magnitude of improvement for incremental syncs with these performance tests, not aiming at high precision.

The two connectors chosen for this test, Google Drive and Github, were chosen because they have different IO profiles.

Google Drive acts like a file storage. It:

• Has a fast API that does not throttle too soon

• Normally stores a lot of binary content of variable size - from small to really large

• Normally stores a small number of records - tens or hundreds of thousands rather than millions

GitHub data is ingested via a more of a classic API, that:

• Throttles quite often

• Contains many records that are much smaller than those in Google Drive

• Does not send binary content at all

Due to these differences, the incremental sync performance will majorly differ.

Both tests will contain these mandatory steps:

1. Do a full sync against a 3rd-party system

2. Modify some documents on the 3rd-party system

3. Run an incremental sync and check the amount of time it takes

This setup is very bare bones but will already give a good indication of the magnitude of performance improvement. Both tests will be slightly different and I will provide results with commentary in the next section.

Setup #1 - Google Drive Connector

Initial setup will be:

• 1 folder is on Google Drive with 1553 files (100 of them are 2MB in size, 1443 are 5KB in size)

• A full sync is executed and this data gets into Elasticsearch

• More files are added into Google Drive to make it 10144 files (100 of them are 2MB in size, all the rest are 5KB in size)

• Incremental sync is executed again to pull the new data

• Then some minor changes are made to files on Google Drive (1 added, 2 deleted)

• Incremental sync is executed again

• Full sync is executed to compare the run time against incremental sync again

The following table contains the results of the described test with commentary:

Sync Description

Run time

Documents Added

Documents Deleted

Comment

Initial Full Sync

0h 4m 0s

1553

0

This is initial sync - it pulls all documents

Incremental Sync after more data was added to Google Drive

0h 20m 9s

7939

0

Run time was high as expected - a lot of documents went in

Incremental Sync after some data was slightly changed in Google Drive

0h 1m 25s

1

2

Run was very fast. It still called Google Drive API a lot, but did not have to ingest 200+MB of data into Elasticsearch

Full Sync to compare performance

0h 23m 23s

10144

0

As expected, it takes a lot of time - all the data is downloaded from Google Drive and is sent to Elasticsearch, even if it did not change. We can assume that it takes 22 minutes to download and then upload the data into Elasticsearch for the setup

In summary, incremental sync significantly improved the performance of the connector because most of the time is spent on the connector downloading the content of the files and sending this content to Elasticsearch. Full sync brings 2 * 100 + 1443 * 5 / 1024 = 207MB of content - both downloaded by connector and ingested into Elasticsearch. If only 1 large file is changed, this amount changes to only 2MB - a magnitude of 100 change. This explains the performance improvement well.

Setup #2: GitHub connector

The GitHub connector is very different since the actual volume of data it syncs is relatively small - issues, pull requests and such are reasonably small, while there are lots of them. Additionally, GitHub has strict throttling policies and throttles connector a lot.

To give a good real-world example we’ll use the Kibana Github repository with the GitHub connector and observe its performance.

Sync Description

Run time

Documents Added

Documents Deleted

Comment

Initial Full Sync

8h 40m 1s

147421

0

---

Incremental Sync ran immediately after

9h 6m 7s

59

0

This sync took even more time to run, mostly because it was constantly throttled. Connector had to fetch all the data from GitHub but sent only 59 records with a total volume of less than 1MB

Next incremental sync

9h 2m 52s

191

1

This sync was triggered immediately after previous incremental sync finished. Run time is the same due to data being almost the same and throttling being a major factor in the connector run time

Key takeaways

• As you can see, there is no performance improvement for incremental sync for the Github connector - there is barely any space for optimization as most of the time is spent by the connector querying the system and waiting for the throttling to stop.

• Documents that are extracted are reasonably small, so network throughput usage is minimal. To improve the connector run time, the incremental sync would actually have to limit the number of queries to Github, but at this point it's not implemented in the connector.

Summary

What is the primary factor that impacts the performance of incremental sync? In simplified terms, it's the raw volume of data that is ingested.

For Sharepoint Online connector there is a special logic to fetch less data via the delta API. This saves good amount of time because the delta API allows connectors to not fetch files that were not changed. Files tend to be large, thus not downloading and ingesting them will save a lot of time.

For other connectors, incremental sync is generic - it just checks document timestamps before ingesting them to Elasticsearch - if this document is already in the index and the timestamp did not change, then it is not ingested. It saves less time than the previous approach that Sharepoint Online employs but works generically for all connectors. Some connectors - ones that contain large documents - benefit from this logic a lot, while others - that get throttled by a 3rd-party system and contain relatively small documents - get no benefit from incremental syncs.

Additionally, if Elasticsearch is under heavy load, incremental sync is less likely to be throttled by Elasticsearch, thus making it more performant under load.

Let's look at the following graph:

In the graph you can see how much time each part of content extraction and ingestion takes on the timeline. In the example above the connector is spending the most time on ingesting the data, even pausing for extraction and content download. In this case incremental sync has a potential of improving the run time of the sync by 30-40%.

Let's look at another example - a system that has throttling and low throughput, but stores very little data in Elasticsearch (Sharepoint Online, GitHub, Jira, Confluence):

This system will not benefit from generic incremental syncs a lot - most of the time is spent extracting content from the 3rd-party system.

And the last example - fast and accessible system that stores huge amounts of data in Elasticsearch (Google Drive, Box, OneDrive, Network Drive):

If there aren't too many items that change in such a system between syncs, this system will benefit a lot from generic incremental syncs.

Currently connectors that potentially get the most of incremental sync are:

• Azure Blob Storage

• Box

• Dropbox

• Google Cloud Storage

• Google Drive

• Network Drive

• OneDrive

• S3

• Sharepoint Online

Other connectors will benefit less from incremental syncs, or will not benefit at all, but there's no one-size-fits-all answer here. Performance heavily depends on the profile of data ingested. The bigger each individual document is, the bigger the benefit.

Elasticsearch offers two types of index templates: legacy and composable. Composable templates introduced in Elasticsearch 7.8 that are set to replace legacy templates, both can still be used in Elasticsearch 8.

This article explores the differences between these templates and how they interact. In particular, we will focus on how you can detect which template will be used when you are creating an index. Let's get started by looking at how to create the different types of index templates.

Index templates in Elasticsearch

Legacy templates can be created using the following API:

Composable templates can be created using this API:

Component templates are a third type, which are typically used for managing multiple templates with similar structures. For example, if you need to create hundreds of templates with similar structures, you can create a component template with the common settings, mappings, and aliases, and then include it in your index templates. Component templates can be created using this API:

Important!

Determining which template an index will use when it is created

To determine which template an index will use upon creation, you can use the _simulate_index API. This API will return the template that will be used, along with any overlapping templates. However, if no composable templates are present, the API will return an empty body. In that case, you can create a dummy index and check the logs of the elected master node to determine which template will be used.

What happens if you have both legacy templates and composable templates?

As noted above, if you have both legacy and composable templates, the legacy template will be ignored as if it did not exist.

In such a case, you would get a warning message like the following when you run the command:

If a newly created composable template matches an existing legacy template with the same or includes an index pattern you will get a warning message like the following:

Use this command if you want to test it:

Notes from real life scenario

Conflicts can be annoying, and they can crash the application. Imagine that you have logstash-dev-*, logstash-prd-*, logstash-stg-* legacy templates that all working fine. If someone adds a single composable template that include index pattern like a logstash-* all legacy templates will be ignored, the fields types can be change and finally it can break the application. Because of that, it’s recommended to switch from legacy to composable templates if you are using Elasticsearch 7 and onwards.

Another good point to keep in mind is that if you run the Logstash in Elasticsearch 8 or higher, Logstash will add it's template as composable template by default. Because manage_template is set to true by default and Logstash template_api is set tocomposable for Elasticsearch 8 and onwards. It will create a Logstash composable template with logstash-* index pattern if the composable template does not exist. Yes, it will ignore all legacy templates covering logstash-* and overlap them.

Template Overlapping

1. What happens if you have two composable templates that point to the same index pattern?

As previously mentioned, if you have two composable templates that point to the same index pattern, the composable template with the highest priority will take precedence.

In this example, you have two templates—ct1 and ct2—both targeting the same index pattern test_index-. However, ct2 has a higher priority (1) than ct1 (0). Therefore, when you create an index that matches the pattern test_index-, the settings and mappings defined in ct2 will be applied before ct1. If there are the same settings in the ct1 and ct2 templates, the ct2 template will overwrite.

2. What happens if you have two legacy templates that point to the same index pattern?

As highlighted above, if you have multiple templates that point to the same index pattern, the templates with lower-order values are merged first. Templates with higher-order values are merged later, overriding templates with lower values.

If two legacy templates have the same order value, they will be sorted by name. For example, in a case with [t2, t1], t1 would be merged first, t2 would be merged later, and t2 would override t1 if there are any same mapping/settings/aliases.

Unfortunately, if you don't have composable templates, this API call responds with an empty body. So how you can check that?

The answer is to create a dummy index and check the Elasticsearch elected-master logs.

From the logs, we can see that "[test_index-1] creating index, cause [api], templates [t2, t1]".

As you can see, both legacy templates t1 and t2 have the same order; so, which one will override the other?

In this case, Elasticsearch will sort the legacy index templates according to their names and apply them. Both templates will be applied, and the first one in the list, which is t2 in this example, will override the template.

Bonus: What happens if you have two legacy templates that point to the same index pattern with same field name but inappropriate type?

Attempting to merge attributes within the legacy template, regardless of the order, is likely to fail since field definitions should remain atomic. This issue is a primary motivator for introducing the new composable templates. See the below example. We thank Philipp Krenn for adding these comments to the article.

Notes and good things to know

1. Using legacy templates in the same order can cause a lot of confusion. That’s why it's recommended to add order to the template.

2. Templates with lower-order values are merged first. Templates with higher order values are merged later, overriding templates with lower values.

3. You can't create two composable templates with the same priority.

Conclusion

In conclusion, understanding how Elasticsearch's index templates work is crucial for effective index management. By knowing how to determine which template an index will use upon creation, you can ensure that your indices are created with the correct settings, mappings, and aliases.

Resources

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-template.html https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates-v1.html https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-simulate-index.html