The problem

Imagine you have two indices, index-a (source) and index-b (target), and you want to find all documents that exist in index-a but are missing from index-b.

A naive approach, querying both indices and comparing results in memory, won't scale. Elasticsearch is designed to handle millions of documents, and loading them all at once isn’t practical.

There are two scenarios:

1. IDs are stable: Both indices use the same _id for the same document (for example, emp_no as the document ID). This is the easy case.

2. IDs are generated: Documents were ingested through different pipelines that assigned random or sequential IDs. You can't compare by _id; you need to match on content.

Let's walk through both.

Step 0 — A lighter CLI for Elasticsearch

All the examples in this post use escli, a small Rust command line interface (CLI) that wraps the Elasticsearch REST API. It reads your cluster URL and credentials from environment variables, so you don’t have to repeat authentication headers on every command.

To see why that matters, here's a typical _search call with raw curl:

With escli, the same request becomes:

The credentials live in a .env file that escli sources automatically — no -H "Authorization: ..." on every call, no risk of leaking secrets in shell history. The request body is passed via stdin (<<<), which makes it easy to pipe in multi-line JSON built dynamically with jq.

Step 1 — Count documents in both indices

Before doing a full scan, get a quick count of each index. If the counts match, the indices are likely in sync, and there’s no need to scan at all.

The _count API returns:

If the counts differ, proceed to the full comparison.

Step 2 — When IDs mean something: Use op_type=create

If both indices use the same _id for the same document, for example, because you indexed documents using a functional business key like emp_no rather than a generated UUID, you can find and fix missing documents in a single _reindex call.

Why functional IDs matter

Using a meaningful field as _id (instead of a random UUID) is a best practice when the data has a natural key. It means:

• The same document always gets the same _id, regardless of which pipeline ingested it.

• You can easily update or delete documents by ID.

• You can use op_type=create to skip documents that already exist in the target.

• No client-side scanning or comparison is needed.

The op_type=create trick

_reindex with op_type=create tries to create each document from the source in the target. If a document with the same _id already exists, Elasticsearch reports it as a version_conflict and moves on. It doesn’t overwrite the existing document. Setting conflicts=proceed tells the API to continue instead of aborting on the first conflict.

The response tells you exactly what happened:

• created: Documents that were missing from index-b and have now been added.

• version_conflicts: Documents that already existed in index-b and were left untouched.

No scanning, no client-side comparison, no intermediate file. Everything happens server-side in about six seconds on a 1M-document dataset.

Step 3 — When IDs are not stable: Business-key comparison

Sometimes you can't rely on _id. A document pipeline that generates IDs at ingestion time will assign a different _id each time the same record is processed. If index-a and index-b were populated by two such pipelines, the same employee record might have _id: "abc123" in one index and _id: "xyz789" in the other, even though the underlying data is identical.

In this case, you need to match documents by content rather than by ID. The key is to identify a set of fields that together form a unique business key.

For an employee dataset, a reasonable business key is (first_name, last_name, birth_date). A document in index-a is "missing" from index-b if no document in index-b has the same combination of those three fields.

3a — Scan the source with PIT + search_after

Open a point in time (PIT) on the source index to get a consistent snapshot, and then paginate through it, fetching only the business-key fields:

The sort key _shard_doc is the most efficient sort for full-index pagination: it uses the internal Lucene document order with no overhead. Repeat with search_after until the response contains zero hits. Always close the PIT when done:

3b — Check each page against the target via _msearch

For each page of source documents, build one _msearch request with one subquery per document. Each subquery uses a bool/must on the three business-key fields and requests size: 0; we only need to know whether a match exists, we don’t need to retrieve the document itself.

The response contains one entry per subquery, in the same order:

total.value == 0 means no document in index-b matches that business key; the document is missing. Collect the corresponding _id from the source page.

3c — Speed it up with split-by-date

If the business key includes a date field, you can partition the source into date slices and run each slice as an independent job. Each slice opens its own PIT with a range filter on birth_date, runs its own msearch loop, and writes its results to a separate file. The parent script launches all slices in parallel and aggregates the results when they’re all done.

But depending on your use case, you might want to partition by a different field; for example, if you have a team field, you could run one slice per team. The key is to find a field that allows you to split the data into reasonably even chunks that can be processed in parallel.

Performance on a 1M dataset

To validate the approaches, the demo generates 1,000,000 documents in index-a and deliberately skips ~5% in index-b (49,594 missing documents), and then runs the full compare → reindex cycle.

Results on a MacBook M3 Pro:

Comparison (compare-indices.sh):

Strategy

Compare

Reindex

Total

How it works

op_type

6s

6s

Full _reindex server-side, skips existing

business-key

1m 38s

4s

1m 42s

PIT scan + _msearch by business key

split-by-date

32s

4s

36s

Same as business-key, 5 slices in parallel

The op_type=create approach is fastest because everything is server-side and requires no client-side scanning. The split-by-date strategy cuts the business-key duration from 1m 38s down to 36s through parallelism: not bad for a comparison across two 1M-document indices.

Decision tree

Conclusion

Elasticsearch doesn't offer a native index diff command, but the right strategy depends on your data model:

• Use functional _ids (a natural business key like emp_no) whenever possible. It unlocks the simplest and fastest approach: _reindex with op_type=create finds and fills gaps in one server-side call.

• When IDs are unstable, match by business key using PIT + _msearch. Partition by a field and run slices in parallel to recover most of the performance. If you find yourself doing this regularly, consider computing a hash of your business key fields and using it as _id at ingestion time. You get the best of both worlds: stable IDs and efficient lookups.

The complete demo, including dataset generation, comparison scripts, and reindex scripts, is available at https://github.com/dadoonet/blog-compare-indices/.

That philosophy hasn’t changed.

What has changed, however, is how little effort it now takes to achieve that goal. With Elasticsearch 9.x, the modern Java client, and Testcontainers 2.x, the experience of writing integration tests feels noticeably smoother, as if a layer of incidental complexity has quietly been removed.

The example accompanying this article is intentionally modest and can be found here.

It doesn’t attempt to demonstrate sophisticated indexing strategies or elaborate data pipelines; instead, it concentrates on the essentials, because the essentials are precisely where the improvements are most visible.

When the tooling stops getting in the way

Anyone who has maintained a test suite for a few years will recognize the pattern: You introduce a new library, a transitive dependency pulls something unexpected, and before long, you’re negotiating between versions of testing engines rather than writing tests.

With Testcontainers 2.x, that negotiation largely disappears. The dependency structure is clearer, the modules are more explicit, and the accidental coupling to older testing frameworks no longer sneaks in behind your back. In practical terms, adding Elasticsearch support to your tests is now as straightforward as declaring:

And, if you’re using JUnit Jupiter integration:

There are no exclusions to sprinkle in, no legacy engines to silence, and no uneasy feeling that something hidden might surface during the next upgrade. The configuration becomes almost unremarkable, which, in the context of build tooling, is a compliment.

A real Elasticsearch node, with security intact

In the demo test, we use the official Elasticsearch 9.3.1 Docker image:

At first glance, this may look similar to older examples, yet the subtle difference lies in what we no longer need to do. We don’t disable security. We don’t bypass SSL. We don’t simplify the environment just to make the test convenient.

Instead, once the container is started, we construct a client that uses the REST API and authenticates properly:

What deserves special mention here is how neat the client construction itself has become. In earlier iterations, creating an Elasticsearch client often meant juggling multiple intermediate objects, configuring transport layers explicitly, wrapping low-level clients, and dedicating some amount of code to what was essentially plumbing. Now, the signal-to-noise ratio is refreshingly high. The builder encapsulates the necessary details, the container provides what the client needs, and the resulting configuration fits comfortably within a few readable lines.

Just as importantly, the ElasticsearchClient is AutoCloseable, which means it integrates naturally with try-with-resources, ensuring proper cleanup without additional ceremony. The lifecycle is explicit, concise, and self-contained, which is exactly what you want in integration tests that should focus on behavior rather than infrastructure management.

The container exposes everything required to build a legitimate, secure connection, and the client integrates with it naturally, which means the test environment mirrors production in all the aspects that matter, without imposing additional mental overhead from the developer.

This alignment between realism and simplicity is, perhaps, one of the most meaningful improvements.

Typed APIs change the character of tests

The evolution of the Elasticsearch Java client has also reshaped how integration tests read and feel. Where older approaches often involved parsing JSON responses or navigating loosely typed structures, the modern client offers a builder-based, strongly typed API that guides you through valid request shapes at compile time.

In the demo, we perform a simple cluster health check:

What’s striking here is not the complexity of the operation, but the absence of friction. There’s no manual extraction from maps, no assertions built on untyped string values, and no detour into low-level response handling. The test code looks indistinguishable from application code, which subtly reinforces the idea that integration tests aren’t a special category of code with different rules, but simply another consumer of the same APIs.

When the boundary between production code and test code becomes thinner, confidence increases almost by default.

Reading the test as a story

If you take a look at the full test case:

you’ll notice that it reads less like a configuration script and more like a short narrative:

• We define the container.

• We start the container.

• We build a client.

• We call a real API.

• We assert the outcome.

The supporting infrastructure fades into the background, leaving the intent of the test clearly visible. That clarity isn’t accidental; it’s the cumulative effect of incremental improvements across Testcontainers and the Elasticsearch client.

The advanced patterns still apply

None of the more advanced techniques discussed in earlier articles, Faster integration tests with real Elasticsearch and Advanced integration tests with real Elasticsearch, have become obsolete. Reusing containers to speed up large test suites, customizing cluster settings, preloading indices, or testing role-based access scenarios remain entirely valid and, in many cases, essential.

What has improved is the baseline experience. The simplest possible integration test, the one that merely needs a real node and a real client, no longer requires defensive configuration or dependency gymnastics. It’s concise, expressive, and production-like by default.

Progress without drama

There was no dramatic rewrite of the ecosystem, no disruptive migration guide that forced a rethinking of everything. Instead, there has been a steady refinement of APIs and dependencies, each release smoothing a rough edge here and removing a surprise there.

The result isn’t flashy, yet it’s tangible. Writing integration tests against Elasticsearch now feels less like assembling a test harness and more like exercising a real system in miniature.

Sometimes progress announces itself loudly. Sometimes it arrives quietly, in the form of code that simply reads better and requires less explanation. In this case, it’s the latter, and for those of us who care about clean, reliable integration tests, that’s more than enough.

And what if we could do something similar with Kibana? Sounds appealing? Stay tuned!

When we built Elasticsearch Serverless, we had to decide how to bill our users. While a charge per query may have been easier to reason about from a consumption perspective, it would be a lot harder to reason about from a resource perspective. Instead, we implemented a simple pricing scheme comprising three dimensions for compute: search, ingest, and machine learning VCUs. This means we charge users for the actual resources we allocate to fulfill your requested workloads.

VCU, ECU, and other terms

Let's start by defining a few terms that will keep coming back throughout this post.

VCU

A VCU is a Virtual Compute Unit, representing a fraction of RAM, CPU, and local disk for caching. We separate compute by the workloads they support, so we have three flavors of VCU:

1. Search VCU

2. Ingest VCU

3. Machine Learning (ML) VCU

VCU’s are charged by the hour.

Regional pricing

We have different prices for different regions and different cloud providers. You can find a full list of prices on this page.

ECU

An ECU is an Elastic Consumption Unit, which is the unit we bill you in. The nominal value of an ECU is $1.00 USD. All of the different components of consumption are charged at a specific rate of ECUs per time unit. For example, one Gigabyte of storage might cost 0.047 ECU per month, so 100 GB of storage will cost you 4.7 ECU = $4.70 for one month. Similarly, if your search workload consumed 10 VCUs in a day and the Search VCU rate in your region is 0.09 ECU, your cost for that day would be $0.90.

Interactive Dataset Size

The amount of data in your project has a direct influence on your costs. We make the distinction of “interactive dataset” primarily for time-series data, as this relates to the amount of data in the Boost Window. For non-time-series data, this is simply the amount of data in the project.

Project settings

We have three project settings that allow you to control your project's usage.

Search power

Search Power controls the speed of searches against your data. With Search Power, you can improve search performance by adding more resources for querying, or you can reduce provisioned resources to cut costs. Choose from three Search Power settings:

On-demand: Autoscales based on data and search load, with a lower minimum baseline for resource use. This flexibility results in more variable query latency and reduced maximum throughput.

Performant: Delivers consistently low latency and autoscales to accommodate moderately high query throughput.

High-availability: Optimized for high-throughput scenarios, autoscaling to maintain query latency even at very high query volumes.

Boost window

For time series use cases, the boost window is the number of days of data that constitutes your interactive dataset size. The interactive dataset is the portion of your data that we keep cached, and that we use to determine how to scale the Search tier for your project. By default, the boost window is seven days.

Data retention

You can set the number of days of data that are retained in your project, which will affect the amount of storage we need. You can do this on a per-data stream basis in your project.

Price components

Serverless Elasticsearch contains a few different pricing components. For most use cases, the components you will care most about are Search, Ingest, and ML VCUs, as well as the Elastic Inference Service's token consumption.

Search VCUs

Search VCU consumption is the most complex part of pricing. We make this simple for you by automatically determining the right amount of VCUs that are needed to fulfill your workloads. For more details on how our autoscaling logic works, see our earlier blog on the topic.

Search VCU inputs

Search VCUs are allocated based on a few factors, but mainly, we can boil it down to three inputs: the interactive dataset size, the search load on the system, and Search Power.

For traditional search use cases, the interactive dataset size will generally be your entire dataset. For time series use cases, it will be the portion of your dataset that fits inside the Boost Window.

Search load measures the amount of load being placed on the system by currently active searches. The main contributing factors are the number of searches per second, the complexity of the searches (the more that needs to be computed, the higher the load), and the size of the dataset that needs to be searched to fulfill the result. If we can get you the right number of results by scanning 10% of the dataset, then the load will be much lower than if we need to scan the full dataset.

Finally, Search Power influences the number of VCUs we allocate. Each Search Power setting defines the baseline capacity of the search tier.

In short: the larger the dataset size and the higher the search load, the more VCUs we need to fulfill your search requests. Search Power allows you to tune to what extent we will scale up and down.

Minimum VCUs

Elasticsearch Serverless is designed to align infrastructure costs directly with your application's demand.

For smaller workloads, the search infrastructure can scale down to zero VCUs during periods of inactivity. If the system detects fifteen minutes of total inactivity, the associated hardware resources are deprovisioned. This makes the platform highly cost-effective for development environments, bursty workloads, or applications with intermittent usage. Note that inactivity means actual inactivity: no user-initiated searches whatsoever. As soon as we need to serve a search of any kind, we need to allocate hardware resources to execute that search.

As your interactive dataset grows, the system eventually reaches a storage threshold where a baseline level of resources is required to maintain data availability and indexing readiness. A minimum VCU allocation is maintained to ensure your data remains "warm" and queryable, even if no active searches are occurring.

VCU consumption is not linear

Because our hardware is allocated in steps, consumption of VCUs does not necessarily scale linearly with workload size. Each scaling step can contain a wide range of workloads, and if your workload is at the bottom of that range, it may have a lot of room to grow before we need to jump to the next scaling step.

This can make estimating based on a non-representative workload hard. For example, you may be consuming 2 VCUs per hour on a small workload. It's entirely possible that you could increase your workload size by a factor of 100 and still fit in that 2 VCU per hour load before we need to start increasing the amount of VCUs we allocate to serve your workload.

We know this makes estimating your cost a little harder, and we are working on ways to make that easier for you. If you need more help estimating your likely price, you can always talk to our customer team and get more personalized assistance.

Ingest VCUs

Ingest VCUs are much simpler than Search VCUs.

Ingest VCU Inputs

Ingest VCUs have essentially three inputs: the number of indices, the ingest rate, and the ingest complexity. We need to allocate a little bit of memory for every index in your system, which is why the number of indices matters. Read indices in data streams do not count for this calculation.

The faster you ingest, the more CPU we will need to process that ingestion. And the more complex your ingest requests, the more CPU we will need. Some factors that make ingest requests more expensive to execute are complicated field mappings or a lot of post-processing.

Minimum Ingest VCUs

We do not have a minimum number of VCUs we allocate to your ingest. If you do not ingest data, we do not need to allocate any VCUs to processing ingestion. There is an exception for a large number of indices (think: thousands of indices), where we do need to keep some resources allocated to be responsive when indexing requests come in.

VCU consumption is not linear

As with Search VCUs, we allocate Ingest VCUs based on step functions. Each step can contain a wide range of workloads: it's entirely possible that if you have a minimal amount of ingest, you could increase your ingest rate by a factor of 100 and still fit in the same step, thus not actually increasing your cost.

AI workloads

When running machine learning tasks in Serverless, we give you three options:

1. You use our Elastic Inference Service (EIS) to run your inference and completion workloads. We take care of everything, and you are charged per token.

2. You use traditional Elasticsearch Machine Learning capabilities to run your workloads. These use our Trained Models capabilities. We will scale up and down based on your machine learning workload requirements.

3. You do it yourself, outside of our systems, and just bring your vectors or other inference results to store and search in Elasticsearch.

EIS

The pricing for EIS is quite straightforward: you get charged a rate per one million consumed tokens. Token consumption is generally easy to predict for inference workloads. For LLM-based tasks, particularly agentic ones, this can be more complex, and some experimentation and trial runs may be useful to determine how many tokens your workloads typically consume.

ML VCUs

Machine Learning VCUs work on one simple input: machine learning workloads. The more inference you require, the more VCUs we will consume. Once you stop performing inference, we will scale down. We will keep a trained model in memory for about 24 hours after you last used it so that we can be responsive, which means that the minimal amount of VCU required to keep that model available will remain up for 24 hours before scaling down entirely.

We generally recommend our customers use EIS instead of our Machine Learning nodes for inference, particularly if your usage is periodic. By switching to EIS, you will not have to wait for machine learning nodes to spin up, and we won't charge you for unused ML node time before scaling down. EIS charges on a per token basis.

Storage

We charge storage per gigabyte per month. Storage does serve as an input into other parts of our system, particularly Search VCUs (see Search VCU above), but the pricing for storage itself is quite straightforward.

Data Out (egress)

We charge you for the data you take out of the system.

To minimize your egress costs, we recommend a few optimizations on your queries:

1. Do not return vectors in your query responses. We do this by default for indices created after October 2025. You can always return vectors in your responses explicitly if necessary.

2. Return only the fields needed for your application. You can do this by using the fields and _source parameters.

Support

We charge support as a percentage of your total ECU usage. We currently have four levels of support:

1. Limited support

2. Base support

3. Enhanced support

4. Premium support

Project subtype profiles

We currently offer two project subtypes for Serverless Elasticsearch, referred to as “General Purpose” and “Vector Optimized”. All Serverless Elasticsearch projects created through the cloud console UI will be created using the “General Purpose” option. You may create a “Vector Optimized” by calling the API directly with the optimized_for parameter (see documentation for all options).

The difference between the two options is the allocation of resources. We allocate approximately four times more resources (aka VCUs) to the “Vector Optimized” profile, which will result in your costs being up to four times higher. This is why we recommend starting on the “General Purpose” profile and only using the “Vector Optimized” profile when your use case demands the use of uncompressed dense vectors with high dimensionality, and quantization and DiskBBQ will not serve your needs.

When Serverless Elasticsearch was envisioned years ago, we thought that vector workloads would require much more resources to remain performant. However, with innovations like semantic_text, sparse_vector models, and Better Binary Quantization (BBQ), we’ve found that many vector workloads perform well on the “General Purpose” profile at a fraction of the cost. Therefore, don’t let the “Vector Optimized” label fool you…you can get excellent price and performance for vector workloads on the “General Purpose” profile.

Monitoring costs

We recognize that keeping track of your costs, especially when you are new to Elasticsearch Serverless, is important to you. We built a few tools just for this purpose, and continue to improve them for even greater visibility.

Cloud console billing usage

The Elastic Cloud Console provides billing details for your cloud account, across all cloud-based resources, including Elasticsearch Serverless. There, you can find a breakdown of all the price components described in this article. Filters allow you to zoom in on specific time periods and resources.

To further monitor your costs, you can also configure custom budget alerts from the Budgets and notifications tab under the Billing and subscriptions page.

AutoOps monitoring

We’re bringing AutoOps to Serverless! One of the key value propositions of Elasticsearch Serverless is that we ensure everything runs smoothly, but that also means you have limited observability into the infrastructure. AutoOps for Serverless gives users visibility into what is driving usage, and, therefore, costs.

AutoOps is rolled out in new Serverless regions regularly, and we're always working to add new monitoring tools. Make sure to check out the region coverage and future planned monitoring tools.

1. Ingesting JSON data into Elasticsearch

2. Using an Ingest Pipeline to parse JSON fields

3. Querying and aggregating JSON fields

1. Ingesting JSON data into Elasticsearch

When ingesting JSON data into Elasticsearch, it is essential to ensure that the data is properly formatted and structured. Elasticsearch can automatically detect and map JSON fields, but it is recommended to define an explicit mapping for better control over the indexing process.

To create an index with a custom mapping, you can use the following API call:

In this example, we create an index called my_index with a custom mapping for a JSON field named json_field.

2. Using an Ingest Pipeline to parse JSON fields

If your JSON data is stored as a string within a field, you can use the Ingest Pipeline feature in Elasticsearch to parse the JSON string and extract the relevant fields. The Ingest Pipeline provides a set of built-in processors, including the json processor, which can be used to parse JSON data.

To create an ingest pipeline with the json processor, use the following API call:

In this example, we create an ingest pipeline called json_parser that parses the JSON string stored in the message field and stores the resulting JSON object in a new field called json_field.

To index a document using this pipeline, use the following API call:

The document will be indexed with the parsed JSON fields:

3. Querying and aggregating JSON fields

Once the JSON fields are indexed, you can query and aggregate them using the Elasticsearch Query DSL. For example, to search for documents with a specific value in the field1 subfield, you can use the following query:

To aggregate the values of the field2 subfield, you can use the following aggregation:

Bonus: How to deal with unparsed JSON data?

If you are in the situation where you have already ingested unparsed JSON data into a text/keyword field, there’s a way to extract the JSON data without having to reindex everything from scratch.

You can leverage the Update by Query API with the ingest pipeline developed in section 2. But before running that update, you’ll first need to update your index mapping similarly to what we did in section 1 to add the json_field mapping, by running the command below:

When done, you can simply run the command below, which will iterate over all documents in your index, extract the JSON from the message field and index the parsed JSON data into the json_field object.

Conclusion

In conclusion, parsing JSON fields in Elasticsearch can be achieved using custom mappings, the Ingest Pipeline feature, and the Elasticsearch Query DSL. By following these steps, you can efficiently index, query, and aggregate JSON data in your Elasticsearch cluster.

You can learn more about UBI in this article.

Data collected with the UBI collector can be used on Kibana to build dashboards that open the window to users’ behavior in our application. In this blog, we will explore how to analyze UBI data in Kibana to gain insights into how our app is being used.

Demo set up

We can easily reproduce the demo in this blog following these steps:

1. Clone the repository

2. Install required libraries:

3. Run the setup script. Make sure to have the following environment variables set beforehand

1. ES_HOST

2. API_KEY

3. KIBANA_HOST

That’s all you need to do. If everything went well, you should see this output from the script execution:

As we can see the script:

• Created two indices with the appropriate mappings

• Indexed 23 documents to these indices

• Uploaded some saved objects to Kibana

Now, let’s take a look at what exactly this script did behind the scenes.

Understanding the uploaded data

First, we put some data in Elasticsearch before creating our visualizations.

You can reproduce the process manually in Kibana DevTools, copying the mappings and sample data and using the PUT <index> and PUT _bulk APIs, respectively.

Ubi_events index

User action data, documents are generated for every click (in this case), and it includes:

• application: The client application that generated the event ("search-ui")

• action_name: Type of user action performed ("click")

• query_id: Links this event to the corresponding search query session

• client_id: A generated, unique ID that represents a user or session without revealing personal data. It is generated instead of using identifiable data like email addresses or usernames. This approach allows us to have privacy advantages such as safe analytics capabilities and secure data sharing without exposing PII, while still having important functionality like session continuity, behavioral analysis, or A/B testing.

• timestamp: ISO 8601 formatted timestamp when the event occurred

• message_type: Category of the event for processing ("CLICK_THROUGH")

• message: Human-readable description of what happened ("Clicked Fahrenheit 451")

• user_query: The original search term that led to this event ("fahrenheit")

• event_attributes: Nested object containing detailed event context:

  • object.device: Device type used by the user ("mobile")

  • object.object_id: Unique identifier of the clicked item

  • object.description: Details about the clicked item (book title, date, author)

  • object.position.ordinal: Ranking position of the item in search results (1st)

  • object.position.page_depth: Which page of results the item appeared on (1st page)

  • object.user.ip: User's IP address

  • object.user.city/region/country: Geographic location data

  • object.user.location: Precise latitude/longitude coordinates

Sample document:

You can download the index mappings here

Ubi_queries index

Search data includes data relevant to each search executed by the users:

• query_response_id: Unique identifier for this specific query response instance

• user_query: The original search term entered by the user ("fahrenheit")

• query_id: Unique identifier for the search query session

• query_response_object_ids: Array of object IDs that were returned as search results (["3", "9"])

• query: The complete Elasticsearch query object in JSON format, including search parameters, fields to search, result size, sorting, and metadata

• client_id: A generated unique ID that represents a user or session without revealing personal data. It is generated instead of using identifiable data like email addresses or usernames. This approach allows us to have privacy advantages such as safe analytics capabilities and secure data sharing without exposing PII, while still having important functionality like session continuity, behavioral analysis, or A/B testing.

• timestamp: Unix timestamp in milliseconds when the query was executed (1753885225098)

Sample document:

You can download the index mappings here.

Sample data

We can use the _bulk API to index some sample data in both indices

This will create 6 documents in the ubi_queries index and 16 in the ubi_events index.

Dashboard object

Before going into details of the visualizations used in this example here, you can download the Saved Object of the full example dashboard and import it into your Kibana instance. This dashboard explores the most searched terms, when searches and events took place, and where they come from (in a map).

Visualize Insights

We are going to create a Kibana dashboard to analyze the most common metrics leveraging Kibana Lens. For a reference on available visualizations, visit this page.

Ubi_events

We will start with some simple Metric visualizations created with Lens: Total events: Counts how many events were triggered in the timeframe. Uses a simple count of the documents in the index, denoted by # Records in the field list.

Event actions: Counts actions by action_name. This is a simple count of documents split by action_name.keyword. In our sample data, we have two types of actions:

• click: Generated when a user clicks in the book link

• search_input: Generated when a user enters text in the search box (debounce 300ms)

Now on table visualizations:

Top clicks: A table with a count of the number of events split by the query they come from. It uses a Top values function on the user_query.keyword. This can give us visibility on which queries generate more interactions on our webpage.

Finally, some other visualizations:

Device types: This visualization breaks down the percentage of events by the device they come from. The device can be one of three categories: Desktop, mobile, or tablet. This visualization is a pie that uses the top values of event_attributes.object.device.keyword, and can give us insights into which type of devices our users have. This can generate alerts if we detect an unexpected, sudden fall of events on a specific type of device, as this might indicate that a recent change in our app resulted in errors when accessing it from a device.

Events map: A map visualization that shows where the events are coming from, which allows us to see the geographical distribution of our users. Right now, this shows where individual documents come from, but this can also be used to see the density of users with a heatmap, for example.

This particular visualization can provide very interesting insights when used with different filters. For example, we can see where different search terms are coming from or where most of our clicks are originating. This can be useful information for making decisions on localization efforts or establishing differences across local markets. The map uses the location at event_attributes.object.user.location.

UBI Events: A saved search with the latest UBI events documents

Ubi_queries

Here we have visualizations from this index:

Total queries: A simple document count of the index to show how many queries have been received in total. This shows the big picture and answers the question of how many total queries we had in the selected time window.

Unique clients: A unique_count of the field client_id to show how many different clients have used our website.

Top queries (tag cloud): A Tag cloud of the top 5 most searched terms. This visualization uses the field user_query.keyword and allows us to easily see the main terms that our users are looking for.

Queries over time: A line chart of queries per hour, which uses a simple count metric in a horizontal axis of the field timestamp

Query terms over time: Similar to the last one, but broken down by the user_query.keyword. This chart can show how many different terms are searched over time.

Top queries: A Top values table showing how many times a term was searched. It uses the user_query.keyword field.

Client queries: A Top values table of the client_id field that counts the total queries and unique queries per client.

No result queries: A Top value table that shows the top query_terms that didn’t match any document, and a Unique Count of the field client_id. This can be very useful to determine what products our website is lacking. For example, in an e-commerce book store, seeing regular searches for a particular book title could lead us to buy copies to sell. Alternatively, it can also indicate shortcomings in our search implementation, for example, if people are using question-based searches that align better with semantic search approaches.

Here you can see the full dashboard:

Analysis of sample data

In our dashboard, we can get some insights:

• Traffic is coming from 3 different cities in the US

• Most of our users access our website from a desktop device, but we have a sizable number of users using a mobile device and even some using a tablet.

• We can see the top query is “asimov,” but at the same time, we do not have any results. This might be a good indicator of what products should be prioritized for stock acquisition.

To further this analysis, we could use Kibana’s Machine Learning capabilities to understand and predict behaviours on our website. Going even one step further, we can create alerts based on these behaviors using the different available connectors.

From a search relevance perspective, user behavior is a useful input for relevance engineering tools like LTR.

Conclusion

Data collected by the UBI collector can be easily used to have a better understanding of our users. The resulting dashboard becomes a live pulse of what our users are searching for and can point to data gaps to drive improvements in our search engine.

Note: The o19s User Behavior Insights (UBI) plugin mentioned in this article is a third-party, community-maintained plugin and is not officially supported by Elastic. For questions or issues related to this plugin, please refer to the o19s UBI project repository at https://github.com/o19s/ubi.

Elastic is built to adapt to all kinds of data your systems emit: logs, metrics, traces, custom telemetry, and more. But when that data doesn’t match the expected shape due to a schema change, a misconfigured agent, or a rogue service emitting unexpected fields, it can fail to process and silently disappear.

That absence is a signal. But it’s also hard to detect, hard to debug, and hard to report on. And worse, it puts the burden on the client to figure out what happened.

That’s why we built failure store: a new way to capture, debug, and analyze failed events directly in the Elastic Stack. In this blog, we’ll go over Elastic’s failure store and explain how it provides visibility into data ingestion issues, helps debug schema changes, and enables teams to monitor data quality and pinpoint failure patterns.

About failure store

Failure store gives you visibility into failed events that were previously only visible to the client sending the data and to dead letter queues. It works by capturing and indexing failed documents into dedicated `::failures` indices that live in your data stream alongside your production data. You can enable it per data stream or across multiple data streams with a single cluster setting.

Why it matters

Teams are often downstream from the source of truth. They don’t write the code; they just keep it all running. When upstream teams ship changes that break mappings or introduce unexpected fields, failures happen. But without access to the original failed data itself, debugging becomes guesswork.

Even worse, when data fails to be indexed, it doesn’t exist in your indexes—which means it’s much harder to measure the impact. You can’t track which streams are failing most often. You can’t quantify how broken your pipelines are. And you certainly can’t alert on what’s missing if the platform never saw it (except for alerts when data goes missing).

The failure store allows a developer to understand which data failed indexing and why, giving observability engineers the tools needed to quickly understand and fix ingestion failures. Triage also happens quickly since failures are stored in Elasticsearch, with no need to gather information from remote clients or shippers.

Get started

Set up for new data streams…

…or enable for existing data streams

Enable failure store for individual data streams in stack management in Kibana or leverage the _data_stream API:

Enable failure store via cluster setting

If you have a large number of existing data streams, you may want to enable their failure stores in one place. Instead of updating each of their options individually, set data_streams.failure_store.enabled to a list of index patterns in the cluster settings. Any data streams that match one of these patterns will operate with their failure store enabled.

A failing document response

After enabling the failure store, requests that previously would fail are now processed differently. The client now receives a 201 created instead of a 400 Bad Request. Especially if you’re using custom applications or our language clients, make sure to update your code accordingly. When a document goes to the failure store, the response will contain the failure_store: used attribute.

Search and filter failure data just like any other logs, with support for ES|QL and Kibana tools:

Data in the failure store comes with all the context to make debugging simple. Each ::failures index contains information about which pipeline failed, along with specific error messages, stack traces, and error types that help you identify patterns.

Are you getting lots of errors and aren't sure where to start? Use ES|QL and ML functions. With the data exposed in ES|QL, errors can be analyzed with ML capabilities such as CATEGORIZE to help parse errors and extract patterns. Read more about data remediation techniques in our documentation.

Control costs and retention using the same data stream lifecycle you're already using for your other data. Absent a custom retention, failure store data will stick around for 30 days.

Monitor data quality over time with failure metrics and sortable dashboards by failure percentage to find new problem areas that require investigation. Read more about data quality monitoring in the documentation.

Learn more

Failure store is available starting in Elastic 9.1 and 8.19 and will be enabled by default on logs-*-* indexes in an upcoming release. To learn more, check the documentation for setup instructions and best practices.

Do you want to start using semantic search for your data, but focus on your model and results instead of on the technical details? We’ve introduced the semantic_text field type that will take care of the details and infrastructure that you need.

Semantic search is a sophisticated technique designed to enhance the relevance of search results by utilizing machine learning models. Unlike traditional keyword-based search, semantic search focuses on understanding the meaning of words and the context in which they are used. This is achieved through the application of machine learning models that provide a deeper semantic understanding of the text.

These models generate vector embeddings, which are numeric representations capturing the text meaning. These embeddings are stored alongside your document data, enabling vector search techniques that take into account the word meaning and context instead of pure lexical matches.

How to perform semantic search

To perform semantic search, you need to go through the following steps:

• Choose an inference model to create embeddings, both for indexing documents and performing queries.

• Create your index mapping to store the inference results, so they can be efficiently searched afterwards.

• Setting up indexing so inference results are calculated for new documents added to your index.

• Automatically handle long text documents, so search can be accurate and cover the entire document.

• Querying your data to retrieve results.

Configuring semantic search from the ground up can be complex. It requires setting up mappings, ingestion pipelines, and queries tailored to your chosen inference model. Each step offers opportunities for fine-tuning and optimization, but also demands careful configuration to ensure all components work together seamlessly.

While this offers a great degree of control, it makes using semantic search a detailed and deliberate process, requiring you to configure separate pieces that are all related to each other and to the inference model.

semantic_text simplifies this process by focusing on what matters: the inference model. Once you have selected the inference model, semantic_text will make it easy to start using semantic search by providing sensible defaults, so you can focus on your search and not on how to index, generate, or query your embeddings.

Let's take a look at each of these steps, and how semantic_text simplifies this setup.

Choosing an inference model

The inference model will generate embeddings for your documents and queries. Different models have different tradeoffs in terms of:

• Accuracy and relevance of the results

• Scalability and performance

• Language and multilingual support

• Cost

Elasticsearch supports both internal and external inference services:

• Internal services are deployed in the Elasticsearch cluster. You can use already included models like ELSER and E5, or import external models into the cluster using eland.

• External services are deployed by model providers. Elasticsearch supports the following:

  • Cohere

  • Hugging Face

  • Mistral

  • OpenAI

  • Azure AI Studio

  • Azure OpenAI

  • Google AI Studio

Once you have chosen the inference mode, create an inference endpoint for it. The inference endpoint identifier will be the only configuration detail that you will need to set up semantic_text.

Creating your index mapping

Elasticsearch will need to index the embeddings generated by the model so they can be efficiently queried later.

Before semantic_text, you needed to understand about the two main field types used for storing embeddings information:

• sparse_vector: It indexes sparse vector embeddings, like the ones generated by ELSER. Each embedding consists of pairs of tokens and weights. There is a small number of tokens generated per embedding.

• dense_vector: It indexes vectors of numbers, which contains the embedding information. A model produces vectors of a fixed size, called the vector dimension.

The field type to use is conditioned by the model you have chosen. If using dense vectors, you will need to configure the field to include the dimension count, the similarity function used to calculate vectors proximity, and storage customizations like quantization or the specific data type used for each element.

Now, if you're using semantic_text, you define a semantic_text field mapping by just specifying the inference endpoint identifier for your model:

That's it. No need for you to define other mapping options, or to understand which field type you need to use.

Setting up indexing

Once your index is ready to store the embeddings, it's time to generate them.

Before semantic_text, to generate embeddings automatically on document ingestion you needed to set up an ingestion pipeline.

Ingestion pipelines are used to automatically enrich or transform documents when ingested into an index, or when explicitly specified as part of the ingestion process.

You need to use the inference processor to generate embeddings for your fields. The processor needs to be configured using:

• The text fields from which to generate the embeddings

• The output fields where the generated embeddings will be added

• Specific inference configuration for text embeddings or sparse embeddings, depending on the model type

With semantic_text, you simply add documents to your index. semantic_text fields will automatically calculate the embeddings using the specified inference endpoint.

This means there's no need to create an inference pipeline to generate the embeddings. Using bulk, index, or update APIs will do that for you automatically:

Inference requests in semantic_text fields are also batched. If you have 10 documents in a bulk API request, and each document contains 2 semantic_text fields, then that request will perform a single inference request with 20 texts to your inference service in one go, instead of making 10 separate inference requests of 2 texts each.

Automatically handling long text passages

Part of the challenge of selecting a model is the number of tokens that the model can generate embeddings for. Models have a limited number of tokens they can process. This is referred to as the model’s context window.

If the text you need to work with is longer than the model’s context window, you may truncate the text and use just part of it to generate embeddings. This is not ideal as you'll lose information; the resulting embeddings will not capture the full context of the input text.

Even if you have a long context window, having a long text means a lot of content will be reduced to a single embedding, making it an inaccurate representation.

Also, returning a long text will be difficult for the users to understand, as they will have to scan the text to check it's what they are looking for. Using smaller snippets would be preferable instead.

Another option is to use chunking to divide long texts into smaller fragments. These smaller chunks are added to each document to provide a better representation of the complete text. You can then use a nested query to search over all the individual fragments and retrieve the documents that contain the best-scoring chunks.

Before semantic_text, chunking was not done out of the box - the inference processor did not support chunking. If you needed to use chunking, you needed to do it before ingesting your documents or use the script processor to perform the chunking in Elasticsearch.

Using semantic_text means that chunking will be done on your behalf when indexing. Long documents will be split into 250-word sections with a 100-word overlap so that each section shares 100 words with the previous section. This overlap ensures continuity and prevents vital contextual information in the input text from being lost by a hard break.

If the model and inference service support batching the chunked inputs are automatically batched together into as few requests as possible, each optimally sized for the Inference Service. The resulting chunks will be stored in a nested object structure so you can check the text contained in each chunk.

Querying your data

Now that the documents and their embeddings are indexed in Elasticsearch, it's time to do some queries!

Before semantic_text, you needed to use a different query depending on the type of embeddings the model generates (dense or sparse). A sparse vector query is needed to query sparse_vector field types, and either a knn search or a knn query can be used to search dense_vector field types.

The query process can be further customized for performance and relevance. For example, sparse vector queries can define token pruning to avoid considering irrelevant tokens. Knn queries can specify the number of candidates to consider and the top k results to be returned from each shard.

You don't need to deal with those details when using semantic_text. You use a single query type to search your documents:

Just include the field and the query text. There’s no need to decide between sparse vector and knn queries, semantic text does this for you.

Compare this with using a specific knn search with all its configuration parameters:

Under the hood: How semantic_text works

To understand how semantic_text works, you can create a semantic_text index and check what happens when you ingest a document. When the first document is ingested, the inference endpoint calculates the embeddings. When indexed, you will notice changes in the index mapping:

Now there is additional information about the model settings. Text embedding models will also include information like the number of dimensions or the similarity function for the model.

You can check the document already includes the embedding results:

The field does not just contain the input text, but also a structure storing the original text, the model settings, and information for each chunk the input text has been divided into.

This structure consists of an object with two elements:

• text: Contains the original input text

• inference: Inference information added by the inference endpoint, that consists of:

  • inference_id of the inference endpoint

  • model_settings that contain model properties

  • chunks: Nested object that contains an element for each chunk that has been created from the input text. Each chunk contains:

    • The text for the chunk

    • The calculated embeddings for the chunk text

Customizing semantic_text

semantic_text simplifies semantic search by making default decisions about indexing and querying your data:

• uses sparse_vector or dense_vector field types depending on the inference model type

• Automatically defines the number of dimensions and similarity according to the inference results

• Uses int8_hnsw index type for dense vector field types to leverage scalar quantization.

• Uses query defaults. No token pruning is applied for sparse_vector queries, nor custom k and num_candidates are set for knn queries.

Those are sensible defaults and allow you to quickly and easily start working with semantic search. Over time, you may want to customize your queries and data types to optimize search relevance, index and query performance, and index storage.

Query customization

There are no customization options - yet - for semantic queries. If you want to customize queries against semantic_text fields, you can perform advanced semantic_text search using explicit knn and sparse vector queries.

We're planning to add retrievers support for semantic_text, and adding configuration options to the semantic_text field so they won't be needed at query time. Stay tuned!

Data type customization

If you need deeper customization for the data indexing, you can use the sparse_vector or dense_vector field types. These field types give you full control over how embeddings are generated, indexed, and queried.

You need to create an ingest pipeline with an inference processor to generate the embeddings. This tutorial walks you through the process.

What's next with semantic_text?

We're just getting started with semantic_text! There are quite a few enhancements that we will keep working on, including:

• Better inference error handling

• Customize the chunking strategy

• Hiding embeddings in _source by default, to avoid cluttering the search responses

• Inner hits support, to retrieve the relevant chunks of information for a query

• Filtering and retrievers support

• Kibana support

Try it out!

semantic_textis available on Elasticsearch Serverless now! It will be available soon on Elasticsearch 8.15 version for Elastic Cloud and on Elasticsearch downloads.

If you already have an Elasticsearch serverless cluster, you can see a complete example for testing semantic search using semantic_text in this tutorial, or try it with this notebook.

We'd love to hear about your experience with semantic_text! Let us know what you think in the forums, or open an issue in the GitHub repository. Let's make semantic search easier together!

First, we'll load the data using Node.js and ensure it remains current through regular updates. Then, we'll leverage the capabilities of Azure Function Apps to automate these updates, thereby ensuring your index is always fresh and reliable.

For this blog post, we will be using the Near Earth Object Web Service (NeoWs), a RESTful web service offering detailed information about near-earth asteroids. By integrating NeoWs with Node.js services integrated as Azure serverless functions, this example will provide you with a robust framework to handle the complexities of managing dynamic data effectively. This approach will help you minimize the risks of working with outdated information and maximize the accuracy and usefulness of your data.

Prerequisites

• This example uses Elasticsearch version 8.13; if you are new to Elasticsearch, check out our Quick Start on Elasticsearch. Any 8.0 version should work for this blog post.

• Download the latest NPM and Node.js version. This tutorial uses Node v21.6.1 and npm 10.5.0.

• An API key for NASA's APIs.

• An active Azure account with access to create a Function App.

• Access to the Azure portal or Azure CLI

Setting up locally

Before you begin indexing and loading your data locally, setting up your environment is essential. First, create a directory and initialize it. Then, download the necessary packages and create a .env file to store your configuration settings. This preliminary setup ensures your local environment is prepared to handle the data efficiently.

You will be using the Elasticsearch node client to connect to Elastic, Axios to connect to the NASA APIs and dotenv to parse your secrets. You will want to download the required packages running the following commands:

After downloading the required packages, you can create a .env file at the root of the project directory. The .env file allows you to keep your credentials secure locally. Check out the example .env file to learn more. To learn more about connecting to Elasticsearch, be sure to take a look at the documentation on the subject.

To create a .env file, you can use this command at the root of your project:

In your .env , be sure to have the following entered in. Be sure to add your complete endpoint:

You will also want to create a new JavaScript file as well:

Creating your index and loading your data in

Now that you have set up the proper file structure and downloaded the required packages, you are ready to create a script that creates an index and loads data into the index. If you get stuck along the way be sure to check out the full version of the file you are creating in this section.

In the file loading_data_into_a_index.js, configure the dotenv package to use the keys and tokens stored in your .env file. You should also import the Elasticsearch client to connect to Elasticsearch and Axios and make HTTP requests.

Since your keys and tokens are currently stored as environment variables, you will want to retrieve them and create a client to authenticate to Elasticsearch.

You can develop a function to retrieve data from NASA's NEO (Near Earth Object) Web Service asynchronously. You will first configure the base URL for the NASA API request and create date objects for today and the previous week to establish the query period. After you format these dates in the YYYY-MM-DD format required for the API request, set up the dates as query parameters and execute the GET request to the NASA API. Additionally, the function includes error-handling mechanisms to aid debugging should any issues arise.

Now, you can create a function to transform the raw data from the NASA API into a structured format. Since the data you get back is currently nested in a complex JSON response. A more straightforward array of objects makes handling data easier.

You will want to create an index to store the data from the API. An index inside Elasticsearch is where you can store your data in documents. In this function, you will check to see if an index exists and create a new one if needed. You will also specify the proper mapping of fields for your index. This function also loads the data into the index as documents and maps the id field from the NASA data to the _id field in Elasticsearch.

You will want to create a main function to fetch, structure, and index the data. This function will also print out the number of records being uploaded and log whether the data is indexed, whether there is no data to index, or whether it failed to get data back from the NASA API. After creating the run function, you will want to call the function and catch any errors that may come up.

You can now run the file from your command line by running the following:

To confirm that your index has been successfully loaded, you can check in the Elastic Dev Tools by executing the following API call:

Keeping your index updated with an Azure Function App

Now that you've successfully loaded your data into your index locally, this data can quickly become outdated. To ensure your information remains current, you can set up an Azure Function App to automatically fetch new data daily and upload it to your Elasticsearch index.

The first step is to configure your Function app in Azure Portal. A helpful resource for getting started is the Azure quick start guide.

After you've set up your function, you can ensure that you have environment variables set up for ELASTICSEARCH_ENDPOINT, ELASTICSEARCH_API_KEY, and NASA_API_KEY. In Function Apps, environment variables are called Application settings. Inside your function app, click on the "Configuration" option in the left panel under "Settings." Under" the "Application settings" tab, click on "+ New application setting."

You will want to make sure the required libraries are installed as well. If you go to your terminal on the Azure Portal, you can install the necessary packages by entering the following:

The packages you are installing should look very similar to the previous install, except you will be using the moment to parse dates, and you no longer need to load an env file since you just set your secrets to be Application settings.

You can click where it says create to create a new function inside your Function App select the template entitled “Timer trigger”. You will now have a file called function.json set for you. You will want to adjust it to look as follows to run this application every day at 10 am.

You'll also want to upload your package.json file and ensure it appears as follows:

The next step is to create a index.js file. This script is designed to automatically update the data daily. It accomplishes this by systematically fetching and parsing new data each day and then seamlessly updating the dataset accordingly. Elasticsearch can use the same method to ingest time series or immutable data, such as webhook responses. This method ensures the information remains current and accurate, reflecting the latest available data.You can can check out the full code as well.

The main differences between the script you run locally and this one are as follows:

• You will no longer need to load a .env file, since you have already set your environment variables

• There is also different logging designed more towards creating a more sustainable script

• You keep your index updated based on the most recent close approach date

• There is an entry point for an Azure Function App

You will first want to set up your libraries and authenticate to Elasticsearch as follows:

Afterward, you will want to obtain the last date update date from Elasticsearch and configure a backup method to get data from the past day if anything goes wrong.

The following function connects to NASA's NEO (Near Earth Object) Web Service to get the data to keep your index updated. There is also some additional error handling that can capture any API errors that might come up.

Now, you will want to create a function to organize your data by iterating over the objects of each date.

Now, you will want to load your data into Elasticsearch using the bulk indexing operation. This function should look similar to the one in the previous section.

Finally, you will want to create an entry point for the function that will run according to the timer you set. This function is similar to a main function, as it calls the functions created previously in the file. There is also some additional logging, such as printing the number of records and informing you if the data was indexed correctly.

Conclusion

Using Node.js and Azure's Function App, you should be able to ensure that your Elasticsearch index is updated regularly. By utilizing Node.js's capabilities in conjunction with Azure's Function App, you can efficiently maintain your index's regular updates. This powerful combination offers a streamlined, automated process, reducing the manual effort involved in keeping your index regularly updated. Full code for this example can be found on Search Labs GitHub. Let us know if you built anything based on this blog or if you have questions on our forums and the community Slack channel.

You've got your issue-tracking, your note-taking, your meeting transcripts, your wiki pages, your video recordings, your chats and IMs and DMs. And don't forget the emails!

It's no wonder that so many enterprises are trying to create workplace search experiences - giving their employees a centralized, one-stop-shop for searching for internal information.

With Elastic's catalog of connectors, this is relatively easy to do. But after you get all your data indexed and ready to be searched, how do you ensure that it is secured? After all, Tess (from Engineering) shouldn't be looking at Bob's (from HR) notes on performance reviews. How can you make sure that each separate user who comes to this unified search bar does gets their own unique view into only the data that they're authorized to view?

Enter, Document Level Security (DLS).

Understanding document level security (DLS) in Elasticsearch

Folks who've followed Elasticsearch for a while may already be aware that DLS has been an Elasticsearch feature for quite a long time. It's part of the larger theme of user authorization, and is really quite simple. You embed metadata in Elasticsearch documents, and then you craft an Elasticsearch query, filtering based on that document metadata, that describes the user's authorization. That query is used to create an Elasticsearch Role.

At query time, when the search user authenticates, their role(s) (if any) is identified, and the embedded query filter (if any) is applied to their searches.

Let's look at a simplistic example. Say we have two documents:

A query that would fetch only my data would be:

That query can be embedded into a Role, like:

So if my user is assigned the role my_role, if I just do

I will only see document 1, but not document 2.

While this example is simple in theory, it has a relatively large number of moving pieces.

• you must ensure that the documents contain the relevant metadata ("my-data": true vs "my-data": false)

• you must trust that the metadata on those documents is accurate

• you must create a Role for every search user with a finely crafted Elasticsearch query

• you must ensure that every role you create correctly maps to the right user at query time

• you mush ensure that all of the above stays up to date.

That last one is particularly difficult. When people in your enterprise join, leave, switch teams, or get promoted, that requires changes - potentially to both your (meta)data AND your Roles. And if you add in data sources that support sharing or access editing, you're definitely needing to make sure that your (meta)data stays up-to-date.

DLS with Elastic connectors

Connector document level security builds off of the Elasticsearch DLS primitives. For many connectors, this includes syncing the relevant metadata and Role Descriptors to support DLS. This results in the documents in your content index automatically containing metadata (usually in a _allow_access_control field) to describe the people/groups who are authorized to search for this document, as well as documents in a special .search-acl-filter-<index-name> index that contain the Role Descriptor JSON necessary to build a concrete Role or an API key for a given search user.

You can find which connectors have DLS available here. For this blog, we're going to reference an example application which utilizes the Sharepoint Online connector. This was the first connector we enabled DLS on, but the example could be easily adapted to work with any DLS-enabled connector.

If your connector is eligible, and you have a Platinum+ Elasticsearch license, you can enable DLS through a toggle on the connector configuration page.

From there, it's just a matter of running a Full sync and an Access Control sync, and Elasticsearch will have all the data it needs.

DLS implementation example

And then what?

Once Elasticsearch has Role Descriptors and document data with sufficient metadata for those role filters, you're ready to build a secure search experience.

We've built an example knowledge search app that we'll use for this blog, and you're welcome to go take a look at its source code. However, we do want to stress that this is an example only - it is not ready to be run in production on its own. Please exercise good judgement and do not run code that you have not read or do not understand.

This application has a pretty simple architecture.

It is composed of a Flask backend and a React frontend. The backend is configured with environment variables to establish a connection with Elasticsearch.

Using this connection, the backend provides three endpoints:

1. GET /api/persona This endpoint lists the identifiers for the "identities" or "personas" that the connector found during the Access Control sync. The frontend uses this list to populate a dropdown of personas so to demonstrate how search results change depending on the selected persona.

2. GET /api/indices This endpoint lists which indices have been included in your Search Application. The frontend uses this list to allow you to choose which ones to search against.

3. GET /api/api_key?persona=<persona> This endpoint creates and returns an Elasticsearch API key based off of a selected persona. In a production system, persona wouldn't be a request argument, but would be inferred from the authentication credentials. This API key is then used by the frontend to issue search requests to Elasticsearch.

Caveats

As stated above, this example should not be used in production. Gaps include:

• It does not implement authentication. A production-ready app would need a way for users to authenticate, and have their identities verified, rather than selecting a user from a dropdown.

• It does not utilze SSL/TLS. The backend currently transmits Elasticsearch API Keys to the frontend over HTTP, not HTTPS.

• The frontend issues /_search requests directly to Elasticsearch. Depending on the production use case, you may not want to expose Elasticsearch to your end user like this. Instead, it may be advisable to issue requests from the frontend to your backend (again, with authentication implemented), and have the backend translate those requests to Elasticsearch queries.

Source Code

Below we link to the critical pieces of code that are necessary to implement search with DLS.

Creating the authenticated user's role descriptor

Code link

You may notice that the query template in this role descriptor is significantly more complex than the simple example provided earlier in this blog. This query does several things:

1. It uses a query template, instead of an explicit query. This makes it easier when reading to separate a long list of permissions from the query syntax.

2. It uses a bool query. This allows us to combine several logical checks.

3. It grants access to any documents that do not contain the _allow_access_control field

4. It grants access to documents where the _allow_access_control field contains a value found in this user's permissions

Creating an API Key from that Role Descriptor

Code link

Searching with the API Key

Code link