A sorted ES|QL query over 346 million log documents

Using real-world Nginx logs, the benchmark searched 346 million docs across 432 GB of total index data. It returned 500 results and took ~500 seconds. That's more than eight minutes.

What’s happening under the hood?

Before this optimization, candidate rows were loaded, decoded, and evaluated by the user filter before TopN could determine which of them could no longer contribute to the final result.

How min-competitive TopN prunes work in ES|QL

TopN learns something useful while the query is still running. Once its heap is full, the worst value currently in the heap becomes a competitive threshold: that is, any row that cannot beat that value cannot make the final result.

ES|QL can feed that threshold back into earlier stages of execution. Two changes make this work:

1. Dynamic Lucene pushdown. As TopN finds better candidates, its competitive threshold improves. ES|QL pushes that threshold down to Lucene, allowing Lucene to skip candidates that cannot beat the current worst TopN value. This avoids loading, decoding, and evaluating rows that have no chance of making the result.

2. Shared threshold across drivers. ES|QL processes data in parallel across multiple drivers. If each driver only uses its own local threshold, one driver cannot benefit from better candidates discovered by another. Sharing the best-known competitive threshold means a strong result found by one driver immediately helps all the others prune their remaining work.

Together, these changes move TopN from a late-stage filter to something that shapes query execution much earlier in the pipeline.

How does the min-competitive TopN optimization work?

The query asks for the top three rows matching a wildcard filter, sorted in descending order, across three segments of 10 rows each. Without the optimization, all 30 rows are loaded, decoded, and evaluated against the filter before TopN sees any of them, at a cost of 30 row loads and 15 TopN operations.

With dynamic pushdown, ES|QL maintains the TopN heap as it scans. The heap fills at the end of the first segment with [98, 51, 47], and its worst entry is pushed down to Lucene as the competitive threshold. That cuts the second segment to four rows out of 10, which improves the heap to [98, 83, 77] and tightens the threshold again, leaving just two rows to load in the third segment. This produces the same three results, with 16 row loads and 10 TopN operations.

Sharing the TopN threshold across drivers

One driver may find excellent recent candidates immediately. Without coordination, another driver doesn't know that and continues working with a weak threshold. Sharing the global worst competitive value lets that discovery benefit all drivers.

Benchmark results on a local node and Elastic Cloud Serverless

The min-competitive TopN optimization ships as generally available (GA) in Elasticsearch 9.6. For real-world Nginx data benchmarks (same query, same dataset, p50 of 51 measurements, with 20 warmup runs discarded):

Local node

Serverless (GCP)

Before optimization

~500s

~520s

Production implementation

~5s

~14s

Runtime improvement

~99%

~97%

Speedup

100x

37x

Benchmark hardware

Local node

Serverless (GCP)

CPU

Apple M4

Small GCP host

RAM

48 GB

8 GB

Java Virtual Machine (JVM) heap

24 GB

4 GB

Storage

Local external TB5 NVMe SSD

Cloud storage

Shards

100

100

When does this query optimization technique help most?

The biggest gains happen when TopN becomes competitive early. If good candidates appear at the start of execution, the threshold rises quickly and subsequent scans get pruned aggressively. Three conditions amplify this:

1. N is small relative to the candidate population. Finding 500 rows out of 346 million documents gives the optimizer far more room to eliminate work than returning a large fraction of the matches does.

2. The post-Lucene work is expensive enough that skipping a row saves something measurable, like avoiding loading large strings, decompression, or wildcard evaluation.

3. Many candidates pass the coarse Lucene scan in the first place. If almost nothing matches the initial filter, there’s not much unnecessary work left to eliminate.

If matching rows are rare or if good candidates are discovered late, the threshold takes longer to become useful and the gain can be much smaller.

The current implementation applies to @timestamp sorted queries. Queries sorted by other fields don’t yet benefit from this optimization.

What's next for ES|QL query performance

The same idea has room to run in several directions, including:

• Other numeric sort fields. The underlying approach isn’t inherently limited to timestamps, so extending it to other numeric sort fields could benefit a much broader range of sorted TopN queries.

• Processing promising data first. How quickly the threshold becomes useful depends on how early TopN sees good candidates. Processing more promising data slices first could establish a strong threshold sooner, allowing more aggressive pruning of subsequent work.

• Early termination. As the threshold strengthens during execution, it becomes possible to recognize when remaining ranges cannot produce a better result and to stop processing them entirely, rather than running through to completion.

• Coordination across nodes. The current optimization shares competitive information across drivers within a node. Broader coordination across nodes or clusters could let strong candidates discovered in one part of a distributed query reduce work elsewhere.

• Late materialization and result streaming. Competitive filtering and late materialization attack the same problem from different angles. Competitive filtering eliminates rows that cannot reach the final TopN as early as possible, while a fetch phase can defer loading expensive field values until the set of likely winners is much smaller. Result streaming could extend this further. If the engine can determine that some leading results can no longer be displaced, those rows could be returned before the full query completes. For interactive investigations, getting the first useful result fast matters as much as total query runtime.

What this means for Elasticsearch performance tuning

The min-competitive TopN optimization in Elasticsearch 9.6 shows how much work can be avoided when information discovered during query execution is fed back into earlier stages. In our Nginx log benchmark, that turned a roughly 500-second query into a roughly 5-second query on a single node.

This is also a starting point. Extending competitive filtering to more sort fields, finding strong candidates earlier, terminating work sooner, and eventually coordinating these decisions across nodes offer opportunities to push the same idea further.

The earlier we know what can still win, the less work we need to do for everything that cannot.

A filter inside the knn clause, such as a tenant ID or a language, runs as a pre-filter, so it's evaluated over every document in every segment before the search starts. When a filter matches enough of the corpus, Elasticsearch runs the vector search first and checks the filter only against the candidates that come back. The candidate pool is sized statistically so that at least k survive. If they fall short, Elasticsearch retries once and then falls back to the pre-filtered search. Your queries stay the same, and you still get k results. It's scheduled to be on by default in Elasticsearch 9.6.

How a kNN filter works in Elasticsearch

As a quick reminder, Elasticsearch has two approximate nearest-neighbor structures for dense_vector fields. HNSW is a proximity graph: the search walks from neighbor to neighbor, keeping the num_candidates closest vectors it has seen so far as its beam. IVF, the bbq_disk index type, clusters vectors around centroids, each with a posting list of its vectors, and scans the posting lists closest to the query; the fraction of the segment it scans, its visit ratio, is derived from num_candidates and k. The posts on HNSW graphs and DiskBBQ cover both in depth.

What sets up everything that follows is that neither is a single index-wide structure. Every Lucene segment carries its own HNSW graph or its own set of IVF posting lists, so a kNN query searches each segment separately and merges the results, first per shard and then on the coordinating node. Anything the search has to prepare before it starts exploring is therefore prepared once for every segment.

Now let's add a filter to the kNN query:

Because the filter sits inside the knn clause, it is a pre-filter: the 10 results are the 10 nearest documents among those matching language: en. To deliver that, the search has to know, for every document it comes across, whether that document passes the filter. In each segment, this happens in two steps.

Step 1: Materializing the filter into a bitset

Neither structure visits documents in doc ID order. An HNSW walk jumps from a node to its graph neighbors, while IVF scans posting lists one centroid at a time, meaning document IDs arrive in whatever order vector proximity dictates. The search needs to answer "does doc 84,219 match?" on demand, and a filter's regular DocIdSetIterator (which can only move forward) cannot do that. Before the search starts, the filter is run to completion over the segment, and its matches are recorded in a bitset. For HNSW, this happens in Lucene's AcceptDocs:

createBitSet drains the filter's iterator to the end and fills either a FixedBitSet sized to the segment's maxDoc or a sparse bitset, depending on how many documents match. IVF does the equivalent through Elasticsearch's own ESAcceptDocs before it scans its first posting list. Either way, the cost depends on the size of the segment and how expensive the filter is to evaluate, not on how much of the segment the search goes on to visit.

Step 2: Skipping documents that fail the filter

Documents that fail the filter do not count toward the results. The search has to visit more of the segment to collect enough that pass: HNSW walks further through the graph, and IVF scans a few extra posting lists. Both keep this bounded. HNSW, for instance, stops walking the graph once it has visited as many vectors as the filter matches, and scores the matching documents directly instead. For a filter that matches most of the corpus, the extra work is small, as the profile in the next section shows.

The takeaway: The extra search work caused by a filter is bounded, and it is rarely where a filtered query's time goes. The materialization in step 1 has no such bound. That is why the interesting case is not a restrictive filter but a broad one, matching, say, 90% of the corpus. It prunes almost nothing and adds little search work, yet it is still materialized at full price in every segment.

Why a filtered kNN search can be slower than an unfiltered one

The search profile shows this cost directly, though not where you might first look for it.

Lucene's kNN queries do almost all of their work in rewrite. For HNSW, AbstractKnnVectorQuery#rewrite materializes the filter, searches every segment, merges the per-segment results, and returns a DocAndScoreQuery holding the final doc IDs and scores. IVF does the same and returns Elasticsearch's KnnScoreDocQuery. By the time the query is executed in the usual sense, the search is over, and all that is left is to hand back a precomputed list of hits.

A top-level knn clause runs in the DFS phase, so its profile lands under profile.shards[].dfs.knn[]. Here is an unfiltered search with k: 10 and num_candidates: 100 against a single HNSW segment of one million vectors, with each breakdown trimmed to its non-zero entries:

rewrite_time, 5.2 ms, is the kNN search. The DocAndScoreQuery entry takes under 6 microseconds: it only returns the ten hits.

Now add a match_phrase filter that matches 90% of the documents. A phrase that matches nearly the whole corpus is not a common filter in practice, but it isolates the effect cleanly: it barely narrows the search, and it is expensive to evaluate. Most real filters that match this broadly cost somewhere between this one and the term filter we compare it to further down.

Three things changed:

• The filter has its own entry, next to the DocAndScoreQuery rather than inside it, because it runs during the kNN query's rewrite. CachingEnableFilterQuery is the wrapper Elasticsearch puts around kNN filters so that the query cache always considers them worth caching; the query you wrote is its child. On a bbq_disk field, the entries are named differently, but the filter shows up the same way.
• into_bit_set is the materialization from step 1: a single call that runs the phrase query over the whole segment and records every match in the bitset. It took 33.4 ms. That time is part of rewrite_time, not added to it; rewrite_time went from 5.2 ms to 40.1 ms, and the filter accounts for 33.6 ms of the difference. (A filter matching fewer than about 1 in 128 documents is collected into a sparse bitset one document at a time instead, so its cost shows up under next_doc.)
• vector_operations_count barely moved. It went up 13%, roughly what you would expect when one visited document in ten fails the filter.

More than 80% of this query's time went into evaluating the filter. One thing to keep in mind when reading profiles like this: profiling bypasses the query cache and the profile always shows the filter's uncached cost, even for a filter that unprofiled requests would find in the cache.

kNN filter cost: Materialization vs. extra search work

Two costs make up the difference between these two profiles, and it is worth separating them because they scale differently.

Filter materialization is paid per segment, proportional to the segment size, before any vector work. How expensive it is depends entirely on the query type:

Filter

Materialization cost

term on a keyword

Cheap - read one postings list, intoBitSet in bulk

range on a numeric or date field

Usually cheap - walks the points (BKD) index; only a field with doc values alone (index: false) has to check every document

match_phrase

Positions decoding and intersection for every document containing the terms; by far the most expensive

bool of several clauses

Cost of each clause, plus conjunction/disjunction bookkeeping

Cached filter

Nearly free on a cache hit, full price on a miss

On the one-million-document segment above, a term filter matching the same 90% of documents spends about 0.3 ms in into_bit_set, roughly 100x less than the phrase filter. And the cost grows with the segment: in the benchmarks below, a match_phrase filter matching 90% of a 10M-document segment takes an HNSW query from 2.3 ms to 395 ms.

Extra search work adds something on top, but as we saw, it is bounded, and for a broad filter it is small: the 13% above. Materialization dominates, and a filtered kNN search on a broad, expensive filter can spend most of its time on work that is not vector comparison. The answer it arrives at also overlaps heavily with the unfiltered one: if a filter matches 90% of the corpus and is unrelated to the query, about nine of the ten nearest neighbors overall already pass it, and the ones that do not are replaced by documents only a few ranks further down.

That is the asymmetry the rest of this post is about. The filter's cost is tied to how many documents it is evaluated over, and evaluating it after the vector search instead of before changes that number by orders of magnitude:

Pre-filtering vs post-filtering in kNN search

What if we just post-filter? That observation is not new, and Elasticsearch has always let you act on it. Move the filter outside the knn clause and it becomes a post-filter: the kNN search runs unconstrained, and the filter is applied to the k results it returns.

We covered this trade-off in detail in How to choose between exact and approximate kNN search in Elasticsearch, where the catch is put plainly:

The problem with using post-filters in kNN is that the filter is being applied after we have gathered the top k results. This means that we can end up with less than k results, as we need to remove the elements that don't pass the filter from the top k results that we've already retrieved from the HNSW graph.

Ask for 10, get back 6. Or 2. Or 0. The vector search does not know the filter exists, so nothing guarantees that 10 of its results survive it. The only remedy available to you is to over-collect by hand: ask for k: 50 and hope 10 survive. That puts you in an awkward position:

• You have to guess the multiplier, and the right guess depends on how selective your filter is for this particular query; something you generally do not know.
• Guess too low and you silently return short result sets. Guess too high and you pay for exploration you did not need, which is the cost you were trying to avoid.
• Either way, k no longer means what the API says it means, so paging, size, and anything downstream has to be reasoned about separately.

So both options are unsatisfying. Pre-filtering is correct but can be pathologically slow. Post-filtering is fast but shifts a statistics problem onto the user and still offers no guarantee.

The table below compares both options with the automatic post-filtering covered in the next section.

Pre-filtering

Manual post-filtering

Automatic post-filtering

Where the filter goes

Inside the knn clause

Outside the knn clause, in a bool query

Inside the knn clause (unchanged)

Filter is evaluated over

Every document in every segment

The k results the vector search returns

The candidates the vector search returns

Returns k results

Yes

Not guaranteed

Yes, by falling back to pre-filtering if needed

Who sizes the candidate pool

Not needed

You, by guessing a multiplier for k

Elasticsearch, from the filter's estimated selectivity

Best suited to

Selective filters

Cases where a short result set is acceptable

Broad filters (selectivity of 0.7 or more by default)

But we can do better

The insight is that the choice between these two does not have to be the user's, and it does not have to be made once for the whole index. Elasticsearch is holding the filter's Weight at rewrite time. It can estimate how selective the filter is, decide whether post-filtering is a good bet for this query on this shard, size the over-collection from that estimate rather than from a guess, and keep a correctness net underneath so that a bad estimate costs latency instead of results.

This is what automatic post-filtering adds, for both HNSW and IVF (bbq_disk). Importantly, none of it changes the query you write: you keep expressing a pre-filter, you keep getting pre-filter semantics, and how that gets satisfied becomes an engine decision.

Four ideas, in order:

1. Estimate the filter's selectivity.
2. Use it to size an over-collection.
3. Retry once if that comes up short.
4. Fall back to the original query if it still does.

Estimating kNN filter selectivity

First, we need a number for "how much of the corpus does this filter let through?" The cheap and surprisingly effective answer is to ask the filter's own scorers what they expect to match, and divide by the number of vectors actually indexed for the field:

Two things make this cheap. ScorerSupplier#cost() is an estimate. For a term query, it is the postings list length, read from metadata, so we get it without materializing anything. And the denominator is the vector count from the codec, not maxDoc, so a field that only some documents have does not skew the ratio.

Two things make it imperfect, and both are handled downstream rather than pretended away. cost() is an upper bound for conjunctions, and selectivity can be overestimated. More fundamentally, selectivity is a global property of the shard, while what actually matters is the filter's pass rate inside the neighborhood of the query vector. For a filter that is independent of vector content, the two match. For one that is correlated with it, they can drift apart in either direction.

Say language: en matches 90% of a shard. For an English query, nearly all of the nearest neighbors are English. The local pass rate is therefore close to 100% and the global estimate is merely conservative. For a query written in Spanish, the nearest neighbors can be dominated by the 10% of documents that are not English, and the local pass rate can fall far below 90% even though the global estimate has not changed. That second case is the one the retry and fallback rounds below exist for.

Sizing the candidate pool with a binomial model

Given selectivity , how many raw candidates should the unfiltered search collect to get at least survivors?

The naive answer is . That gets you survivors on average, which means it falls short roughly half the time. Averages are the wrong tool here: we need a high-probability guarantee.

So model each candidate as independently passing the filter with probability . The number of survivors out of candidates is then a binomial variable:Now instead of asking for the mean to equal , ask for , to sit standard deviations *below* the mean:Solving that for exactly gives a quadratic. Substituting into the variance term instead yields a closed form that is marginally conservative, which is the right direction to err: is a confidence dial: round one succeeds with probability . The implementation uses , or about 99.4%.

The assumption worth naming explicitly is independence. We assume it because, absent any signal about how the filter correlates with vector content, it is the only thing we can assume. The retry and fallback rounds below exist precisely to cover the cases where it is wrong.

How many extra candidates post-filtering collects

In code, that formula plus its guard rails:

The 1.2x floor matters because, as , the variance term vanishes and the formula would return exactly , leaving no slack for the handful of documents that do get filtered out. The NUM_CANDS_LIMIT cap (10,000) keeps a very restrictive filter from requesting an unbounded candidate pool.

For and , the resulting oversample:

Selectivity

margin

Candidates

Oversample

0.99

0.79

12 (floor)

1.2x

0.95

1.81

13

1.3x

0.90

2.64

15

1.5x

0.80

3.95

18

1.8x

0.70

5.18

22

2.2x

0.55

7.15

32

3.2x

Note how modest these are and how they shrink relative to k as k grows, because the standard deviation grows as while the mean grows as . At and the oversample is only 2.2x, versus 3.2x at . Asking for 15 results instead of 10 is nothing compared to materializing an expensive filter over a 10M-document segment.

There is a second budget that must not be conflated with k, and getting this wrong is an easy way to accidentally retune the search. For HNSW, num_candidates is the beam width, and it is an independent knob. The delegate keeps the user's value, floored at the grown k (a beam narrower than the number of results requested cannot surface them):

A consequence worth spelling out: an HNSW delegate hands back more candidates than the grown k. Each segment's graph walk keeps its whole beam (up to num_candidates candidates), and every one of them is checked against the filter. On HNSW, the grown k mostly matters when num_candidates is close to k; with the usual wider beam, round one has plenty of candidates to draw from.

For IVF, num_candidates is only meaningful relative to k: the codec derives its visit ratio from the ratio of the two. Carrying num_candidates over unchanged while k grows would silently reduce exploration, and IVF scales it to preserve the ratio:

Retrying when post-filtering comes up short

The binomial model is calibrated for ~99.4% success, and its independence assumption can be wrong. Therefore, round one sometimes comes up short. The retry round is what turns "usually enough" into "enough."

It runs only if the shortfall looks like bad luck rather than a bad model. If round one returned far fewer survivors than global selectivity predicts, that is evidence that the filter is correlated with the query neighborhood; the case the binomial model explicitly cannot handle. More rounds will not fix a wrong model, so we exit early instead:

Fewer than half the predicted survivors means the filter is hostile to this query's vector neighborhood, and pre-filtering is the right tool. The check is skipped for k < 5 , where the expected count is too small for the ratio to mean anything.

A retry is not just a re-run with a bigger k ; that would re-find the same candidates. It has to search somewhere new, so it carries three pieces of state forward:

An exclusion set. Every document seen in round one, both the ones that passed the filter and the ones that did not, is excluded via an ExcludeDocsQuery. HNSW composes it into the accept-docs; IVF pushes it into posting-list iteration so the codec skips those documents outright.

Seed entry points. For HNSW, restarting the graph walk from the top layer would retrace the same descent. Instead, the retry seeds from the nearest round-one matches (up to four per segment) so it starts already in the right neighborhood and explores outward:

IVF ignores seeds (for now); it already knows which centroids are closest and simply rescans them with the excluded documents skipped.

A re-sized target. remaining is the shortfall in survivors. It is run back through computeScaledK with the same selectivity to inflate it for the attrition the retry will also suffer.

There is exactly one retry. A second would be chasing a distribution the model has already been shown to be wrong about, and the fallback below is both cheaper and strictly correct.

Falling back to pre-filtering

The fallback round makes automatic post-filtering safe to enable by default. If post-filtering does not produce the full candidate pool, its results are thrown away, and the original pre-filtered query runs instead.

Returning null from postFilterRewrite sends rewrite down the ordinary path:

- If at least candidates passed the filter, return the nearest .

- If fewer than half the expected number passed (), exit early to the pre-filtered query.

One detail that matters for score correctness. For quantized fields, approximate scores get refined by an exact rescoring pass. Normally, IVF does that inside its own rewrite, but on a post-filter delegate it must happen after filtering, or we would rescore documents that are about to be discarded. The delegate skips it, and the orchestrator calls back in through finalizeTopK once the pool is final. Without that hook, post-filtered results would carry raw quantized scores while fallback results carried exact ones, mixing two score domains inside a single search.

A filtered kNN query, end to end

Say you run this against a 10M-document shard where language: en matches 9M of them:

1. Rewrite. PostFilterKnnQuery#rewrite builds the filter Weight, conjoined with a FieldExistsQuery on embedding, so documents without a vector never count, but does not execute it.

2. Estimate. ScorerSupplier#cost() over the segments gives ~9M against 10M indexed vectors: . That clears the default threshold of 0.7, so post-filtering is on. (Had it come in at 0.4, we would stop here and run the ordinary pre-filtered query.)

3. Size round one. , so a filter-less delegate is built asking for 15 results instead of 10. Its exploration budget follows the rules above: on HNSW, num_candidates stays at 100, the beam width; on IVF, it scales to 150 so the visit ratio does not change.

4. Search unfiltered. The delegate runs across all segments with no accept-docs bitset: no filter materialization and no extra exploration. Each segment keeps every candidate its own search collected, so the orchestrator does not have to re-derive them. On HNSW that is the segment's whole beam, up to 100 candidates; on IVF it is a small multiple of the 15 requested, since IVF over-collects to absorb documents that appear in more than one posting list.

5. Apply the filter to the candidates, not to 10 million documents. applyFilter walks each segment's candidates in doc-ID order and tests them through Lucene#asSequentialAccessBits. For a filter exposing a TwoPhaseIterator, that is one approximation advance per candidate, plus a matches() call whenever the approximation lands on it. This is the whole point: the filter is evaluated once per candidate rather than once per document in the segment.

It also means the filter no longer needs to be cached to be fast. With pre-filtering, the usual way to avoid paying for materialization on every query is the filter cache, which stores the filter's per-segment bitset for reuse. That only helps once a filter has been seen often enough to get cached, and every new segment starts cold. Post-filtering has nothing worth caching: applyFilter tells the filter how few candidates it will be checked against, and Lucene's query cache declines to build a bitset for a filter that matches millions of documents when only a tiny fraction of them will be looked at. The filter is just as fast on its first use as on its hundredth.

6. Check against the target. The target is the 10 results the user asked for. With a filter that passes 90% of documents, roughly nine in ten candidates pass, far more than 10 across the segments. We deduplicate and keep the 10 nearest.

Suppose instead that only 7 distinct candidates had passed, as can happen when the filter is correlated with the query, like the Spanish-language query from earlier. 7 is above the hostile-filter threshold (), so a retry fires: exclude every candidate already seen, whether it passed or not, seed from up to 4 of the nearest passing candidates per segment, and ask for computeScaledK(3, 0.9) = 5 more to cover the shortfall of 3. If that still leaves us short of 10, the pre-filtered query runs and post-filtering has cost nothing but the candidate collection.

7. Finalize. Select the top k by score via introselect and wrap the result in a KnnScoreDocQuery.

The filter was evaluated only on the candidates the vector search returned, instead of on ten million documents. The vector search was unconstrained, and the returned documents are the same 10 that pre-filtering would have found, with a guarantee that if they were not, you would have gotten the pre-filtered answer instead.

Post-filtering vs. pre-filtering benchmarks on up to 10M vectors

Benchmark setup

To measure the ceiling here, we ran a forced comparison: every configuration executed twice, once pre-filtered and once post-filtered, with the adaptive logic bypassed so that neither strategy could bail out to the other. That is a deliberately harsher test than production behavior; the real implementation picks per query and falls back, but it maps out where each strategy wins.

Grid: 5 datasets from 523K to 10M vectors, each indexed with both IVF and HNSW, in both single- and multi-segment layouts, against 5 filter types at 7 selectivities, run both pre-filtered and post-filtered: 5 × 2 × 2 × 5 × 7 × 2 = 1400 measurements, or 700 matched pre/post pairs. The seven selectivities are 0.55, 0.7, 0.8, 0.9, 0.95, 0.99, and 1.0, where 1.0 means no filter. Fixed at num_candidates=1000, k=10, 1% visit ratio, uncached filters, 300 queries per point. IVF with clusterSize=384; HNSW with m=16, efConstruction=200; 1-bit quantization throughout.

Filters were uncached. That is the cold case for pre-filtering: a warm filter cache would narrow the gap for filters that repeat across queries. It is also the only case for post-filtering, which never relies on the cache in the first place.

Not every dataset has a numeric, keyword, or text field for the range, term, and phrase filters to target. Where one was missing, we added it and filled it with random content. We then sampled the corpus to pick filter values that match the target D% of documents:

• range is a range query over a numeric field, with bounds chosen so that D% of the documents fall inside them.
• term matches a keyword value that D% of the documents carry.
• phrase is a phrase query over a text field, for a phrase that D% of the documents contain. Evaluating it means checking term positions, not just term presence, which makes it the most expensive filter here to materialize.
• range_term combines a range and a term filter in a bool query.
• random matches a randomly chosen D% of the documents.

Results across corpus sizes and filter types

For filtering, the property of a dataset that matters most is its size, since materializing a filter costs time in proportion to the number of documents in each segment. We use the five datasets to show how the two strategies compare as the corpus grows, and take the detailed numbers below from the largest, cohere-msmarco-10M (10M vectors, 1024 dimensions, float32).

Post-filtering comes out ahead almost everywhere: at every corpus size, with both index types, and in both layouts. How far ahead depends on how expensive the filter is to materialize, and in most layouts the gap widens as the corpus grows. The 16 cells out of 100 below 1.0x are all cheap term or range filters on multi-segment layouts, and the lowest is 0.68x.

On the cohere-msmarco-10M dataset, 104 of the 120 filtered pairs are faster with post-filtering, including all 60 on a single segment, and 2 more are ties (within 2%). The remaining 14 are cheap range and term filters on multi-segment layouts, where pre-filtering is ahead by at most 1.33x (1.76 ms against 2.34 ms).

The latency-versus-selectivity curves show the mechanism directly:

The blue post-filter lines are nearly flat. With post-filtering, latency is driven mainly by the kNN search itself, which runs unfiltered and therefore costs the same no matter how much the filter matches. Applying the filter to the candidates adds little on top, and the over-collection stays small even as the filter narrows. For a given filter type, latency is almost indifferent to selectivity: in the median configuration it varies by about 10% across the whole 0.55-0.99 range, against about 50% for pre-filtering. The red pre-filter lines sit above them by roughly the cost of materializing the filter, meaning the gap is as large as the filter is expensive, and everything converges at 1.0, where there is no filter to evaluate.

Recall is essentially strategy-independent: every one of the 140 pairs is within 0.01 recall, so the trade-off really is pure latency.

Limitations: Correlated filters and multi-segment layouts

Two caveats are worth stating plainly.

These filters are largely uncorrelated with the vectors. A filter built on random content passes documents independently of where they sit in vector space, which is exactly the assumption the binomial model makes. It shows in the results: within any one configuration, recall differs by at most 0.04 across the five filter types, and median recall is 0.74-0.75 at every selectivity. Correlated filters, like the language example earlier, are handled by the retry, early exit, and fallback rounds, which keep results correct even when the selectivity estimate is off. Measuring how much of the speedup carries over to strongly correlated filters is a natural next benchmark.

Multi-segment layouts shrink post-filtering's edge. Compare the single- and multi-segment panels of the heatmap. Fixed per-segment costs are paid more times, and pre-filtering amortizes its bitset across a smaller maxDoc per segment. Post-filtering still wins on the expensive filters; it just wins by less.

How to enable and tune automatic post-filtering

Automatic post-filtering is scheduled to be enabled by default in Elasticsearch 9.6 and in upcoming Elastic Cloud Serverless releases. Once it is, there is nothing to change in your queries: you keep writing filters inside the knn clause, and each shard decides, query by query, whether post-filtering is worth attempting.

Tuning the post-filter selectivity threshold

That decision is controlled by a single index setting, index.dense_vector.post_filter_selectivity_threshold, which defaults to 0.7. A query goes through the post-filter pipeline when its estimated selectivity is greater than or equal to the threshold. The threshold is therefore a floor on how broad a filter must be before post-filtering is tried:

Threshold

Behavior

1.0

Off. No filter can qualify.

0.9

Only very broad filters - matching 90%+ of the corpus - post-filter.

0.7 (default)

Filters matching 70%+ post-filter.

0.0

Every filtered kNN query attempts post-filtering.

To tune it for an index, lower it if your filters tend to be expensive to evaluate, or set it to 1.0 to opt out entirely. You can set this up in the index settings:

Scope and limitations

Worth knowing about scope:

• Applies to both HNSW and bbq_disk (IVF) fields, for float, bfloat16, and byte element types, as well as bit vectors on HNSW.
• Nested vector fields are handled, with one wrinkle: because a nested kNN search keeps only one hit per parent, a parent that already produced a match is excluded from the retry as a whole block, while a parent whose child was merely filtered out stays eligible - a deeper child of it may still pass.
• Query semantics are unchanged. You still write a pre-filter and get pre-filter semantics. Nothing in the request body changes; this is purely an execution-strategy decision.
• The profile shows which strategy ran. When a query is post-filtered, the filter's entry has no into_bit_set: the filter is checked candidate by candidate, meaning it reports one advance per candidate the vector search returned. On the phrase-filter example from earlier, that is 100 advance calls (one per candidate in the beam), taking well under a millisecond in total, instead of a 33.4 ms into_bit_set. A second hit-list entry, a KnnScoreDocQuery holding the final top k, also appears next to the vector search's own. For the individual decisions (retries, early exits, fallbacks, and the selectivity estimates behind them), set logger.org.elasticsearch.search.vectors.PostFilterKnnQuery: DEBUG.

What's next for filtered kNN search in Elasticsearch

The pieces we are most interested in improving:

Better selectivity estimates. ScorerSupplier#cost() is free but crude, and it overestimates conjunctions. Elasticsearch already tracks richer field statistics; wiring those in, or sampling the filter over a bounded slice of a segment, would let the threshold be set less conservatively.

Detecting correlation up front. The hostile-filter early exit is reactive: we find out the filter is correlated with the query neighborhood only after spending a round discovering it. Estimating the filter's local pass rate before committing (from the seed documents of the graph descent, say, or from cached per-query statistics) would turn that into a cheap up-front routing decision and let us safely lower the threshold.

Cost-aware routing. The benchmarks say the right threshold depends on filter type as much as selectivity: a match_phrase filter is worth post-filtering at far lower selectivity than a term filter, because its materialization cost is orders of magnitude higher. A cost model over the filter's query shape, rather than selectivity alone, would let the engine make that call per query instead of relying on a single per-index threshold.

Per-segment decisions. Routing is currently per shard, but selectivity and segment size vary between segments. Deciding per segment would let a small segment pre-filter while a large one post-filters in the same query.

The broader point generalizes beyond kNN. Elasticsearch has spent a lot of effort making vector comparisons fast (quantization, SIMD, better HNSW graphs, and a disk-friendly IVF index), and has been rewarded with queries where vector comparison is no longer the bottleneck. When most of a filtered kNN search goes into materializing an expensive filter, the optimization worth having is not a faster distance function. It is noticing that the filter did not need to run over ten million documents to answer a question about ten.

Doc values were built to make aggregations, sorting, and grouping possible on a document engine, and they do that job well. Columnar Mode changes what the columns are for, and this post walks through the five properties that separate storing columns from being a columnar database.

Five properties separate a column store from a columnar database

Doc values were an optimization on top of _source

For most of the last decade, the original JSON document was the source of truth and the columns were a derived convenience. That ordering has consequences throughout the engine.

Because the engine could always fall back to _source, per-field storage was allowed to be lossy. Text fields had no doc values at all, since text could be reread from the stored document when needed. Even synthetic source, which reconstructs a document from its fields rather than storing a copy, sometimes reads from row-shaped structures to stay faithful to the JSON that arrived, with values that exceed ignore_above and fields that arrived unmapped going into stored fields.

The result is a clear contract; whatever JSON you send, you get back, and the columns accelerate everything else. For an engine whose job is to return your documents, that’s the right way round. Columnar Mode inverts it. Every field stores itself exactly once as doc values, doc values cannot be turned off, text fields get doc values, too, and the document is reconstructed from the columns when something asks for it.

How dictionary encoding handles high-cardinality data

Sorted doc values, the default for keyword fields, store a dictionary of distinct values plus one ordinal per document pointing into it. This is an excellent trade when values repeat. A host.name field drawn from a hundred machines, or a status code that’s almost always 200, compresses beautifully and groups quickly.

It works like the index cards in a warehouse. When 50 crates hold the same product, one card and 50 pointers beats writing the product name 50 times. When 50 crates hold the same product, you can store the product name in the index with the list of 50 crate IDs. When every crate holds something unique, you might as well just put the product name on the crates; the index will help you find what crate you want, but it won't save ink.

High-cardinality fields describe a lot of real data, including URLs and trace identifiers, along with message bodies. Columnar Mode, which skips the dictionary and compresses the values in blocks instead, uses binary doc values for high-cardinality strings. Which of the two a field gets isn’t something you configure. The engine decides per field, based on the values it sees, so each column is encoded for the data it actually holds instead of one default applied to every field. Pure columnar systems have long carried cardinality in their type system, but usually as something you declare, and you own the consequences when the data shifts underneath it. Here, it’s the engine's job.

Why every field builds an inverted index by default

By default, a keyword field also builds an inverted index and a numeric field also builds a BKD tree. That happens on every field because at write time the engine doesn’t know which capability you’ll want at read time, significantly increasing the footprint of each field. Those structures also have to be rebuilt during segment merges, which costs CPU exactly when ingest is heaviest.

Our time series engine (TSDB) is proof of what happens when you stop paying for capability that the workload doesn’t use. Replacing the indices on @timestamp and dimension fields with doc value skippers, which are sparse structures holding the minimum and maximum value for each block of documents, removed 10 bytes of the original 25 bytes per OpenTelemetry (OTel) data point. There was no measurable query regression on time range and dimension filters, and indexing CPU dropped by about 10%, as a bonus.

Columnar Mode generalizes that default. Fields aren’t indexed unless something needs them to be, with only text-mapped fields keeping their inverted index for fast free-text search.

Metadata fields like _id and _routing were row-shaped

The fields you never think about followed the same document-first design. The _id field was a stored field plus an inverted index. Custom _routing was a stored field. Sequence numbers were kept for optimistic concurrency control, regardless of whether a workload ever updated a document.

TSDB deals with all three. It synthesizes _id from the _tsid and @timestamp values that already identify a data point, using a segment-level bloom filter to catch duplicates, which removes 5 bytes per data point with no loss of functionality. It trims sequence numbers once replication no longer needs them, which removes 4 bytes. Add a codec block size increase from 128 to 512 elements for another 2 bytes, and those four changes contribute across versions 9.1 through 9.4 to the 21 bytes that took OTel metrics from 25 bytes per data point down to 3.75.

Columnar Mode makes those ideas general rather than metrics-specific. By general availability (GA), all metadata fields will store themselves as doc values, while we plan to follow up and add a sort id mode synthesizing the identifier from the index sort fields, in addition to derived fields that will generalize what _tsid does for time series to any set of fields.

Columnar query execution in the ES|QL compute engine

A column store only pays off if the engine reads it as columns. Aggregations inherited the document-at-a-time shape from search, which is the natural fit for an engine built around documents. Reading columns instead lets the engine hand a whole block of values to a single instruction, and that’s where the numbers below come from.

The ES|QL compute engine changed that shape, and TSDB again shows the size of the effect:

• Vectorized execution of time series aggregations was worth up to 8x on its own.

• Decoding on-disk data straight into the primitive arrays the engine aggregates over, with no intermediate copies, was worth roughly another 10x.

• Constant blocks turned repeated values into a form of in-memory run-length encoding.

• Filter pushdown moved filters down to Lucene, where skippers can discard whole blocks unopened.

Together with the rest of the block-level query work, query latency improved by up to 160x compared to earlier versions.

That work continues. Skipper-aware operators, aggregations that group on ordinals and convert to real values as late as possible, and richer per-block summaries are all in progress, and they benefit every index mode because every mode reads doc values underneath.

What Columnar Mode changes for logs and analytical data

Storing values in columns is a storage detail. A columnar database needs five things:

1. The columns are the only copy of the data.

2. Each column is encoded for the data it actually holds.

3. Metadata is columnar, too.

4. Fields add indices only when something needs them.

5. The query engine processes blocks of values rather than records or documents.

TSDB reached all five for metrics in Elasticsearch 9.4, which is why the numbers in this post come from metrics rather than from a slide. Columnar Mode applies the same treatment to logs and security telemetry, along with analytical data. It’s in technical preview in Elasticsearch 9.5, with GA targeted for 9.7.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

The changes are about storage and performance, along with the out-of-the-box (OOTB) experience. Up until 9.5.0, Elasticsearch operated as a document-based search engine by default. It could be set up to behave like a columnar system storage-wise, but that wasn’t the OOTB experience. Columnar index modes make a number of fundamental changes that allow Elasticsearch to optimize columnar analytic and search use cases:

• Fields are stored once as doc values only and are no longer indexed by default.

• New multi-value semantics. The original ordering of multiple values per field per document (for example, in arrays) is preserved by default.

• Mappings are always flat, and object and passthrough fields in mappings are always auto-flattened.

How columnar index modes fit into Elasticsearch

Many of the columnar index mode changes originate from time series data streams (TSDSs). As part of making TSDB a competitive metrics solution, we improved doc values format on disk and only store dimensions and metric fields once as doc values. We also improved query performance. TSDB is already columnar today. Essentially, this makes TSDB’s storage mode columnar. The lessons learned from TSDB are now being applied more broadly to Elasticsearch. 

Note that columnar index modes are opt-in and columnar, and document-based indices can coexist in the same cluster. An enterprise search use case can use a document-oriented index mode, while a logging use case can use logsdb_columnar index mode and be fully columnar, all in the same cluster. In fact, there are currently seven index modes, and indices can all use them in the same cluster.

How columnar storage stays fast without an inverted index

Indexed fields, either an inverted index for string-based fields or block k-dimensional (BKD) tree for numeric fields, allow Elasticsearch to query or filter by field very efficiently. However, the cost for this is an additional expensive data structure that uses a lot of disk space and is expensive to build at index time and at merge time. With the columnar index modes, fields are no longer indexed by default, so what did we do for query performance to be still acceptable on fields that were no longer indexed?

One major change was improving doc values scanning performance. This is key and is the cornerstone that any columnar system relies on. Previously, the scanning of doc values was essentially document by document. Lucene’s doc values API only allowed for looking up one value at a time. Historically, this fit the execution model of a search engine. In our own doc value format, we build the capability to allow bulk reading of values for Elasticsearch Query Language (ES|QL) queries. Also over recent minor Lucene releases, Lucene doc values API added support for bulk reading. Without this, fast columnar scanning wouldn’t have been possible.

Secondly, we fully adopted doc value skippers, a hierarchical skiplist over doc values. Contrary to an inverted index or a BKD tree, doc values skippers are lightweight data structures. At its core, a skipper allows queries to skip over a range of documents that don’t match a query. It can do this because it stores information, like min and max values. So, for example, when a range query is executed, an interval of documents can be skipped based on the intermediate result and a doc value skipper’s min and max values. The effectiveness of doc value skippers depends on the order in which documents are laid out on disk. This is why index sorting should be enabled or altered to match the use case.

By significantly improving our columnar scanning and doubling down on doc values skippers, we’re able to avoid indexing fields by default. Note that a field can still be indexed; the index mapping attribute just defaults to false in columnar mode. The exceptions to this rule are text-based fields, which are still indexed by default. This is because text fields provide free text search, which includes text analysis, along with phrase and wildcard matching. This is different from just filtering.

How Elasticsearch handles high and low cardinality fields

When setting up a schema with string fields, an important configuration parameter is often cardinality; that is, whether many unique values or a few unique string values are expected. Many systems have dedicated field or column types that target low and high cardinality string fields.

Fields that have low cardinality are typically stored with a dictionary, containing all unique values. Then, for each row offset, the offset into the dictionary containing the term the row has is stored. This is often called an ordinal. For low cardinality fields, this works well, as storing an ordinal per document takes up much less space. Encoding techniques, like delta encoding, offset encoding, and bitpacking, work well for ordinals to compact the per-document storage to just a few bits. 

However, for high cardinality fields, the dictionary and ordinal approach can work counterintuitively. If a larger percentage of the documents have a unique value, building the dictionary becomes expensive and storage savings diminish. The dictionary then becomes another level of indirection for reading values. This is why most systems in that case store values in a columnar fashion using block-based compression. For example, values of multiple rows are stored in 128KB blocks using a sliding-window dictionary-based compression algorithm (like zstandard or lz4). This, in general, is a simple and effective method to store higher cardinality fields and avoids building and maintaining a dictionary. 

With document-based Elasticsearch, there are two ways to map a string: using either the keyword field mapping or one of the text-based field mappings. The former stores an inverted index and dictionary-based doc values. The latter only stores an inverted index. This is why, typically, a text field mapper is often used in combination with a keyword mapper as a multi-field. Also, keyword field mapper (as the name suggests) is meant for keywords or fields that have a lower cardinality and uses dictionary-based doc values implementation. However, in practice, keyword field mappers are also used for high cardinality.

In columnar mode, every field is only stored once by default. For keyword fields, this means only doc values are stored with no inverted index. Text-based fields now also store doc values and an inverted index by default. Text-based field mappers are different from the keyword field mapper, as these are not automatically used via Elasticsearch’s dynamic mapping logic for columnar indices, and therefore text fields keep storing an inverted index by default.

For both keyword- and text-based fields, we didn’t choose to expose a cardinality mapping attribute. It’s not always possible to know ahead of time whether a field is low or high cardinality. When flushing and merging segments to disk, Elasticsearch sees all values and can determine the cardinality of a field. This is why we’re choosing to automatically determine whether the usage of a dictionary and ordinal-based encoding is beneficial over block-based compression using a simple cardinality threshold. If a field is below this threshold, dictionary and ordinal-based encoding is used; otherwise block-based compression is used. This simplifies configuration of the mappings and makes it possible to automatically optimize storage as data evolves, since some segments may use dictionaries while others may use blocks for the same field. However, this is currently not ready yet and so, as part of 9.5.0, in columnar mode, both keyword- and text-based field mappers store values in doc values in a block-based compressed layout on disk.

The two approaches compare as follows:

Dictionary and ordinal encoding

Block-based compression

Suits

Low cardinality fields

High cardinality fields

What’s stored

A dictionary of unique values, plus one ordinal per document

Values for many documents compressed together in blocks

Compression

Delta encoding, offset encoding, and bitpacking reduce each ordinal to a few bits

Sliding-window dictionary compression, such as zstandard or lz4, typically over 128KB blocks

Read path

Resolve the ordinal, and then look up the value in the dictionary

Decompress the block, and then read the value directly

Cost as cardinality rises

Dictionary grows large, savings shrink, and the extra indirection stays

Stable, with no dictionary to build or maintain

Used in columnar mode tech preview

Not yet

Yes, for both keyword and text fields

Columnar mapping attributes: multi_value, nullability, on_failure

The columnar index modes provide more control over how data is stored as doc values. By default, Elasticsearch is lenient and accepts all non-malformed values (for example, nulls and multiple values per field and document). If documents have fields with multiple values per document or no value, doc values store additional data structures to deal with them and therefore implicitly increase costs. 

With columnar, new mapping attributes provide additional control. Note that these new mapping attributes are currently only available with the columnar index modes but will eventually also be available for all index modes.

Three new mapping attributes are involved:

Attribute

Default

Enforces

On violation

Available

multi_value

true

One value per document per field

Document indexing fails

9.5.0

nullability

true

Field must have a value

Document indexing fails

9.5.0

on_failure

fail

How the above failures are handled

Sets fail or ignore behavior

Next minor release

multi_value: Enforcing single-valued fields

By default, Elasticsearch accepts multiple values per document. To understand the implications of this, we first should take a look at how Elasticsearch (using Lucene’s doc values) stores a dense numeric field where all documents have a single value:

With this layout, all values are stored in blocks. The number of values per block depends on the index mode but is typically 128 values and is always the same within an index. All values in a block are encoded using various encoding techniques, like delta encoding and bit packing, so each block can have a different size, depending on how well the encoding techniques compress the values. This is why a block index is required. 

Lucene has the notion of a docid (internal numbering for a document), which is essentially a row identifier.

1. Queries produce matching docids.

2. In case of a dense field, the block id can be resolved from the docid directly.

3. The offset of a block can be resolved from the block index.

4. Once that has been looked up, the target block gets decoded and all values are available.

5. Finally, from docid, the ordinal within the decoded values array can be resolved, which produces the final value.

Now let’s have a look at how the data layout changes when documents have multiple values per document:

To determine how many values belong to a single docid, an offset lookup is required.

In this case, a docid has one or more offsets. Each offset points to a block index. Values for a single document are adjacent but can stretch over blocks. In general, compaction of values works well in blocks because values are similar. However, multi-value fields can cause the compaction of values to be less efficient if the number of values per field and document is large and values aren’t similar. This and the additional storage of offsets result in multi-value fields typically having a higher storage footprint on disk.

If a field is truly single-valued, you may want to enforce this property. The multi_value mapping attribute makes that possible now. An example is a log level field. Logs typically have one log level (such as debug, info, or error). Enforcing that this field is single-valued in your mappings can help avoid accidentally using more storage than anticipated. A mapping snippet example that disallows the field log.level to have multiple values per document:

Note that even if a field allows multiple values, this doesn’t mean an offset lookup is stored. This only happens when a Lucene segment has at least one document with two or more values. The multi_value mapping attribute exists just for enforcement.

nullability: Requiring every document to have a value

By default, Elasticsearch accepts documents with fields that have no value or null value. Just as  multiple values per document require additional accounting, documents with no value require additional accounting to identify which of them have at least one value.

Doc values store a docid to offset lookup (known as IndexedDISI in Lucene) in case not all documents have a value in a segment. The offset either points directly to the block index for single-valued fields or the offset lookup in case of multi-valued fields. This lookup is compact compared to the value blocks being stored. However, if it were to be created for fields that should have at least one value per document, that would be a waste.

The nullability mapping attribute allows you to control whether documents are allowed to have no value. Just like the multi_value mapping attribute, the nullability attribute exists for enforcement.  Following is a mapping snippet example that requires the log.level field to have a value:

on_failure: What happens when validation fails

What happens if a document has multiple values for a field and if the multi_value mapping attribute is set to false or when a field is mapped with nullability set to false and a document doesn’t have that field? At the moment, indexing such documents will fail with a bad request error.

As part of the next minor release, the on_failure mapping attribute will be available. This allows you to indicate how to handle these validation failures, on a per-mapped field basis. This will support two values:

1. Fail: Fail indexing of the entire document with a client error. This is the current behavior in Elasticsearch 9.5.0.

2. Ignore: Ignore the validation error, mark the field as ignored, and store values for that field in a hidden field so that it can be introspected when requesting the source. 

Trying the columnar index modes

The columnar index modes are still under active development, but we encourage you to give them a test drive. As we prepare the columnar index modes for general availability (GA), we’ll add more performance and efficiency improvements. We believe that by adapting a columnar mindset, many use cases will benefit from being more cost effective or having better performance characteristics.

Columnar mode is Elasticsearch's analytics-optimized columnar storage mode, built for scan-heavy workloads, like log analytics. In this mode, keyword fields don't get an inverted index by default, so term and wildcard queries scan doc values. DocValuesSkippers (zone maps) already trim how much data a scan touches, but these rewrites cut the cost of what's left. 

How Lucene's query rewrite mechanism works

In Lucene, every query has the option to implement a rewrite method that returns another query. This method returns a query with the same semantics but a different implementation. The query engine repeatedly calls the rewrite method until the returned query doesn’t change. This final query is the one that’s actually evaluated. Importantly, the rewrite can see the actual query arguments and specialize the implementation based on these.

For example, in a query looking for documents where a string field contains the value "foo", the rewrite method knows that the term we’re searching for is "foo". In theory, rewrite could replace the general query class with something specific to "foo". For example, the original query class ScanningBinaryDocValuesTermQuery could be replaced with FooQuery. Now this rule probably wouldn't be helpful, but it gives a sense for the level of specialization that’s achievable with rewrite rules.

Rewrite rules and query optimization in database systems

It's worth placing rewrite rules in the larger context of database systems. Lucene and Elasticsearch aren’t the first systems to use transformation rules to optimize queries. Most (or maybe all) database systems use some kind of rule system during query optimization. The most influential rewrite rule system was in IBM's Starburst database. This system's core contribution was extensibility; for example, it was possible to add new data types and storage methods, along with (most importantly to us) optimizer rewrite rules.

Each rule consisted of two parts:

1. A condition function: A predicate determining whether the rule applies to the current query graph.

2. An action function: The transformation that rewrites the query plan into a more optimal form.

A rule engine applied matching rules until a stopping condition was met.

Though Lucene's rewrite method is superficially different from these condition and action functions, it achieves the same goal. It checks whether certain conditions match, and if they do, it applies the rewrite by returning a new query. If conditions don’t match, the rewrite returns this, replacing the query with itself; that is, choosing not to apply the rule.

Why these rules live in Lucene, not the ES|QL query optimizer

Elasticsearch actually contains a separate rewrite rule system within the Elasticsearch Query Language (ES|QL) optimizer. This operates on the high-level structure of a query; for example, doing predicate pushdown to avoid unnecessary computation on documents that will be filtered out. But it’s still useful to have the rule system within Lucene. Since Lucene acts as the storage layer for ES|QL (and classic _search) queries, it’s easier to express rewrites that take advantage of the physical data format in Lucene rather than in a higher-level optimizer.

A query rewrite rule for wildcard queries: Simpler code, no automaton

Wildcard queries support the ? and * operators to match any character once or any character multiple times. These operators can appear any number of times in a wildcard query. As with regexes, to evaluate whether a string matches a wildcard query, we build an automaton from the query string and then use the string bytes to do state transitions through the automaton. This is relatively fast, but if you have to evaluate it for every document, the latency really adds up.

But maybe we don't always have to run an automaton. Consider a query like *foo*. How would you implement this if you were writing a simple query engine to find matching strings in a list of strings? Pretty much every programming language has the tool you want built in: a method that finds a substring within a given string. This function doesn't need a complicated automaton; it probably just consists of a couple of for loops.

Now of course we couldn't use this function to implement an arbitrary wildcard query, but we don't have to. The rule rewrite system isn't for the general form. It's for implementing special cases, and it can see the specific query. It knows that we’re looking for *foo* and realizes that this specific case doesn't require the heavyweight automaton machinery. And it can do the same for any query that starts and ends with a *, with some term in the middle.

The following pseudo-code shows the pattern. At the top, we have the generic WildcardQuery. It has two notable fields: the query string (for example, *foo*) and the automaton built for that query. The matches method checks whether the field value for a given docId is a match by using it to evaluate the state transitions of the automaton. More interestingly, its rewrite method checks whether the query matches our special case. We show this with a regex that checks whether the query string starts with a *, has any non-*characters at least once, and then ends in a *. If so, we return the special case as a ContainsQuery and pass in the inner query string (since it doesn't care about the *s). The ContainsQuery then just does a simple contains check to see whether the term bytes are somewhere within the value bytes.

Benchmarking the wildcard rewrite on ClickBench Q20

The wildcard rewrite is straightforward, but does it actually work? Yes, we can use the ClickBench benchmark, which has several queries of this form. Query 20 (Q20) is FROM hits | WHERE URL LIKE "*google*" | STATS count = COUNT(*). It's exactly the query shape that this rule matches: a string match against the wildcard query *google*. And since the query is just counting, we can see exactly how well this technique works. It turns out to be quite effective. Q20 saw a 1.75x improvement on median latency of hot query times, with no filter cache. All benchmarks in this post were run on an Intel Core i9-13900H.

Adding SIMD to the substring search: 1.75x to 2.3x

But can we do better? Yes, switching to a simple contains check opens up a new possibility. Instead of using the two for loops, we can swap scalar logic for single instruction, multiple data (SIMD) logic. Elasticsearch uses the Panama vector API (see our post on SIMD in Elasticsearch), which lets us implement the contains check in SIMD. This works particularly well for longer strings that can take advantage of the wide SIMD registers; for strings under 24 characters, we still use the scalar approach. With this change, we saw another 1.32x improvement, resulting in a total speedup of 2.3x over the automaton-based approach.

A query rewrite rule for empty strings: Less data, no decompression

One benefit of Lucene-based rules is that they’re low level and can fit to the data format. That’s the case for this rule, which applies to string data. 

How columnar storage encodes string data

In Elasticsearch's standard mode, string values are stored by document; this is a row-major format. But in columnar mode, unsurprisingly, the data is stored in columnar-major format. A column of string data is stored in chunks. Each chunk contains many string values and consists of an array of integer offsets and a (Zstd-compressed) blob of the strings' bytes. For a string at index i, offsets[i] points to the offset in the decompressed byte blob where the string starts. So the length of string i can be computed from offset[i+1]-offsets[i]. (There's a dummy extra offset at the end, so we can easily compute the length of the last string). The following diagram shows how a chunk with the strings ‘Feta’, ‘Asiago’, ‘’, ‘Stilton’, and ‘Brie’ is encoded.

Why a term query has to decompress the chunk

Now that we understand the columnar format, let's get back to query optimization. First, consider a term query for the query foo. We’re looking for documents where a given string field exactly matches the string foo. So how do we implement this on a string column in the above format? The algorithm is straightforward:

The bottleneck is the Zstd decompression step. But there's not much we can do about that; if we want to check the bytes, we have to decompress the chunks. But remember, we aren't trying to optimize the general case, we’re looking for special cases. (In reality, you don't just try to think up special cases. These optimizations came about by first running a useful query, realizing that it could be faster, and then looking for ways to improve it.)

Rewriting the empty string query as a length check

One special case we found that’s worth improving is a query for the term "". Admittedly, it's a silly term, but empty strings are all over the place. Since they're rarely useful, we usually filter them out with a query like term != "". Thankfully, this is a query we can optimize.

Consider the above algorithm for the empty string term. The line if value == term is a bit weird; we’re asking Does this value equal the empty string? We can do that, but there are no bytes to compare, so the check unwinds:

1. We only need to know whether the value has length 0.

2. If we only need the length, we don't need to look up the value in the decompressed chunk.

3. If we never look up a value, we don't need any bytes from the chunk at all.

4. If we need no bytes from the chunk, we don't need to decompress it.

All we need are the lengths, and those live in the offsets array. It's compressed, too, but with cheap integer compression rather than Zstd, which is much faster.

With this realization, we can rewrite empty string term queries. The one new operation we need is docValues.loadLength(docId), which reads directly from the offset array without touching the compressed bytes. After the previous example, this should look familiar. The most interesting part is TermEqualsQuery.rewrite; it finds the empty string special case and replaces the query with the simpler version that only checks the length.

Benchmarking the empty string rewrite: 1.6x faster

Now let's see how this stacks up. There aren't any pure-scan ClickBench queries that use this rule as directly as Q20 does for the previous rule, so we'll make our own. Consider the query: FROM hits | WHERE SearchPhrase != '' | STATS count(*). On this query, we see a 1.6x speedup, which is a great improvement for a fairly uncomplicated change. Better yet, ES|QL can take advantage of loadLength directly. Any time that ES|QL accesses a string's BYTE_LENGTH, without needing the string itself, the request uses this same specialized length loading to avoid unnecessary decompression.

What makes a good query rewrite rule

The two rules covered here follow the same shape: identify that a query is a special case, and then swap it for a cheaper implementation. But they reduce cost in different ways. 

Wildcard rule

Empty string rule

Query shape detected

*term*

field == ""

Replaced with

SIMD substring search

Length check on the offsets array

Cost reduced

Algorithmic work

Data access

Speedup

2.3x

1.6x

The underlying pattern is worth noting: finding a query that leaves performance on the table, finding a special case that can be optimized, and swapping in a cheaper implementation. The hard parts are finding queries that uncover these opportunities for optimization and then identifying the special cases. The actual fix is often relatively straightforward, as both rules here show. Our work on columnar mode has provided many opportunities to run interesting queries and hunt down exactly these kinds of wins.

That's also why extensibility in a rule system is so important. These rules can't be built into a database from the start; they're found through an incremental discovery process. Lucene's rewrite system makes that practical. As columnar mode grows to handle new workloads, rules like these will keep emerging.

To try columnar mode and the optimizations described in this article, use Elastic Cloud Serverless or Elasticsearch 9.5 or later, where columnar mode is available as a technical preview.

Why dynamic mapping breaks down with schemaless data

Elasticsearch wants a schema. Every field you index has a mapping that defines its type, its analyzer (for text), and how it is stored. That schema makes storage compression efficient, with fast search and cheap aggregation. It becomes a liability the moment your data stops looking like a database table.

Consider the shapes that show up in real systems:

• Log events where every service adds its own attributes. One emits labels.region, another labels.k8s.pod, and another labels.tenant_id.

• OpenTelemetry resource attributes, where the set of keys is defined by whatever agent happened to send the span.

• User-supplied metadata bags, feature flags, or tagging systems, where the keys are open-ended by design.

If you map each key as its own field, the mapping grows unbounded. This is the classic "mapping explosion." Thousands of dynamically created fields inflate the cluster state, slowing down every mapping update and eventually hitting the field limit. Each field also carries overhead in the index. You pay a structural cost for keys you didn’t plan for and may query only once.

The naive escape hatch is to store the whole object as a string and give up on querying its contents. That trades one problem for another: you keep the data but lose the ability to filter or group by anything inside it.

The flattened field type is the better path. You can index an entire JSON object under a single mapped field, keeping the keys queryable with almost none of the mapping-explosion cost. 

This post covers how a flattened field stores the data on a disk and how ES|QL reads it back.

How flattened fields index dynamic JSON keys under one mapping

Map one field as flattened:

Then write arbitrary nested JSON into it:

There is exactly one field in the mapping, labels, no matter how many keys appear across your documents. The cluster state doesn’t grow when a new key shows up. The subkeys remain individually searchable. You can reference labels.region or labels.k8s.pod in queries, even though neither was ever declared.

The catch and central tradeoff is that every leaf value is a keyword. The number 4 above is indexed as the string "4". There is no numeric typing, no date parsing, and no range math on dynamic keys. Flattened fields exchange per-field richness for schema flexibility. That fact explains almost every design decision that follows.

How Elasticsearch stores flattened field JSON keys on disk

There are two types of queries on flattened fields: an unkeyed query on the root flattened field, and a keyed query on a specific subfield. Following the previous example, a query of the form labels: "us-east-1" matches a value under any key, while the querylabels.region: "us-east-1" matches a value under the specific regionkey.

To support these two distinct query formats, the flattened mapper writes each leaf value into two distinct Lucene fields.

Take this document:

The mapper produces:

• A root field under labels, holding the bare values:

• A keyed field under labels._keyed, holding the flattened key concatenated with its value:

In the keyed field, nested objects are dot-flattened into a single key (k8s.pod), and the key is joined to its value with a reserved NULL byte (\0) as the separator. Keys that contain a NULL byte are rejected at parse time, so the separator is always unambiguous. To find the value, you split on the first NULL.

These two fields make both query shapes work:

• labels: "us-east-1" matches a value under any key, so it searches the root field.

• labels.region: "us-east-1" matches a value under a specific key. It rewrites the query to the term region\0us-east-1 and searches the keyed field.

Query restrictions on flattened field subkeys

Because every key's terms live in one sorted list, the keyed field cannot answer every query shape a plain keyword field can. Three restrictions follow:

1. No fuzzy, regexp, or wildcard on a specific subkey. Nothing about the layout makes them impossible,  but the pattern would have to be combined with the key prefix so it can’t walk past the key boundary. The mapper doesn’t do that at this writing.

2. Any range query on a subkey needs at least one bound. Elasticsearch already has a query for "this field has some value here, whatever it is": the exists query. On a flattened subkey it runs as a prefix query on key\0, which sweeps every term belonging to that key. A range with neither bound would sweep exactly the same terms. Rather than support two spellings of one scan, the mapper rejects the boundless range and asks for the exists query.

3. A range query on a subkey needs the field to be indexed. A flattened field can be mapped with index: false, which skips the inverted index and keeps only doc values, the columnar per-document storage covered in the next section. Exact-match queries survive that. With no terms to look up, Elasticsearch scans the doc values column instead, which is slower but gives the same answer. Range queries have no equivalent fallback, so a range on a subkey of an unindexed flattened field throws an Exception.

All three are limits on the Lucene query the mapper is willing to build, and where you notice them depends on how you query.

On the search API, where you name labels.region directly, they come back as errors.

In ES|QL you will not see them as errors at all. There, the same limits decide only whether a predicate is pushed into Lucene or runs in the compute engine on the extracted column. 

This is pushed to a term query on the keyed field:

This is not, so the filter runs per row on the extracted keyword:

Same answer, more work. That distinction is the subject of the second half of this post.

Under the hood: how range queries stay inside key boundaries

This part is internal. You don’t need it to use the field, but it explains where the bounds rule comes from.

For a handful of documents in one segment, the shared term list looks like this:

Each key owns a contiguous slice of that list. For example, all values for key "region" are clustered together in an ordered sublist. A range with both bounds set encodes each bound the same way a term is encoded, so a lower bound of "us-east" on the region key becomes region\0us-east and an upper bound of "us-west" becomes region\0us-west. Both endpoints already carry the key prefix, so the scan can’t leave the region slice. Nothing special is needed.

The half-open case is problematic. Handing Lucene a lower bound of region\0us-east with no upper bound would scan to the end of the term list, straight through tenant_id\0acme and every other key that sorts after region. So the mapper substitutes a sentinel for the missing side:

• A missing lower bound becomes key\0, inclusive. That’s the encoding of the empty value, and it’s the first term in the key's slice.

• A missing upper bound becomes key\1, exclusive. Byte 0x01 is the next byte after the 0x00 separator, so it sorts after every key\0value term and before the first term of any other key.

A one-sided range is therefore boxed into [key\0, key\1), which makes it exactly as safe as a closed one.

The upper sentinel also covers the case where one key is a prefix of another. If an index holds both region and regionx, then region\1 still sorts below regionx\0eu-west-1, because 0x01 is smaller than the x that follows the shared region prefix. A range on region cannot leak into regionx.

How flattened fields use the inverted index and doc values

Each leaf value can be written into two Lucene structures. Both are enabled by default, but can be disabled by the mapping configuration.

• The inverted index (when the field is indexed). Two untokenized keyword terms are indexed per value, one on the root path and one on the keyed path. This powers term, prefix, and range searches.

• Doc values (when doc_values is enabled). A columnar, document-ordered structure. This powers sorting, aggregations, and ES|QL reads.

The inverted index answers the question: "Which documents contain this term?" 

Doc values answer another question: "For this document, what are the values?" Doc values are laid out column by column so a scan touches only the bytes it needs. A flattened field uses both inverted indexes and doc values, so it can serve search and analytics from the same field.

The nature of the index means that the root field is only present in some cases. When the inverted index is disabled by the mapping, any value search requires a linear scan of the doc values for the searched value. In this case, when performing a search on the root field, there isn’t much additional overhead compared to just scanning the keyed field and ignoring the key markers. So the flattened mapper skips writing the root field, relying on the keyed field for both root and keyed queries.

Why flattened fields switched from dictionary to binary doc values

Historically, flattened field doc values used Lucene's SortedSetDocValues, which is a dictionary-compressed format. This means that every key\0value value indexed across all documents per segment is stored in one big sorted, deduplicated set of values. Each document tracks a list of ordinals into that value set.

This dictionary approach provides great compression for low-cardinality fields that tend to have repeated values. It is byte-efficient to store a value only once, and then refer to it by a single integer value. However, there is overhead associated with building and maintaining that dictionary of values, and that overhead is wasted effort when operating on high-cardinality fields that don’t repeat values.

Because flattened fields are a catch-all type, their cardinality tends to be very high. So while the dictionary approach works, it’s not the most compact or scan-friendly layout for this data, especially in time-series indices where flattened bags are common and storage pressure is real.

Recent versions switched the storage to use Lucene’s BinaryDocValues. This format just stores a literal binary blob for each document, which is compressed using Zstandard by our doc values codec when written to disk.

This new binary format provides an additional benefit: it allows us to maintain original array ordering without any overhead. Flattened fields support the mapping parameter preserve_leaf_arrays, which affects how multivalued fields are returned when using synthetic source. When configured to preserve_leaf_arrays: exact, returned values preserve the order, duplicates, and nulls from the original source value.

The dictionary encoding inherent to sorted-set doc values means the returned values are sorted, deduplicated, and de-nulled. To implement preserve_leaf_arrays, flattened fields have traditionally used an additional sidecar field, tracking the required metadata to reconstruct the original source value. However, the nature of binary doc values means that this sidecar field is no longer needed. The values are just stored and returned as indexed.

Limitations of schema on read with flattened fields

The limitations below follow from the keyword-only rule and the shared keyed field:

• No numeric, date, or Boolean typing on dynamic keys. 100 and "100" are the same term.

• No fuzzy, regexp, or wildcard on a specific subkey.

• No multi-fields (fields) or copy_to on the flattened field.

• A depth_limit (default 20) on how deeply nested the object can be.

If you need real typing for a known key, flattened now supports explicitly mapped subfields: you declare individual keys with real types under a properties block, and those keys are indexed by their own typed mapper instead of the keyed field. You get numeric ranges on labels.status_code, while everything else in labels stays dynamic and keyword-only. This is the escape hatch for the handful of keys you actually know about.

Schema on read: querying flattened field JSON keys in ES|QL

Search has supported flattened fields for years. ES|QL, the newer, piped query language built on a columnar compute engine, now reads flattened fields, too. Support began in the Technical Preview of Elasticsearch 9.5.0. It has two distinct pieces.

What ES|QL returns when you select a flattened field

ES|QL uses a dedicated data type for flattened fields, rather than folding it into keyword. When you select the root, you get the whole object back as a JSON string:

Keys come back sorted. You can carry this value through a query, count it, group by it, and run the multi-value and comparison functions on it. But an opaque JSON blob is not usually what you want to filter or aggregate on. For that, you need to reach inside it and process its contents.

How FIELD_EXTRACT reads JSON keys from flattened fields

There is no dotted-path syntax for dynamic keys in ES|QL. You cannot write labels.region for an unmapped key, because to the engine the flattened root is a single leaf value, not a set of columns. Instead you use a function:

The absence of a dotted-path syntax is a tentative limitation. The keyed field already addresses individual subkeys, so a more natural syntax for reaching into a flattened root is something we are planning to support.

FIELD_EXTRACT(field, path) takes a flattened field and a key, and returns a keyword. The rules are easier to see against a document. Take this one:

And this query:

The result is:

Four things to take from this:

• The dot is part of the key, not a navigation operator. The mapper already collapsed the nested object into the flat key k8s.pod, so "k8s.pod is a direct lookup, not a walk from k8s to pod.

• Matching is exact. "k8s" returns null because there is no leaf stored at k8s, only at k8s.pod and k8s.node. For the same reason, "host" will not find "host.name", and matching is case-sensitive, so "Region" will not find "region".

• Arrays come back multi-valued. "tags" yields a multi-valued keyword you can MV_EXPAND, count, or filter on. A missing key yields null.

• Everything is a keyword. "retries" comes back as the string "4", and a Boolean leaf comes back as "true" or "false".

JSONPath syntax is rejected outright, at parse time rather than per row. Both FIELD_EXTRACT(labels, "['k8s.pod']") and FIELD_EXTRACT(labels, "tags[0]") fail with field_extract path must be a literal flattened sub-field name.

Once extracted, the value is an ordinary keyword column. You can filter on it, group by it, sort by it, or use it as the join key in a LOOKUP JOIN:

How ES|QL pushes flattened field predicates into the columnar store

The obvious way to implement FIELD_EXTRACT would be to read the whole flattened root out of storage, render it as a JSON string, hand it to the compute engine, and parse it once per row to pull out a single key. But that means reading every key to use once and paying for a JSON parse on every document. So this obvious implementation is not performant.

ES|QL avoids this whenever possible. Between storage and the compute engine sits the block loader, the step that turns stored data into the columnar blocks the engine operates on. FIELD_EXTRACT hooks into that step instead of running after it.

When ES|QL loads a column, the flattened field type inspects the request. If the request is an extraction of a single constant key and the field has doc values, it routes straight to the keyed doc-values loader, which reads the key\0value entries for only that key out of the columnar structure. The column that arrives at the compute engine already contains only that key's values. The JSON string is never built and never parsed, and the other keys in the object are never read.

When extraction can’t be fused into the block loader, for example, because the key is computed per row or the root is the output of another function such as CASE, ES|QL falls back to the parse-per-row path. The results are identical either way. Only the cost changes.

The comparison can push down further. A predicate like FIELD_EXTRACT(labels, "region") == "us-east-1" can be pushed to Lucene as a term query against the synthetic keyed field, the same region\0us-east-1 term the search path uses. So a filter on an extracted subkey can be answered by the inverted index (if available), and the projection can be answered by doc values, exactly like a first-class field, even though the key was never in the mapping.

Ordering comparisons push down, too. The four, single-sided comparators (>, >=, <, <=) and closed BETWEEN-style ranges all become a range query on that same synthetic keyed field. This is where the key\0 / key\1 sentinels earn their keep: the single-sided forms are only pushable because the mapper can box an open bound inside the key. The pushed range is treated as a candidate, and the predicate is re-evaluated on the extracted keyword column afterwards, so multi-valued keys don’t slip through.

The values are keywords, so the ordering is lexicographic, not numeric. FIELD_EXTRACT(labels, "retries") > "10" compares strings, which means "9" is greater than "10". If you need numeric ranges on a key, map it explicitly under properties, or cast the value in ESQL.

Explicitly mapped subfields behave differently on purpose. Because they carry real types, comparison semantics diverge from the keyword path, so they are loaded and compared through their own typed mapper rather than fused into the keyed loader. And when you select a flattened root that has mapped subfields, ES|QL loads it from _source, so every leaf renders as a string and no keys are dropped silently.

When to use flattened fields vs. dynamic mapping in Elasticsearch

Use flattened fields when:

• The set of keys is open-ended or unknown ahead of time.

• You would otherwise cause a mapping explosion.

• Keyword-level filtering and grouping on the values is enough, and you don’t need numeric or date semantics on the dynamic keys.

• You have a few keys that do need real types. Map those explicitly under properties, and let the rest stay dynamic.

Avoid it, or map fields normally, when the schema is stable and you need full-text analysis, numeric aggregation, or date math across the board.

Remember that flattened is not a dumping ground for JSON you have given up on. It’s a real columnar-and-inverted store for schemaless data, and with ES|QL support, it’s now a first-class analytical citizen. You can keep the messy, unmapped parts of your data messy, and still filter, group, join, and aggregate across them as needed.

Vector search is revolutionizing the way we conduct similarity searches, particularly in fields like AI and machine learning. With the increasing adoption of vector embedding models, the ability to efficiently search through millions of high-dimension vectors becomes critical.

When it comes to powering vector databases, Elastic and OpenSearch have taken notably different approaches. Elastic has invested heavily in optimizing Apache Lucene together with Elasticsearch to elevate them as the top-tier choice for vector search applications. In contrast, OpenSearch has broadened its focus, integrating other vector search implementations and exploring beyond Lucene's scope. Our focus on Lucene is strategic, enabling us to provide highly integrated support in our version of Elasticsearch, resulting in an enhanced feature set where each component complements and amplifies the capabilities of the other.

This blog presents a detailed comparison between Elasticsearch 8.14 and OpenSearch 2.14 accounting for different configurations and vector engines. In this performance analysis, Elasticsearch proved to be the superior platform for vector search operations, and upcoming features will widen the differences even more significantly. When pitted against OpenSearch, it excelled in every benchmark track — offering 2x to 12x faster performance on average. This was across scenarios using varying vector amounts and dimensions including so_vector (2M vectors, 768D), openai_vector (2.5M vectors, 1536D), and dense_vector (10M vectors, 96D), all available in this repository alongside the Terraform scripts for provisioning all the required infrastructure on Google Cloud and Kubernetes manifests for running the tests.

The results detailed in this blog complement the results from a previously published and third-party validated study that shows Elasticsearch is 40%–140% faster than OpenSearch for the most common search analytics operations: Text Querying, Sort, Range, Date Histogram and Terms filtering. Now we can add another differentiator: Vector Search.

Up to 12x faster out-of-the-box

Our focused benchmarks across the four vector data sets involved both Approximate KNN and Exact KNN searches, considering different sizes, dimensions and configurations, totaling 40.189.820 uncached search requests. The results: Elasticsearch is up to 12x faster than OpenSearch for vector search and therefore requires fewer computational resources.

Figure 1: Grouped tasks for ANN and Exact KNN across different combinations in Elasticsearch and OpenSearch.

The groups like knn-10-100 means KNN search with and . In HNSW vector search, determines the number of nearest neighbors to retrieve for a query vector. It specifies how many similar vectors to find as a result. sets the number of candidate vectors to retrieve at each segment. More candidates can enhance accuracy but require greater computational resources.

We also tested with different quantization techniques and leveraged engine-specific optimizations, the detailed results for each track, task and vector engine are available below.

Exact KNN and Approximate KNN

When dealing with varying data sets and use cases, the right approach for vector search will differ. In this blog all tasks stated as knn-* like knn-10-100 use Approximate KNN and script-score-* refer to Exact KNN, but what is the difference between them, and why are they important?

In essence, if you're handling more substantial data sets, the preferred method is the Approximate K-Nearest Neighbor (ANN) due to its superior scalability. For more modest data sets that may require a filtration process, Exact KNN method is ideal.

Exact KNN uses a brute-force method, calculating the distance between one vector and every other vector in the data set. It then ranks these distances to find the nearest neighbors. While this method ensures an exact match, it suffers from scalability challenges for large, high-dimensional data sets. However, there are many cases in which Exact KNN is needed:

• Rescoring: In scenarios involving lexical or semantic searches followed by vector-based rescoring, Exact KNN is essential. For example, in a product search engine, initial search results can be filtered based on textual queries (e.g., keywords, categories), and then vectors associated with the filtered items are used for a more accurate similarity assessment.

• Personalization: When dealing with a large number of users, each represented by a relatively small number (like 1 million) of distinct vectors, sorting the index by user-specific metadata (e.g., user_id) and brute-force scoring with vectors becomes efficient. This approach allows for personalized recommendations or content delivery based on precise vector comparisons tailored to individual user preferences.

Exact KNN therefore ensures that the final ranking and recommendations based on vector similarity are precise and tailored to user preferences.

Approximate KNN (or ANN) on the other hand employs methods to make data searching faster and more efficient than Exact KNN, especially in large, high-dimensional data sets. Instead of a brute-force approach, which measures the exact nearest distance between a query and all points leading to computation and scaling challenges, ANN uses certain techniques to efficiently restructure the indexes and dimensions of searchable vectors in the data set. While this may cause a slight inaccuracy, it significantly boosts the speed of the search process, making it an effective alternative for dealing with large data sets.

In this blog all tasks stated as knn-* like knn-10-100 use Approximate KNN and script-score-* refer to Exact KNN.

Testing methodology

While Elasticsearch and OpenSearch are similar in terms of API for BM25 search operations, since the latter is a fork of the former, it is not the case for Vector Search, which was introduced after the fork. OpenSearch took a different approach than Elasticsearch when it comes to algorithms, by introducing two other engines — nmslib and faiss — apart from lucene, each with their specific configurations and limitations (e.g., nmslib in OpenSearch does not allow for filters, an essential feature for many use cases).

All three engines use the Hierarchical Navigable Small World (HNSW) algorithm, which is efficient for approximate nearest neighbor search, and especially powerful when dealing with high-dimensional data. It's important to note that faiss also supports a second algorithm, ivf, but since it requires pre-training on the data set, we are going to focus solely on HNSW. The core idea of HNSW is to organize the data into multiple layers of connected graphs, with each layer representing a different granularity of the data set. The search begins at the top layer with the coarsest view and progresses down to finer and finer layers until reaching the base level.

Both search engines were tested under identical conditions in a controlled environment to ensure fair testing grounds. The method applied is similar to this previously published performance comparison, with dedicated node pools for Elasticsearch, OpenSearch, and Rally. The terraform script is available (alongside all sources) to provision a Kubernetes cluster with:

• 1 Node pool for Elasticsearch with 3 e2-standard-32 machines (128GB RAM and 32 CPUs)

• 1 Node pool for OpenSearch with 3 e2-standard-32 machines (128GB RAM and 32 CPUs)

• 1 Node pool for Rally with 2 t2a-standard-16 machines (64GB RAM and 16 CPUs)

Each "track" (or test) ran for 10 times for each configuration, which included different engines, different configurations and different vector types. The tracks have tasks that repeat between 1000 and 10000 times, depending on the track. If one of the tasks in a track failed for instance due to a network timeout, then all tasks were discarded, so all results represent tracks that started and finished without problems. All test results are statistically validated, ensuring that improvements aren’t coincidental.

Detailed findings

Why compare using the 99th percentile and not the average latency? Consider a hypothetical example of average house prices in a certain neighborhood. The average price may indicate an expensive area, but on closer inspection, it may turn out that most homes are valued much lower, with only a few luxury properties inflating the average figure. This illustrates how the average price can fail to accurately represent the full spectrum of house values in the area. This is akin to examining response times, where the average can conceal critical issues.

Tasks

• Approximate KNN with k:10 n:50

• Approximate KNN with k:10 n:100

• Approximate KNN with k:100 n:1000

• Approximate KNN with k:10 n:50 and keyword filters

• Approximate KNN with k:10 n:100 and keyword filters

• Approximate KNN with k:100 n:1000 and keyword filters

• Approximate KNN with k:10 n:100 in conjunction with indexing

• Exact KNN (script score)

Vector engines

• lucene in Elasticsearch and OpenSearch, both on version 9.10

• faiss in OpenSearch

• nmslib in OpenSearch

Vector types

• hnsw in Elasticsearch and OpenSearch

• int8_hnsw in Elasticsearch (HNSW with automatic 8 bit quantization: link)

• sq_fp16 hnsw in OpenSearch (HNSW with automatic 16 bit quantization: link)

Out-of-the-box and Concurrent Segment Search

As you probably know, Lucene is a highly performant text search engine library written in Java that serves as the backbone for many search platforms like Elasticsearch, OpenSearch, and Solr. At its core, Lucene organizes data into segments, which are essentially self-contained indices that allow Lucene to execute searches more efficiently. So when you issue a search to any Lucene-based search engine, your search will end up being executed in those segments, either sequentially or in parallel.

OpenSearch introduced concurrent segment search as an optional flag, and does not use it by default, you must enable it using a special index setting index.search.concurrent_segment_search.enabled as detailed here, with some limitations.

Elasticsearch on the other hand searches on segments concurrently out-of-the-box, therefore the comparisons we make in this blog will take into consideration, on top of the different vector engines and vector types, also the different configurations:

• Elasticsearch ootb: Elasticsearch out-of-the-box, with concurrent segment search;

• OpenSearch ootb: without concurrent segment search enabled;

• OpenSearch css: with concurrent segment search enabled

Now, let’s dive into some detailed results for each vector data set tested:

2.5 million vectors, 1536 dimensions (openai_vector)

Starting with the simplest track, but also the largest in terms of dimensions, openai_vector - which uses the NQ data set enriched with embeddings generated using OpenAI's text-embedding-ada-002 model. It is the simplest since it tests only Approximate KNN and has only 5 tasks. It tests in standalone (without indexing) as well as alongside indexing, and using a single client and 8 simultaneous clients.

Tasks

• standalone-search-knn-10-100-multiple-clients: searching on 2.5 million vectors with 8 clients simultaneously, k: 10 and n:100

• standalone-search-knn-100-1000-multiple-clients: searching on 2.5 million vectors with 8 clients simultaneously, k: 100 and n:1000

• standalone-search-knn-10-100-single-client: searching on 2.5 million vectors with a single client, k: 10 and n:100

• standalone-search-knn-100-1000-single-client: searching on 2.5 million vectors with a single client, k: 100 and n:1000

• parallel-documents-indexing-search-knn-10-100: searching on 2.5 million vectors while also indexing additional 100000 documents, k:10 and n:100

The averaged p99 performance is outlined below:

Here we observed that Elasticsearch is between 3x-8x faster than OpenSearch when performing vector search alongside indexing (i.e. read+write) with :10 and :100 and 2x-3x faster without indexing for the same k and n. For :100 and :1000 (standalone-search-knn-100-1000-single-client and standalone-search-knn-100-1000-multiple-clients Elasticsearch is 2x to 7x faster than OpenSearch, on average.

The detailed results show the exact cases and vector engines compared:

Recall

knn-recall-10-100

knn-recall-100-1000

Elasticsearch-8.14.0@lucene-hnsw

0.969485

0.995138

Elasticsearch-8.14.0@lucene-int8_hnsw

0.781445

0.784817

OpenSearch-2.14.0@lucene-hnsw

0.96519

0.995422

OpenSearch-2.14.0@faiss

0.984154

0.98049

OpenSearch-2.14.0@faiss-sq_fp16

0.980012

0.97721

OpenSearch-2.14.0@nmslib

0.982532

0.99832

10 million vectors, 96 dimensions (dense_vector)

In dense_vector with 10M vectors and 96 dimensions. It is based on the Yandex DEEP1B image data set. The data set is created from the first 10 million vectors of the "sample data" file called learn.350M.fbin. The search operations use vectors from the "query data" file query.public.10K.fbin.

Both Elasticsearch and OpenSearch perform very well on this data set, especially after a force merge which is usually done on read-only indices and it’s similar to defragmenting the index to have a single "table" to search on.

Tasks

Each task warms up for 100 requests and then 1000 requests are measured

• knn-search-10-100: searching on 10 million vectors, k: 10 and n:100

• knn-search-100-1000: searching on 10 million vectors, k: 100 and n:1000

• knn-search-10-100-force-merge: searching on 10 million vectors after a force merge, k: 10 and n:100

• knn-search-100-1000-force-merge: searching on 10 million vectors after a force merge, k: 100 and n:1000

• knn-search-100-1000-concurrent-with-indexing: searching on 10 million vectors while also updating 5% of the data set, k: 100 and n:1000

• script-score-query: Exact KNN search of 2000 specific vectors.

Both Elasticsearch and OpenSearch performed well for Approximate KNN. When the index is merged (i.e. has just a single segment) in knn-search-100-1000-force-merge and knn-search-10-100-force-merge, OpenSearch performs better than the others when using nmslib and faiss, even though they are all around 15ms and all very close.

However, when the index has multiple segments (a typical situation where an index receives updates to its documents) in knn-search-10-100 and knn-search-100-1000, Elasticsearch keeps the latency in about ~7ms and ~16ms, while all other OpenSearch engines are slower.

Also when the index is being searched and written to at the same time (knn-search-100-1000-concurrent-with-indexing), Elasticsearch maintains the latency below 15ms (at 13.8ms), being almost 4x faster than OpenSearch out-of-the-box (49.3ms) and still faster when concurrent segment search is enabled (17.9ms), but too close to be significative.

As for Exact KNN, the difference is much larger: Elasticsearch is 6x faster than OpenSearch (~260ms vs ~1600ms).

Recall

knn-recall-10-100

knn-recall-100-1000

Elasticsearch-8.14.0@lucene-hnsw

0.969843

0.996577

Elasticsearch-8.14.0@lucene-int8_hnsw

0.775458

0.840254

OpenSearch-2.14.0@lucene-hnsw

0.971333

0.996747

OpenSearch-2.14.0@faiss

0.9704

0.914755

OpenSearch-2.14.0@faiss-sq_fp16

0.968025

0.913862

OpenSearch-2.14.0@nmslib

0.9674

0.910303

2 million vectors, 768 dimensions (so_vector)

This track, so_vector, is derived from a dump of StackOverflow posts downloaded on April, 21st 2022. It only contains question documents — all documents representing answers have been removed. Each question title was encoded into a vector using the sentence transformer model multi-qa-mpnet-base-cos-v1. This data set contains the first 2 million questions.

Unlike the previous track, each document here contains other fields besides vectors to support testing features like Approximate KNN with filtering and hybrid search. nmslib for OpenSearch is notably absent in this test since it does not support filters.

Tasks

Each task warms up for 100 requests and then 100 requests are measured. Note the tasks were grouped for sake of simplicity, since the test contains 16 search types * 2 different k values * 3 different n values.

• knn-10-50: searching on 2 million vectors without filters, k:10 and n:50

• knn-10-50-filtered: searching on 2 million vectors with filters, k:10 and n:50

• knn-10-50-after-force-merge: searching on 2 million vectors with filters and after a force merge, k:10 and n:50

• knn-10-100: searching on 2 million vectors without filters, k:10 and n:100

• knn-10-100-filtered: searching on 2 million vectors with filters, k:10 and n:100

• knn-10-100-after-force-merge: searching on 2 million vectors with filters and after a force merge, k:10 and n:100

• knn-100-1000: searching on 2 million vectors without filters, k:100 and n:1000

• knn-100-1000-filtered: searching on 2 million vectors with filters, k:100 and n:1000

• knn-100-1000-after-force-merge: searching on 2 million vectors with filters and after a force merge, k:100 and n:1000

• exact-knn: Exact KNN search with and without filters.

Elasticsearch is consistently faster than OpenSearch out-of-the-box on this test, only in two cases OpenSearch is faster, and not by much (knn-10-100 and knn-100-1000). Tasks involving knn-10-50, knn-10-100 and knn-100-1000 in combination with filters show a difference of up to 7x (112ms vs 803ms).

The performance of both solutions seems to even out after a "force merge", understandably, as evidenced by knn-10-50-after-force-merge, knn-10-100-after-force-merge and knn-100-1000-after-force-merge. On those tasks faiss is faster.

The performance for Exact KNN once again is very different, Elasticsearch being 13 times faster than OpenSearch this time (~385ms vs ~5262ms).

Recall

knn-recall-10-100

knn-recall-100-1000

knn-recall-10-50

Elasticsearch-8.14.0@lucene-hnsw

1

1

1

Elasticsearch-8.14.0@lucene-int8_hnsw

1

0.986667

1

OpenSearch-2.14.0@lucene-hnsw

1

1

1

OpenSearch-2.14.0@faiss

1

1

1

OpenSearch-2.14.0@faiss-sq_fp16

1

1

1

OpenSearch-2.14.0@nmslib

0.9674

0.910303

0.976394

Elasticsearch and Lucene as clear victors

At Elastic, we are relentlessly innovating Apache Lucene and Elasticsearch to ensure we are able to provide the premier vector database for search and retrieval use cases, including RAG (Retrieval Augmented Generation). Our recent advancements have dramatically increased performance, making vector search faster and more space efficient than before, building upon the gains from Lucene 9.10. This blog presented a study that shows when comparing up-to-date versions Elasticsearch is up to 12 times faster than OpenSearch.

It's worth noting both products use the same version of Lucene (Elasticsearch 8.14 Release Notes and OpenSearch 2.14 Release Notes).

The pace of innovation at Elastic will deliver even more not only for our on-premises and Elastic Cloud customers but those using our stateless platform. Features like support for scalar quantization to int4 will be offered with rigorous testing to ensure customers can utilize these techniques without a significant drop in recall, similar to our testing for int8.

Vector search efficiency is becoming a non-negotiable feature in modern search engines due to the proliferation of AI and machine learning applications. For organizations looking for a powerful search engine capable of keeping up with the demands of high-volume, high-complexity vector data, Elasticsearch is the definitive answer.

Whether expanding an established platform or initiating new projects, integrating Elasticsearch for vector search needs is a strategic move that will yield tangible, long-term benefits. With its proven performance advantage, Elasticsearch is poised to underpin the next wave of innovations in search.

So, why not allow non-normalized vectors in dot-product and thus enable maximum-inner-product? What's the big deal?

Negative values and Lucene optimizations

Lucene requires non-negative scores, so that matching one more clause in a disjunctive query can only make the score greater, not lower. This is actually important for dynamic pruning optimizations such as block-max WAND, whose efficiency is largely defeated if some clauses may produce negative scores. How does this requirement affect non-normalized vectors?

In the normalized case, all vectors are on a unit sphere. This allows handling negative scores to be simple scaling.

Figure 1: Two opposite, two dimensional vectors in a 2d unit sphere (e.g. a unit circle). When calculating the dot-product here, the worst it can be is -1 = [1, 0] * [-1, 0]. Lucene accounts for this by adding 1 to the result.

With vectors retaining their magnitude, the range of possible values is unknown.

Figure 2: When calculating the dot-product for these vectors [2, 2] \* [-5, -5] = -20

To allow Lucene to utilize blockMax WAND with non-normalized vectors, we must scale the scores. This is a fairly simple solution. Lucene will scale non-normalize vectors with a simple piecewise function:

Now all negative scores are between 0-1, and all positives are scaled above 1. This still ensures that higher values mean better matches and removes negative scores. Simple enough, but this is not the final hurdle.

The triangle problem

Maximum-inner-product doesn't follow the same rules as of simple euclidean spaces. The simple assumed knowledge of the triangle inequality is abandoned. Unintuitively, a vector is no longer nearest to itself. This can be troubling. Lucene’s underlying index structure for vectors is Hierarchical Navigable Small World (HNSW). This being a graph based algorithm, it might rely on euclidean space assumptions. Or would exploring the graph be too slow in non-euclidean space?

Some research has indicated that a transformation into euclidean space is required for fast search. Others have gone through the trouble of updating their vector storage enforcing transformations into euclidean space.

This caused us to pause and dig deep into some data. The key question is this: does HNSW provide good recall and latency with maximum-inner-product search? While the original HNSW paper and other published research indicate that it does, we needed to do our due diligence.

Experiments and results: Maximum-inner-product in Lucene

The experiments we ran were simple. All of the experiments are over real data sets or slightly modified real data sets. This is vital for benchmarking as modern neural networks create vectors that adhere to specific characteristics (see discussion in section 7.8 of this paper). We measured latency (in milliseconds) vs. recall over non-normalized vectors. Comparing the numbers with the same measurements but with a euclidean space transformation. In each case, the vectors were indexed into Lucene’s HNSW implementation and we measured for 1000 iterations of queries. Three individual cases were considered for each dataset: data inserted ordered by magnitude (lesser to greater), data inserted in a random order, and data inserted in reverse order (greater to lesser).

Here are some results from real datasets from Cohere:

Figure 3: Here are results for the Cohere’s Multilingual model embedding wikipedia articles. Available on HuggingFace. The first 100k documents were indexed and tested.

Figure 4: This is a mixture of Cohere’s English and Japanese embeddings over wikipedia. Both datasets are available on HuggingFace.

We also tested against some synthetic datasets to ensure our rigor. We created a data set with e5-small-v2 and scaled the vector's magnitudes by different statistical distributions. For brevity, I will only show two distributions.

Figure 5: Pareto distribution of magnitudes. A pareto distribution has a “fat tail” meaning there is a portion of the distribution with a much larger magnitude than others.

Figure 6: Gamma distribution of magnitudes. This distribution can have high variance and makes it unique in our experiments.

In all our experiments, the only time where the transformation seemed warranted was the synthetic dataset created with the gamma distribution. Even then, the vectors must be inserted in reverse order, largest magnitudes first, to justify the transformation. These are exceptional cases.

If you want to read about all the experiments, and about all the mistakes and improvements along the way, here is the Lucene Github issue with all the details (and mistakes along the way). Here’s one for open research and development!

Conclusion

This has been quite a journey requiring many investigations to make sure maximum-inner-product can be supported in Lucene. We believe the data speaks for itself. No significant transformations required or significant changes to Lucene. All this work will soon unlock maximum-inner-product support with Elasticsearch and allow models like the ones provided by Cohere to be first class citizens in the Elastic Stack.