In this reranking experiment, Elasticsearch retrieved the products, Jev judged each query/product pair, and a small deterministic policy converted those judgments into the final order. The best Jev policy raised normalized Discounted Cumulative Gain at the 10 results (nDCG@10) from 0.9351 to 0.9565 and Exact MRR from 0.9201 to 0.9616. 

The result is useful, but what’s more interesting is the architecture. Jev returns typed decisions and probabilities, and then the application can decide what relevant means and can weight probabilities to match application goals.

At $0.042 per million input tokens with output free, Jev’s worth testing anywhere that large language model (LLM) reranking would be too slow or too expensive. Elasticsearch still does the retrieval, and a small amount of application code turns Jev's scores into the final ranking.

What are System One models, and how do they differ from LLMs?

Let’s back up, though. We shouldn’t gloss over System One models, which areTypeSafe’s term for a new kind of model. The main difference between them and LLMs is that System One models, like Jev, make decisions rather than generating strings. For example, you could ask Jev whether a snippet answers a user’s question, and it would give a likelihood (using their Noul type). Or you could ask Jev to make a choice among a list of intents for a search query (using the Choice type). Notice that none of these are outputting text.

As a side effect, Jev is fast and cheap. TypeSafe claims 70 to 500ms end-to-end response times and charges $0.042 per million input tokens (output tokens are $0.000 per million tokens, not a typo). With speed and cost like that, we started wondering whether this could work for search in cases where an LLM is too slow and too expensive.

Reranking was one idea we had. First, retrieve documents and let Jev score the result. Then rerank based on those scores.

How we benchmarked ecommerce search reranking with Jev

We evaluated the approach on 250 US-English queries and 4,754 judged query/product pairs from Amazon's public Shopping Queries Dataset. This dataset is useful for a few reasons, but it’s particularly interesting since query latency is especially important for ecommerce searches.

The search flow was implemented like this:

• Elasticsearch retrieved up to 40 candidates. (We measured on both lexical and lexical/semantic hybrid retrieval, both unfiltered.)

• Jev judged each query/product pair.

• Application code calculated a ranking score.

• Candidate scores feed the final ranking.

Prerequisites

• An Elasticsearch deployment with a product index and an inference endpoint for semantic retrieval.

• A Jina reranker endpoint, if you want to reproduce the benchmark comparator. This run used .jina-reranker-v3.

• A TypeSafe API key and a Jev model available to your account. This run used jev-1.13.0.

• Python 3.11 or later for the companion notebook.

Implementing reranking with Elasticsearch and Jev

Hybrid search retrieval with semantic_text and RRF

We kept Elasticsearch in charge of retrieval, with information that was relevant for both search and display kept as separate fields, plus a combined semantic search field using, of course, semantic_text:

The combined field is a literal combination of product fields:

We duplicated text intentionally. The semantic_text field type told Elasticsearch to send the field value to Elastic Inference Service (EIS) at indexing time and to generate and store the embeddings and to chunk the text if necessary. At query time, Elasticsearch again embedded the query with the same EIS endpoint and searched with the embedding.

For the request, we combine results using Elasticsearch’s reciprocal rank fusion (RRF) retriever:

The size here is illustrative: limiting the results to 40, the more expensive second stage has a bounded candidate set.

For the benchmark, however, we didn’t use the 40 results as illustrated above. Instead, the benchmark evaluates reranking separately from retrieval. For each query, the Shopping Queries  Dataset supplies a group of products with human relevance judgments. We gave that same complete product group to every strategy, including BM25, hybrid search, Elasticsearch’s Jina reranker, and the Jev policies, and then we compared how each strategy ordered it. 

These aren’t the same as Elasticsearch’s live top 40 results. Holding the products constant means that differences in the results measure ordering quality, not which products each retrieval method found. This is important, because otherwise the comparison would mix retrieval recall with reranking quality. The test instead asks a narrower question: Given the same products, which method orders them best?

Sending query and product fields to Jev

We tested multiple setups with Jev, across both the Choice and Score types, but each request contained the query and product fields that a human could otherwise use to judge the relevance of a product against a query (meaning, no product ID) and nothing that would unduly influence the model (no Elasticsearch score, judgment, or original position).

Classifying product relevance with Choice and Noul questions

Our first implementation used a TypeSafe Choice over the same four relationships that were defined by the dataset:

(Although we didn’t spend much time on this criteria, there’s likely some room for instruction optimization; for example, by using a tool like optimize_anything.)

In return, Jev returned to us the selected option and a probability for each option, along with a confidence score.

For bts map of the soul 7 and a branded T-shirt, the stored Choice response was genuinely uncertain:

The held-out ESCI label was Exact, which shows something important. The distribution exposes the ambiguity, but Jev isn’t infallible.

We also asked four Noul questions with signals that may be useful in a reranking step:

All four questions went into a single request:

Again, Jev returned scores, not text, so we didn’t need to extract scores from prose or prompt for JSON. For the same bts map of the soul 7 query from above, Jev returned:

Turning Jev scores into a reranking policy

Although Jev made the semantic judgments, the application code decided how to take those judgments and rerank results.

We derived four benchmark strategies from the same Choice response. Giving them names here makes the results easier to follow.

Ranking by exact probability

The Jev exact probability policy ranked each candidate using only the probability that its relationship was Exact:

Ranking by exact probability is the most direct way to optimize for getting an Exact product to the top. It ignores the difference between a likely Substitute, Complement, and Irrelevant result whenever their Exact probabilities are equal.

Ranking by relationship expected utility

The Jev relationship expected utility policy used the complete Choice distribution. It multiplied each relationship probability by an application-defined value and added the results:

The relationship expected utility policy scores each candidate across the complete distribution of choices. The utility weights are our own values, not a standard expressed by the dataset, so they can be tuned and tested (though for this benchmark we didn’t spend much time on the tuning). The goal was to prefer Exact products, allow for Substitutes, and push down Complements and Irrelevant products.

The expected utility policy also separates slam dunk Exact matches from borderline ones. In other words, a candidate that Jev marks as 99% Exact should be higher than one that’s 51% Exact and 49% Substitute.

Composite policy with constraint and compatibility signals

The Jev composite policy started with relationship expected utility and then adjusted it using the auxiliary Noul probabilities for whether the candidate was the requested main item and satisfied explicit constraints, in addition to supporting any requested compatibility:

The score is first adjusted to account for whether the item is the one requested in the query. This isn’t a full discounting if Jev labels the product as being an accessory (the most the score can be discounted is 40%), because that first relationship score still provides an important signal.

Then we discount if the product doesn’t match all of the constraints expressed by the query (for example, red running shoes size 10 has three constraints). We, again, don’t allow for a full discounting of the score; in this case, to guard against a product listing with incomplete metadata dragging the score down.

Finally, the compatibility factor applied only when deterministic code detected compatibility language such as "fits," "works with," or "replacement." For the same reasons as above, this doesn’t fully discount the score.

(We also collected same_core_purpose as a diagnostic signal but didn’t include it in the composite ranking score because it substantially overlaps with the Exact/Substitute/Complement/Irrelevant relationship judgment.)

Direct Score policy: One Jev Score question per product

The fourth strategy, the Jev direct Score policy, used a TypeSafe Score with an ordered relevance rubric:

The direct Score policy is the smallest useful Jev reranker we tested. It performed well, though not as well as the best policy built from explicit relationship judgments.

Benchmark results: Jev reranking vs. text-similarity reranker vs. BM25 and hybrid search

The benchmark used the US-English portion of the Shopping Queries Dataset, usually called ESCI after its Exact, Substitute, Complement, and Irrelevant labels. Each selected query came with a complete official candidate list.

We used 50 queries from the training split for development and then froze the questions, model, endpoints, and scoring policy before running a separate 250-query test sample, which contained 4,754 query/product pairs. (If it seems confusing that we have 250 queries, but we spoke earlier about reranking up to 40 results, it’s because for this benchmark we were limited in the results that were annotated, which averages to around 19 results per query.)

The comparison included BM25, hybrid lexical and semantic retrieval with reciprocal rank fusion, and the four Jev strategies defined above:

• Jev direct Score: The expected value of one ordered Score question.

• Jev exact probability: P(exact) from the relationship Choice.

• Jev relationship expected utility: The weighted value of the complete relationship distribution.

• Jev composite policy: Relationship expected utility adjusted by the auxiliary policy signals.

Ranking quality: nDCG@10 and Exact MRR

The main metric was nDCG@10. It rewards putting higher-value results near the top, with 1.0 representing the ideal ordering for that candidate set. Exact MRR measures how high the first Exact product appeared.

Strategy

Ordinal nDCG@10

Δ vs. hybrid search (95% CI)

Random-to-perfect gap captured

Exact MRR

Jev relationship expected utility

0.9565

+0.0214 [+0.0125, +0.0307]

69.1%

0.9550

Jev direct Score

0.9557

+0.0206 [+0.0106, +0.0308]

68.5%

0.9457

Jev policy composite

0.9555

+0.0204 [+0.0114, +0.0295]

68.3%

0.9559

Jev exact probability

0.9533

+0.0182 [+0.0102, +0.0263]

66.8%

0.9616

Elasticsearch hybrid BM25 + semantic RRF

0.9351

Baseline

53.8%

0.9201

Elasticsearch BM25

0.9234

−0.0117 [−0.0180, −0.0057]

45.5%

0.9015

Random order

0.8595

−0.0756 [−0.0939, −0.0586]

0.0%

0.7426

It’s probably worth discussing something that may have stood out to you: the random ordering is quite high. Why is that? Of the 4,716 judged pairs, 2,960 of them were Exact and only 439 were marked as irrelevant. Under those label distributions, the expected random ordinal nDCG@10 is 0.8634; the observed 0.8595 is ordinary. A follow-up benchmark would test on a judgment list that had a higher distribution of observed irrelevant results.

The best overall result belonged to the composite policy, while the direct Score version also improved nDCG over hybrid retrieval. The exact probability policy did best at Exact MRR, which isn’t a surprise.

Takeaways: Reranking with Jev and multiple signals

There are caveats to this test: it’s just one test, on one dataset, across 250 queries. And, as mentioned before, it does lean toward Exact judgments, or, at least, non-Irrelevant judgments.

But, still, it does show that using Elasticsearch as a retriever and Jev as a scorer is a viable approach to reranking. Also interesting in this benchmark is what we do with those scores. We’re blending them rather than just using the scores directly. This allows us to take into account business needs or simply blend multiple signals. And this all makes sense, because just like we don’t use a single score for textual relevance, ranking benefits from multiple scores, as well.

For new users, this is the fastest way to get high-quality vector search running. If you already use Elasticsearch, the new offering is vector search on the platform where your data already lives, with no new system to adopt. Elasticsearch Vector Database supports a range of scenarios, from grounding a large language model (LLM), to giving an AI agent retrieval and memory, to serving hundreds of billions of vectors. Spin up a new project and get started in minutes.

One engine, every vector use case

Elasticsearch Vector Database is built for anyone building applications using vectors:

• RAG: Retrieve the right context for your LLM with dense and sparse vector retrieval, or go with hybrid search combining both vector and lexical retrieval. The quality of your generation improves with the quality of your retrieval.

• AI agents: Give agents fast, filtered retrieval over documents and conversation memory, with the low latencies that multistep agent loops demand.

• Semantic search: Match on meaning, not keywords, with one field type and zero pipeline code.

• Recommendations and similarity: Find nearest neighbors across products, images, or whatever content you have, at scale.

Everything your vector workload needs, optimized out of the box

Building a vector-based application means wiring together several separate pieces: setting up and hosting embedding models, indexing your documents through them, storing the vectors efficiently, applying the embedding model to each query, matching against the vector store, and finally, retrieving the documents behind the matches. Elasticsearch Vector Database handles all of it for you, with no additional configuration or setup.

Vector indexing with vectordb_document index mode

vectordb_document index mode, a new index configuration purpose-built for vector-first workloads, is on by default, so you get the settings that experts would choose. Here's what it turns on:

• bfloat16 by default: Vectors are stored at half the size of float32 with negligible impact on recall, cutting your disk footprint roughly in half before quantization even enters the picture.

• Source vectors excluded: In Elasticsearch, your embeddings already live in the index structures used for search; keeping a second raw copy in _source just inflates storage and slows down fetching results. We exclude the duplicate so responses return faster and you store less.

• The right files preloaded into cache: The data structures that vector queries touch first are warmed into memory ahead of time, so your first (and your thousandth) query is lightning fast.

• Parallel merging: Merging consolidates segments into better-organized vector structures, which lifts both recall and latency, and running those merges multi-threaded means you get there faster.

Vector storage, compression, and auto-tuning

• Your vectors are compressed automatically. Better Binary Quantization (BBQ) shrinks vector memory footprints by up to 32x while preserving recall, and DiskBBQ reduces memory requirements further for large-scale workloads. 

• Opt in to auto-calibration, which tunes each segment's quantization to your data and retunes on every merge as data drifts. When tested across 18 datasets, queries per second (QPS) improved by an average of 16.7%, with recall gains in most of them.

Embeddings on managed GPU inference

• Generate embeddings with native Jina AI embedding and reranking models, or bring third-party models, all on managed GPUs via Elastic Inference Service (EIS) with no model servers to operate. Or self-host, if you prefer your own.

• The semantic_text field type automatically handles chunking and embedding, along with querying, the simplest path to semantic search in the market. 

Hybrid search and filtered vector search

• Hybrid search is built in, combining full-text and vector retrieval in a single query. Blend the results with reciprocal rank fusion (RRF) or any other blending mechanism you want. Vector search is usually the hardest part of hybrid search to configure well. With Elasticsearch Vector Database, you have it handled, and your whole hybrid stack gets better. 

• With filtered vector search, apply metadata filters as part of vector retrieval itself and not as an afterthought that wrecks recall.

Enterprise on day one

You also get role-based access control (RBAC), audit logging, and the compliance certifications that pure-play vector databases generally lack.

Affordable at scale and predictable

Elasticsearch Vector Database is built to stay affordable as you grow: BBQ and DiskBBQ compression that keeps storage linear and memory low means scaling to hundreds of billions of vectors doesn't blow up your bill. And what you do pay is built from numbers you already know: how much data you store and how much you index, along with how much search capacity you need. Estimate your document count and vector dimensions, plus your query load, and you can work out what you'll pay before you create the project. You can also understand your bill line by line at the end of the month. There are no opaque compute units and no surprise charges for background operations.

How to get started with Elasticsearch Vector Database

Create a serverless vector database project

Create a new serverless Vector Database project in Elastic Cloud. Point your data at the endpoint, and you're ready to index.

Create an index using semantic_text

Vector index mode handles the vector configuration. Using semantic_text means that embeddings and chunking setup are managed for you, as is index setup, on managed GPU inference, with no embedding pipeline to build.

Ingest documents

Index text, and the embeddings are generated for you.

Run a semantic search query

Query the same semantic field you just created:

And you get results back:

Semantic search is just the start. Run fully textual queries or combine both into hybrid queries. You can even craft your own vector queries for full control. Follow the semantic search quickstart in the docs for the full instructions.

What's next for vector search in Elasticsearch

We're already working on the next improvements:

• Better multi-tenant handling: If your data needs to stay separated per tenant, we'll give you a way to do it faster and with less code.

• Automatic index optimization: From "brand new index" to "fully optimized," with as little tinkering as possible.

• Continuous infrastructure improvements: Ongoing tuning of Vector Database's settings and infrastructure so you're always getting the best throughput and fastest responses.

Try Elasticsearch Vector Database on Elastic Cloud Serverless

Go from an empty project to a hybrid, filtered vector query in minutes, with production-grade defaults doing the tuning for you. Build fast, scalable AI apps, not infrastructure.

Start on Elastic Cloud Serverless, or dive into the full documentation and API reference. You can also access the new offering on AWS Marketplace, Google Cloud Marketplace and Microsoft Marketplace.