What you'll discover

In this post, you'll learn how to:

• Use Elastic APM's built-in views to explore search latency and throughput and to explore errors.

• Define Service Level Objective (SLOs) for search, including latency targets and availability, along with search quality.

• Create SLOs in Kibana that track your search health over time, with burn rate alerting.

• Build operational dashboards with Elasticsearch Query Language (ES|QL) that show latency percentiles and time breakdowns and that include trends.

• Set up alerts for latency regressions and error spikes, along with zero-results rate increases.

• Establish an incident response pattern for search degradation.

What you'll need

• Search instrumentation from Blogs 2–4 (search spans with search.* attributes in Elastic).

• Kibana access with permissions to create SLOs and alert rules.

• Basic understanding of SLOs. (We'll explain the search-specific parts.)

• An Elastic cluster with an Enterprise subscription, an Elastic Cloud trial, or a local deployment with the trial activated.

Why search monitoring matters beyond cluster health

Search is the primary navigation path for a significant share of visitors, and search-initiated sessions tend to show stronger purchase intent than browse sessions. A search outage is a revenue event, rather than a minor feature degradation. A latency regression from 100ms to 500ms changes user behavior before anyone files a ticket.

Most platform teams monitor search at the infrastructure level, checking whether the Elasticsearch cluster is healthy and whether nodes are responding. They also determine whether the disk is full. These are all necessary but not sufficient. A cluster can be green while search quality silently degrades; for example, queries returning stale data after a bad index deployment or latency creeping up as the index grows. This could also include zero-results rates climbing because a synonym list wasn't updated.

The gap is between "search is up" and "search is working well."

A note on examples: As we have throughout this series, we use ecommerce search for concrete examples, but these reliability patterns apply equally to any search application, including content platforms, internal knowledge bases, job boards, and support portals.

OpenTelemetry search spans as monitoring signals

If your team followed Blogs 2–4 in this series, every search request already emits an OTel span with search.* attributes. These include the query text, result count, Elasticsearch execution time, and error status. Those spans land in traces-generic.otel-default in Elastic.

Following along with code? The reference project has all the instrumentation from Blogs 2–4. Generate traffic, and then follow along with the SLO and dashboard setup below. See queries/blog6_reliability.esql for ready-to-run queries.

The search team built that instrumentation for product analytics; that is, understanding what users search for and measuring click-through rates (CTRs) and conversion rates. They’ve prioritized relevance work, but the same spans contain everything you need for operational monitoring. Span duration gives you a latency signal, and search.result_count == 0 value reflects quality. Span errors point to availability signals.

This post shows how to put this operational value to work, beginning with what you can see right now in Kibana and then building SLOs and alerting on top of it, along with incident response.

Search monitoring out of the box with Elastic APM

Before building anything new, let's look at what Elastic APM already gives you out of the box.

If your search API is instrumented with Elastic Distribution of OpenTelemetry (EDOT) (as in Blog 2), it appears automatically as a service in the Elastic APM UI. Open Observability > APM > Services in Kibana, and select your search service (named search-analytics-demo if you're using the reference project). You'll immediately see the following:

Elastic APM service overview for search

The service overview page shows latency distribution and throughput over time, along with error rate, and doesn’t require configuration. You can see at a glance whether search is healthy, and the time-series charts make regressions obvious. If latency crept up after yesterday's deployment, you'll see it here.

Trace waterfall: breaking down search request latency

Click into any transaction, and you'll see the trace waterfall, which is a visual breakdown of every span in the request. For a search API call, this typically shows:

The waterfall makes the invisible visible. You can see that the 503ms API response breaks down into HTTP handling, a 241ms query rules lookup, and a 260ms Elasticsearch query,  plus the custom search span (36ms) carrying all of our search.* attributes. Click any span, and the metadata flyout shows exactly what was captured: search.query: "usb hub", search.result_count: 33, search.took_ms: 43, the index name, hit IDs, and more.

Blog 2 discussed the gap between search.took_ms and span duration. The waterfall shows you exactly where that gap lives, without writing any queries.

Automatic search error capture with OpenTelemetry

One of the most valuable things OTel auto-instrumentation gives you is automatic error capture. When an Elasticsearch query fails because of issues like a tripped circuit breaker or a timeout, or if an index isn’t found,  the span records the exception type and message, along with the stack trace. Blog 4 mentioned this as a side benefit of span-based conversion tracking; here it becomes an operational lifeline.

The errors tab on your service page automatically aggregates these, grouped by error type and frequency. The instrumentation captures the details for you, so you don't need custom error handling or logging. During an incident, this is often the fastest way to understand what's actually failing.

Service map: search API and Elasticsearch dependencies

The service map shows dependencies between your search API and Elasticsearch, making it easy to see whether a latency problem is in your service or in the cluster it depends on.

All of this is available the moment you deploy the instrumentation from Blog 2, without building any dashboards or writing any queries. This is the foundation everything else in this post builds on.

Defining service level objectives for search

An SLO defines good enough in measurable terms. You define what working means, measure it continuously, and alert when you're burning through your error budget too fast, instead of reacting when something breaks.

Elastic Observability has a built-in SLO framework that handles Service Level Indicator (SLI) calculation and budget tracking. It also takes care of burn rate alerting. You create SLOs directly in Kibana. No ES|QL or custom pipelines are required for the core indicators.

Three SLOs every search service needs

Navigate to Observability > SLOs in Kibana, and click Create SLO. The SLO creation workflow walks you through three steps: Define the SLI (what to measure), set the objective (the target), and describe the SLO.

1. Search latency SLO: 99% of queries under 250ms

Indicator type: Elastic APM latency target: 99% of searches complete in under 250ms.

The Elastic APM latency indicator is purpose-built for this. Select your search service (search-analytics-demo), and set the threshold to 250ms.  Elastic handles the rest, including calculating the percentage of transactions below the threshold and tracking your error budget over time.

Note: The Elastic APM latency indicator measures all HTTP transactions for the search-analytics-demo service, including health checks and click/cart/checkout endpoints, along with static asset requests, not just the POST /api/search endpoint. For a search-only latency SLO, use a custom Kibana Query Language (KQL) indicator with name: "search" AND attributes.search.query: * on the traces-generic.otel-default index. The Elastic APM indicator is still valuable for whole-service health. For full coverage, combine both.

This measures end-to-end span duration; that is, what the user actually experiences. If your Elasticsearch query takes 50ms but the user waits 300ms because of network overhead or slow application logic, this SLO catches it. Use search.took_ms in the Elastic APM waterfall to diagnose where the latency lives when the SLO starts burning.

2. Search availability SLO: 99.9% success rate

Indicator type: Elastic APM availability target: 99.9% of searches succeed.

The Elastic APM availability indicator calculates the percentage of successful transactions for your service. When the Elasticsearch client throws an exception or the search endpoint returns a 5xx, the span's status records an error and this SLO counts it.

Note: Like the latency SLO, the Elastic APM availability indicator covers all HTTP transactions on search-analytics-demo, not just POST /api/search. Click/cart/checkout errors will consume this budget. For a search-only availability SLO, use a custom KQL indicator with name: "search" AND attributes.search.query: * for good events and name: "search" as the total query.

An 0.1% error budget on 100,000 daily searches means that you can tolerate 100 errors per day. That's tight, but search errors are hard failures and the user gets nothing. Availability SLOs should be stricter than latency SLOs.

3. Search quality SLO: tracking zero-results rate

Indicator type: Custom KQL target: 85% of searches return at least one result (zero-results rate < 15%); index: traces-generic.otel-default; good query: name: "search" AND attributes.search.result_count > 0; total query:name: "search" AND attributes.search.query: *.

Note on KQL versus ES|QL: The SLO framework uses KQL for its indicator filters rather than ES|QL. KQL uses field: valuesyntax and is the same language you see in the Kibana search bar. The ES|QL queries throughout this series are for ad hoc analysis and dashboards; KQL here is the SLO indicator's document filter. Both query the same traces-generic.otel-default index.

This is the SLO that surprises most teams. A search that returns an empty result set isn't an error; HTTP status is 200 and the span status is OK. Plus, no exception was thrown. But from the user's perspective, it failed. They asked for something and got nothing.

The quality SLO uses the custom KQL indicator type because it relies on our custom search.result_count attribute, which the built-in Elastic APM indicators don't know about. But the SLO framework handles everything else, including budget tracking and burn rate calculation, along with alerting.

A sudden spike in zero-results rate, such as from 12% to 40% over an hour, is almost always an infrastructure event, like a failed index deployment or a mapping change that broke queries. It could also be a synonym list misconfiguration. That's an operational problem, not a relevance problem.

Reading your search health in the SLO overview

Once the latency, availability and quality SLOs are created, the SLO overview page shows your search health at a glance:

Each SLO shows the current value, the target, the remaining error budget, and the burn rate. Green means healthy:  Search availability is at 100%, and Search quality is just above its 85% target. Red means violated: Search latency is at 50% against a 99% objective, with the burn rate breached at 200x the sustainable rate. When a budget bar starts shrinking faster than expected, you know something changed, even before users complain.

Clicking into the SLO detail shows burn rate across multiple time windows (1h, 6h, 24h, 72h) and the historical SLI trend. It also shows remaining error budget. For the latency SLO, the Elastic APM latency indicator tracks the percentage of transactions below your 250ms threshold. For the quality SLO, the custom KQL indicator uses traces-generic.otel-default with the good query filtering for attributes.search.result_count > 0. This is where the custom search.* attributes from Blog 2 pay off, since they're the foundation of meaningful SLOs.

Burn rate alerting for search SLOs

When you create an SLO through the Kibana UI, a default burn rate alert rule is automatically created. This is where the real operational value lives.

Burn rate alerts improve on threshold alerts ("error rate > 1%"), which are noisy and miss slow degradation. : Burn rate alerts measure how fast you're consuming your error budget relative to the SLO window.

A burn rate of 1.0 means that you're spending budget at exactly the sustainable rate, but a burn rate of 10.0 means that you're burning 10x too fast and you'll exhaust the budget in 1/10th of the window.

The default burn rate rule uses a multi-window approach, with four severity levels:

Severity

Burn rate

Long window

Short window

What it means

Critical (page)

> 14.4x

1 hour

5 minutes

Exhausts budget in ~50 hours

High (ticket)

> 6.0x

6 hours

30 minutes

Exhausts budget in ~5 days

Medium (review)

> 3.0x

24 hours

120 minutes

Exhausts budget in ~10 days

Low (awareness)

> 1.0x

72 hours

360 minutes

Trending toward exhaustion

The short window prevents alerting on brief spikes that self-resolve, and the long window catches sustained degradation. Together, they balance responsiveness with alert fatigue.

Routing search alerts to PagerDuty, Slack and Jira

Alerts are only useful if they reach the right people in the right tools. Elastic's alerting framework supports a wide range of connectors out of the box, including:

• Incident management: PagerDuty, Opsgenie, xMatters for on-call routing.

• Chat: Slack, Microsoft Teams for team notifications.

• Case management: Jira, ServiceNow for automatic ticket creation when SLOs breach.

• Custom: Webhooks for integrating with any system via HTTP.

You can also use Elastic's built-in cases to track incidents directly within Kibana, linking alerts and traces in one place, along with investigation notes, with push to Jira or ServiceNow when escalation is needed.

A typical routing setup:

Alert

Severity

Channel

Latency SLO burn rate > 14.4

Page

PagerDuty

Availability SLO burn rate > 14.4

Page

PagerDuty + Slack

Quality SLO burn rate > 6

Ticket

Jira (auto-create) + Slack

CTR anomaly (machine learning [ML] job)

Notification

Slack (search team)

Anomaly detection for search quality

Some search degradations are gradual shifts that slip past threshold-based alerts, rather than sudden spikes. A relevance regression after a model update might reduce CTR by 15% over a week, and latency might creep up by 5ms per day as the index grows. These are real problems, but they don't trigger burn rate alerts until it's too late.

Elastic's anomaly detection is built for exactly this. It learns normal patterns in your search metrics and flags deviations automatically, and you don’t have to configure any thresholds. 

Detecting search quality degradation with ML anomaly detection

• Latency anomalies:

 Elastic APM anomaly detection can be enabled directly from the Elastic APM UI for your search service. It learns the typical latency distribution, including daily and weekly patterns, and alerts when behavior deviates. A gradual 5ms/day creep will eventually register as anomalous before it hits your SLO threshold.

• CTR drops:

A relevance regression is invisible to traditional monitoring; latency is fine and errors are zero, plus the result counts are normal, but the ranking changed and users aren't clicking. Anomaly detection on click volume per query is a practical proxy: When a query that normally receives 20 first-click events per hour drops to 5, something likely changed.

To set this up: In Kibana → Machine Learning → Anomaly Detection, create a new job. Use the Multi-metric wizard, and select traces-generic.otel-default as the index. Configure a count detector on attributes.search.first_click split by attributes.search.query. This creates a per-query click-volume baseline and alerts when individual query engagement drops outside the expected range.

Note: This job detects click-volume anomalies per query, not CTR (which requires dividing clicks by searches). Click volume is a useful proxy (a CTR regression usually manifests as a drop in absolute click count), but be aware that a traffic surge with flat click volume would show as a CTR drop without triggering this alert. For true CTR anomaly detection, use a scheduled ES|QL transform to materialize hourly CTR values and run anomaly detection on the computed ratio.

Route the resulting ML alert rule to your Slack search channel.

• Throughput shifts:

A sudden drop or unexpected surge in search volume can indicate upstream problems (like load balancer changes or traffic shifts) or downstream issues (such as search becoming unresponsive or users retrying).

Configure ML anomaly alert rules to route these to your notification channels. These complement your SLO burn rate alerts; burn rates catch budget consumption, and anomaly detection catches pattern changes.

Building a search monitoring dashboard with ES|QL

SLOs tell you whether search is healthy. When they indicate a problem, you need a dashboard that tells you why.

The search team and the on-call team need different views of the same data. A search engineer wants query-level detail, such as which queries have low CTR and which ones return nothing. They’re also interested in where to invest in relevance. But an on-call SRE wants the operational picture, including whether search is fast and whether it’s up. It also wants to know whether search is degrading, and if so, since when.

Search monitoring panels for the on-call dashboard

Build it in Kibana dashboards using Kibana Lens panels. Lens supports ES|QL as a data source, so the queries from Blogs 2–4 can power dashboard panels directly. The key panels include:

• Search throughput over time:  A sudden drop is often the first sign of a problem.

• Latency percentiles (p50, p95, p99) over time: When they diverge (p50 flat, p99 spikes), you have a subset of slow queries.

• Error rate over time: Spikes here mean hard failures.

• Zero-results rate over time: A step change upward, especially correlated with a deployment, means something changed in the index or query pipeline.

The ES|QL for each panel follows the patterns from earlier blogs. For example, a latency percentile panel:

This includes three lines on one chart. When they diverge, such as when p50 stays flat but p99 spikes, you likely have a subset of queries that are slow while the majority are fine. That's a different diagnosis than all queries slowing down (cluster-level pressure).

Drill-down panels: slowest queries and top zero-result queries

For investigation, add a few detail panels, such as:

Slowest queries: A table showing the queries with the highest p95 latency and their search volume. During an incident, this narrows the problem from "search is slow" to "these specific queries are slow."

• Top zero-result queries: A table showing which queries most frequently return nothing. When zero-results rate spikes, this panel immediately shows which queries are responsible.

These drill-down panels use the same ES|QL patterns as Blogs 2 and 3, just surfaced on a persistent dashboard instead of run ad hoc.

Search incident response using OpenTelemetry traces

As an example, an alert fires, noting that search latency has spiked. What happens now?

The trace data from Blog 2's instrumentation gives you a structured path from symptom to root cause.

Step 1: Assess scope

Start at the on-call dashboard, and get answers to the basics:

• When did it start? Narrow the time range to the degradation window.

• How bad is it? Is p50 affected (all queries slow) or just p99 (a subset)?

• Is it just search? Check the Elastic APM service map to determine whether the Elasticsearch dependency is also degraded.

Step 2: Find the problem queries

If the problem is a subset of queries (p99 spike but p50 is fine), use the slowest queries panel or run:

Adjust the 100 ms threshold to match your environment's normal range. It can be lower for a fast cluster or higher if your data volume makes 100ms typical. This narrows the problem from "search is slow" to "these specific queries are slow." That's the difference between restarting the cluster and investigating a specific query pattern.

Step 3: Drill into the trace waterfall

Pick a slow query, and open it in the Elastic APM trace view. The waterfall shows exactly where time was spent. (Refer back to the trace waterfall GIF above to see a real example of a POST /api/search trace broken down into its component spans.)

The overhead gap between search.took_ms (Elasticsearch time) and span duration (end-to-end time) is your diagnostic tool:

Scenario

search.took_ms

Span duration

Diagnosis

Elasticsearch slow

400ms

430ms

Elasticsearch problem: Check slow log, cluster metrics.

App slow

50ms

350ms

Application / network overhead: Check serialization, network.

Both slow

400ms

700ms

Multiple issues: Investigate both.

If the problem is in Elasticsearch, drill into the Search Profile API or cluster monitoring. If it's application overhead, look at the spans around the search span in the waterfall.

Step 4: Correlate with events

Check whether the degradation correlates with:

• Deployments: Did someone deploy a new version of the search service or push a new index?

• Cluster events: Is Elasticsearch under memory pressure, or are there long garbage collection pauses? Or maybe the disk is I/O saturated?

• Network: Is latency between the search service and Elasticsearch elevated?

Elastic Observability's unified platform makes this correlation straightforward because traces, logs, metrics, and infrastructure data all live in the same Kibana instance. You're adding filters in the same interface, rather than switching between tools.

Going further: Infrastructure metrics and cost attribution

This post focuses on what you get from trace data; that is, the spans your search API already emits. But OTel and Elastic Observability support a wider instrumentation picture that becomes valuable as your search infrastructure matures.

• Infrastructure metrics. Adding host and container metrics (like CPU, memory, disk I/O, and network) alongside your traces lets you correlate search performance with infrastructure use. When p99 latency spikes, you can immediately see whether the Elasticsearch nodes are under memory pressure and whether garbage collection  pauses are increasing. You can also check whether disk I/O is saturated, and you can do all this in the same Kibana interface. The Elastic Agent collects these automatically for your infrastructure, and the infrastructure monitoring UI surfaces them alongside your Elastic APM data.

• Total cost attribution (TCA). With infrastructure metrics flowing alongside traces, you can start attributing infrastructure costs to specific services and operations. How much compute does your search service consume? How does that correlate with query volume? If a new ranking model doubles CPU usage per query, you can see the cost impact directly. This is particularly valuable for teams running search on cloud infrastructure where costs scale with resource consumption; understanding the cost per search helps justify infrastructure investment and identify optimization opportunities.

• Logs correlation. OTel auto-instrumentation injects trace context (such as trace ID and span ID) into your application logs. This means that when you're investigating a slow search in the trace waterfall, you can click through to the exact log lines from that request, including Elasticsearch slow log entries and application debug output. It also includes error details that don't fit in span attributes. The logs correlation feature automatically ties them together.

These are natural next steps once you have traces working. Each one extends the same unified platform, without new tools or separate pipelines.

How search analytics and search monitoring share one data pipeline

Here's how it all fits together:

The data flows from the instrumentation you built in Blog 2. This one investment supports two audiences: The search team gets product analytics (Blogs 2–5), and the SRE team gets operational monitoring (this post). Neither team needs separate data pipelines.

Getting started with search monitoring in Elastic

This is the last post in the series, and it brings us full circle. Blog 1 describes the vision: Instrument search once with OTel, and send spans to Elastic. Then use ES|QL to answer any question about search behavior. Blogs 2–4 build the instrumentation and analytics, and Blog 5 shows how to feed that data back into relevance improvements. This post shows how the same data powers operational monitoring, including SLOs, alerting, anomaly detection, and incident response.

The key takeaway for search engineers is that the instrumentation you built for analytics already generates the signals. The SLOs and alerts are built-in capabilities of Elastic Observability, as are the dashboards. You're closer to production-grade search monitoring than you might think. Observability isn't a separate discipline you need to learn from scratch. 

If you've been following along and built the instrumentation from Blogs 2–4, start here:

1. Open Elastic APM: Look at your search service, and explore a trace waterfall. You can also check the errors tab.

2. Create three SLOs: Latency (Elastic APM latency), availability (Elastic APM availability), and quality (custom KQL for zero-results).

3. Enable anomaly detection: One click in the Elastic APM UI for latency anomalies.

4. Build the on-call dashboard: Four Lens panels with the queries from this post.

By the end of an afternoon of work, your search service can have the same observability coverage as any other critical production system.

Get started

Working code

• Reference project: Working code for the entire blog series; clone, configure, and run.

Elastic APM and traces

• Elastic APM Overview: Elastic APM concepts and trace analysis.

• Elastic APM UI: Service overview, transactions, dependencies, errors.

• Service Maps: Dependency visualization and health.

• OpenTelemetry Integration: OTel ingestion in Elastic.

SLOs and alerting

• SLOs in Elastic Observability: Creating and managing SLOs.

• Create an SLO: SLI types (like Elastic APM latency, Elastic APM availability, or custom KQL), time windows, budgeting.

• SLO Burn Rate Alerts: Multi-window burn rate alerting.

• Alert Connectors: Slack, PagerDuty, webhook, email integrations.

• Alerting Framework: Rule types and configuration.

• Jira Connector: Automatic ticket creation from alerts.

• ServiceNow Connector: ITSM integration.

• Elastic Cases: Built-in incident tracking with external push.

Anomaly detection

• Anomaly Detection Overview: Unsupervised time series anomaly detection.

• Elastic APM Anomaly Detection: Enable ML for latency, throughput, error rate.

• ML Anomaly Alert Rules: Alerting on detected anomalies.

Dashboards and ES|QL

• Kibana dashboards: Building operational dashboards.

• Lens: Visualization editor.

• ES|QL in Lens: ES|QL-powered dashboard panels.

• ES|QL Overview: Language reference.

Elasticsearch operations

• Slow Log Configuration: Threshold-based query slow logging.

• Search Profiling: Profile API for query execution analysis.

• Monitoring Elasticsearch: Cluster stats, search rate, latency.

From this series

• Modern search analytics with OpenTelemetry: The vision.

• Instrument your search API : Search spans and search.* attributes.

• Measuring search quality : CTR, MRR, click distribution.

• From clicks to conversions: Conversion tracking and revenue attribution.

• Personalizing search from behavior: Judgment lists, rank features, Learning To Rank (LTR).

This is the final post in a six-part series on search analytics with OpenTelemetry and Elastic. Start from the beginning: Modern search analytics with OpenTelemetry, or to start building, jump to Instrument your search API.

What you'll discover

In this post, you'll learn how to:

• Build judgment lists from click data to evaluate and improve relevance.

• Apply basic search tuning (field weights, boosts, query rules) informed by analytics.

• Create rank features from behavioral signals, like popularity and conversion rate.

• Understand LTR and how click data becomes training data.

• Close the feedback loop between analytics and relevance improvement.

What you'll need

• Search analytics data from the previous blogs in this series (search, click, and conversion spans in Elastic).

• An Elasticsearch index with product data (the reference project includes sample data with rank features).

• Familiarity with Elasticsearch queries (BM25 [Elasticsearch's default text scoring algorithm], rank_feature, function scores).

Search relevance techniques at a glance

Technique

Effort

What it improves

When to use

Field weight tuning

Low

Relevance for attribute-rich queries

First step; quick wins

Query rules

Low–medium

Specific high-value queries

Known bad results for specific terms

Rank features

Medium

Blending behavioral signals with text score

Popularity, conversion, freshness boosts

LTR

High

Systematic ranking from click data

When you have ≥5k labeled query-doc pairs

From search analytics to search relevance improvements

Over the past three posts, you built a full instrumentation pipeline, including search spans with search.* attributes and click tracking with position data and click-through rate (CTR) / Mean Reciprocal Rank (MRR) metrics. This pipeline also includes conversion spans tying searches to revenue. And it all sits in traces-generic.otel-default, queryable with Elasticsearch Query Language (ES|QL).

Now you have dashboards, and you know that your CTR is 28%. You also know which queries generate revenue and which ones users abandon. Plus, you can tell your product manager exactly where the funnel leaks.

Now what?

The real value of search analytics is using behavioral data to improve relevance and close feedback loops. It’s also important to make search learn from its users. Your experience tells you that measurement is necessary but not sufficient, and a dashboard that shows poor ranking doesn't improve that ranking. 

This post covers four practical areas:

1. Judgment lists: The foundation for evaluating and improving relevance.

2. Basic search tuning: Field weights, boosts, and query rules informed by analytics.

3. Rank features for personalization: Feeding behavioral signals back into ranking.

4. LTR: Training machine learning (ML) models on click data to optimize ranking automatically.

Each one draws directly from the search.* attributes you're already collecting, and they don’t require any new instrumentation.

What are judgment lists?

Before diving into specific techniques, it's worth understanding the concept that ties them all together: judgment lists.

A judgment list is a set of query-document pairs with relevance grades: For a given query, how relevant is each document? They look like this:

Query

Document

Grade

Label

"wireless headphones"

SKU-001 (Sony WH-1000XM5)

3

Highly relevant

"wireless headphones"

SKU-042 (AirPods Max)

2

Relevant

"wireless headphones"

SKU-099 (Wired earbuds)

0

Not relevant

Judgment lists serve three purposes:

• Evaluating current relevance. Given a set of queries and known-good results, how well does your search rank them? Metrics like Normalized Discounted Cumulative Gain (NDCG) and the Rank Eval API use judgment lists to score your ranking quality. This gives you a baseline before making changes.

• Measuring the impact of changes. When you adjust field weights or add synonyms, judgment lists tell you whether the change helped or hurt. The same is true when you change boosting rules. Run the same evaluation before and after: If NDCG went up, the change improved relevance for those queries.

• Training LTR models. LTR algorithms need labeled training data, such as, For this query, these documents are relevant and those aren't. Judgment lists are that training data.

Manual vs. automated judgment lists

Traditionally, judgment lists are created by human assessors who manually rate documents for a set of test queries. This works well for a small number of high-value queries (your top 50 searches, for example), but it doesn't scale. A large catalog with thousands of distinct queries and frequent inventory changes makes manual assessment impractical.

This is where your click data becomes valuable. Every click is an implicit relevance judgment; that is, a signal that for a given query, a given document was relevant enough to engage with. Conversions are even stronger signals. By aggregating this data, you can build judgment lists automatically from real user behavior, at a scale that manual assessment can't match.

The trade-off is noise. Clicks are influenced by position bias (users click higher-ranked results more often regardless of relevance) and presentation effects. They’re also influenced by accidental clicks. We'll cover techniques to handle this noise later in this post. But the key insight is that approaches like personalization and LTR lean toward automated judgment lists because it's not scalable to manually create lists for every query or user segment, nor for every inventory change.

The judgment lists guide on Search Labs covers the concept in depth, including how to structure lists for evaluation with the Rank Eval API.

How search analytics identify search relevance problems

The most immediate use of your analytics data is to identify and fix specific relevance problems. This only requires using data to direct manual improvements, no personalization or ML.

Finding problem queries with search analytics

The ES|QL queries from the click in your application give you per-query CTR and MRR. Sort by search volume descending and CTR ascending to find your highest-impact relevance failures:

This is the CTR-by-query query re-sorted to surface problems first. The WHERE searches > 5 filter removes one-off queries that would dominate the low-CTR list with small sample noise.

• Zero-CTR queries with results.  Your ranking is returning content, but none of it’s compelling. These queries often benefit from synonym expansion. You can also improve them with boosting rules or pinned results.

• Low-CTR, high-volume queries. These are the biggest relevance investment opportunities, and they affect the most users.

• Low-MRR, high-CTR queries. Users find what they need but have to scroll for it. In these instances, the relevant documents exist, but they're ranked wrong.

Try it: Close the loop in the reference project

If you have click data in the reference project(python generate_traffic.py --blog 3 --sessions 100), you can run the problem-query query above against traces-generic.otel-default and observe your lowest-CTR queries.

The reference project's app.py already uses rank_feature boosting: rank_features.popularity, rank_features.conversion_rate, rank_features.margin_score, and rank_features.freshness are indexed on every product and blended into the BM25 score at query time. To see the effect of adjusting a boost:

1. Open reference/app.py, and find the "should" clause in _build_search_query().

2. Change the "boost" value on rank_features.popularity from 2 to 5.

3. Restart the server (python app.py) and rerun generate_traffic.py --blog 3 --sessions 50.

4. Compare the top-queries and click position distribution in ES|QL before and after.

This is informed manual tuning using the behavioral data you collected, not ML. That's the pattern for the rest of this post: First, ES|QL tells you what's wrong. Then, configuration changes and, eventually, LTR models fix it.

Field weights, boosts, and scoring for search relevance

The simplest tuning lever in Elasticsearch is adjusting how different fields contribute to the relevance score. A multi_match query across title, description, and brand fields can weight the title higher because a match there is usually more relevant. Your analytics data tells you where these weights are wrong: If users consistently click products with titles that don't match the query but with descriptions that do, your title boost may be too aggressive.

Beyond field weights, you can incorporate business metrics directly into scoring. A common example is boosting by profit margin; that is, products with higher margins rank slightly higher when relevance scores are similar. The rank_feature field type is designed for exactly this: Index a numeric signal (margin, popularity, recency) alongside each document, and the rank_feature query blends it with text relevance at query time. For more complex scoring combinations, like decay functions, weighted field values, and scripts, the function_score query gives you full control. The boosting by profit and popularity guide on Search Labs walks through this in detail, including the trade-offs between additive and multiplicative boosting approaches.

Query rules for targeted search relevance fixes

For targeted interventions, query rules let you pin, boost, or exclude specific results for specific queries, and no model training is required. They're the search equivalent of a manual override.

Your analytics data tells you exactly where to apply them. When you see a zero-CTR query, like "returns policy", that consistently returns product results instead of the returns page, you can pin the returns page at position 1 for that query. If you notice a high-revenue query where the best-selling product appears at position 4, you can boost it.

Query rules are valuable precisely because they're simple. They solve known problems immediately while you build toward more sophisticated approaches. The query rules tutorial on Search Labs walks through the setup.

Building judgment lists from click data

The basic tuning above handles individual problems. To improve relevance systematically, you need judgment lists, and your click data can build them automatically.

Converting click data into relevance grades

The simplest approach is to count clicks per query-document pair and assign graded relevance:

This gives you a table of (query, document, click_count, avg_position) tuples. The grading step maps click counts to relevance levels:

Click count

Suggested grade

Label

0

0

Not relevant (never clicked for this query)

1

1

Marginally relevant

2–3

2

Relevant

4+

3

Highly relevant

The thresholds depend on your traffic volume. For a high-traffic site, you might need 10+ clicks before calling something "highly relevant." For lower traffic, even two or three clicks is a meaningful signal. The point is that click frequency across users is a stronger signal than any single click.

You can strengthen judgments further by incorporating conversion data, a document that gets clicked and added to cart is a stronger relevance signal than one that gets clicked and abandoned:

A document with five clicks and three cart additions is a stronger candidate for grade 3 than one with five clicks and zero cart additions.

Handling position bias in click data

There's a problem with raw click counts: position bias. Users click position 1 more often because they see it first, not necessarily because it's the most relevant result. Blog 3 introduced this concept when we discussed click position distribution and referenced the foundational work by Joachims et al. (2005).

Position bias matters for judgment lists because it means raw click data over-weights whatever the current ranking happens to surface first. If you train an LTR model on biased judgments, it learns to replicate the existing ranking, which defeats the purpose.

Two practical approaches to handle this:

• Position-normalized click rates. Instead of raw click counts, calculate click-through rate per position. A document clicked 3 out of 10 times when shown at position 5 is arguably more relevant than one clicked 5 out of 10 times at position 1. Position 5 gets less visibility, so a higher click rate there is a stronger relevance signal.

• Skip-above heuristics. When a user clicks position 3 but skips positions 1 and 2, those skipped documents are implicitly judged "less relevant" for that query. This is the core insight from Joachims et al.: Skipped-above results provide negative training signals. You can extract these pairs from your click data:

Queries where the minimum click position is greater than 1 are sessions where the user skipped the top result(s). The documents at those skipped positions, for those queries, are candidates for grade 0 in your judgment list, because the user saw them and chose something lower.

For production judgment list generation, the LTR tutorial on Search Labs walks through the complete pipeline. The training LTR models with user behavior data guide covers the specific workflow of deriving training data from click-through behavior, including the Clicks Over Expected Clicks (COEC) algorithm for debiasing. And the LTR Jupyter notebooks provide working Python code for feature extraction and model training.

Rank features for search relevance personalization

Basic tuning applies the same ranking to every user. Personalization means adapting results based on who’s searching (their history, preferences, or segment). Rank features are the most accessible way to do this in Elasticsearch.

Building behavioral signals from click streams

You can aggregate your click and conversion data at the document level to produce signals that reflect overall or segment-specific popularity:

These document-level signals (click popularity, cart rate, and conversion rate) are indexed as rank_feature fields on each product document. The workflow:

1. Run the ES|QL query above periodically (daily or weekly).

2. Write the results back to a click_popularity or conversion_rate rank_feature field on each product document.

3. Use a rank_feature query to blend text relevance with behavioral signals.

The rank_feature query applies a saturation function by default: Initial popularity gains matter most, diminishing as values get large. This prevents a single viral product from dominating all queries. You can tune the function's pivot point to control how much influence the feature has relative to text relevance.

Personalizing rank features by user segment

The query above produces global popularity; that is, it’s the same for every user. Personalization comes from segmenting these signals. If you're tracking enduser.pseudo.id or user.id, you can compute features per user cohort:

• Category affinity: How often does this user click products in "electronics" versus "clothing"?

• Price sensitivity: Does this user tend to click and convert on higher- or lower-priced items?

• Brand preference: Which brands does this user engage with most?

These become additional rank features, applied at query time based on who's searching. The personalized search with LTR guide on Search Labs walks through training per-user ranking models. For a lighter approach, the cohort-aware ranking guide shows how to use multiplicative boosting to personalize at the segment level without ML, just analytics-derived weights applied to rank features at query time.

Combining multiple signals with the linear retriever

When you're blending text relevance with semantic search and behavioral rank features, you need a way to combine them. Elasticsearch's linear retriever gives you precise control over how different query types contribute to the final ranking. It computes a weighted sum of normalized scores, so you can say text relevance matters 60%, semantic similarity 30%, popularity 10% and adjust those weights based on your analytics. This is unlike Reciprocal Rank Fusion (RRF), which only considers relative rank positions.

This is particularly useful for personalization because you can vary the weights per user segment. A returning customer might get more weight on purchase history, while a first-time visitor gets more weight on global popularity.

If you're also using semantic search with embedding models, the Elastic Inference Service (EIS) now offers GPU-accelerated embedding generation, including multilingual models, directly within Elastic Cloud, making it straightforward to add a semantic retriever alongside your text and behavioral signals.

How Learn To Rank uses click streams to optimize ranking

Rank features handle individual signals, but LTR handles all of them at once. It trains an ML model to combine features (like text relevance, popularity, CTR, recency, margin, or user affinity) into a single ranking function optimized for your users.

What you need for Learn To Rank in Elasticsearch

Elasticsearch has supported native Learning to Rank since version 8.12 as an Enterprise subscription feature. The pipeline:

1. Judgment list. Query-document pairs with relevance grades (built from click data, as above).

2. Feature extraction. Numeric signals for each query-document pair (BM25 score, popularity, CTR, recency, price, margin, user segment features).

3. Model training. Typically XGBoost or LambdaMART, trained on your judgment list with features.

4. Deployment. Upload the trained model to Elasticsearch via Eland, and use it as a rescorer.

The click and conversion data from this series feeds steps 1 and 2, and the judgment list is your training labels. The document-level features from the rank features section above (click popularity, conversion rate, margin) are additional features alongside text relevance scores.

Why automated judgment lists from click data scale better

This is where the scalability argument for automated judgment lists becomes concrete. A manually curated judgment list might cover your top 100 queries well, but personalized ranking needs judgment data across thousands of queries and multiple user segments. You can't hire assessors to rate results for "wireless headphones" separately for electronics enthusiasts, budget shoppers, and professional audio engineers.

Automated judgment lists from click data scale to every query your users actually run and update as inventory and user behavior change. They can also be segmented by cohort. The training LTR models with user behavior data guide demonstrates this end-to-end workflow, showing how to go from raw click events to trained ranking models.

For teams looking to close this loop even further, the agentic autotuning approach demonstrates using an AI agent to continuously monitor search quality and generate judgment lists from user interactions. It automatically retrains LTR models, turning the feedback loop into an autonomous system.

What Learn To Rank requires: Traffic, pipeline, and evaluation

LTR requires enough traffic to generate meaningful judgment lists and engineering time to build and maintain the pipeline. It also requires ongoing evaluation to ensure that the model improves over time. But for search applications with sufficient volume, it's the most effective way to make ranking learn from user behavior. The LTR introduction on Search Labs covers the full scope of what's involved.

Elasticsearch Relevance Studio: Visual search relevance tuning

Between query rules (manual, targeted) and LTR (ML, systemic), there's a middle ground: visual relevance tuning. Elasticsearch Relevance Studio is a tool for comparing and tuning search configurations side by side. It lets you adjust boost values, field weights, and query structures while seeing the results update in real time.

Analytics data, especially CTR and MRR, tells you which queries to focus on. Start from a ranked list of problem queries (the low-CTR, high-volume queries from the ES|QL analysis above), and work through them systematically in Relevance Studio, instead of guessing which searches need tuning.

The workflow:

1. Identify problem queries. Run the CTR-by-query and MRR-by-query analyses.

2. Open those queries in Relevance Studio. See current results alongside the tuned version.

3. Adjust field weights and boosting. Experiment with configuration changes.

4. Evaluate with judgment lists. Use the Rank Eval API to confirm that the change improves NDCG for your test queries.

5. Measure the impact in production. Rerun the analytics after deploying changes, and compare CTR / MRR.

This before-and-after loop is where analytics and tuning connect. With the data analytics you can prioritize the queries that affect the most users and verify that changes actually helped. Without this data, tuning is guesswork, and you're adjusting weights without knowing which queries matter or how to measure improvement. 

If you are looking at tuning and evaluating relevance you should also investigate the Elasticsearch Relevance Studio project which lets you directly compare search strategies. 

Closing the loop between search analytics and search relevance

Search relevance improvement follows a consistent loop: measure quality with ES|QL analytics, apply changes (query rules, field weights, rank features or LTR models), then measure again.

The key insight is that the same instrumentation that measures search quality also generates the data to improve it. Your search spans produce the queries to analyze, and your click spans produce the judgment data for evaluation and LTR training. Plus, your conversion spans tell you which improvements matter most to the business.

Measuring search relevance improvements after changes

After deploying a change, like a new query rule or updated field weights, or deploying an LTR model, you need to know whether it helped:

Metric

What it measures

How to compare

CTR trend

Whether more users are clicking results after the change

Run the CTR-by-query ES|QL query for the week before and after; compare per-query percentages

MRR trend

Whether clicks are shifting toward higher-ranked positions

Compare mean reciprocal rank per query before and after using the MRR-by-query analysis

Conversion rate trend

Whether more clicks are turning into purchases or cart additions

Run the click-to-conversion ES|QL query for both periods; compare cart rate percentages

Revenue per query

Whether the business impact of the change is positive

Compare revenue attributed to affected queries via conversion spans before and after deployment

Run the same ES|QL queries before and after. If your change was a query rule for "laptop bag", compare that query's CTR and MRR from the week before to the week after.

A/B testing with experiment attributes

The feature_flag.key attribute  is designed for exactly this. Route a percentage of traffic to a new ranking configuration, and set feature_flag.key to the experiment name and optionally feature_flag.result.variant to the variant (for example, "control" or "variant-boost-v2"). Then propagate it to click spans the same way you propagate `query_id`, and compare metrics between groups:

This gives you an A/B comparison of CTR by experiment variant. You don’t need a separate experimentation platform for basic comparisons, although you’ll want a proper framework for statistical rigor on sample sizes and significance.

Monitoring for search relevance regressions

Once you've identified your high-value queries, that is, the ones that drive revenue and have been tuned for good engagement, you need to protect them. Relevance regressions on your top 20 revenue-generating queries are business-critical incidents.

This is where monitoring comes in. In the next blog in the series, we'll cover setting up alerts on these metrics: CTR drops on high-revenue queries and MRR regressions after deployments. We’ll also cover conversion rate anomalies. The same ES|QL queries that power your analytics dashboards can drive alerting rules; the feedback loop covers measurement and improvement. It also provides operational protection.

Search relevance improvement roadmap: Week 1 to Quarter 1

Here's a practical starting point. You don't need to implement LTR on day one. The approaches build on each other:

• Week 1: Evaluate and fix known problems. Run the CTR-by-query analysis. Find your zero-CTR queries with high search volume, and fix the worst ones with synonyms or pinned results via query rules. You could also use adjusted field weights. Use judgment lists (even a small manual one for your top queries) to confirm that the changes improve NDCG before deploying. You should get immediate, targeted impact.

• Month 1: Add behavioral rank features. Extract document-level click popularity and conversion rates from your analytics. Index them as rank_feature fields, and blend behavioral signals with text relevance using the linear retriever or `rank_feature` queries. Then consider simple business boosts like margin. This lifts baseline quality across all queries without manual intervention per query.

• Quarter 1: Automate with LTR. Once you have enough click data (typically several weeks of production traffic), build judgment lists automatically from click and conversion data. Train an LTR model that combines text features, behavioral features, and business features, and then deploy it as a rescorer. The ranking now learns from user behavior and improves as you collect more data.

At each stage, measure the impact with the same metrics. CTR and MRR are your scorecards, as is conversion rate. If a change doesn't move them, it didn't help, regardless of how sophisticated the approach.

What's next: Search reliability engineering and SLOs

We've covered the full sequence: instrument search, measure quality, track conversions, and improve relevance. The missing piece is making sure it all keeps working.

Next, we close the series with search reliability engineering, including Service Level Objectives (SLOs) for search quality and alerting on metric regressions, along with operational dashboards that catch problems before users notice them. The same metrics you've been building become the basis for search health monitoring, turning your analytics pipeline into a reliability system.

Get started

Working code

• Reference project: Working code for the entire blog series; clone, configure, and run.

What you'll discover

In this post, you'll learn how to:

• Add conversion tracking (add-to-cart and purchase spans) with search.* attributes that tie back to the originating search.

• Build a full search-to-revenue funnel: search → click → add-to-cart → purchase.

• Write Elasticsearch Query Language (ES|QL) queries to calculate conversion rates, revenue per query, and average order value.

• Identify where users drop off and which team should own each drop-off point.

• Attribute revenue to specific search queries for prioritizing relevance work.

What you'll need

• Click tracking from Blog 3 (search + click spans flowing to Elastic via OTel-native ingestion).

• Backend endpoints for add-to-cart and checkout events (example code provided).

• Basic understanding of ecommerce conversion funnels.

Why search revenue attribution matters

Your product manager walks into a meeting and asks, "Which searches are driving the most revenue?"

You can tell them what users search for (Blog 2) and what they click (Blog 3). But you can't tell them what they buy. The gap between "clicked a result" and "purchased a product" is where the business case for search investment lives, and right now it's invisible.

This post closes the loop. By adding two more span types (add-to-cart and purchase), you get a full funnel from search to revenue, built on the same search.* attributes and ES|QL queries you've been using since Blog 2.

What search conversion tracking answers for your team

Here's what conversion tracking lets you answer and why each question matters to different people on your team.

• Which searches drive revenue? This is the product manager's question. When you can attribute dollars to specific queries, you can prioritize relevance work by business impact. A query with mediocre click-through rate (CTR) but high conversion value is more important than a high-CTR query that never leads to a purchase.

• Where do users drop off? The funnel from search to purchase has four stages: search, click, add-to-cart, and purchase. Each drop-off points to a different problem. High click-to-cart drop-off suggests that product pages aren't convincing. High cart-to-purchase drop-off is checkout friction rather than a search problem. Knowing where users abandon tells you which team should fix it.

• Which queries to protect? Once you know that "laptop bag" generates $12,000/month in attributed revenue, you treat it differently. Any relevance change that touches high-revenue queries gets extra scrutiny. You can set up monitoring (Blog 6, coming soon) to alert when conversion rates drop for your top-earning searches.

Two new instrumentation points total about 20 lines of code, and they follow the same pattern as you’ve used before. You add attributes to spans, and query them with ES|QL.

Following along with code? The reference project has conversion tracking ready to enable. Uncomment the Blog 4 sections in app.py and app.js, restart, and then generate traffic with python generate_traffic.py --blog 4.

The four-stage search conversion funnel

Before we write any code, here's the shape of what we're building. Each stage is an instrumentation point, and each creates spans in traces-generic.otel-default:

The thread running through the entire chain is search.query_id. The same identifier you derived from the trace ID in Blog 2 and used to link clicks to searches in Blog 3 now carries through to add-to-cart and purchase events. This is what makes revenue attribution possible: You can trace a purchase back to the search that started the journey.

Add conversion tracking spans with OpenTelemetry

Add-to-cart span instrumentation

When a user adds a product to their cart from a search results page (or from a product detail page they reached via search), you create a cart.add span. This captures the moment that intent turns into action.

This follows the same pattern as click tracking in Blog 3; that is, an independent span linked to the originating search via query_id. The new attributes are cart.quantity and cart.price which let you aggregate revenue at query time.

Purchase span instrumentation

When the user completes checkout, you create a checkout.complete span. This is the revenue event, the one that answers the product manager's question.

• Spans capture errors automatically. This is a side benefit of using OTel spans for conversion events. If an add-to-cart or checkout call throws an unhandled exception, the span's status is automatically set to ERROR and the exception details are recorded. These errors are business-critical (a broken checkout flow means lost revenue), and they show up immediately in Elastic APM's error tracking, service maps, and alerting. You get conversion analytics and operational monitoring from the same instrumentation, with no extra code.

• checkout.total_amount is the revenue number. This is what you'll aggregate in ES|QL to get revenue-by-query. It represents the order total, not the price of a single item.

• search.query_id is conditional. Not every purchase originates from search. Users browse categories and follow promotional links. Then they return to their cart days later. The if event.query_id: guard ensures that you only attribute purchases to search when there's a genuine connection. Purchases without a query_id still get recorded; they just don't appear in search attribution queries.

• search.query is set on both cart and purchase spans. This is a deliberate denormalization. You could join back to searches via query_id to get the query text, but while ES|QL does support LOOKUP JOIN, it’s likely not the right choice here due to the need to optimize the lookup index. By putting the query text directly on conversion spans, your revenue-by-query and cart-by-query queries are single, straightforward aggregations.

Span attributes for cart and purchase events

Add-to-cart attributes:

Attribute

Type

Required

Purpose

search.action

string

yes

"add_to_cart"

search.result_click_id

string

yes

Product document ID

search.result_click_position

int

yes

Position in results when added

search.query_id

string

yes

Links to originating search

enduser.pseudo.id

string

yes

Client/device identifier (OTel Semantic Conventions  [SemConv])

cart.quantity

int

yes

Quantity added

cart.price

float

optional

Product price

search.query

string

recommended

The search query text (for cart-by-query analysis)

Purchase attributes:

Attribute

Type

Required

Purpose

search.action

string

yes

"purchase"

checkout.order_id

string

yes

Unique order identifier

checkout.total_amount

float

yes

Order total

checkout.item_count

int

yes

Number of items purchased

enduser.pseudo.id

string

yes

Client/device identifier (OTel SemConv)

search.query_id

string

recommended

Links to originating search (last search in journey)

search.query

string

recommended

The search query text (for revenue-by-query analysis)

These follow the same search.* namespace from Blogs 2 and 3, with new cart.* and checkout.* prefixes for conversion-specific data. With OTel-native ingestion, all attributes are stored under attributes.* with dot notation preserved. That means no more mapping strings to labels.* and numbers to numeric_labels.*. A search.query attribute is queryable as attributes.search.query. Likewise, checkout.total_amount is queryable as attributes.checkout.total_amount.

User identity: connecting search to purchase across sessions

You'll notice that enduser.pseudo.id appears on every span type in this series. It's the minimum identity level; that is, a persistent identifier stored in the browser's localStorage that ties events to a device across sessions.

For conversion tracking, identity becomes more important. You need to connect a search on Monday to a purchase on Tuesday, or correlate cart additions across tabs. Our schema supports three identity levels, aligned with OTel semantic conventions:

Attribute

Persistence

Purpose

enduser.pseudo.id

Permanent (localStorage)

Device/browser identifier (OTel SemConv)

session.id

Per-visit (sessionStorage)

Groups events within a single visit (OTel SemConv)

user.id

Account (auth system)

Authenticated user (OTel SemConv)

For the funnel queries in this post, enduser.pseudo.id is sufficient. It links the journey from search to purchase within a browser. If your users authenticate, adding user.id enables cross-device attribution (searched on mobile, purchased on desktop) and richer personalization. session.id helps disambiguate when the same client has multiple active sessions.

All three are optional on interaction spans. Start with enduser.pseudo.id, and add the others when your use case requires them. The important thing is consistency: Use the same identifiers across search, click, and conversion spans so the joins work.

Frontend integration

The front end needs to propagate query_id through the user journey. When the user clicks a search result, you already have query_id from the search response (Blog 3). The key is carrying it forward.

The front end sends client_id as an HTTP field name; and the back end maps it to the OTel semantic convention enduser.pseudo.id when setting span attributes. The query_id propagation is the critical piece. Store it alongside the product in the cart data structure so it survives navigation between pages. We'll discuss the design challenges of this in the attribution section below.

Using the reference project? The reference app's frontend/app.js wires up add-to-cart buttons, but the checkout flow isn’t implemented in the browser UI. It goes through the traffic generator. To simulate conversion events, run: python generate_traffic.py --blog 4 --sessions 100. This sends a realistic mix of searches, clicks, cart additions, and purchases to all three backend endpoints.

Verify that conversion events are arriving

Before building funnel queries, confirm that both span types are flowing to Elastic:

Add-to-cart events:

Purchase events:

If these return rows, you have the full funnel instrumented. If not, check the same things as always: OpenTelemetry Protocol (OTLP) endpoint, auth token, and span export.

Funnel analysis with ES|QL

Count search, click, cart and purchase events in a single query

You can count all four funnel stages in a single ES|QL query using the same COUNT(CASE(...))pattern from the CTR query in Blog 3:

Example output:

searches

clicked

carts

purchases

click_rate

cart_rate

purchase_rate

146

41

28

12

28.1%

19.2%

8.2%

You get all four counts and three conversion rates from a single query. The WHERE clause pulls all four span types into one result set, and COUNT(CASE(...)) counts each type separately. This is the same technique that made the CTR query in Blog 3 so clean.

Notice that we use search.first_clickfor the click stage rather than counting all click events. This gives you the number of searches that received at least one click (the same definition used for CTR in Blog 3). Without this, a search with three clicks would inflate the click count to three while only counting as one search, making the funnel numbers misleading. Each stage now represents a unique progression: how many searches happened, how many of those got clicked, how many led to a cart addition, and how many resulted in a purchase.

You can turn this into a funnel visualization using Kibana Lens. Run each stage count as a separate ES|QL query, save them as dashboard panels, and arrange them as a horizontal bar chart with the four stages on the y-axis and counts on the x-axis. The drop-off at each step becomes immediately visible.

The dashboard above (from the first blog in the series) shows this in practice: The Conversion Funnel panel in the lower left uses a horizontal bar chart to visualize the four stages, and the Top Queries by Revenue table alongside it shows revenue attribution. You can build these directly from the ES|QL queries in this post.

What drop-off rates tell you and who owns each bottleneck

Each transition in the funnel tells you something specific:

• Search to click (CTR): You measured this in Blog 3. Low CTR means that results aren't compelling. This is a relevance problem.

• Click to cart: The user engaged with a result but didn't add it to their cart. This could mean that the product page isn't persuasive or the price isn't competitive. It could also mean that the item was out of stock. It's often not a search problem. It’s possible that the search worked (the user clicked), but something downstream lost them.

• Cart to purchase: The user committed to buying but didn't complete checkout. Complicated forms, unexpected shipping costs, and payment issues cause this checkout friction. This is almost never a search problem, but it's useful to know where the funnel leaks so you don't waste time optimizing relevance when checkout is the bottleneck.

The diagnostic pattern is straightforward:

Drop-off point

Likely cause

Who owns it

Search to click

Relevance / ranking

Search team

Click to cart

Product page / pricing / availability

Product / merchandising

Cart to purchase

Checkout UX / payment / shipping

Checkout / growth

This is one of the most valuable things that full-funnel data gives you: the ability to point at the right problem. When the VP asks, "Why aren't searches converting?", you can show whether the bottleneck is relevance, product pages, or checkout.

Revenue attribution by search query

Here's the query your product manager actually wants:

This gives you a ranked list of queries by the revenue they generated. The search.query attribute on purchase spans (the denormalization we discussed earlier) makes this a single aggregation query, without joins or subqueries.

How to act on search revenue  data

• Protect high-revenue queries. If "laptop bag" generates the most revenue, any relevance change that affects that query gets extra scrutiny. You might add it to a regression test suite or pin specific results with query rules. You might even set up an alert when its conversion rate drops (Blog 6).

• Prioritize relevance investment. The queries at the top of this list are where relevance improvements have the most business impact. A 10% CTR improvement on a query that generates $500/month in revenue is worth more than a 50% improvement on one that generates $20.

• Identify missed opportunities. Cross-reference with the top-queries data from Blog 2. A query with high search volume but no purchase attribution is either a browsing query (informational intent) or a conversion gap worth investigating.

Top revenue queries: where to focus relevance investment

To focus your relevance team's efforts, pull the top revenue-generating queries:

Cross-reference this with the per-query CTR from Blog 3. A query with high revenue but low CTR is underperforming; even small relevance improvements have outsized business impact. A query with high CTR but no purchase attribution might be informational (such as users browsing but not buying). The queries at the top of both lists deserve the most attention from your relevance team.

Average order value by search query

Average order value (AOV) tells you how much purchases are worth, in addition to which queries convert to those purchases. Queries with high AOV are your premium-intent searches; ranking improvements there have the biggest per-purchase impact:

A query with high AOV but low volume is a different opportunity than high volume + low AOV. The first means premium intent from a small audience (consider featured results or dedicated landing pages); the second means broad reach with budget buyers (price sensitivity may be limiting conversion more than relevance).

Search revenue attribution: limitations and workarounds

Revenue attribution from search is valuable, but it's imperfect. Understanding the limitations helps you set appropriate expectations and design around them.

Last-touch attribution in multi-search journeys

Users rarely search once and buy. A typical journey might look like:

1. Search "laptop bag": Browse results and click a few.

2. Search "laptop bag leather": Refine the search.

3. Search "laptop sleeve 15 inch": Try a different angle.

4. Add to cart from the third search's results.

5. Purchase.

With the instrumentation above, this purchase attributes to the third search, the one whose query_id was on the cart item. The first two searches contributed to the journey but get no credit.

This is last-touch attribution, and it's the simplest model that works within a single query_id linkage. It's not perfect, but it's concrete and unambiguous. The alternative, that is,tracking every query_id in a user's session and distributing credit, adds significant complexity to both instrumentation and analysis.

For most teams, last-touch is a good starting point. If you need multi-touch attribution later, the raw data is there. You can query all searches and clicks for a given client_id within a time window and reconstruct the full journey:

This reconstructs a user's full search journey in chronological order. It's useful for debugging individual sessions, even if you don't build automated multi-touch attribution.

Cross-session attribution limits with query_id

A user searches for "wireless headphones" on Monday, clicks a few results, leaves, and comes back on Wednesday to buy. The query_id from Monday's search is long gone, since it was a property of that specific search request.

This is a fundamental limitation of query_id-based attribution. It works within a session (or more precisely, within the scope where the front end retains the query_id). It doesn't work across sessions.

For cross-session attribution, you'd need a different approach, typically a user-level event store where you associate product views, cart additions, and purchases with a persistent user ID and then look back in time to find the originating search. That's a more complex analytics pipeline and is outside the scope of what we're building here.

The practical impact is that your search-attributed revenue will be an undercount. Some purchases that were genuinely influenced by search won't carry a query_id. This is fine for relative comparisons (such as, Which queries generate more revenue than others?), even if the absolute numbers are conservative.

How to persist query_id from search to checkout

A few practical decisions affect how far your query_id propagation reaches:

• Store query_id in the cart. When a user adds a product to their cart, persist the query_id alongside the item. This way, even if the user navigates away and comes back to checkout later (within the same session), the attribution survives.

• Don't overwrite query_id on re-search. If a user adds a product from search A and then searches again and adds another product from search B, each cart item should keep its own query_id. The purchase event carries the last search's query_id as a summary, but per-item attribution gives you richer data.

• Accept the limitations. Not every purchase will have search attribution. Direct navigation, category browsing, promotional links, and returning customers who go straight to their cart will all produce purchases without a query_id. That's correct behavior, not missing data.

Spans vs. log events for conversion tracking

Conversion events can emit both an OTel span and a UBI-compatible log event, the same dual-signal pattern used for click tracking in Blog 3. The log event is actually richer for conversions. It can contain the full items list with per-item query_id attribution, which doesn't map cleanly to flat span attributes.

The span gives you the simple, aggregatable view (total revenue by query), and the log gives you the detailed, per-item view (which specific products from which specific searches). For the funnel queries in this post, spans are sufficient. If you need item-level attribution analysis, the log events in logs-generic.otel-default have the detail you need, and ES|QL queries them the same way, just against a different index pattern.

What's next: turning conversion data into relevance improvements

We now have the complete instrumentation picture, with four span types: search (Blog 2), search.result.click (Blog 3), cart.add, and checkout.complete (this post). These capture the full user journey from query to purchase. Every span lives in traces-generic.otel-default, and every metric is queryable with ES|QL. The search.query_id thread ties the entire funnel together.

But measuring the funnel is only half the story. The real payoff is using this data to make search better.

In a later blog in this series, we take everything we've built and turn it into relevance improvements. Click positions and conversion data become judgment lists for Learning To Rank, and per-query CTR and revenue become rank features for boosting. Plus, high-revenue queries get protective monitoring. And tools like Elasticsearch Relevance Studio give you a visual interface for tuning the searches that matter most, using exactly the data you're now collecting.

The instrumentation you've built in Blogs 2–4 is a feedback loop, beyond analytics: Measure, improve, and measure again.

Get started with search conversion tracking

• Reference project: Working code for the entire blog series; clone, configure, and run.

• Elastic Distribution of OpenTelemetry for Python: EDOT for Python.

• OpenTelemetry with Elastic: How to send OTel data to Elastic APM.

• ES|QL documentation: Query language reference.

• UBI Standard: Reference schema for search event structure.

• Query rules: Pin, boost, or exclude results for specific queries.

What you'll discover

In this post, you'll learn how to:

• Add client-side click tracking that links clicks back to their originating search via search.query_id.

• Calculate CTR; that is, the percentage of searches that produce at least one click.

• Calculate MRR; that is, how far down the results users click on average.

• Analyze click position distribution to see the full shape of user engagement.

• Write ES|QL queries for all three metrics against traces-generic.otel-default.

• Identify which specific queries need relevance tuning.

What you'll need

• A working OTel instrumentation setup from Blog 2 (search spans with search.* attributes flowing to Elastic via OTel-native ingestion).

• A front end that can send click events (JavaScript example provided).

• Familiarity with the attributes.* field mapping from Blog 2.

Why search logs alone can't measure search quality

In the second blog of the series, we instrumented search requests and ran six ES|QL queries against the data. We can see what users search for, which queries return nothing, and how fast search is.

But there's a blind spot. A search that returns 15 results looks healthy from the server side. Every metric we have says it worked. But if nobody clicks any of those results, your ranking has a problem, and none of the queries from Blog 2 will tell you.

This is the gap between results returned and results that are useful. Search volume, zero-results rate, and latency measure the mechanics of search, but they don't measure whether search is actually helping users find what they need.

To answer that question, you need a second instrumentation point: click tracking.

If you’re following along with code, the reference project has click tracking ready to enable. Uncomment the Blog 3 sections in app.py and frontend/app.js, restart, and then generate traffic with python generate_traffic.py --blog 3.

How click data measures search relevance

Before we write any code, here's what click data lets you measure and why each metric matters:

• CTR answers the most basic engagement question: What percentage of searches result in at least one click? If your CTR is low, users are seeing results but not finding them compelling enough to engage. Establish your own baseline once you have data; CTR varies considerably across product category, query type, and industry vertical.

• MRR goes deeper: When users do click, where in the results are they clicking? An MRR of 1.0 means every user clicks the top result (a perfect ranking). An MRR of 0.5 means the average click is at position 2. Low MRR with high CTR is particularly telling. It means that users are finding what they need, but your ranking is making them work for it.

• Click position distribution shows the full shape of where users click. A healthy search engine shows most clicks at position 1 with a sharp drop-off. A flat distribution across positions 1–5 means that your ranking isn't differentiating well. Per-query distributions reveal exactly which searches need relevance tuning.

Together, these metrics move you from Did search work? (Blog 2) to Did search work well?, and they pinpoint exactly where to invest in relevance improvements. Later in the series, we'll show how to turn these metrics into concrete actions: building judgment lists for Learning To Rank (LTR), tuning relevance with tools like Elasticsearch Relevance Studio, and evaluating changes with the Rank Eval API. But first, you need the data.

All three metrics require just one new instrumentation point: about 15 lines of code.

Add click tracking

Click tracking captures what happens after the results appear. When a user clicks a search result, we create a new span with attributes describing the interaction, including which document they clicked, where it appeared in the results, and which search produced it.

Here's the code:

Let's unpack what matters.

Click spans are separate traces

This is the key architectural difference from Blog 2. Search spans are created synchronously during the API request: The user searches, the span opens, Elasticsearch responds, and the span closes. Click spans are asynchronous. The user searches, gets results, browses the page, and might click 30 seconds later (or they might never click).

That means click spans aren't children of the search span's trace. They're independent traces, linked to the originating search through search.query_id. This is the same query_id we derived from the trace ID in Blog 2, and it now serves as the join key between searches and clicks across traces-generic.otel-default.

Choosing the right OTel signal for clicks

OTel gives you three signal types, and clicks could be modeled as any of them. Each has strengths:

Signal

Index

Weight

Best for…

Spans

traces-generic.otel-default

Full trace context

Same-index queries with search spans

Logs

logs-*

Lighter weight

Log-centric pipelines, high volume

Span events

logs-generic.otel-default

Lightest instrumentation

Attaching to existing spans

• Spans land in traces-generic.otel-default alongside search spans, are fully queryable in ES|QL, appear in Kibana APM views, and carry timing information. Since our search spans are already in traces-generic.otel-default, using spans for clicks means you can query searches and clicks together in a single ES|QL statement, and no cross-index joins are needed.

• Log records are also independently queryable in ES|QL, living in logs-*. If you use the same search.* attribute names, the analytics queries are almost identical; just change the index pattern. Logs are lighter weight (no trace context overhead) and are a natural fit if your team already has a log-centric observability pipeline. One of Elastic's strengths here is that traces, logs, and metrics all land in the same platform and are all queryable with ES|QL, so choosing logs over spans doesn't mean giving up any query capability.

• Span events are lightweight at instrumentation time (attached to an existing span in the OTel API). In Elastic's OpenTelemetry Protocol (OTLP) ingestion pipeline, span events are written as separate documents to logs-* data streams (for example logs-generic.otel-default), and they’re independently queryable in the same way as logs. They’re a good, lightweight option but might require more code changes than logs, which can even pull in logs from legacy code.

In this series, we use spans because they keep searches and clicks in the same index with the simplest query path. But if you're at high volume and want to optimize for cost, or if your organization already routes OTel logs to Elasticsearch, the log-based approach works well; the search.* attribute schema is the same either way, and ES|QL queries against logs-* follow the same patterns you'll see below.

The search.first_click attribute

search.first_click is a Boolean set only on the first click for a given query_id. It exists for one reason: accurate CTR calculation without post-processing.

CTR is defined as the percentage of searches with at least one click. Without search.first_click, you'd need to deduplicate clicks by query_id at query time: grouping, counting distinct values, and subquerying. By marking the first click at instrumentation time, the ES|QL query becomes a simple count.

The set above is demo-only. It grows unbounded and breaks with multiple API replicas. The reference implementation uses a thread-safe time-to-live (TTL) dict with a 30-minute expiry window (_is_first_click() in app.py). For production with multiple replicas, use a shared external cache (Redis, Memcached) keyed by query_id with a TTL matching your session window.

Where to track first click: Front end vs. back end

The search.first_click deduplication could live in either the front end or the back end. Both are valid, and here are the trade-offs:

• Frontend tracking is simpler to implement. The browser already knows the current query and whether the user has clicked before. No server-side state is required, you don’t have to worry about multiple replicas, and it works without any backend changes. The downside is that browser state is ephemeral; a page refresh, multiple tabs, or an ad blocker can interfere with accurate tracking.

• Backend tracking (our approach) gives you a single source of truth. All click events flow through one place, so the deduplication is consistent regardless of what the client does. It also means that the analytics logic is colocated with the instrumentation code, which simplifies reasoning about data quality. The trade-off is that the back end needs to maintain state: the _clicked_queries set. For a single-instance API, this is trivial; for multiple replicas behind a load balancer, you'd use a shared TTL cache (Redis or similar).

We chose backend tracking here because we want the analytics data to be authoritative. This click data will later feed into relevance tuning and judgment lists, where accuracy matters. But if you're starting simple or running a client-side–only setup, frontend tracking is a perfectly reasonable first step. The search.first_click attribute works the same way regardless of where you set it.

Sending click events from the browser

The browser sends click events to the back end when a user clicks a result. It needs three things from the search response: the document ID, the position, and the query_id.

CLIENT_ID is generated once and stored in localStorage. It survives page reloads and gives you a stable enduser.pseudo.id without requiring a login. The back end maps client_id → enduser.pseudo.id on the span.

Positions are 1-indexed; that is, the first result is position 1, not 0.

Click tracking OTel attributes and ES|QL field mapping

Attribute

Type

Required

Purpose

search.action

string

yes

Event type: "click"

search.result_click_id

string

yes

Document ID clicked

search.result_click_position

int

yes

Position in results (1-indexed)

search.query_id

string

yes

Links to originating search

enduser.pseudo.id

string

yes

Client/device identifier

search.first_click

boolean

recommended

true if first click for this query_id

search.result_click_type

string

recommended

Object type: "product", "article"

search.query

string

recommended

The search query text (for queryability)

These follow the same search.* namespace we established in Blog 2. With OTel-native ingestion, attributes map directly to attributes.* fields in ES|QL:

OTel attribute

ES|QL field

search.action

attributes.search.action

search.result_click_id

attributes.search.result_click_id

search.result_click_position

attributes.search.result_click_position

search.query_id

attributes.search.query_id

search.first_click

attributes.search.first_click

With OTel-native ingestion, search.first_click is stored as a native Boolean; you query it with == true, not == "true", and no string coercion is needed.

Verify that clicks are arriving

Before calculating metrics, confirm that click spans are flowing to APM:

If this returns rows, you're ready for analytics. If not, check the same things as we looked at in Blog 2: OTLP endpoint, auth token, and span export.

Note: Results in this post are illustrative, generated by running python generate_traffic.py --blog 3 --sessions 50 on the reference project. Running Blog 3 traffic adds click events and additional search sessions on top of the 62 from Blog 2, so cumulative search counts will exceed 62. Your exact numbers will vary based on session count and the random nature of the traffic simulator. The metric calculations and ES|QL patterns are what to focus on.

CTR

CTR is the primary signal for search relevance, answering the question that Blog 2 couldn't: Are users engaging with the results?

CTR = searches with at least one click / total searches * 100

CTR is a binary per-search metric: Either a search got clicked or it didn't. The maximum is 100%.

Overall search CTR with ES|QL

Result: 41 clicked searches out of 146 total. CTR: 28.1%

This is a single query that pulls both search spans and first-click spans from traces-generic.otel-default. The OR in the WHERE clause brings both into one result set. COUNT(CASE(...)) counts each type separately, and EVAL does the division.

This works because of search.first_click. Without it, you'd be counting raw clicks (a user who clicks three results on one search would inflate the count). The deduplication happened at instrumentation time; the query stays simple.

CTR by search query: Finding your worst relevance failures

The overall number is useful for dashboards. The per-query breakdown is where you find problems.

This shows CTR broken down by query text. 

What CTR tells you (and what it doesn't)

• High searches + zero clicks: These are the worst relevance failures. Fix these first.

• High searches + low CTR: Results appear, but they aren't compelling. Check ranking.

• Low CTR + high zero-results rate: This is a double problem; either no results or bad results.

• CTR trend over time: This measures the impact of relevance changes.

CTR doesn't measure satisfaction. A user who clicks position 1, bounces back, and then clicks position 3 still counts as one clicked search. For a fuller picture, you need to know where they're clicking. That's what MRR measures.

CTR vs. clicks per search

CTR (what we just calculated) is capped at 100%. This is the industry-standard definition.

Clicks per search is total clicks divided by total searches. It can exceed 1.0. For example, a search where the user clicks three results scores 3.0. It measures engagement depth, which is useful but different. If you need it, count all click spans (not just first_click) divided by search spans.

MRR

MRR tells you where users click. It measures how far down the results list users go before finding something worth clicking.

MRR = average of (1 / click_position) across all clicks

The reciprocal rank transforms click positions into a 0–to–1 scale, where higher is better:

Click position

Reciprocal rank

1

1.000

2

0.500

3

0.333

5

0.200

10

0.100

Overall search MRR with ES|QL

MRR can be calculated two ways, depending on what you want to measure:

• All-click MRR: Averages the reciprocal rank of every click and reflects overall click quality, including repeated interactions.

• First-click MRR: Averages only the first click per search (using search.first_click == true). It’s more comparable to traditional information retrieval (IR) evaluation and aligns with how you computed CTR.

For consistency with CTR and alignment with judgment-list workflows in Blog 5, we prefer first-click MRR:

Result: MRR = 0.495

This is decent but shows room for improvement. An MRR of 0.495 means the average first click lands around position 2. It isn’t a crisis, but there are queries where ranking can be improved.

MRR by search query: Finding your worst-ranked results

Like CTR, the per-query breakdown is where the actionable data lives. To surface the worst-ranked queries first, sort ascending.

This reveals which queries have the worst ranking. A query with multiple clicks and low MRR means that the ranking is consistently poor for that search; that is, users find results, but they have to dig for them.

What is a good MRR score for search?

• MRR > 0.8: This ranking is solid; users usually click position 1–2.

• MRR 0.5–0.8:  This is decent, but there’s room for improvement.

• MRR < 0.5: This is a ranking problem, and users are scrolling past top results.

• MRR drop after a change: This is ranking regression that should be investigated immediately.

• Low MRR + high CTR: Users are finding things, but they have to work for it.

That last pattern is particularly interesting. High CTR with low MRR means your results are relevant (users are clicking), but your ranking isn't surfacing the best results first. It's an optimization opportunity, not a crisis.

MRR limitations: Position bias and multi-click sessions

MRR is heavily influenced by the gap between position 1 and position 2 (1.0 versus 0.5). Positions 5 and beyond barely move the average. This means that MRR is most sensitive to whether your top result is good, which is often exactly what you want to optimize.

MRR also only measures clicks, not satisfaction. With all-click MRR, a user who clicks position 1, bounces, and then clicks position 3 contributes two data points, but only the second was useful. The first-click MRR queries above avoid this by counting only the first click per search via search.first_click == true.

Click position distribution

Click position distribution shows you the full picture of where in the results users are engaging.

Results:

Position

Clicks

1

21

2

10

3

7

4

4

5

3

This is a reasonable distribution: 21 of 45 clicks (47%) land on position 1, with a tapering tail. If you paste this query into Discover's ES|QL editor, Kibana auto-generates a bar chart that makes the shape immediately visible.

How to read click position distribution for search relevance

• Sharp dropoff after position 1: The ranking is effective, and the top result is usually right.

• Flat across positions 1–5: The ranking isn't differentiating well, and all positions are equally likely to be clicked.

• Spike at position 3+ for specific queries: Those queries have ranking problems.

• No clicks beyond position 5: Users don't scroll far. Top 5 ranking matters most.

Click position distribution by search query

To see the shape for specific queries:

A query where all clicks land on position 1 has perfect ranking. A query where clicks spread across positions 1–5 needs relevance tuning.

Position bias and click models

One caveat: Click position distribution is influenced by position bias; that is, users see position 1 first, so it gets clicked more regardless of relevance. A click at position 1 isn't necessarily more relevant, just more visible.

This is a well-studied problem in information retrieval. Click models are statistical models that attempt to separate genuine relevance from position bias in click data. The foundational work by Joachims et al. (2005) showed that users are significantly biased toward higher-ranked results, and, in proposed methods like skip-above analysis (if a user clicks position 3 but skips positions 1 and 2), those skipped results are likely less relevant for that query.

For the metrics in this post, you don't need to implement a full click model. The key insight is practical: Compare distributions between queries rather than treating absolute position counts as ground truth. If query A has 80% of clicks at position 1 and query B has clicks spread across positions 1–5, query B's ranking is worse, even accounting for position bias. Later in the series, when we look at building judgment lists for LTR, position bias correction becomes more important, and the click data you're collecting here is exactly what those models need as input.

CTR, MRR, and click distribution: Reading search quality metrics together

These three metrics offer three different angles on search result quality:

Metric

What it measures

Our value

Interpretation

CTR

Do users click at all?

28.1%

Moderate: Roughly a third of searches get engagement, but there’s room to improve.

MRR

Where do they click?

0.495

Decent: The average click is around position 2, but ranking can be improved.

Distribution

What's the shape?

47% at position 1

Reasonable drop-off: The top result wins most but isn’t dominant.

Together, they tell a coherent story. For our demo data, search is performing adequately: Users are engaging with results and can find what they need, but the ranking has room to improve. The CTR of 28% and MRR of 0.495 are realistic starting points for a new search implementation without tuning.

Where they're most valuable is in combination at the query level. The queries to fix first are those with high volume + low CTR + low MRR; that is, lots of users are searching, few are clicking, and those who do click are scrolling deep. That's where relevance investment has the highest return.

Using click data for LTR and relevance tuning

These metrics don't just tell you how search is performing; they're the foundation for making it better. The click data you're now collecting feeds directly into relevance improvement workflows:

• Judgment lists for LTR: Click positions and frequencies become graded relevance labels for training machine learning (ML) ranking models. A document clicked at position 1 across many queries is a strong positive signal.

• Relevance tuning tools: Per-query CTR and MRR tell you exactly which queries to focus on in tools like Relevance Studio or the Rank Eval API, which scores your ranking against expected results.

• Query rules and boosting: Zero-CTR queries with results are candidates for pinning, boosting, or synonym rules.

We'll cover these applications in detail in Blog 5. For now, the important thing is that the instrumentation you've built here is doing double duty: It measures search quality and provides the training data to improve it.

Next in the series: Conversion tracking from search to purchase

We can now measure whether users find results (Blog 2) and whether they engage with them (this post). But a click isn't a conversion. A user who clicks a product and then abandons the page didn't get what they needed.

In the next post, we add conversion tracking, the third instrumentation point that closes the loop from search to purchase. It’s the same pattern: Add search.* attributes to add-to-cart and checkout spans, query with ES|QL, and answer the question your product manager actually cares about: Which searches drive revenue?

Get started

• Reference project: Working code for the entire blog series (clone, configure, and run).

• Elastic Distribution of OpenTelemetry Python: EDOT Python.

• OpenTelemetry with Elastic: How to send OTel data to Elastic APM.

• ES|QL documentation: Query language reference.

• UBI standard: Reference schema for search event structure.

This is the third post in a series on search analytics with OpenTelemetry and Elastic. Next up: From clicks to conversions: Conversion tracking, funnel analysis, and revenue attribution.

What you'll discover

In this post, you'll learn how to:

• Set up OpenTelemetry in an example Python FastAPI back end.

• Add custom search.* attributes to your search spans in ~20 lines of code.

• Understand how OTel-native ingestion maps attributes to queryable data.

• Write six ES|QL queries against real trace data: top queries, zero-results rate, which queries return nothing, average and max latency, slow query investigation, and search volume over time.

• Turn those queries into saved Kibana visualizations.

What you'll need

• An Elastic Cloud deployment (or self-managed with OTel-native ingestion enabled). Examples tested on Elastic Stack 9.x.

From concept to code: Building the search analytics instrumentation

In the first post, we made the case for using OpenTelemetry to capture search analytics. The idea: Add search.* attributes to your existing OTel spans, send them to Elastic APM, and query them with ES|QL.

Now let's build it.

Want working code? A companion reference project accompanies this series. It's a minimal FastAPI app with the exact instrumentation described below. Clone it, add your Elastic Cloud credentials, and you'll have search analytics data flowing in 10 minutes. Each blog stage maps to a commented-out code block you can enable as you progress.

An OpenTelemetry primer for search API developers

If you've been building search systems but haven't worked with OpenTelemetry before, here's the minimum you need to know.

OTel is an open standard for collecting observability data, traces, metrics, and logs from your applications. It's vendor-neutral: You instrument your code once, and send data to any compatible back end.

The core concept is the span. A span represents a single operation, for example, an API call, a database query, or a search request. Every span has a start time, an end time (the difference is the span duration), and attributes, which are key-value pairs that describe what happened.

Spans nest inside each other to form traces. A trace is a tree of spans that represents one end-to-end request. When a user searches, the trace might look like: browser request → API handler → search logic → Elasticsearch query. Each step is a span, and the parent-child relationships show you exactly where time was spent. This is distributed tracing, which works across services and network boundaries, so a single trace can follow a request from front end to back end to database and back.

Why traces instead of logs? You could log "search query=headphones results=15 took=120ms" and parse it later. But a log line is flat; it can't show you that the 120ms Elasticsearch time sat inside a 250ms API call, revealing 130ms of overhead in your application layer. Traces give you hierarchy, timing, and correlation across services. For search analytics, that means you can see not just what happened but also where time was spent and how operations relate to each other.

For this post, we don't need to understand the full OTel ecosystem. We just need three things:

1. Create a span when a search request happens.

2. Add attributes to that span, describing the search (such as search.query or result_count).

3. Send the span to Elastic, where we can query it with ES|QL.

That's it. If you can call span.set_attribute("key", value), you can build search analytics.

What you'll build

By the end of this post, every search request in your API will emit an OTel span that looks like this:

And you'll run six ES|QL queries against real data to answer questions that your team is already asking, using about 20 lines of instrumentation code in total.

Install the OTel SDK

We're using Python and FastAPI here. The same pattern applies to any language with an OTel SDK; the concepts are identical, only the imports change.

Elastic provides the Elastic Distribution of OpenTelemetry Python (EDOT), which bundles the standard OTel SDK with sensible defaults, early access to Elastic-contributed improvements, and a single configure_opentelemetry() call that handles all the boilerplate. We recommend it:

Three packages, two roles:

• elastic-opentelemetry EDOT: The OTel API, SDK, and OpenTelemetry Protocol (OTLP) exporter in one package, preconfigured for Elastic.

• opentelemetry-instrumentation-fastapi: Auto-instruments HTTP endpoints (automatic spans for every request).

• opentelemetry-instrumentation-elasticsearch: Auto-instruments Elasticsearch client calls (automatic spans for every query).

The auto-instrumentation packages are doing real work here. Without writing a single line of tracing code, you already get HTTP request spans and Elasticsearch query spans. What we're adding is the search-specific context that turns generic traces into analytics.

Using the standard OTel SDK instead? Replace elastic-opentelemetry with opentelemetry-api, opentelemetry-sdk, and opentelemetry-exporter-otlp-proto-http. You'll need to wire up the TracerProvider, OTLPSpanExporter, and BatchSpanProcessor manually (about 10 extra lines). Everything else in this post works the same either way. See the Elastic OTel guide for the full setup.

Configure the connection

OTel uses environment variables for connection configuration. You need four to get started:

Variable

Purpose

Example

`OTEL_EXPORTER_OTLP_ENDPOINT`

Managed OTLP (mOTLP) endpoint URL

`https://my-deployment.ingest.us-central1.gcp.elastic-cloud.com`

`OTEL_EXPORTER_OTLP_HEADERS`

Authentication

`Authorization=ApiKey <your-api-key>`

`OTEL_SERVICE_NAME`

Service name (shown in Kibana APM)

`search-analytics-demo`

`OTEL_RESOURCE_ATTRIBUTES`

Other resource attributes

`service.version=1.0.0`

Where to find these values: In Elastic Cloud, your mOTLP endpoint follows the pattern https://<deployment>.ingest.<region>.gcp.elastic-cloud.com. You can find it in the Elastic Cloud console under your deployment's details or in Kibana at the APM integration page (/app/home#/tutorial/apm) under the OpenTelemetry tab. Your API key can be created from Kibana's Stack Management > API Keys or via the Elasticsearch Create API Key API. For self-managed deployments, you can use the EDOT Collector as an intermediary that receives OTLP and forwards to Elasticsearch.

Set them in your environment or .env file:

Initialize the tracer

With EDOT and environment variables configured, initialization wires up three things: the tracer provider and two auto-instrumentation packages that automatically create spans for every HTTP request and every Elasticsearch query:

configure_opentelemetry() reads the OTEL_* environment variables and sets up the tracer provider, exporter, and batch processor with Elastic-optimized defaults, including using the HTTP exporter automatically, which is what the mOTLP endpoint requires. If you see connection errors with vanilla OTel, the most common cause is accidentally using the gRPC exporter instead of HTTP.

The two instrumentors patch the FastAPI and elasticsearch-py libraries at startup so every request and every Elasticsearch call automatically generates a span, without any code changes to individual endpoints.

Instrument your search API

Here's where it gets interesting. This is the code that turns a generic API endpoint into a search analytics source:

A few things worth unpacking.

• start_as_current_span creates the span and sets it as the active span in the current context. This matters because the Elasticsearch client instrumentation picks up the active span and nests its own spans underneath it. You get a span hierarchy automatically.

• search.query_id is derived from the trace ID. Every trace already has a unique identifier, and we're reusing it as the query identifier. There’s no UUID generation or database sequence. When we add click tracking later, clicks will reference this same query_id to link back to the search that produced the results.

• search.result_count does double duty. It tells you how many results came back, and when it's zero, you know you have a content gap. There’s no need for a separate boolean flag: Just filter on result_count == 0 in your queries.

Note that the application name is no longer set as a span attribute; it's the service.name resource attribute, configured once via OTEL_SERVICE_NAME. This is the standard OTel approach: Resource attributes describe the service, and span attributes describe the operation.

Normalize search queries before analyzing them

Notice that we're storing request.query as is. That means "Laptop Bag", "laptop bag", and " laptop bag " will be counted as three different queries when you aggregate with STATS ... BY attributes.search.query.

For cleaner analytics, normalize before setting the attribute:

Lowercasing and trimming whitespace is enough for most cases. If you need the original phrasing (for display or debugging), store it in a separate attribute, like search.query.original. But start simple. You can always add the raw version later if you find you need it.

What a search API trace looks like

Once this is running, a single search request produces this trace:

The auto-instrumented spans give you HTTP latency and Elasticsearch query detail. Your search span in the middle ties them together with the business context: what the user searched for, how many results came back, how long Elasticsearch took.

The search span attributes you need to capture

Here's the full set of attributes we're capturing on the search span:

Attribute

Type

When set

Purpose

`search.query`

string

Before query

The query as the user entered it

`search.query_id`

string

Before query

Unique identifier, derived from trace ID

`search.result_count`

int

After query

Total matching results (0 = zero-result search)

`search.took_ms`

int

After query

Elasticsearch execution time in milliseconds

`search.query_response_hit_ids`

string[]

After query

Document IDs returned (optional; enables per-result analytics)

`feature_flag.key`

string

Before query

A/B test flag name (optional; pair with `feature_flag.result.variant` for the assigned variant; enables per-variant click-through rate (CTR) comparison)

We use the search.* namespace following OTel's convention of domain-specific prefixes (http.*, db.*, messaging.*). While there aren't standardized search conventions in OTel yet, search.* is self-describing and vendor-neutral. The naming is informed by the User Behavior Insights (UBI) Standard, which defines a schema for search events. We reference it for structure without coupling to it. Where established OTel conventions exist, like feature_flag.key (flag name) and feature_flag.result.variant (assigned variant) for A/B experiments, we reuse them rather than inventing custom attributes.

You'll notice that enduser.pseudo.id isn't in the search span table above. We don't need it for query analytics, but you'll add it as soon as you introduce click tracking in our third blog; it ties click events back to a specific browser session, enabling per-user CTR and Mean Reciprocal Rank (MRR). Blog 3 adds enduser.pseudo.id (browser-generated, persistent across sessions) to link clicks back to searches. Our fourth blog focussing on revenue attribution documents session.id and user.id as optional extensions for authenticated users who want cross-device attribution.

How OpenTelemetry attributes become queryable ES|QL fields

Before we start querying, you need to understand how OTel attributes map to Elasticsearch fields. With OTel-native ingestion into Elastic, the mapping is straightforward.

OTel attribute

Type

ES|QL field

`search.query`

string

`attributes.search.query`

`search.result_count`

int

`attributes.search.result_count`

`search.took_ms`

int

`attributes.search.took_ms`

`search.query_id`

string

`attributes.search.query_id`

`feature_flag.key`

string

`attributes.feature_flag.key`

With OTel-native ingestion, attribute names preserve their dot notation under attributes.*. All types live in the same namespace; there’s no split between string and numeric fields. Booleans are stored as native booleans, not strings. If you've used Elastic APM's classic ingestion before, you'll appreciate the simplicity: What you set in code is what you query.

Running ES|QL queries in Kibana Discover

With spans flowing to Elastic, open Kibana and go to Discover (in the left sidebar under Analytics, or use the global search bar and type "Discover"). By default, you'll see the KQL query bar, a filter language familiar from Kibana dashboards. Click Try ES|QL in the top right to switch to the ES|QL editor. Unlike KQL (which filters documents) or the JSON query DSL (which requires nested objects), ES|QL is a piped language: Each | step transforms the previous output, making aggregations like STATS count BY field read naturally from left to right.

The editor gives you a full-width text area where you type piped queries. Results appear as both a table and an auto-generated chart; Kibana picks a sensible visualization based on your query shape. For STATS ... BY queries, you'll get a bar chart automatically.

Set the time range wide enough to capture your data (top-right date picker). If you're just getting started, try "Last 30 days".

Six ES|QL queries for search analytics

Everything below runs against traces-generic.otel-default, the index where Elastic's OTel-native ingestion automatically stores trace data. You don't need to create this index; it's provisioned by Elastic when the first OTLP span arrives.

These queries ran against our live demo cluster: 62 searches, ~20 distinct queries, latency range 77ms–153ms.

Note: Results below are illustrative. When you run the reference project and generate traffic with python generate_traffic.py --blog 2 --sessions 50, your exact numbers and top queries will vary based on session count and random query selection. The query patterns and ES|QL syntax are what matter here.

Query 1: Are spans arriving?

Start simple. Count your search spans.

Result: 62.

If this returns zero, your spans aren't arriving. Check your OTLP endpoint and API key and that init_otel() is being called before any requests. The name == "search" filter ensures that you're counting your custom spans, not the auto-instrumented Elasticsearch client spans (which are also named "search").

Query 2: What are users searching for?

The first question every search team asks.

Results:

Query

Searches

Average results

laptop

9

8

headphones

7

12

running shoes

6

5

"laptop" was the most popular query, with nine searches. There were around 20 distinct queries total (including zero-result ones).

The avg_results column tells you whether popular queries are actually returning content. A query with high volume and low results is a relevance problem worth investigating. If your query has high volume and high results, check whether users are actually clicking. We'll get to that in the next blog focussing on measuring search quality with click data.

Query 3: What percentage of searches return nothing?

Result: 17.7% (11 out of 62 searches returned nothing).

We're using attributes.search.result_count == 0, a straightforward numeric comparison. No separate boolean attribute is needed when you already have the count.

A zero-results rate above 10% is worth investigating. Every zero-result search is a user who asked for something and got nothing back. Some of those are junk queries, but others reveal real content gaps or query parsing failures.

Query 4: Which queries return nothing?

The rate tells you there's a problem. This query tells you where.

Results:

Query

Occurrences

quantum physics calculator

4

unicorn saddle

3

holographic projector

2

time machine parts

2

Three different failure modes: "quantum physics calculator" and "unicorn saddle" are out-of-catalog queries you'll never be able to serve. This is useful to know but nothing to fix. "holographic projector" might be a real emerging category worth considering. "time machine parts" is probably noise. In a production catalog, these would be mixed with legitimate zero-result queries that are fixable, like missing synonyms, phrasing mismatches, or product gaps.

Repeated zero-result queries are the highest-priority fixes. One-off failures are usually noise.

Query 5: How fast is search?

Result: Average 81ms, max 153ms.

search.took_ms captures Elasticsearch's self-reported execution time, the took field from the search response. This is different from span duration, the wall-clock time from when the span started to when it ended (as we covered in the primer above). Span duration measures end-to-end time, including network round trips, serialization, and application logic. You want both: Comparing them tells you where overhead lives. If took_ms is 50ms but the span duration is 200ms, the extra 150ms is network or application overhead, not a query problem.

This attribute also keeps your analytics portable. If you're using OTel log records instead of spans (a lighter-weight alternative we mention in the first blog in the series), there's no span duration. took_ms is the only timing signal you have.

We'll go deeper on search performance monitoring (Service Level Objectives [SLOs], alerting on latency regressions, and using this data for operational dashboards) in the last blog in the series focussing on Search Reliability Engineering.

Want to find the slow queries specifically?

A query with high average latency and high result count is hitting many documents; consider query optimization. High latency with low results might mean complex filters or slow aggregations. Outlier max values are often cold caches or cluster issues.

Query 6: How does search volume change over time?

Counts, rates, and latencies tell you the what. Volume over time tells you the when: When did traffic spike, when did it drop, and when did that zero-results rate jump?

DATE_TRUNC(5 minutes, @timestamp) rounds each timestamp down to the nearest 5-minute boundary. The result is a time series that Kibana's Lens can render as a bar chart or line, showing your search traffic pattern for any time window.

Narrow the bucket for higher granularity (1 minute), widen it for trend analysis (1 hour, 1 day). When you add this to a dashboard alongside your zero-results rate, you can answer: Did zero-results spike because traffic changed or because something broke?

Verify that your search API instrumentation is working

If you're using the reference project, the full setup is:

Then trigger a search:

Wait 5–10 seconds for the BatchSpanProcessor to flush, and then open Kibana → Discover → switch to ES|QL mode and run:

You should see rows with attributes.search.query, attributes.search.result_count, and attributes.search.took_ms.

If no rows appear, check in order:

1. OTEL_EXPORTER_OTLP_ENDPOINT points to the mOTLP endpoint, not your Elasticsearch URL.

2. OTEL_EXPORTER_OTLP_HEADERS includes Authorization=ApiKey <your-key>.

3. OTEL_TRACES_SAMPLER=always_on is set (default sampler may drop spans).

4. Kibana → Observability → APM → Services shows search-analytics-demo (confirms export is working).

Turn ES|QL results into Kibana visualizations

The bar chart that Discover auto-generates from your ES|QL results is a good start, but you can customize it. Click the pencil icon in the top-right corner of the chart to open the inline Lens editor.

From here you can:

• Change chart type (bar, line, area, pie, table, metric).

• Adjust axes and add breakdown dimensions.

• Save the visualization to a dashboard.

This is the path from ad hoc ES|QL exploration to a persistent dashboard panel. You don't need to build visualizations from scratch; Discover and Lens handle the chart rendering from your query results.

Lens is Elastic's drag-and-drop visualization editor, and it's more capable than this quick workflow suggests. You can build multilayer charts, combine metrics with breakdowns, add reference lines, and design full dashboards that mix ES|QL panels with traditional aggregation-based visualizations. For search analytics, that means you can put top queries, zero-results trends, and latency percentiles side by side in a single view.

To go deeper:

• Lens documentation: A full guide to the visualization editor.

• ES|QL in Lens: Using ES|QL queries as data sources for dashboard panels.

• Kibana Dashboards: Building and sharing operational dashboards.

• Use an Agent built in Kibana to make visualizations for you

We'll build a full search analytics dashboard in a later post.

How sampling affects search analytics accuracy

Most application performance monitoring (APM) configurations sample traces to control costs, capturing 10% or 25% of requests. For application monitoring, that's fine. For search analytics, it's a problem.

If you're sampling at 10%, your "total searches" count is 90% lower than reality. Your zero-results rate is still accurate (it's a ratio), but volume counts are off.

Two approaches:

Approach 1: Configure 100% sampling for search endpoints. Your search API probably handles far fewer requests than your main application, so the data volume increase is manageable. The simplest way is through environment variables:

These are head-based sampling decisions made at the start of each trace. They apply globally to the service, which is fine if your search API is a dedicated service. If search shares a service with other endpoints and you need per-endpoint sampling rules, you can implement a custom Sampler in the OTel SDK that inspects the span name or attributes before deciding.

More sophisticated routing (sampling differently per endpoint, dropping noisy spans, or making decisions after a trace completes [tail-based sampling]) typically involves deploying an OTel Collector (such as the EDOT Collector) as an intermediary between your application and Elastic. That's a valuable architecture pattern, but it's beyond the scope of this post. See the OTel Collector documentation and Elastic's EDOT Deployment for more on collector-based sampling and routing architectures.

Approach 2: Upscale in your queries. If you know the sampling rate, multiply: EVAL estimated_total = total_searches * 10. Ratios and averages stay correct; only absolute counts need adjustment.

For more on sampling strategies generally, see the OTel sampling documentation.

For the queries in this post, we used 100% sampling.

What's next: Adding click tracking to search analytics

The six ES|QL queries in this post answer: What do users search for, what returns nothing, how fast is search, and when does traffic spike? They're all derived from a single instrumentation point: the search span.

But they can't tell you whether users are finding what they need. A search that returns 15 results looks healthy from the server side. But if nobody clicks any of those results, your ranking has a problem.

In the next post, we add click tracking, a second span that captures which result the user clicked and where it appeared in the list. If you've been running the reference project, you already have 62 search spans; the next post builds directly on that data. With searches and clicks linked together, we'll calculate:

• Click-through rate (CTR): What percentage of searches result in a click.

• Mean Reciprocal Rank (MRR): How far down the results users have to scroll.

• Click position distribution: The shape of where users click.

The pattern is the same: You add attributes to spans and query them with ES|QL. You get a richer view of the same data without introducing new infrastructure.

Resources to get started with search analytics on OpenTelemetry

• Reference project: Working code for the entire blog series; clone, configure, run.

• EDOT Python: Elastic distribution of OpenTelemetry for Python.

• OpenTelemetry with Elastic: How to send OTel data to Elastic APM.

• OpenTelemetry Python SDK: Upstream SDK documentation and instrumentation guides.

• ES|QL documentation: Query language reference.

• UBI Standard: Reference schema for search event structure.

This is the second post in a series on search analytics with OpenTelemetry and Elastic. Next up: Measuring search quality: Click tracking, CTR, MRR, and click position analysis.

AI shopping agents fail not because the model is wrong but because the agent arrives at every conversation not knowing your catalog, vocabulary, or business rules. It has to discover all of this context through tool calls, and that discovery is the cost. Precomputing a structured context layer from signals you already hold (vocabulary, policies, user profiles, session behavior) cuts the exploratory work the agent does before it can answer and makes its behavior governed and predictable.

In the analogous document-retrieval case, precomputing context reduced input tokens by up to 75% on a controlled benchmark; we expect comparable savings in ecommerce because the exploration pattern is the same, though the exact figure will vary by catalog and query mix. The signals are richer in ecommerce than in almost any other domain, and most retailers already have them. The question is whether or not these signals are assembled in a form that the agent can use before it starts reasoning. Elastic’s broad mix of search capabilities, semantic, hybrid, keyword, filtering, and aggregations make it a compelling choice for not only building your core retrieval tools but also as the vital context engine.

Why AI shopping agents fail in production

Retailers investing in AI shopping assistants are discovering an uncomfortable gap between what the demos promise and what the first few months in production deliver.

The assistant takes four seconds to respond. It confidently recommends a product in a size range that doesn't exist for that item. It tells a returning customer about a coat style they bought 18 months ago and returned. It filters by a category name that doesn't match the internal taxonomy and returns zero results. The customer gives up and abandons the chat altogether.

These aren't model failures. The frontier models powering these agents are capable of extraordinary reasoning when they have the right information in front of them. The problem is that the agent arrives at the conversation knowing nothing about the retailer's catalog, customer, or business rules that govern what should and shouldn't be recommended. It has to learn all of this through the conversation itself, making exploratory tool calls to discover what departments exist, what filter values are valid, and what the brand's policies are on certain product types. Every one of those discovery calls costs latency and tokens before the agent has said a single useful thing to the customer.

Latency matters in ecommerce in a way it doesn't in many other contexts. Shoppers expect response times measured in seconds, not the minutes that enterprise knowledge-base agents routinely take. It’s well established that slower responses reduce engagement and conversion in online retail. An AI shopping assistant that thinks visibly for five seconds before answering a question about gift ideas isn't a feature; it's friction.

The fix isn't a faster model or a bigger context window. The agent's latency and cost problem is a context problem. This can be solved by carefully computing context before the agent call, not during it.

What an AI shopping agent knows before it searches

Imagine you’re the agent. You have a search tool connected to the catalog, and this query arrives:

"an outfit for an autumn wedding"

What do you actually do with that?

Start with what you don't know. An outfit for a man or a woman? Is "autumn" a color, a season, a style of fabric, or just when the wedding happens? And who is this shopper, someone who has bought from you for years, or a stranger? Do they buy expensive designer brands or do they always hunt out a bargain on sale? You have none of these answers. So you do what an agent does when it's working blind: You ask loads of follow-up questions, guess, or fire off a series of exploratory searches to find out what departments and filter values even exist, watching the seconds tick by before you've offered the customer anything at all.

Hold onto that feeling of working blind. The rest of this article is about what changes when the agent is handed the answers first.

Why ecommerce is different from general RAG

Most of the published work on reducing agent costs focuses on document retrieval: question-answering over corpora of articles, reports, or knowledge-base entries. A recent experiment from the Elastic team (Cutting agent costs with pre-computed context) demonstrated that pre-extracting structured facts from documents before the agent call reduced input token consumption by up to 75% and improved answer accuracy from 60% to 92% on a hard factual benchmark. That improvement came in stages, with the largest jump driven by feeding the agent's own wrong answers back into the extraction step rather than by precomputing context alone, which is a distinction we'll return to when we discuss governance.

Ecommerce applies the same principle to a fundamentally different structure. A product catalog isn't a document corpus. It's a highly structured index of items with strict field semantics, a domain-specific vocabulary of brand names, color codes, and category hierarchies, and a layer of business rules that override pure relevance in specific situations.

The failure modes that result are distinct from document retrieval augmented generation (RAG) failures:

• Vocabulary mismatch: A customer asks for a "navy jumper." The agent constructs a filter against a field where the canonical value is NAVY and the category is stored as Knitwear & Jumpers. Without a vocabulary mapping, the agent either guesses colors and categories and gets it wrong, or makes multiple exploratory calls to discover what values exist before it can filter correctly.

• Hallucinated filter values: Without knowing which filter dimensions are valid for a given query, agents can construct queries against fields that don't exist or with values that return zero results. A filter like category: knitwear looks reasonable; masterCategoryNames: "Knitwear & Jumpers" is what the index actually contains. If the agent doesn’t know, it either hallucinates or has to do a separate tool call to find out, causing another LLM loop, which costs time and tokens.

• Context-free personalization: The same query from two different customers, one who typically shops in the premium range and dresses for formal occasions, and one who buys primarily casualwear under £40, should return different results. Without profile context, the agent treats every query identically, which is worse than a well-tuned keyword search because it creates the impression of a personal assistant while delivering generic answers.

The context layer: What signals it needs

The reason ecommerce is particularly well suited to precomputed context is that retailers already hold an unusually rich set of signals. The challenge isn't data availability; it's assembly.

The signals fall into two groups. Two of them, the catalog vocabulary and the business policies, are the genuinely original work and the heart of this approach. The rest, live facet state, user profiles, and session history, are valuable but closer to table stakes, signals that most teams already understand how to fetch. Here's what most mid-to-large retailers have and what each signal prevents.

Signal

What it prevents

Effort to build

Catalog vocabulary

Vocabulary mismatch and hallucinated filter values; the agent guessing at colors, categories, or brand names instead of resolving them to canonical field values

One-time engineering effort (full-catalog aggregation); incremental to maintain as new categories and brands are added

Business policies

Recommendations that ignore legal or trading requirements, for example, missing age verification on alcohol queries or routing that misses a gluten-free range

Human-authored and governed, not automated; ongoing review as new policy types are added

Live facet state

Recommending filters that return zero results or out-of-stock options for the current query

Runs in parallel with vocabulary and policy lookups; leans on existing catalog and retrieval infrastructure

User profile

Making a returning customer restate sizes, budget, or brand preferences they've already given

Fastest signal to retrieve, a single document lookup by user ID

Session and purchase history

Re-recommending an item the customer already dismissed or bought and returned

Most aspirational layer; depends on customer relationship management (CRM) and analytics integration, best added once the agent is already live

The reason we say semantic metadata and not just metadata is that we’re trying to match the semantic (meaning) of the intent rather than the exact words. If a user is searching for “teal,” we should be able to understand that this is a color and which colors exist in our products that are semantically similar to teal, even if none of them are actually teal. So, if we search for “teal,” we might want to return:

ProductColours = “aquamarine, turquoise”

Hopefully, you can see how this semantic metadata is bridging the gap between user intent and the agent’s knowledge of the products.

Catalog vocabulary: The foundation of context engineering

Catalog vocabulary is the layer that does the most work, and it's the one most worth getting right first.

A vocabulary index maps natural language to the exact field values and category paths used in the product index. It answers questions like: What does "navy" map to? Which categories fall under "knitwear"? Is "Autograph" a brand or a range? What's the correct spelling of a competitor brand the agent might encounter in a query?

What makes this more than a synonym list is how it's queried. The interesting thing a shopper says is rarely an exact field value. They say "something cozy for fall," not colour: NAVY and masterCategoryNames: "Knitwear & Jumpers". So the vocabulary index needs to resolve fuzzy, natural language intent into precise, exact-match filters, and that requires both kinds of matching at once: semantic search to understand that "cozy" leans toward knitwear and fleece, and exact keyword matching to pin the result to the canonical values the product index actually stores. A metadata index that supports both on the same documents is, in effect, a translation layer between how customers talk and how the catalog is structured.

This is also where the index earns the description "semantic metadata layer" rather than "lookup table." Each entry is a small natural language description of a facet value or schema concept, so the agent can match against meaning and then read back the exact filter to use. For a typical fashion retailer, this covers hundreds of color values, brand aliases, category synonyms, and size-range conventions. Building the semantic metadata layer from a full-catalog aggregation is a one-time engineering effort; maintaining it is incremental as new categories and brands are added. Without it, an agent encountering an unfamiliar term must either guess or make exploratory tool calls to discover what's there.

Business policies in the context layer

The second original layer is policy. Some queries carry implicit business requirements that pure relevance cannot handle. A query for "wine gift for a friend" should trigger an age-verification reminder in markets where it's legally required. A query for "gluten-free food gift" should route away from general confectionery toward the specific gluten-free range. A query mentioning "wedding guest outfit" in spring should apply different weighting than the same query in November.

These are policies, and most retail search teams already write them. They just call them boost rules, merchandising overlays, or synonym configurations. The difference in an agentic context is that instead of being applied silently as query modifications, they're surfaced as readable hints the agent can use when deciding how to frame its answer and which products to surface. The agent doesn't have to infer your trading rules from the catalog; it's handed them.

The critical point: These policies encode business intent, not just relevance. A policy that routes alcohol queries through an age-appropriate flow isn't a retrieval optimization; it's a trading requirement. That's why this layer must be human-authored and governed, not generated automatically from traffic patterns, a point we return to in the governance section.

Together, vocabulary and policy are what make the agent behave like it understands your business rather than just your data. The remaining three signals sharpen the experience, but they're more familiar engineering.

Facet, profile, and session signals in the context layer

• Live facet state: Before the agent recommends filters, it should know which filters are available and how many results each returns for this specific query. An agent that suggests "filter by size 8" without knowing that size 8 is out of stock for this query undermines the customer's trust immediately. A facet state query against the product index, run in parallel with the vocabulary and policy lookups, returns the counts, ranges, and available values specific to the current query.

• User profile: A persistent profile (sizes, color preferences, budget range, brand affinities) lets a returning customer skip restating what they've already told you. It's typically the fastest signal to retrieve, a single document lookup by user ID.

• Session and purchase history: Within a session, the agent should know what the customer has already seen, dismissed, or added to their basket, so it doesn't re-recommend a dismissed item or repeat itself. Longer-term purchase history extends this, and signals like return history are richer still, but using them well depends on data most retailers hold in systems that aren't yet wired into their search path. This is the most aspirational layer and the one best approached last, once the earlier layers are delivering value. There must also be balance when building the initial context not to overinflate the size, which will slow down the first reply and increase token costs. There is, therefore, a careful balance to strike between providing information like purchase history in the initial context or providing it as a tool for the agent to use during conversation, but users will expect that, if they’re logged in, the agent should know what they’ve purchased. The precise optimal context is likely to be specific to each implementation and customer experience and will require careful testing.

How context engineering works before the LLM call

The pattern that makes this work is simple to describe and moderately involved to implement:

Without this context build, the agent makes the same discoveries on its own, but through LLM-driven tool calls that each cost a full inference round trip. As a rough rule of thumb, an exploratory tool (for example, GetFilterValues) call tends to land somewhere in the region of 600ms to 1 second of latency in practice; an agent that discovers the vocabulary, checks policy hints, and retrieves facet state through three separate tool calls before it even begins answering can, therefore, add two to three seconds to the response time, and that's before it performs the actual product search. These are order-of-magnitude estimates, not benchmarked figures, and the real numbers depend heavily on the model, the network path, and how the tools are implemented.

Replacing those discovery calls with a parallel fetch (multiple context-building queries can be done in parallel) that runs before the LLM is invoked removes most of that cost. The context build takes roughly the same wall time as a single LLM tool call, but it replaces three or four of them. The LLM would either need to repeat failed searches or do its own context-building tool calls sequentially to get enough context to be successful. Precomputing context is also deterministic rather than subject to the model's tool-selection choices, which gives the business more control to fine-tune the experience.

The token reduction follows the same logic: Each exploratory tool call returns raw data the model must process. A preassembled context summary replaces that raw data with structured facts the model can consume in one pass. At the scale of usage possible in public-facing retail websites, this token cost saving can be significant. This work on document search (Cutting agent costs with pre-computed context) showed up to 75% input token reduction on a controlled benchmark. That benchmark was document retrieval rather than ecommerce, and its authors are explicit that the multiplier isn't a fixed number you can expect everywhere. We expect the direction to hold in ecommerce, because the exploration pattern is the same; it just runs against a structured catalog rather than a document corpus. The magnitude is something each team should measure against its own traffic.

The AI shopping agent with full context

Remember the query that left you guessing: an outfit for an autumn wedding. Run it again, but this time, before you have to think, you’re handed a precomputed context as a short brief:

• This shopper’s name is Sarah, female, age 32, and she buys womenswear, size 12.

• She typically buys your mid-tier ranges.

• "Autumn" here matches these specific color labels: “rust”, “burgundy”, “forest green”, “camel”.

• Matching departments: “Womenswear”, “Menswear”.

• Matching tags: "occasion dresses”, “trouser suits”, “wedding”.

• She already has a burgundy bag in her basket.

• House rule for wedding-guest looks: Complete the outfit. Show hats and accessories, not just the dress.

Suddenly, you’re not guessing; you’re styling for this customer. And the interesting part is how those facts combine rather than just stack. The season proposes a whole autumn palette; the burgundy bag already in her basket narrows that palette to the few tones that coordinate with it; the house rule tells you to finish the look with a matching fascinator rather than stopping at the dress. Together, these facts let you answer like someone who knows both this customer and this shop, in a single pass, with nothing invented.

That short brief is exactly what the signal stack produces: the shopper's profile, the resolved vocabulary, the live basket, and the business policy. These are assembled in parallel and placed in front of the agent before its first move, so the "What do I even do with this?" problem never has to be solved one expensive tool call at a time.

Governing the context layer without automated drift

One difference between the approach described here and automated knowledge extraction systems is worth addressing directly: In ecommerce, the context index cannot self-update without human review.

The policies that govern how an agent responds to gift queries, alcohol queries, or queries from customers in certain age brackets aren't just relevance configurations; they're trading decisions with potential legal and brand implications. An automated system that generates new policies from traffic patterns, without review, is a compliance risk before it's a technical asset.

This is actually the right constraint for most retail organizations, and it aligns with how search teams already work. Merchandisers write boost rules. Search teams maintain synonym configurations. Content teams approve what language appears in automated recommendations. The context policy layer is the same kind of governed configuration; it just serves a different consumer, namely, the agent's reasoning step rather than the query pipeline.

It's worth noting where this differs from the document-retrieval work referenced earlier. In that experiment, the biggest accuracy gain came from an automated feedback loop that fed the agent's wrong answers straight back into the extractor. That works well for factual question-answering, where "right" and "wrong" are unambiguous. In ecommerce, the equivalent signals still surface automatically, but a human decides what to do with them, because the changes carry trading and compliance weight. The loop is the same shape; the publication step has a person in it.

The governance loop that works in practice has two tiers:

• Automatic signal surfacing: Zero-result queries, repeated reformulations on the same topic, and sessions that end without a purchase after an agent interaction are all signals that something in the context layer is missing or wrong. These surface automatically as candidates for improving the experience, for example: a vocabulary term that didn't resolve or a policy that didn't fire on a query type it should have covered. To do this, you need a thorough log of conversations, including the reasoning and tool call trace in a platform like Elastic. This allows you to analyze the performance of the agent using both structured tools, for example, percentage increase in thumbs-down conversations and semantically. You could also run an automated review of conversations about "gifts" around December to characterize the thumbs-up/down ratio across an AB test of two agents.

• Human authorship and review: The search or merchandising team reviews candidates and authors the appropriate vocabulary entry or policy. Policies go through approval before publication. This typically mirrors the workflow that already exists for synonym changes or boost rule modifications; the tooling is the only new part.

How to implement context engineering in phases

• Phase 1: Vocabulary layer (highest leverage, bounded engineering task).

• Phase 2: Facet state and initial policies (leans on the same catalog and retrieval primitives).

• Phase 3: User profiles and session signals (requires CRM and analytics integration; best added when the agent is active).

• Phase 4: Governed feedback loop (shifts to organizational alignment; surfaces gaps for merchandising teams).

The full signal stack described above doesn't need to be built at once, and the order isn’t arbitrary. The highest-leverage starting point is also the lowest in implementation complexity: the vocabulary layer. A semantic metadata index built from a full-catalog aggregation (canonical color values, brand aliases, category paths, field names) is a bounded engineering task, and an agent that can resolve "navy jumper" to color: NAVY, masterCategoryNames: "Knitwear & Jumpers" before its first tool call is materially better than one that discovers this through trial and error. If you build nothing else, build this.

Facet state and the first set of policies can follow close behind, often in parallel, because they lean on the same catalog and the same retrieval primitives. The later layers, such as user profiles, session signals, and the governed feedback loop, are where the work shifts from search engineering to organizational alignment. To implement CRM system integration, merchandising workflow changes, and the analytics needed to surface gaps can take a significant amount of work. Those layers are more valuable once the agent is already in regular use and generating the traffic signals that make the governed loop worth running. The important feature is that each layer stands on its own, so a retailer gets real value from phase one without committing to phase six.

What infrastructure does a context layer need?

Precomputing context at the depth described here places specific requirements on the underlying platform. It's worth being explicit about these, because the temptation in early agent builds is to reach for the simplest available tool for each capability.

• Semantic search to match natural language queries against the vocabulary index and surface the right canonical values. Fuzzy keyword matching alone won't resolve ambiguity between similar brand names or color terms.

• A percolator to implement the policy layer. A percolator reverses the usual search direction: Instead of matching a query against stored documents, it stores the queries and matches an incoming piece of text (here, the customer's message) against them. That's exactly what policy matching needs, because each policy is essentially a saved pattern that says "When a query looks like this, surface this hint."

• Real-time aggregations over the full product catalog to produce accurate facet state at query time. Precomputed facet snapshots go stale quickly in active catalogs; query-time aggregations are the more reliable source.

• Document retrieval by key for user profiles: fast, single-document lookups by user ID that must complete within the context build window.

• Structured and semantic logging and analytics over query traces and agent interactions, which are the raw material for the governed loop's automatic signal surfacing.

These are standard capabilities of a mature search and analytics platform, rather than six separate systems, and Elasticsearch provides all of them in one place. That matters less as a procurement point than as an architectural one: When semantic matching, percolation, aggregations, profile lookups, and analytics all run against the same catalog in the same cluster, the context layer stays consistent with the search layer by construction. Splitting these across a separate vector store and a separate analytics platform is a legitimate choice, but it adds operational surface and introduces a consistency problem between two systems that are reasoning about the same products. The context infrastructure is simplest to run when it lives where the product data already lives and can be updated without a separate extract, transform, load (ETL) step.

Conclusion: Context engineering is a search team's job to own

The retailers who run effective AI shopping experiences at scale aren't the ones with the largest models or the most generous token budgets. They're the ones who have done the work to make their catalog, vocabulary, and tpolicies legible to an agent before it starts reasoning.

The good news is that most of the work is already done. The vocabulary is implicit in the catalog. The policies exist as merchandising rules and compliance guidelines. The user profiles are in the CRM. The session signals are in the analytics stream. The gap isn't data; it's the assembly layer that turns those signals into a structured context the agent can consume before it starts reasoning.

The search team already owns the vocabulary, policies, and merchandising workflows. The context layer is the right home for work the search team is already doing, in a form that serves the agent as well as the query pipeline. And, because it grows every time a gap is found and filled, it behaves less like a setup cost and more like an asset that compounds.

To begin building a context layer in Elastic, you can start a trial of Elastic Cloud or run locally. You should become familiar with configuring semantic search, and if you’re interested in how to build, store, and match search policies at query time, you will enjoy this blog.