Alexander Dávila - Elasticsearch Labs

Training LTR models in Elasticsearch with judgement lists based on user behavior data

Alexander Dávila — Wed, 15 Oct 2025 00:00:00 GMT

A big challenge when using Learning-to-rank models is to create a high-quality judgment list to train the model on. Traditionally, this process involves a manual evaluation of query-document relevance to assign a grade to each one. This is a slow process that does not scale well and is hard to maintain (imagine having to update a list with hundreds of entries by hand).

Now, what if we could use real user interactions with our search application to create this training data? Using UBI data lets us do just that. Creating an automatic system that can capture and use our searches, clicks, and other interactions to generate a judgment list. This process can scale and be repeated far more easily than a manual interaction and would tend to yield better results. In this blog, we will explore how we can query UBI data stored in Elasticsearch to calculate meaningful signals to generate a training dataset for an LTR model.

You can find the full experiment here.

Why UBI data can be useful to train your LTR model

UBI data offers several advantages over a manual annotation:

Volume: Given that UBI data comes from real interactions, we can collect much more data than we can generate manually. This is assuming we have enough traffic to generate this data, of course.
Real User intent: Traditionally, a manual judgment list comes from an expert evaluation of the available data. On the other hand, UBI data reflects real user behavior. This means we can generate better training data that will improve our search system's accuracy, because it's based on how users actually interact with and find value in your content rather than theoretical assumptions about what should be relevant.
Continuous updates: Judgment lists need to be refreshed over time. If we create them from UBI data, we can have current data that results in updated judgment lists.
Cost effectiveness: Without the overhead of manually creating a judgment list, the process can be repeated efficiently any number of times.
Natural query distribution: UBI data represent real user queries, which can drive deeper changes. For example, do our users use natural language to search in our system? If so, we might want to implement a semantic search or hybrid search approach.

It does come with some warnings, though:

Bias amplification: Popular content is more likely to receive clicks, just because it gets more exposure. So this might end up amplifying popular items and possibly drowning out better options.
Incomplete coverage: New content lacks any interactions, so it might be difficult for it to be high in the results. Rare queries can also lack sufficient data points to create meaningful training data.
Seasonal variations: If you expect user behaviour to change drastically over time, historical data might not tell you much about what is a good result.
Task ambiguity: A click doesn’t always guarantee that the user found what they were looking for.

Grades calculation

Grades for LTR training

To train LTR models, we need to provide some numerical representation of how relevant a document is for a query. In our implementation, this number is a continuous score going from 0.0 to 5.0+, where higher scores indicate higher relevance.

To show how this grading system works, consider this manually created example:

Query

Document content

Grade

Explanation

"best pizza recipe"

"Authentic Italian Pizza Dough Recipe with Step-by-Step Photos"

4.0

Highly relevant, exactly what the user is looking for

"best pizza recipe"

"History of Pizza in Italy"

1.0

Somewhat in topic, it is about pizza but is not a recipe

"best pizza recipe"

"Quick 15-Minute Pizza Recipe for Beginners"

3.0

Relevant, a good result but it maybe misses the mark on being the “best” recipe.

"best pizza recipe"

"Car Maintenance Guide"

0.0

Not relevant at all, completely unrelated to the query

As we can see here, the grade is a numerical representation of how relevant a document is to our sample query of “best pizza recipe”. With these scores, our LTR model can learn which documents should be presented higher in the results.

How to calculate the grades is the core of our training dataset. There are multiple approaches to do this, each with its own strengths and weaknesses. For example, we could assign a binary score of 1 for relevant 0 for not relevant or we could just count the number of clicks in a resulting document for each query.

In this blog post, we will be using a different approach, taking into account the user behavior as our input and calculating a grade number as the output. We will also be correcting bias that could occur from the fact that higher results tend to be more clicked, regardless of the relevancy of the document.

Calculating the grades - COEC algorithm

The COEC (Clicks over Expected Clicks) algorithm is a methodology for calculating judgment grades from user clicks. As we stated earlier, users tend to click on higher-positioned results even if the document is not the most relevant to the query; this is called Position Bias. The core idea for using the COEC algorithm is that not all clicks are equally significant; a click on a document at position 10 indicates that the document is much more relevant to the query than a click on a document at position 1. To quote the research paper about the COEC algorithm (linked above):

“It is well known that the click-through rate (CTR) of search results or advertisements decreases significantly depending on the position of the results.”

You can further read about position bias here.

To address this with the COEC algorithm, we follow these steps:

1. Establish position baselines: We calculate the click-through rate (CTR) for each search position from 1 to 10. This means we determine what percentage of users typically click on position 1, position 2, and so on. This step captures the users’ natural position bias. We calculate the CTR using:Where:

= Position. From 1 to 10

= Total clicks (on any document) at position p across all queries

= Total impressions: How many times any document appeared at the position p across all queries

Here, we expect higher positions to get more clicks.

2. Calculate Expected Clicks (EC):

This metric establishes how many clicks a document “should” have received based on the positions it appeared in and the CTR for those positions We calculate EC using:Where:

= All queries where the document d appeared

= Position of the document d in the query q results

3. Count actual clicks: We count the actual total clicks a document received across all queries where it appeared, hereafter called A(d).

4. Compute the COEC score: This is the ratio of Actual clicks (A(d)) over the Expected clicks (EC(d)):This metric normalizes for position bias like this:

A score of 1.0 means the document performed exactly as expected given the positions it appeared in.
A score above 1.0 means the document performed better than expected by looking at its positions. So this document is more relevant for the query.
A score under 1.0 means the document performed worse than expected by looking at its positions. So this document is less relevant for the query.

The end result is a grade number that captures what users are looking for, taking into account position-based expectations extracted from real interactions with our search system.

Technical implementation

We will be creating a script to create a judgment list to train an LTR model.

The input for this script is the UBI data indexed in Elastic (queries and events).

The output is a judgment list in a CSV file generated from these UBI documents using the COEC algorithm. This judgment list can be used with Eland to extract relevant features and train an LTR model.

Quick start

To generate a judgment list from the sample data in this blog, you can follow these steps:

1. Clone the repository:

git clone https://github.com/Alex1795/elastic-ltr-judgement_list-blog.git cd elastic-ltr-judgement_list-blog

2. Install required libraries

For this script, we need the following libraries:

pandas: to save the judgment list
elasticsearch: To get the UBI data from our Elastic deployment

We also need Python 3.11

pip install -r requirements.txt

3. Update the environment variables for your Elastic deployment in a .env file

ES_HOST
API_KEY

To add the environment variables, use:

source .env

4. Create the ubi_queries, ubi_events indices, and upload the sample data. Run the setup.py file:

python setup.py

5. Run the Python script:

python judgement_list-generator.py

If you follow these steps, you should see a new file called judgment_list.csv that looks like this:

This script calculates the grades applying the COEC algorithm discussed before using the calculate_relevance_grade() function that is shown below.

Data architecture

Ubi queries

Our UBI queries index has information about the queries executed in our search system. This is a sample document:

{ "client_id": "client_002", "query": "italian pasta recipes", "query_attributes": { "search_type": "recipe", "category": "food", "cuisine": "italian" }, "query_id": "q002", "query_response_id": "qr002", "query_response_object_ids": [ "doc_011", "doc_012", "doc_013", "doc_014", "doc_015", "doc_016", "doc_017", "doc_018", "doc_019", "doc_020" ], "timestamp": "2024-08-14T11:15:00Z", "user_query": "italian pasta recipes" }

Here we can see data from the user (client_id), from the results of the query (query_response_object_ids), and the query itself (timestamp, user_query)

Ubi click events

Our ubi_events index has data from each time a user clicked a document in the results. This is a sample document:

{ "action_name": "click", "application": "recipe_search", "client_id": "client_001", "event_attributes": { "object": { "description": "Authentic Italian Pizza Dough Recipe with Step-by-Step Photos", "device": "desktop", "object_id": "doc_001", "position": { "ordinal": 1, "page_depth": 1 }, "user": { "city": "New York", "country": "USA", "ip": "192.168.1.100", "location": { "lat": 40.7128, "lon": -74.006 }, "region": "NY" } } }, "message": "User clicked on document doc_001", "message_type": "click", "query_id": "q001", "timestamp": "2024-08-14T10:31:00Z", "user_query": "best pizza recipe" }

Judgment list generation script

General script overview

This script automates the generation of the judgment list using UBI data from Queries and Click events stored in Elasticsearch. It executes these tasks:

Fetches and processes the UBI data in Elasticsearch.
Correlates UBI events with its queries.
Calculates the CTR for each position.
Calculates the expected clicks (EC) for each document.
Counts the actual clicks for each document.
Calculates the COEC score for each query-document pair.
Generates a judgment list and writes it in a CSV file.

Let’s go over each function:

connect_to_elasticsearch()

def connect_to_elasticsearch(host, api_key): """Create and return Elasticsearch client""" try: es = Elasticsearch( hosts=[host], api_key=api_key, request_timeout=60 ) # Test the connection if es.ping(): print(f"✓ Successfully connected to Elasticsearch at {host}") return es else: print("✗ Failed to connect to Elasticsearch") return None except Exception as e: print(f"✗ Error connecting to Elasticsearch: {e}") return None

This function returns an Elasticsearch client object using the host and api key.

fetch_ubi_data()

def fetch_ubi_data(es_client: Elasticsearch, queries_index: str, events_index: str, size: int = 10000) -> Tuple[List[Dict], List[Dict]]: """ Fetch UBI queries and events data from Elasticsearch indices. Args: es_client: Elasticsearch client queries_index: Name of the UBI queries index events_index: Name of the UBI events index size: Maximum number of documents to fetch Returns: Tuple of (queries_data, events_data) """ logger.info(f"Fetching data from {queries_index} and {events_index}") # Fetch queries with error handling try: queries_response = es_client.search( index=queries_index, body={ "query": {"match_all": {}}, "size": size } ) queries_data = [hit['_source'] for hit in queries_response['hits']['hits']] logger.info(f"Fetched {len(queries_data)} queries") except Exception as e: logger.error(f"Error fetching queries from {queries_index}: {e}") raise # Fetch events (only click events for now) with error handling try: events_response = es_client.search( index=events_index, body={ "query": { "term": {"message_type.keyword": "CLICK_THROUGH"} }, "size": size } ) events_data = [hit['_source'] for hit in events_response['hits']['hits']] logger.info(f"Fetched {len(events_data)} click events") except Exception as e: logger.error(f"Error fetching events from {events_index}: {e}") raise logger.info(f"Data fetch completed successfully - Queries: {len(queries_data)}, Events: {len(events_data)}") return queries_data, events_data

This function is the data extraction layer; it connects with Elasticsearch to fetch UBI queries using a match_all query and filters UBI events to get ‘CLICK_THROUGH’ events only.

process_ubi_data()

def process_ubi_data(queries_data: List[Dict], events_data: List[Dict]) -> pd.DataFrame: """ Process UBI data and generate judgment list. Args: queries_data: List of query documents from UBI queries index events_data: List of event documents from UBI events index Returns: DataFrame with judgment list (qid, docid, grade, keywords) """ logger.info("Processing UBI data to generate judgment list") # Group events by query_id clicks_by_query = {} for event in events_data: query_id = event['query_id'] if query_id not in clicks_by_query: clicks_by_query[query_id] = {} # Extract clicked document info object_id = event['event_attributes']['object']['object_id'] position = event['event_attributes']['object']['position']['ordinal'] clicks_by_query[query_id][object_id] = { 'position': position, 'timestamp': event['timestamp'] } judgment_list = [] # Process each query for query in queries_data: query_id = query['query_id'] user_query = query['user_query'] document_ids = query['query_response_object_ids'] # Get clicks for this query query_clicks = clicks_by_query.get(query_id, {}) # Generate judgment for each document shown for doc_id in document_ids: grade = calculate_relevance_grade(doc_id, query_clicks, document_ids, queries_data, events_data) judgment_list.append({ 'qid': query_id, 'docid': doc_id, 'grade': grade, 'query': user_query }) df = pd.DataFrame(judgment_list) logger.info(f"Generated {len(df)} judgment entries for {df['qid'].nunique()} unique queries") return df

This function handles the judgment list generation. It starts processing the UBI data by associating UBI events and queries. Then it calls the calculate_relevance_grade() function for each document-query pair to obtain the entries for the judgment list. Finally, it returns the resulting list as a pandas dataframe.

calculate_relevance_grade()

def calculate_relevance_grade(document_id: str, clicks_data: Dict, query_response_ids: List[str], all_queries_data: List[Dict] = None, all_events_data: List[Dict] = None) -> float: """ Calculate COEC (Click Over Expected Clicks) relevance score for a document. Args: document_id: ID of the document clicks_data: Dictionary of clicked documents with their positions for current query query_response_ids: List of document IDs shown in search results (ordered by position) all_queries_data: All queries data for calculating position CTR averages all_events_data: All events data for calculating position CTR averages Returns: COEC relevance score (continuous value, typically 0.0 to 5.0+) """ # If no global data provided, fall back to simple position-based grading if all_queries_data is None or all_events_data is None: logger.warning("No global data provided, falling back to position-based grading") # Simple fallback logic if document_id in clicks_data: position = clicks_data[document_id]['position'] if position > 3: return 4.0 elif position >= 1 and position <= 3: return 3.0 if document_id in query_response_ids: position = query_response_ids.index(document_id) + 1 if position <= 5: return 2.0 elif position >= 6 and position <= 10: return 1.0 return 0.0 # Calculate rank-aggregated click-through rates position_ctr_averages = {} position_impression_counts = {} position_click_counts = {} # Initialize counters for pos in range(1, 11): # Positions 1-10 position_impression_counts[pos] = 0 position_click_counts[pos] = 0 # Count impressions (every document shown contributes) for query in all_queries_data: for i, doc_id in enumerate(query['query_response_object_ids'][:10]): # Top 10 positions position = i + 1 position_impression_counts[position] += 1 # Count clicks by position for event in all_events_data: if event.get('action_name') == 'click': position = event['event_attributes']['object']['position']['ordinal'] if position <= 10: position_click_counts[position] += 1 # Calculate average CTR per position for pos in range(1, 11): if position_impression_counts[pos] > 0: position_ctr_averages[pos] = position_click_counts[pos] / position_impression_counts[pos] else: position_ctr_averages[pos] = 0.0 # Calculate expected clicks for this specific document expected_clicks = 0.0 # Count how many times this document appeared at each position for any query for query in all_queries_data: if document_id in query['query_response_object_ids']: position = query['query_response_object_ids'].index(document_id) + 1 if position <= 10: expected_clicks += position_ctr_averages[position] # Count total actual clicks for this document across all queries actual_clicks = 0 for event in all_events_data: if (event.get('action_name') == 'click' and event['event_attributes']['object']['object_id'] == document_id): actual_clicks += 1 # Calculate COEC score if expected_clicks > 0: coec_score = actual_clicks / expected_clicks else: coec_score = 0.0 logger.debug( f"Document {document_id}: {actual_clicks} clicks / {expected_clicks:.3f} expected = {coec_score:.3f} COEC") return coec_score

This is the function that implements the COEC algorithm. It calculates the CTR for each position, then it compares the actual clicks for a document-query pair, and finally calculates the actual COEC score for each one.

generate_judgment_statistics()

def generate_judgment_statistics(df: pd.DataFrame) -> Dict: """Generate statistics about the judgment list.""" stats = { 'total_judgments': len(df), 'unique_queries': df['qid'].nunique(), 'unique_documents': df['docid'].nunique(), 'grade_distribution': df['grade'].value_counts().to_dict(), 'avg_judgments_per_query': len(df) / df['qid'].nunique() if df['qid'].nunique() > 0 else 0, 'queries_with_clicks': len(df[df['grade'] > 1]['qid'].unique()), 'click_through_rate': len(df[df['grade'] > 1]) / len(df) if len(df) > 0 else 0 } return stats

It generates useful statistics from the judgment list, such as total queries, total unique documents, or the grade distribution. This is purely informational and does not change the resulting judgment list.

Results and impact

If you follow the instructions in the Quick start section, you should see a resulting CSV file containing a judgment list with 320 entries (you can see a sample output in the repo). With these fields:

qid: unique ID of the query
docid: unique identifier for a resulting document
grade: the calculated grade for the query-document pair
query: The user query

Let’s look at the results for the query “Italian recipes”:

qid

docid

grade

query

q1-italian-recipes

recipe_pasta_basics

0.0

Italian recipes

q1-italian-recipes

recipe_pizza_margherita

3.333333

Italian recipes

q1-italian-recipes

recipe_risotto_guide

10.0

Italian recipes

q1-italian-recipes

recipe_french_croissant

0.0

Italian recipes

q1-italian-recipes

recipe_spanish_paella

0.0

Italian recipes

q1-italian-recipes

recipe_greek_moussaka

1.875

Italian recipes

We can see from the results that for the query “Italian recipes”:

The risotto recipe is definitely the best result for the query, receiving 10 times more clicks than expected
Pizza Margherita is a great result too.
The Greek mousaka (surprisingly) is a good result as well and performs better than its position on the results would suggest. This means a few users looking for Italian recipes got interested in this recipe instead. Maybe these users are interested in Mediterranean dishes in general. At the end, what this tells us is that this could be a good result to be shown under the other two ‘better’ matches we discussed above.

Conclusion

Using UBI data lets us automate the training of LTR models, creating high-quality judgment lists from our own users. UBI data provides a big dataset that reflects how our search system is being used.By using the COEC algorithm to generate the grades, we account for inherent bias while at the same time, it reflects what a user considers a better result. The method outlined here can be applied to real use cases to provide a better search experience that evolves with real usage trends.

Elasticsearch plugin for UBI: Analyze user data in Kibana

Eduard Martin,Alexander Dávila — Fri, 26 Sep 2025 00:00:00 GMT

In this article, we’ll show you how to capture and analyze user analytics data using the UBI (User Behavior Insights) standard in Elasticsearch.

You can learn more about UBI in this article.

Data collected with the UBI collector can be used on Kibana to build dashboards that open the window to users’ behavior in our application. In this blog, we will explore how to analyze UBI data in Kibana to gain insights into how our app is being used.

Demo set up

We can easily reproduce the demo in this blog following these steps:

1. Clone the repository

git clone https://github.com/Alex1795/ubi-dashboard-elasticsearch_blog.git cd ubi-dashboard-elasticsearch_blog

2. Install required libraries:

pip install -r requirements.txt

3. Run the setup script. Make sure to have the following environment variables set beforehand

ES_HOST
API_KEY
KIBANA_HOST

python setup.py

That’s all you need to do. If everything went well, you should see this output from the script execution:

As we can see the script:

Created two indices with the appropriate mappings
Indexed 23 documents to these indices
Uploaded some saved objects to Kibana

Now, let’s take a look at what exactly this script did behind the scenes.

Understanding the uploaded data

First, we put some data in Elasticsearch before creating our visualizations.

You can reproduce the process manually in Kibana DevTools, copying the mappings and sample data and using the PUT and PUT _bulk APIs, respectively.

Ubi_events index

User action data, documents are generated for every click (in this case), and it includes:

application: The client application that generated the event ("search-ui")
action_name: Type of user action performed ("click")
query_id: Links this event to the corresponding search query session
client_id: A generated, unique ID that represents a user or session without revealing personal data. It is generated instead of using identifiable data like email addresses or usernames. This approach allows us to have privacy advantages such as safe analytics capabilities and secure data sharing without exposing PII, while still having important functionality like session continuity, behavioral analysis, or A/B testing.
timestamp: ISO 8601 formatted timestamp when the event occurred
message_type: Category of the event for processing ("CLICK_THROUGH")
message: Human-readable description of what happened ("Clicked Fahrenheit 451")
user_query: The original search term that led to this event ("fahrenheit")
event_attributes: Nested object containing detailed event context:
- object.device: Device type used by the user ("mobile")
- object.object_id: Unique identifier of the clicked item
- object.description: Details about the clicked item (book title, date, author)
- object.position.ordinal: Ranking position of the item in search results (1st)
- object.position.page_depth: Which page of results the item appeared on (1st page)
- object.user.ip: User's IP address
- object.user.city/region/country: Geographic location data
- object.user.location: Precise latitude/longitude coordinates

Sample document:

{ "application": "search-ui", "action_name": "click", "query_id": "2dd48446-7ca8-4510-89f4-2ebb67ed240b", "client_id": "8c1915fe-8ee0-4487-b801-3b1d67c25cf6", "timestamp": "2025-07-30T14:25:52.698Z", "message_type": "CLICK_THROUGH", "message": "Clicked Fahrenheit 451", "user_query": "fahrenheit", "event_attributes": { "object": { "device": "mobile", "object_id": "ZwoTM5gBPJ218VOaBpj4", "description": "Fahrenheit 451(1953-10-15) by Ray Bradbury", "position": { "ordinal": 1, "page_depth": 1 }, "user": { "ip": "192.168.1.100", "city": "New York", "region": "New York", "country": "United States", "location": { "lat": 40.7128, "lon": -74.006 } } } } }

You can download the index mappings here

Ubi_queries index

Search data includes data relevant to each search executed by the users:

query_response_id: Unique identifier for this specific query response instance
user_query: The original search term entered by the user ("fahrenheit")
query_id: Unique identifier for the search query session
query_response_object_ids: Array of object IDs that were returned as search results (["3", "9"])
query: The complete Elasticsearch query object in JSON format, including search parameters, fields to search, result size, sorting, and metadata
client_id: A generated unique ID that represents a user or session without revealing personal data. It is generated instead of using identifiable data like email addresses or usernames. This approach allows us to have privacy advantages such as safe analytics capabilities and secure data sharing without exposing PII, while still having important functionality like session continuity, behavioral analysis, or A/B testing.
timestamp: Unix timestamp in milliseconds when the query was executed (1753885225098)

Sample document:

{ "query_response_id": "03e8af3e-8725-49d9-99ad-36bf2a8e96d1", "user_query": "fahrenheit", "query_id": "f8b2f5bc-cb3c-49d4-86bc-19212a782ba7", "query_response_object_ids": [ "3", "9" ], "query": """{"from":0,"size":20,"query":{"multi_match":{"query":"fahrenheit","fields":["author^1.0","name^1.0"]}},"_source":{"includes":["name","author","image_url","url","price","release_date"],"excludes":[]},"sort":[{"_score":{"order":"desc"}}],"ext":{"query_id":"f8b2f5bc-cb3c-49d4-86bc-19212a782ba7","user_query":"fahrenheit","client_id":"8c1915fe-8ee0-4487-b801-3b1d67c25cf6","object_id_field":null,"query_attributes":{}}}""", "client_id": "8c1915fe-8ee0-4487-b801-3b1d67c25cf6", "timestamp": 1753885225098 }

You can download the index mappings here.

Sample data

We can use the _bulk API to index some sample data in both indices

This will create 6 documents in the ubi_queries index and 16 in the ubi_events index.

Dashboard object

Before going into details of the visualizations used in this example here, you can download the Saved Object of the full example dashboard and import it into your Kibana instance. This dashboard explores the most searched terms, when searches and events took place, and where they come from (in a map).

Visualize Insights

We are going to create a Kibana dashboard to analyze the most common metrics leveraging Kibana Lens. For a reference on available visualizations, visit this page.

Ubi_events

We will start with some simple Metric visualizations created with Lens: Total events: Counts how many events were triggered in the timeframe. Uses a simple count of the documents in the index, denoted by # Records in the field list.

Event actions: Counts actions by action_name. This is a simple count of documents split by action_name.keyword. In our sample data, we have two types of actions:

click: Generated when a user clicks in the book link
search_input: Generated when a user enters text in the search box (debounce 300ms)

Now on table visualizations:

Top clicks: A table with a count of the number of events split by the query they come from. It uses a Top values function on the user_query.keyword. This can give us visibility on which queries generate more interactions on our webpage.

Finally, some other visualizations:

Device types: This visualization breaks down the percentage of events by the device they come from. The device can be one of three categories: Desktop, mobile, or tablet. This visualization is a pie that uses the top values of event_attributes.object.device.keyword, and can give us insights into which type of devices our users have. This can generate alerts if we detect an unexpected, sudden fall of events on a specific type of device, as this might indicate that a recent change in our app resulted in errors when accessing it from a device.

Events map: A map visualization that shows where the events are coming from, which allows us to see the geographical distribution of our users. Right now, this shows where individual documents come from, but this can also be used to see the density of users with a heatmap, for example.

This particular visualization can provide very interesting insights when used with different filters. For example, we can see where different search terms are coming from or where most of our clicks are originating. This can be useful information for making decisions on localization efforts or establishing differences across local markets. The map uses the location at event_attributes.object.user.location.

UBI Events: A saved search with the latest UBI events documents

Ubi_queries

Here we have visualizations from this index:

Total queries: A simple document count of the index to show how many queries have been received in total. This shows the big picture and answers the question of how many total queries we had in the selected time window.

Unique clients: A unique_count of the field client_id to show how many different clients have used our website.

Top queries (tag cloud): A Tag cloud of the top 5 most searched terms. This visualization uses the field user_query.keyword and allows us to easily see the main terms that our users are looking for.

Queries over time: A line chart of queries per hour, which uses a simple count metric in a horizontal axis of the field timestamp

Query terms over time: Similar to the last one, but broken down by the user_query.keyword. This chart can show how many different terms are searched over time.

Top queries: A Top values table showing how many times a term was searched. It uses the user_query.keyword field.

Client queries: A Top values table of the client_id field that counts the total queries and unique queries per client.

No result queries: A Top value table that shows the top query_terms that didn’t match any document, and a Unique Count of the field client_id. This can be very useful to determine what products our website is lacking. For example, in an e-commerce book store, seeing regular searches for a particular book title could lead us to buy copies to sell. Alternatively, it can also indicate shortcomings in our search implementation, for example, if people are using question-based searches that align better with semantic search approaches.

Here you can see the full dashboard:

Analysis of sample data

In our dashboard, we can get some insights:

Traffic is coming from 3 different cities in the US
Most of our users access our website from a desktop device, but we have a sizable number of users using a mobile device and even some using a tablet.
We can see the top query is “asimov,” but at the same time, we do not have any results. This might be a good indicator of what products should be prioritized for stock acquisition.

To further this analysis, we could use Kibana’s Machine Learning capabilities to understand and predict behaviours on our website. Going even one step further, we can create alerts based on these behaviors using the different available connectors.

From a search relevance perspective, user behavior is a useful input for relevance engineering tools like LTR.

Conclusion

Data collected by the UBI collector can be easily used to have a better understanding of our users. The resulting dashboard becomes a live pulse of what our users are searching for and can point to data gaps to drive improvements in our search engine.

Note: The o19s User Behavior Insights (UBI) plugin mentioned in this article is a third-party, community-maintained plugin and is not officially supported by Elastic. For questions or issues related to this plugin, please refer to the o19s UBI project repository at https://github.com/o19s/ubi.