synonym-power-boost-blog-banner.jpg

synonym-power-boost-blog-thumb.jpg

The same, but different: Boosting the power of Elasticsearch with synonyms

Made To Measure: How to Use the Ranking Evaluation API in Elasticsearch

Part of the success of Elasticsearch has probably been its large number of supported language clients right from the start. Its powerful REST API can be accessed using HTTP and JSON, which makes it easy to integrate with using any language that provides those capabilities.

Nevertheless, there hasn’t been an official Elastic Java REST client for a long time. This was partly due to the existence of the <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/client.html">native Java Client</a> (also sometimes called the TransportClient), that doesn’t communicate with the cluster via REST but via a binary protocol. However, there are several drawbacks of this client, like security aspects or tighter coupling to JVM and cluster versions. For more details, we recommend reading the <a href="https://www.elastic.co/blog/state-of-the-official-elasticsearch-java-clients">State of the Official Elasticsearch Java Clients</a> blog post, where we also described our future plans around the Java clients for Elasticsearch.

<h3>Enter the Low-Level Client </h3>
As a first step to improve the situation, the low-level REST client was released with version 5.0 of the Elastic Stack.
 It solved the compatibility and dependency problems by providing strict access via the REST API. However, this came at the price of losing all the comforts of the Java API, which people were used to when using the transport client, like search request and query builders and response objects that are convenient to use in Java. That’s why next we started working on the high-level REST client: a client that sends requests using the low-level client, but that is also able to accept 
higher level request objects and returns response objects similar to those the native client already uses.

<h3>Let's Level Up</h3>
Right after the release of version 5.0 we started working on the high-level client, with the primary goal to make migration from the current transport client easy and to initially support the core APIs index, bulk, get, delete and search. It soon became clear that in order to progress and to ease migration at the same time, we would still need the
 current search and query builders and support the current response objects. The work then mostly involved implementing parsing Elasticsearch REST responses back to Java objects, a tedious issue if you consider the richness especially of the search response with all its varying aggregation responses. One of the 
	<a href="https://github.com/elastic/elasticsearch/issues/23331">tracking issues</a> gives a bit of an insight into the work that has gone into this.

For this reason, the high-level client still depends on Elasticsearch, like the Java API does today. This may not be ideal, as it still ties users of the client to depend on a certain version of Elasticsearch, but this decision allows users to migrate away more easily from the transport client. We would like to get rid of this direct dependency in the future, but since this is a separate long-term project, we didn’t want this to affect the timing of the client’s first release.

The first release of the Java high-level REST client went out with the <a href="https://www.elastic.co/blog/elastic-stack-6-0-0-beta1-released">6.0.0-beta1 release</a> of the Elastic Stack. It allows you to communicate via REST with Elasticsearch nodes running 6.0.0-beta1, but it is not fully compatible with 5.x nodes. The <a href="https://www.elastic.co/blog/elastic-stack-5-6-0-released">5.6.0 release</a> of the Elastic Stack recently went out and includes the 5.6.0 version of the high-level client, which works against Elasticsearch nodes running version 5.6. It is unfortunately not compatible with previous 5.x versions of Elasticsearch.

<h3>What’s Up Next?</h3>
The high-level client will eventually replace the transport client in the future, hence we invite all our Java users to try it out and migrate to it if possible. The current release supports the following <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-supported-apis.html">Elasticsearch APIs</a>:

 index, get, delete, update, bulk, search, scroll and clear scroll. Support for other APIs will be added step by step, also according to the feedback received from the community. As described in our <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-level-migration.html">migration guide</a>, it is possible meanwhile to use the low-level REST client to perform requests that are not yet supported by the high-level one.

The documentation for the high-level REST client can be found <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high.html">here</a>, and its javadoc <a href="https://artifacts.elastic.co/javadoc/org/elasticsearch/client/elasticsearch-rest-high-level-client/5.6.0/index.html">here</a>. We are looking forward to your feedback! Even better: become an <a href="https://www.elastic.co/blog/elastic-stack-6-0-0-alpha1-released">Elastic Pioneer</a> and receive a special Elastic gift package as our thanks for trying out the high level client in one of the 6.0 pre-releases and report any bug you find on our GitHub page.

Banner Image Credit: <a href="https://pixabay.com/users/Hans-2/">Hans</a> via <a href="https://pixabay.com/photos/elevator-lift-staircase-futuristic-1271928/">Pixabay</a> (CC0)

elevator-elasticsearch-high-level-java-client-fullbleed.jpg

The Elasticsearch Java High-Level Rest Client is Out

<html><head></head><body>This is the second post in a three-part series about Instant Aggregations. Read how it all started in <a href="https://www.elastic.co/blog/the-tale-of-caching-and-why-it-matters">The Tale of Caching and Why It Matters</a> from Simon Willnauer and Shay Banon and how the magnificent end to the story in <a href="https://www.elastic.co/blog/instant-aggregations-rewriting-queries-for-fun-and-profit">Instant Aggregations: Rewriting Queries for Fun and Profit</a>. Enjoy the trilogy! When writing software, adding cool new features is of course always great fun. But sometimes it&#x2019;s also important to work on internal changes in the code base that enable those shiny new additions in the future. For example, there were <a href="/blog/the-tale-of-caching-and-why-it-matters">a number of cool new ideas</a> for Elasticsearch floating around that were essentially blocked by the lack of having a good intermediate representation for search requests arriving through the REST layer, which prevented early query optimizations and delayed parsing to the shard level. For the upcoming release of Elasticsearch 5.0 we embarked on a large refactoring to change the way search requests work internally. In this blog post we want to highlight some of the changes and challenges that came along with the refactoring of queries and the search request, how it helped us improve our testing and how it enables great new features like &quot;Instant Aggregations&quot;, that will be highlighted in a following blog post.<h2>How search requests were sent across the cluster prior to the Refactoring</h2>When you send a search request to a node in the cluster, the node receiving the request coordinates the search request from then on. The coordinating node identifies what shards the search request needs to be executed on, and forwards it to the nodes that hold those shards via the internal node-to-node <a href="/guide/en/elasticsearch/reference/current/modules-transport.html">transport layer</a>. Each shard returns a set of documents, whose ids will be sent back to the coordinating node that is responsible for reducing the matching documents obtained to the top matching ones that need to be returned to the client. Once those documents are identified, they are fetched as part of the fetch phase and finally returned.Before Elasticsearch 5.0, each node received the original search request, parsed the query and used other information available from the shard (like mappings) to actually create the Lucene query that was then executed as part of the query phase.
 
<img src="https://api.contentstack.io/v2/assets/57d6d80eb37b0d4368c02ae7/download?uid=blt97f5df8ef0af08af" alt="parse 1" width="850">The body of the search request is not parsed on the coordinating node but rather serialized via the transport layer untouched, as an opaque byte array which holds nothing more than its json representation. This was historically done to avoid having to write serialization code for every single query and every single section of a search request, as elasticsearch uses its own binary serialization protocol. We eventually came to the conclusion that this was the wrong trade-off. <h2>So what&apos;s the problem with this?</h2>While simply forwarding incoming requests on the coordinating node as described above is simple and fast (at least on the coordinating node), there are some drawbacks:<ul>
 <li>
 Each search request gets parsed multiple times, once for each shard of the target index. That means that even if with the default of 5 shards, a query will be parsed 5 times across the cluster (even multiple times in the same node if that node holds more than one shard that is relevant for the query). This is a potential waste of cpu-cycles as the search request could be parsed earlier and only once.
 </li>
</ul><ul>
 <li>
 For the same reason, if a query is malformed, it will be rejected with an error by each shard. This is why in Elasticsearch 1.x you get multiple parse failures in your response when you make a mistake in your query. It is all the same error, but it gets thrown by each shard. On 2.x we already deduplicate those kind of errors, but they are still thrown from all shards.
 </li>
</ul><ul>
 <li>
 The search request cannot be rewritten for optimizations on the coordinating node without impacting performance, as this would mean having to parse it twice.
 </li>
</ul><ul>
 <li>
 Parsing and query creation are tightly coupled, making unit testing of each individual step difficult without having the whole context of a shard.
 </li>
</ul>All these drawbacks arise because before 5.0 there is no intermediate representation of a query within elasticsearch, only the raw JSON request and the final Lucene query. The latter can only be created on the shards, because Lucene queries are not serializable and often depend on context information only available on the shard. By the way, this is actually true not only for the &quot;query&quot; part of the search request, but for many other things like aggregations, suggestions etc.. that can be part of a search request.
 So, wouldn&apos;t it be nice to be able to parse queries to an intermediate representation once and early? That way we would be able to eventually rewrite them, and optimize their execution, for example by shortcutting them to queries that are less expensive.<h2>What we did</h2>For the queries we achieved this by splitting the query parsing into two phases:<ul>
 <li>
 Json/Yaml parsing that happens on the coordinating node, which allows us to represent a search request in a serializable intermediate format. This phase is independent from mappings and data present in the shards, to make sure that it can be performed on any node. This happens in a method called &#x201C;fromXContent()&#x201D; that every query has and which produces a new intermediate query object (all implementing the QueryBuilder interface).
 </li>
 <li>
 Conversion from elasticsearch serializable queries to lucene queries ready to be executed on the shard. This phase depends on mappings and information present in the shards, so it has to happen separately on each shard to be able to run the query against its data. The method handling this in the <a href="https://github.com/elastic/elasticsearch/blob/5.0/core/src/main/java/org/elasticsearch/index/query/AbstractQueryBuilder.java">QueryBuilder</a> interface is called `toQuery()`.
 </li>
</ul><img src="https://api.contentstack.io/v2/assets/57d6d82fdc33a3807b6eb256/download?uid=blt0b742cc4635f9791" alt="parse 2" width="850">As a result of the split, the code is better organized as parsing is decoupled from lucene query generation. Having a serializable intermediate format for queries meant that we had to write serialization code for all of the queries and search sections supported in elasticsearch. Also, every query implements `equals()` and `hashCode()`, so that they can be compared with each other for easier caching and to aid their testing.Once we moved parsing to the coordinating node, we could also move some validation to earlier stages of a search request and throw one single error earlier. Validation applies to malformed queries in terms of invalid json, queries that are missing some of their required values, or queries with invalid values provided.In order to make all those changes in a safe way that doesn&#x2019;t break existing behavior, we wrote extensive unit tests. For each query, we added <a href="/blog/elasticsearch-testing-qa-increasing-coverage-randomizing-test-runs">randomized tests</a> to verify that it can be properly parsed and serialized, added tests that the resulting lucene query is the expected one and also test explicitly that all of the required values are checked and validated correctly. In order to do so, we created a <a href="https://github.com/elastic/elasticsearch/blob/master/test/framework/src/main/java/org/elasticsearch/test/AbstractQueryTestCase.java">base test class</a> which provides the code necessary for test setup and has some abstract methods that only need to be filled in for each individual query. This base class is shipped as part of our test framework helping downstream developers of custom queries with testing.Most of our query parsing code had low unit test coverage before the refactoring. For example, the `org.elasticsearch.index.query` package that contains all the QueryBuilder and QueryParser classes had only 47% test coverage (actually, with randomized testing we cover more over time, these numbers refer to the coverage in one CI run) before the refactoring, going to above 77% coverage after the refactoring branch was merged.<h2>How we did it</h2>This query refactoring was really a <a href="https://github.com/elastic/elasticsearch/issues/10217">long running task</a> that needed a proper feature branch. Initially we thought it could be enough to only refactor the query part of the search request, but in the end we went for refactoring almost all parts of the search request. The branch stayed alive for several months and we took care of merging master in daily (or at least twice a week). In some cases we found bugs that we fixed upstream rather than on the branch, to get them out with regular releases as soon as possible.As a team working on the feature branch, we decided to essentially replicate the development model of how we work on Elasticsearch master: instead of trying to plow through all changes needed and having an enormous bulk review at the very end, each change was submitted and reviewed as a small, manageable pull request against the branch. Having a closely knit team dedicated to this work, we could agree early on the goals of the changes, coding and architectural standards. This later made finding a reviewer familiar with those goals easy and made review cycles faster.Given the amount of code that this refactoring touched, the challenging part was to keep the branch healthy while the task was in progress. We had to sit down and come up with some incremental steps to run tests while the refactoring was still in progress. It was all a matter of introducing the new functionality gradually, while leaving the old infrastructure &#xA0;alive until there was a complete replacement for it. Initially we introduced the two phase parsing but it all still happened on the data nodes rather than on the coordinating node. When all queries were migrated, we were able to move parsing to the coordinating node and start working on the remaining sections of the search request (e.g. rescore, suggestions, highlighting and so on).<h2>Lessons learnt</h2>After several months of wading through code that was originally written by multiple authors and at very different times through the evolution of the code base, we learned a couple of things:<ul>
 <li>
 just because you&#x2019;ve refactored a handful of queries already doesn&#x2019;t mean you have any idea of what the other about <a href="https://github.com/elastic/elasticsearch/issues/10217">50 classes</a> look like.
 </li>
 <li>
 in order to keep your branch healthy, it is vital to use CI to frequently run your test suite and deliver improvements in small incremental steps, even though this means doing more work in total.
 </li>
 <li>
 changing architectural decisions years into development does have a non-negligible cost attached, but it&#x2019;s the right thing to do, pays off in terms of maintainability and enables new features.
 </li>
</ul><h2>Conclusion</h2>All in all, the improvements made with this large refactoring become obvious when we need to go back to the old 2.x branch now and then to backport something from the current main development line. Not only has the parsing infrastructure become much more efficient, but introducing the intermediate query representation and decoupling parsing from query creation has lead to much cleaner code and the new test infrastructure makes debugging of parsing-related problems and writing new test much easier. But the most important thing is, that we are now able to analyze, rewrite and optimize search request on the coordinating node much more easily. Features like &#x201C;instant aggregations&#x201D; that were long talked about, but never tackled because of the months of work required, suddenly became possible to implement quickly. In an upcoming blog post we will shed some more light on what this feature is and how it works.</body></html>

elasticsearch-the-great-query-refactoring-full-bleed.jpg

elasticsearch-the-great-query-refactoring-thumbnail.jpeg

The Great Query Refactoring: Thou shalt only parse once

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Christoph Büscher

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Christoph Büscher

The same, but different: Boosting the power of Elasticsearch with synonyms

Made To Measure: How to Use the Ranking Evaluation API in Elasticsearch

The Elasticsearch Java High-Level Rest Client is Out

The Great Query Refactoring: Thou shalt only parse once

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards