<p>Part of the success of Elasticsearch has probably been its large number of supported language clients right from the start. Its powerful REST API can be accessed using HTTP and JSON, which makes it easy to integrate with using any language that provides those capabilities.
</p>
<p>Nevertheless, there hasn’t been an official Elastic Java REST client for a long time. This was partly due to the existence of the  <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/client.html">native Java Client</a> (also sometimes called the TransportClient), that doesn’t communicate with the cluster via REST but via a binary protocol. However, there are several drawbacks of this client, like security aspects or tighter coupling to JVM and cluster versions. For more details, we recommend reading the <a href="https://www.elastic.co/blog/state-of-the-official-elasticsearch-java-clients">State of the Official Elasticsearch Java Clients</a> blog post, where we also described our future plans around the Java clients for Elasticsearch.
</p>
<h3>Enter the Low-Level Client </h3>
<p>As a first step to improve the situation, the low-level REST client was released with version 5.0 of the Elastic Stack.
 It solved the compatibility and dependency problems by providing strict access via the REST API. However, this came at the price of losing all the comforts of the Java API, which people were used to when using the transport client, like search request and query builders and response objects that are convenient to use in Java. That’s why next we started working on the high-level REST client: a client that sends requests using the low-level client, but that is also able to accept 
higher level request objects and returns response objects similar to those the native client already uses.
</p>
<h3>Let's Level Up</h3>
<p>Right after the release of version 5.0 we started working on the high-level client, with the primary goal to make migration from the current transport client easy and to initially support the core APIs index, bulk, get, delete and search. It soon became clear that in order to progress and to ease migration at the same time, we would still need the
 current search and query builders and support the current response objects. The work then mostly involved implementing parsing Elasticsearch REST responses back to Java objects, a tedious issue if you consider the richness especially of the search response with all its varying aggregation responses. One of the 
	<a href="https://github.com/elastic/elasticsearch/issues/23331">tracking issues</a> gives a bit of an insight into the work that has gone into this.
</p>
<p>For this reason, the high-level client still depends on Elasticsearch, like the Java API does today. This may not be ideal, as it still ties users of the client to depend on a certain version of Elasticsearch, but this decision allows users to migrate away more easily from the transport client. We would like to get rid of this direct dependency in the future, but since this is a separate long-term project, we didn’t want this to affect the timing of the client’s first release.
</p>
<p>The first release of the Java high-level REST client went out with the <a href="https://www.elastic.co/blog/elastic-stack-6-0-0-beta1-released">6.0.0-beta1 release</a> of the Elastic Stack. It allows you to communicate via REST with Elasticsearch nodes running 6.0.0-beta1, but it is not fully compatible with 5.x nodes. The <a href="https://www.elastic.co/blog/elastic-stack-5-6-0-released">5.6.0 release</a> of the Elastic Stack recently went out and includes the 5.6.0 version of the high-level client, which works against Elasticsearch nodes running version 5.6. It is unfortunately not compatible with previous 5.x versions of Elasticsearch.
</p>
<h3>What’s Up Next?</h3>
<p>The high-level client will eventually replace the transport client in the future, hence we invite all our Java users to try it out and migrate to it if possible. The current release supports the following <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-supported-apis.html">Elasticsearch APIs</a>:
</p>
<p> index, get, delete, update, bulk, search, scroll and clear scroll. Support for other APIs will be added step by step, also according to the feedback received from the community. As described in our <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-level-migration.html">migration guide</a>, it is possible meanwhile to use the low-level REST client to perform requests that are not yet supported by the high-level one.
</p>
<p>The documentation for the high-level REST client can be found <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high.html">here</a>, and its javadoc <a href="https://artifacts.elastic.co/javadoc/org/elasticsearch/client/elasticsearch-rest-high-level-client/5.6.0/index.html">here</a>. We are looking forward to your feedback! Even better: become an <a href="https://www.elastic.co/blog/elastic-stack-6-0-0-alpha1-released">Elastic Pioneer</a> and receive a special Elastic gift package as our thanks for trying out the high level client in one of the 6.0 pre-releases and report any bug you find on our GitHub page.
</p>
<p><em>Banner Image Credit: <a href="https://pixabay.com/users/Hans-2/">Hans</a></em> <em>via <a href="https://pixabay.com/photos/elevator-lift-staircase-futuristic-1271928/">Pixabay</a></em> (CC0)<br>
</p>

elevator-elasticsearch-high-level-java-client-fullbleed.jpg

The Elasticsearch Java High-Level Rest Client is Out

<p>Elasticsearch has a powerful <a href="/guide/en/elasticsearch/reference/current/search-search.html">_search API</a> that allows it to search against all indices on &nbsp;the local cluster. We recently released Elasticsearch 5.3.0 including a new functionality called <a href="/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html">Cross-Cluster Search</a> that allows users to compose searches not just across local indices but also across clusters. This means that one can search against data that belongs to other remote clusters. Historically, the Tribe Node was used when searches needed to span multiple clusters, yet it works very differently. In this blog post, we will cover why we implemented Cross-Cluster Search, how it works and compares to the Tribe Node, and why we are convinced it is a step in the right direction for the future of federated search using Elasticsearch.</p><h3>Keep it simple</h3><p>When we sat down to try and redesign the next-generation Tribe Node, we re-evaluated the problems that it was trying to solve. The goal was to make federated search possible without all the limitations that the Tribe Node provides, and without adding a specific API for it, so that maintaining such feature would also be easier. We realized that some of the features that the Tribe Node offers besides federated search are commodities. The Tribe Node supports many Elasticsearch APIs allowing, for instance, to retrieve the cluster state or nodes stats via a Tribe Node, which will return the information collected from all the remote clusters and merged into a single view. Merging information coming from different sources is nothing complicated though, and is easily performed on the client side by sending multiple requests to the different clusters. The hard problem that must be addressed on the server side is federated search. It involves a distributed scatter and gather to be executed on nodes belonging to multiple clusters as well as the result merging and reduction requiring internal knowledge. That is why we decided to focus on solving this specific problem in a sustainable and robust way by adding support for Cross-Cluster Search to the existing _search API.</p><h3>Search API Detour</h3><p>The _search API allows Elasticsearch to execute searches, queries, aggregations, suggestions, and more against multiple indices, each one composed by one or more shards. When a Client sends a search request to an Elasticsearch cluster, the node that receives the request acts as the coordinating node for the whole execution of the request. It identifies which indices, shards, and nodes the search has to be executed against. While executing, all data nodes holding a shard that is queried receive requests in parallel, then each node executes the query phase locally and sends the results back to the coordinating node. The coordinating node waits for all shards to respond in order to reduce the results down to the top-N hits that need to be fetched from the shards. A second execution phase then fetches the top-N documents from the shards in order to return the results back to the Client.</p><h3>How Cross-Cluster Search Works</h3><p>Since Elasticsearch 5.3.0, it is possible to register remote clusters through the <a href="/guide/en/elasticsearch/reference/current/cluster-update-settings.html">cluster update settings API</a> under the <code>search.remote</code> namespace. Each cluster is identified by a cluster alias and a list of seed nodes that are used to discover other nodes belonging to the remote cluster, as follows:</p><pre class="prettyprint">PUT _cluster/settings
  {
   "persistent": {
     "search": {
       "remote": {
         "cluster_one": {
           "seeds": ["remote_node_one:9300"]
         },
         "cluster_two": {
           "seeds": ["remote_node_two:9300"]
         }
       }
     }
   }
  }</pre><p>Once one or more remote clusters are registered, it is possible to execute search requests against their indices using the _search API of the cluster the remotes are configured on. In contrast to a so-called local index, remote indices must be disambiguated with their corresponding configured cluster alias separated by a colon (e.g. <code>cluster_two:index_test</code>).</p><p>Whenever a search request expands to an index on a remote cluster, the coordinating node resolves the shards of these indices on the remote cluster by sending one _search_shards request per cluster. Once the shards and remote data nodes are fetched, searches are executed just like any other search on the local cluster explained above, using the exact same code-paths which improves testability and robustness dramatically. </p><p>From a search execution perspective, there is no difference between local indices and indices that belong to remote clusters as long as the coordinating node can reach some nodes belonging to the remote clusters. Finally, the hits returned as part of the search response which belong to remote clusters have their index name prefixed with their cluster alias.</p><p>When registering a remote cluster, Elasticsearch discovers by default up to 3 nodes per configured remote cluster through the seed nodes in the configuration. In contrast to nodes in the local cluster, where any node connects to any other node, cross cluster search nodes are only connected in an unidirectional fashion. Those are the nodes that the coordinating node will communicate with as part of a Cross-Cluster Search request. It is possible to control how many nodes are discovered when registering the remote clusters, as well as mark nodes belonging to remote clusters as gateways in order to control exactly which nodes will receive connections from the outside world. In case a node holds data that has to be accessed as part of a cross cluster search request, that node will not receive a direct connection from the remote coordinating node, but rather from another node marked as gateway in its own cluster, which will act as a proxy.</p><p>Additionally, it is possible to control which nodes can act as coordinating nodes as part of a cross cluster search request through the <code>search.remote.connect</code> setting. This is useful to control which nodes in a cluster can send requests to remote clusters. If a node that is not allowed to connect to remote clusters receives a search request that involves remote clusters, an error will be returned.</p><h3>What about the Tribe Node?</h3><p>Searching against multiple clusters isn’t something new to Elasticsearch. In fact, users have been doing that up until now using a Tribe Node.</p><p>The Tribe Node is a separate node whose main job is to sniff the cluster states of the remote clusters and merge them altogether. In order to do that, it joins all the remote clusters which makes it a very special node that doesn’t belong to a cluster of its own, yet it joins multiple clusters.</p><p>When a Tribe Node receives a search request, it knows right away which nodes to forward it to as it holds the cluster state of all the registered remote clusters, it’s in fact a node in all of the clusters. That means that the additional “find out where the remote shards are and which node they belong to” step required with Cross-Cluster Search is not required with a Tribe Node. It is important to note that Cross-Cluster Search doesn’t require additional nodes, given that any node can act as the coordinating node for a search request, regardless of whether or not the request spans across multiple clusters. That also means that no additional nodes join the remote clusters, hence cluster state updates don’t have to be sent to those, which could potentially slow down the remote clusters when using a Tribe Node. That is because the Tribe Node receives and has to acknowledge every single cluster state update from every remote cluster.</p><p>Furthermore, when merging cluster states from the remote clusters, the Tribe Node cannot keep indices that have the same name in its own cluster state, although they belong to multiple clusters. This is quite a big limitation which Cross-Cluster Search addresses, as well as being able to dynamically register, update, and remove remote clusters, which requires a node restart when using the Tribe Node.</p><p>Also, being the tribe node such a special node, it turned out very hard to maintain code-wise over time, as it is the exception to the main assumption for an Elasticsearch node, that a node must belong to one and only one cluster.</p><h3>Further Search Improvements</h3><p>Elasticsearch’s retrieval capabilities are outstanding given that even with significant cluster size users can search across as many shards as they feel like. But wait, that’s not necessarily the case — at least in the current state of affairs. Elasticsearch comes with a limitation in the form of a soft-limit <code>action.search.shard_count.limit</code> which rejects search requests that expand to more than 1000 shards. Why would we want to limit this? Well, the reason here is obvious given the implementation detail mentioned in the search detour section above. We fan out to all shards and maintain all shards results in a dense data structure until all shards responded. Imagine you are running large aggregations, the coordinating node has to maintain a non-trivial amount of memory per search request to the entire duration of the initial phase.</p><p>Now with the addition of cross cluster search where we are emphasizing searches across many, many shards, having such a soft limit isn’t providing a good user experience after all. In order to eliminate the impact of querying a large number of shards, the coordinating node now reduces aggregations in batches of 512 results down to a single aggregation object while it’s waiting for more results to arrive. <a href="https://github.com/elastic/elasticsearch/pull/23946">This upcoming improvement</a> was the initial step to eventually remove the soft limit altogether. The upcoming 5.4.0 release will also allow to reduce the top-N documents in batches. With these two major memory consumers under control, we now default the <code>action.search.shard_count.limit</code> setting to <a href="https://github.com/elastic/elasticsearch/pull/24012">unlimited</a>. This allows users to still limit and protect their searches in the number of shards while &nbsp;providing a good user experience for other users.</p><h3>Conclusions</h3><p>Cross-Cluster Search is the new way of executing federated search, which will replace the Tribe Node in the future. It works very differently compared to the Tribe Node and is not subject to most of the drawbacks that the Tribe Node comes with. We invite you to try it out by <a href="/downloads/elasticsearch">downloading Elasticsearch 5.3.x</a> (or <a href="/cloud/as-a-service">deploying on Elastic Cloud</a>) and give us feedback.</p>

Tribe Nodes & Cross-Cluster Search: The Future of Federated Search in Elasticsearch

<html><head></head><body><p>Java programmers have two choices when communicating with Elasticsearch: they can use either the REST API over HTTP, or the internal Java API used by Elasticsearch itself for node-to-node communication.
</p><p>So, what&apos;s the difference between these two APIs? &#xA0;When a user sends a REST request to an Elasticsearch node, the coordinating node parses the JSON body and transforms it into its corresponding Java object. &#xA0;From then on, the request is sent to other nodes in the cluster in a binary format -- the Java API -- using the transport networking layer. &#xA0;A Java user uses the Transport Client to build these Java objects directly in their application, then makes requests using the same binary format passed across the transport layer, skipping the need for the parsing step needed by REST.
</p><h2>What are the problems with this approach?</h2><p>This solution is quite powerful, and didn&#x2019;t require us to write specific Java client code for Elasticsearch as the Java API was already used and maintained internally. The Java API is also theoretically more performant than REST, as it skips the parsing step and allows clients to use the binary protocol. <a href="/blog/benchmarking-rest-client-transport-client">Benchmarks, however, &#xA0;show</a> that the performance of the HTTP client is close enough to that of the Transport client that the difference can be pretty much disregarded.
</p><p>Over time, we came to realize that the Java API has some downsides:
</p><h2>Backwards compatibility</h2><p>We are very careful with backwards compatibility on the REST layer where breaking changes are made only in major releases. On the other hand, we make breaking changes to Elasticsearch&#x2019;s internal classes all the time, which is necessary in order to move the project forward. Those changes result in changes to the binary format, which we compensate for by having version-specific serialization code. &#xA0;It is this compatibility layer that allows you to do a rolling upgrade of an Elasticsearch cluster.
</p><p>That said, running a mixed version cluster is something we only recommend during the upgrade process, not as the status quo. Having older versions of nodes or clients in the cluster limits the support of newer features, as the older client simply doesn&apos;t know how to write or read requests in the newer binary format. &#xA0;
</p><p>When you upgrade your cluster, you should upgrade all nodes and clients to have the same version. The requirement to upgrade all Java clients makes upgrades harder, because it affects all the applications that communicate with Elasticsearch. And all of those changes we made to the internal Java API? &#xA0;Your application has to be adapted to cope with them. &#xA0;Fortunately, Java has a compiler which complains when an interface has changed, so updating your Java app shouldn&apos;t be too complicated. &#xA0;That said it can still be a pain in the neck to do for every upgrade.
</p><p>The REST interface is much more stable and can be upgraded out of step with the Elasticsearch cluster.
</p><h2>JVM version</h2><p>We also recommend that the client and the server are on the same Java version. This used to be a strict requirement before Elasticsearch 2.0, when we used Java serialization for exceptions. &#xA0;These days, having exactly the same Java version probably isn&apos;t as important as it used to be, but given how low level the binary format of the Java API is, it is advisable to use the same JVM version on all nodes and clients.
</p><p>The REST client can use the same version of the JVM that is used by your application.
</p><h2>Dependencies</h2><p>The Java API is not published as a separate artifact, which means that, your project has to depend on the whole Elasticsearch, including all the server code that is not really needed on the client side. This means you have some strange dependencies like lucene and log4j2, which you may not need in your application, and can actually end up conflicting with libraries that you have in your classpath. This problem has been reduced with the recent removal of the Guava dependency, but it still exists.
</p><p>The low-level REST client today has minimal dependencies: Apache HTTP Async Client and its transitive dependencies (Apache HTTP Client, Apache HTTP Core, Apache HTTP Core NIO, Apache Commons Codec and Apache Commons Logging).
</p><h2>Security</h2><p>Ideally, we would like to have a single entry-point to the cluster for users: the REST API, which can be secured via HTTPS. The transport layer should only be used for internal node-to-node communication. Knowing that users can only enter via the REST layer and not via the transport layer would greatly simplify how we write code, give us more freedom to add new features and also simplify how an Elasticsearch cluster can be secured.
</p><p>We reached the point where the disadvantages of the java API outweighed the main benefit, which is not having to write and maintain a separate client.
</p><h2>Low-level Java REST client</h2><p>This is why we released our first Java REST Client with version 5.0.0. It was the first step towards making the Java client work in the same way as all the other language clients. The first release only included what we call a low-level client, which has the following features:
</p><ul style="list-black">
	<li>
	compatibility with any Elasticsearch version
	</li>

	<li>
	load balancing across all available nodes
	</li>
	<li>
	failover in case of node failures and upon specific response codes
	</li>
	<li>
	failed connection penalization
	</li>
	<li>
	persistent connections
	</li>
	<li>
	trace logging of requests and responses
	</li>
	<li>
	optional automatic discovery of cluster nodes (also known as sniffing)
	</li>
</ul><p>The documentation is available <a href="/guide/en/elasticsearch/client/java-rest/current/index.html">here</a>.
</p><p>It is called the low-level client because it does little to help the Java users to build requests or to parse responses. It handles path and query-string construction for requests, &#xA0;but it treats JSON request and response bodies as opaque byte arrays which have to be handled by the user.
</p><p>The next step is releasing a high level client that accepts proper request objects, &#xA0;takes care of their marshalling, and &#xA0;returns parsed response objects. We&#x2019;ve been considering a few different approaches on how to get there. We thought about starting from scratch: having a client with minimal dependencies with its own requests and responses. That would be a nice green field project to work on, but it would require a big effort and make migrating applications from the Java API quite a headache.
</p><p>We decided against this especially as this approach would considerably delay the first release of the high level REST client. Instead, the initial version of the high level Java REST client will still depend on Elasticsearch, like the Java API does today. This will allow us to reuse the existing Java API requests and responses, which will ease migration for users.
</p><p>We will not implement the existing <a href="https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/client/Client.java">Client</a> interface though - we have plans to improve this interface, so client calls in your application will need to be migrated, but requests and responses should stay exactly the same as today. The first release will not support all the APIs but we are planning to support the most important ones to start with: index, bulk, get, delete and search.
</p><p>The Java REST client is the future for Java users of Elasticsearch. Please get involved and try out the high-level client as soon as it becomes available, as your feedback will help us to make it better faster. As soon as the REST client is feature complete and is mature enough to replace the Java API entirely, we will deprecate and finally remove the transport client and the Java API.
</p></body></html>

State of the official Elasticsearch Java clients

<html><head></head><body><p><em>This is the second post in a three-part series about Instant Aggregations. Read how it all started in </em><a href="https://www.elastic.co/blog/the-tale-of-caching-and-why-it-matters">The Tale of Caching and Why It Matters</a> <em>from Simon Willnauer and Shay Banon and how the magnificent end to the story in</em> <a href="https://www.elastic.co/blog/instant-aggregations-rewriting-queries-for-fun-and-profit">Instant Aggregations: Rewriting Queries for Fun and Profit</a>. <em>Enjoy the trilogy!</em><br><em></em></p><p>When writing software, adding cool new features is of course always great fun. But sometimes it&#x2019;s also important to work on internal changes in the code base that enable those shiny new additions in the future. For example, there were <a href="/blog/the-tale-of-caching-and-why-it-matters">a number of cool new ideas</a> for Elasticsearch floating around that were essentially blocked by the lack of having a good intermediate representation for search requests arriving through the REST layer, which prevented early query optimizations and delayed parsing to the shard level. </p><p>For the upcoming release of Elasticsearch 5.0 we embarked on a large refactoring to change the way search requests work internally. In this blog post we want to highlight some of the changes and challenges that came along with the refactoring of queries and the search request, how it helped us improve our testing and how it enables great new features like &quot;Instant Aggregations&quot;, that will be highlighted in a following blog post.</p><h2>How search requests were sent across the cluster prior to the Refactoring</h2><p>When you send a search request to a node in the cluster, the node receiving the request coordinates the search request from then on. The coordinating node identifies what shards the search request needs to be executed on, and forwards it to the nodes that hold those shards via the internal node-to-node <a href="/guide/en/elasticsearch/reference/current/modules-transport.html">transport layer</a>. Each shard returns a set of documents, whose ids will be sent back to the coordinating node that is responsible for reducing the matching documents obtained to the top matching ones that need to be returned to the client. Once those documents are identified, they are fetched as part of the fetch phase and finally returned.</p><p>Before Elasticsearch 5.0, each node received the original search request, parsed the query and used other information available from the shard (like mappings) to actually create the Lucene query that was then executed as part of the query phase.
  
</p><p><img src="https://api.contentstack.io/v2/assets/57d6d80eb37b0d4368c02ae7/download?uid=blt97f5df8ef0af08af" alt="parse 1" width="850"></p><p>The body of the search request is not parsed on the coordinating node but rather serialized via the transport layer untouched, as an opaque byte array which holds nothing more than its json representation. This was historically done to avoid having to write serialization code for every single query and every single section of a search request, as elasticsearch uses its own binary serialization protocol. We eventually came to the conclusion that this was the wrong trade-off. </p><h2>So what&apos;s the problem with this?</h2><p>While simply forwarding incoming requests on the coordinating node as described above is simple and fast (at least on the coordinating node), there are some drawbacks:</p><ul>
  <li>
    Each search request gets parsed multiple times, once for each shard of the target index. That means that even if with the default of 5 shards, a query will be parsed 5 times across the cluster (even multiple times in the same node if that node holds more than one shard that is relevant for the query). This is a potential waste of cpu-cycles as the search request could be parsed earlier and only once.
  </li>
</ul><ul>
  <li>
    For the same reason, if a query is malformed, it will be rejected with an error by each shard. This is why in Elasticsearch 1.x you get multiple parse failures in your response when you make a mistake in your query. It is all the same error, but it gets thrown by each shard. On 2.x we already deduplicate those kind of errors, but they are still thrown from all shards.
  </li>
</ul><ul>
  <li>
    The search request cannot be rewritten for optimizations on the coordinating node without impacting performance, as this would mean having to parse it twice.
  </li>
</ul><ul>
  <li>
    Parsing and query creation are tightly coupled, making unit testing of each individual step difficult without having the whole context of a shard.
  </li>
</ul><p>All these drawbacks arise because before 5.0 there is no intermediate representation of a query within elasticsearch, only the raw JSON request and the final Lucene query. The latter can only be created on the shards, because Lucene queries are not serializable and often depend on context information only available on the shard. By the way, this is actually true not only for the &quot;query&quot; part of the search request, but for many other things like aggregations, suggestions etc.. that can be part of a search request.</p><p>
  So, wouldn&apos;t it be nice to be able to parse queries to an intermediate representation once and early? That way we would be able to eventually rewrite them, and optimize their execution, for example by shortcutting them to queries that are less expensive.</p><h2>What we did</h2><p>For the queries we achieved this by splitting the query parsing into two phases:</p><ul>
  <li>
    Json/Yaml parsing that happens on the coordinating node, which allows us to represent a search request in a serializable intermediate format. This phase is independent from mappings and data present in the shards, to make sure that it can be performed on any node. This happens in a method called &#x201C;fromXContent()&#x201D; that every query has and which produces a new intermediate query object (all implementing the QueryBuilder interface).
  </li>
  <li>
    Conversion from elasticsearch serializable queries to lucene queries ready to be executed on the shard. This phase depends on mappings and information present in the shards, so it has to happen separately on each shard to be able to run the query against its data. The method handling this in the <a href="https://github.com/elastic/elasticsearch/blob/5.0/core/src/main/java/org/elasticsearch/index/query/AbstractQueryBuilder.java">QueryBuilder</a> interface is called `toQuery()`.
  </li>
</ul><p><img src="https://api.contentstack.io/v2/assets/57d6d82fdc33a3807b6eb256/download?uid=blt0b742cc4635f9791" alt="parse 2" width="850"></p><p>As a result of the split, the code is better organized as parsing is decoupled from lucene query generation. Having a serializable intermediate format for queries meant that we had to write serialization code for all of the queries and search sections supported in elasticsearch. Also, every query implements `equals()` and `hashCode()`, so that they can be compared with each other for easier caching and to aid their testing.</p><p>Once we moved parsing to the coordinating node, we could also move some validation to earlier stages of a search request and throw one single error earlier. Validation applies to malformed queries in terms of invalid json, queries that are missing some of their required values, or queries with invalid values provided.</p><p>In order to make all those changes in a safe way that doesn&#x2019;t break existing behavior, we wrote extensive unit tests. For each query, we added <a href="/blog/elasticsearch-testing-qa-increasing-coverage-randomizing-test-runs">randomized tests</a> to verify that it can be properly parsed and serialized, added tests that the resulting lucene query is the expected one and also test explicitly that all of the required values are checked and validated correctly. In order to do so, we created a <a href="https://github.com/elastic/elasticsearch/blob/master/test/framework/src/main/java/org/elasticsearch/test/AbstractQueryTestCase.java">base test class</a> which provides the code necessary for test setup and has some abstract methods that only need to be filled in for each individual query. This base class is shipped as part of our test framework helping downstream developers of custom queries with testing.</p><p>Most of our query parsing code had low unit test coverage before the refactoring. For example, the `org.elasticsearch.index.query` package that contains all the QueryBuilder and QueryParser classes had only 47% test coverage (actually, with randomized testing we cover more over time, these numbers refer to the coverage in one CI run) before the refactoring, going to above 77% coverage after the refactoring branch was merged.</p><h2>How we did it</h2><p>This query refactoring was really a <a href="https://github.com/elastic/elasticsearch/issues/10217">long running task</a> that needed a proper feature branch. Initially we thought it could be enough to only refactor the query part of the search request, but in the end we went for refactoring almost all parts of the search request. The branch stayed alive for several months and we took care of merging master in daily (or at least twice a week). In some cases we found bugs that we fixed upstream rather than on the branch, to get them out with regular releases as soon as possible.</p><p>As a team working on the feature branch, we decided to essentially replicate the development model of how we work on Elasticsearch master: instead of trying to plow through all changes needed and having an enormous bulk review at the very end, each change was submitted and reviewed as a small, manageable pull request against the branch. Having a closely knit team dedicated to this work, we could agree early on the goals of the changes, coding and architectural standards. This later made finding a reviewer familiar with those goals easy and made review cycles faster.</p><p>Given the amount of code that this refactoring touched, the challenging part was to keep the branch healthy while the task was in progress. We had to sit down and come up with some incremental steps to run tests while the refactoring was still in progress. It was all a matter of introducing the new functionality gradually, while leaving the old infrastructure &#xA0;alive until there was a complete replacement for it. Initially we introduced the two phase parsing but it all still happened on the data nodes rather than on the coordinating node. When all queries were migrated, we were able to move parsing to the coordinating node and start working on the remaining sections of the search request (e.g. rescore, suggestions, highlighting and so on).</p><h2>Lessons learnt</h2><p>After several months of wading through code that was originally written by multiple authors and at very different times through the evolution of the code base, we learned a couple of things:</p><ul>
  <li>
    just because you&#x2019;ve refactored a handful of queries already doesn&#x2019;t mean you have any idea of what the other about <a href="https://github.com/elastic/elasticsearch/issues/10217">50 classes</a> look like.
  </li>
  <li>
    in order to keep your branch healthy, it is vital to use CI to frequently run your test suite and deliver improvements in small incremental steps, even though this means doing more work in total.
  </li>
  <li>
    changing architectural decisions years into development does have a non-negligible cost attached, but it&#x2019;s the right thing to do, pays off in terms of maintainability and enables new features.
  </li>
</ul><h2>Conclusion</h2><p>All in all, the improvements made with this large refactoring become obvious when we need to go back to the old 2.x branch now and then to backport something from the current main development line. Not only has the parsing infrastructure become much more efficient, but introducing the intermediate query representation and decoupling parsing from query creation has lead to much cleaner code and the new test infrastructure makes debugging of parsing-related problems and writing new test much easier. </p><p>But the most important thing is, that we are now able to analyze, rewrite and optimize search request on the coordinating node much more easily. Features like &#x201C;instant aggregations&#x201D; that were long talked about, but never tackled because of the months of work required, suddenly became possible to implement quickly. In an upcoming blog post we will shed some more light on what this feature is and how it works.</p></body></html>

elasticsearch-the-great-query-refactoring-full-bleed.jpg

elasticsearch-the-great-query-refactoring-thumbnail.jpeg

The Great Query Refactoring: Thou shalt only parse once

<html><head></head><body><p>Index aliases are supported in elasticsearch for some time now. Using aliases you can easily abstract your physical indices away, so that only logical indices are exposed to the users and a lot of interesting things become possible. For instance you can <a href="/blog/changing-mapping-with-zero-downtime/">change your mapping without downtime</a>, or even scale out without reindexing your data when your index has reached its full capacity. Aliases can be created, modified and deleted through proper aliases api; elasticsearch <code>1.1.0</code> makes it even easier to create them though. The idea is that aliases should be used in every situation, right from the beginning of every project, which is why we made it possible to specify them while creating an index, as well as when creating index templates. </p>
<h2>Aliases upon index creation</h2>
<p>As of <code>1.1.0</code>, aliases can be provided as part of the create index api. Here is an example:</p>
<pre class="prettyprint noescape">curl -XPUT localhost:9200/logs-2014-04-14 -d &apos;
{
  &quot;aliases&quot; : {
    &quot;april_2014&quot; : {},
    &quot;year_2014&quot; : {}
  },  
  &quot;settings&quot; : {
    &quot;number_of_shards&quot; : 3,
    &quot;number_of_replicas&quot; : 0 
  } 
}
&apos;
</pre>
<p>The above request creates a new index called <code>logs-2014-04-14</code> and associates the aliases <code>april_2014</code> and <code>year_2014</code> with it.</p>
<h2>Aliases support in index templates</h2>
<p>From <code>1.1.0</code> it is also possible to specify aliases when registering <a href="/guide/en/elasticsearch/reference/current/indices-templates.html">index templates</a>. This comes in very handy especially when dealing with time based data, for instance daily indices. You can just create a couple of index templates as follows:</p>
<pre class="prettyprint noescape">curl -XPUT localhost:9200/_template/template_2014 -d &apos;
{
  &quot;template&quot; : &quot;logs-2014-*&quot;,
  &quot;aliases&quot; : {
    &quot;logs-2014&quot; : {}
  } 
}
&apos;
</pre>
<pre class="prettyprint noescape">curl -XPUT localhost:9200/_template/template_2014_04 -d &apos;
{
  &quot;template&quot; : &quot;logs-2014-04-*&quot;,
  &quot;aliases&quot; : {
    &quot;logs-2014-04&quot; : {}
  }
}
&apos;
</pre>
<p>As a result, whenever you create an index for year 2014, it will get automatically added to the <code>logs-2014</code> alias. Also, whenever an index for april 2014 is created, it will get added to the existing <code>logs-2014-04</code> alias that holds all indices for april 2014.</p>
<p>The above examples contain simple aliases but let&#x2019;s not forget that an alias can hold routing keys and a filter too. Also, because index templates allow you to associate an alias with an index whose name is not known yet, it is possible to use the <code>{index}</code> placeholder as part of the alias name, which will get automatically replaced with the index the template is getting applied to at index creation time.</p>
<pre class="prettyprint noescape">curl -XPUT localhost:9200/_template/template_2014 -d &apos;
{
  &quot;template&quot; : &quot;logs-2014-*&quot;,
  &quot;aliases&quot; : {
    &quot;{index}-alias&quot; : {}
  }
}
&apos;
</pre>
<p>Note that it is only possible to use the whole index name as part of the alias name. Although it would be nice to be able to use a portion of the index name and do much more complex things, we decided to keep it simple for now, but there&#x2019;s already a <a href="https://github.com/elasticsearch/elasticsearch/issues/5359">github issue</a> to address this. A big thank you to <a href="https://github.com/jbrook/">James Brook</a> for his contribution on these improvements! We look forward to your feedback on the <a href="https://groups.google.com/forum/?fromgroups#!forum/elasticsearch">mailing list</a> or <a href="https://twitter.com/elasticsearch">Twitter</a>!</p></body></html>

Aliases for the Win

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Luca Cavanna

<p>Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.</p>

Newsletter GDPR Text - Ukrainian

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

Intentionally Blank

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。</p>

Newsletter GDPR Text - TW

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p><p>By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.</p>

GDPR Text + Google Cloud

<p>Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.</p>

Newsletter GDPR Text - Italian

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Luca Cavanna

The Elasticsearch Java High-Level Rest Client is Out

Tribe Nodes & Cross-Cluster Search: The Future of Federated Search in Elasticsearch

State of the official Elasticsearch Java clients

The Great Query Refactoring: Thou shalt only parse once

Aliases for the Win

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards