<html><head></head><body>This is the second post in a three-part series about Instant Aggregations. Read how it all started in <a href="https://www.elastic.co/blog/the-tale-of-caching-and-why-it-matters">The Tale of Caching and Why It Matters</a> from Simon Willnauer and Shay Banon and how the magnificent end to the story in <a href="https://www.elastic.co/blog/instant-aggregations-rewriting-queries-for-fun-and-profit">Instant Aggregations: Rewriting Queries for Fun and Profit</a>. Enjoy the trilogy! When writing software, adding cool new features is of course always great fun. But sometimes it&#x2019;s also important to work on internal changes in the code base that enable those shiny new additions in the future. For example, there were <a href="/blog/the-tale-of-caching-and-why-it-matters">a number of cool new ideas</a> for Elasticsearch floating around that were essentially blocked by the lack of having a good intermediate representation for search requests arriving through the REST layer, which prevented early query optimizations and delayed parsing to the shard level. For the upcoming release of Elasticsearch 5.0 we embarked on a large refactoring to change the way search requests work internally. In this blog post we want to highlight some of the changes and challenges that came along with the refactoring of queries and the search request, how it helped us improve our testing and how it enables great new features like &quot;Instant Aggregations&quot;, that will be highlighted in a following blog post.<h2>How search requests were sent across the cluster prior to the Refactoring</h2>When you send a search request to a node in the cluster, the node receiving the request coordinates the search request from then on. The coordinating node identifies what shards the search request needs to be executed on, and forwards it to the nodes that hold those shards via the internal node-to-node <a href="/guide/en/elasticsearch/reference/current/modules-transport.html">transport layer</a>. Each shard returns a set of documents, whose ids will be sent back to the coordinating node that is responsible for reducing the matching documents obtained to the top matching ones that need to be returned to the client. Once those documents are identified, they are fetched as part of the fetch phase and finally returned.Before Elasticsearch 5.0, each node received the original search request, parsed the query and used other information available from the shard (like mappings) to actually create the Lucene query that was then executed as part of the query phase.
 
<img src="https://api.contentstack.io/v2/assets/57d6d80eb37b0d4368c02ae7/download?uid=blt97f5df8ef0af08af" alt="parse 1" width="850">The body of the search request is not parsed on the coordinating node but rather serialized via the transport layer untouched, as an opaque byte array which holds nothing more than its json representation. This was historically done to avoid having to write serialization code for every single query and every single section of a search request, as elasticsearch uses its own binary serialization protocol. We eventually came to the conclusion that this was the wrong trade-off. <h2>So what&apos;s the problem with this?</h2>While simply forwarding incoming requests on the coordinating node as described above is simple and fast (at least on the coordinating node), there are some drawbacks:<ul>
 <li>
 Each search request gets parsed multiple times, once for each shard of the target index. That means that even if with the default of 5 shards, a query will be parsed 5 times across the cluster (even multiple times in the same node if that node holds more than one shard that is relevant for the query). This is a potential waste of cpu-cycles as the search request could be parsed earlier and only once.
 </li>
</ul><ul>
 <li>
 For the same reason, if a query is malformed, it will be rejected with an error by each shard. This is why in Elasticsearch 1.x you get multiple parse failures in your response when you make a mistake in your query. It is all the same error, but it gets thrown by each shard. On 2.x we already deduplicate those kind of errors, but they are still thrown from all shards.
 </li>
</ul><ul>
 <li>
 The search request cannot be rewritten for optimizations on the coordinating node without impacting performance, as this would mean having to parse it twice.
 </li>
</ul><ul>
 <li>
 Parsing and query creation are tightly coupled, making unit testing of each individual step difficult without having the whole context of a shard.
 </li>
</ul>All these drawbacks arise because before 5.0 there is no intermediate representation of a query within elasticsearch, only the raw JSON request and the final Lucene query. The latter can only be created on the shards, because Lucene queries are not serializable and often depend on context information only available on the shard. By the way, this is actually true not only for the &quot;query&quot; part of the search request, but for many other things like aggregations, suggestions etc.. that can be part of a search request.
 So, wouldn&apos;t it be nice to be able to parse queries to an intermediate representation once and early? That way we would be able to eventually rewrite them, and optimize their execution, for example by shortcutting them to queries that are less expensive.<h2>What we did</h2>For the queries we achieved this by splitting the query parsing into two phases:<ul>
 <li>
 Json/Yaml parsing that happens on the coordinating node, which allows us to represent a search request in a serializable intermediate format. This phase is independent from mappings and data present in the shards, to make sure that it can be performed on any node. This happens in a method called &#x201C;fromXContent()&#x201D; that every query has and which produces a new intermediate query object (all implementing the QueryBuilder interface).
 </li>
 <li>
 Conversion from elasticsearch serializable queries to lucene queries ready to be executed on the shard. This phase depends on mappings and information present in the shards, so it has to happen separately on each shard to be able to run the query against its data. The method handling this in the <a href="https://github.com/elastic/elasticsearch/blob/5.0/core/src/main/java/org/elasticsearch/index/query/AbstractQueryBuilder.java">QueryBuilder</a> interface is called `toQuery()`.
 </li>
</ul><img src="https://api.contentstack.io/v2/assets/57d6d82fdc33a3807b6eb256/download?uid=blt0b742cc4635f9791" alt="parse 2" width="850">As a result of the split, the code is better organized as parsing is decoupled from lucene query generation. Having a serializable intermediate format for queries meant that we had to write serialization code for all of the queries and search sections supported in elasticsearch. Also, every query implements `equals()` and `hashCode()`, so that they can be compared with each other for easier caching and to aid their testing.Once we moved parsing to the coordinating node, we could also move some validation to earlier stages of a search request and throw one single error earlier. Validation applies to malformed queries in terms of invalid json, queries that are missing some of their required values, or queries with invalid values provided.In order to make all those changes in a safe way that doesn&#x2019;t break existing behavior, we wrote extensive unit tests. For each query, we added <a href="/blog/elasticsearch-testing-qa-increasing-coverage-randomizing-test-runs">randomized tests</a> to verify that it can be properly parsed and serialized, added tests that the resulting lucene query is the expected one and also test explicitly that all of the required values are checked and validated correctly. In order to do so, we created a <a href="https://github.com/elastic/elasticsearch/blob/master/test/framework/src/main/java/org/elasticsearch/test/AbstractQueryTestCase.java">base test class</a> which provides the code necessary for test setup and has some abstract methods that only need to be filled in for each individual query. This base class is shipped as part of our test framework helping downstream developers of custom queries with testing.Most of our query parsing code had low unit test coverage before the refactoring. For example, the `org.elasticsearch.index.query` package that contains all the QueryBuilder and QueryParser classes had only 47% test coverage (actually, with randomized testing we cover more over time, these numbers refer to the coverage in one CI run) before the refactoring, going to above 77% coverage after the refactoring branch was merged.<h2>How we did it</h2>This query refactoring was really a <a href="https://github.com/elastic/elasticsearch/issues/10217">long running task</a> that needed a proper feature branch. Initially we thought it could be enough to only refactor the query part of the search request, but in the end we went for refactoring almost all parts of the search request. The branch stayed alive for several months and we took care of merging master in daily (or at least twice a week). In some cases we found bugs that we fixed upstream rather than on the branch, to get them out with regular releases as soon as possible.As a team working on the feature branch, we decided to essentially replicate the development model of how we work on Elasticsearch master: instead of trying to plow through all changes needed and having an enormous bulk review at the very end, each change was submitted and reviewed as a small, manageable pull request against the branch. Having a closely knit team dedicated to this work, we could agree early on the goals of the changes, coding and architectural standards. This later made finding a reviewer familiar with those goals easy and made review cycles faster.Given the amount of code that this refactoring touched, the challenging part was to keep the branch healthy while the task was in progress. We had to sit down and come up with some incremental steps to run tests while the refactoring was still in progress. It was all a matter of introducing the new functionality gradually, while leaving the old infrastructure &#xA0;alive until there was a complete replacement for it. Initially we introduced the two phase parsing but it all still happened on the data nodes rather than on the coordinating node. When all queries were migrated, we were able to move parsing to the coordinating node and start working on the remaining sections of the search request (e.g. rescore, suggestions, highlighting and so on).<h2>Lessons learnt</h2>After several months of wading through code that was originally written by multiple authors and at very different times through the evolution of the code base, we learned a couple of things:<ul>
 <li>
 just because you&#x2019;ve refactored a handful of queries already doesn&#x2019;t mean you have any idea of what the other about <a href="https://github.com/elastic/elasticsearch/issues/10217">50 classes</a> look like.
 </li>
 <li>
 in order to keep your branch healthy, it is vital to use CI to frequently run your test suite and deliver improvements in small incremental steps, even though this means doing more work in total.
 </li>
 <li>
 changing architectural decisions years into development does have a non-negligible cost attached, but it&#x2019;s the right thing to do, pays off in terms of maintainability and enables new features.
 </li>
</ul><h2>Conclusion</h2>All in all, the improvements made with this large refactoring become obvious when we need to go back to the old 2.x branch now and then to backport something from the current main development line. Not only has the parsing infrastructure become much more efficient, but introducing the intermediate query representation and decoupling parsing from query creation has lead to much cleaner code and the new test infrastructure makes debugging of parsing-related problems and writing new test much easier. But the most important thing is, that we are now able to analyze, rewrite and optimize search request on the coordinating node much more easily. Features like &#x201C;instant aggregations&#x201D; that were long talked about, but never tackled because of the months of work required, suddenly became possible to implement quickly. In an upcoming blog post we will shed some more light on what this feature is and how it works.</body></html>

elasticsearch-the-great-query-refactoring-full-bleed.jpg

elasticsearch-the-great-query-refactoring-thumbnail.jpeg

The Great Query Refactoring: Thou shalt only parse once

<html><head></head><body>
	We&apos;re please to bring your our second article on on our testing and QA processes for Elasticsearch. If you missed last Thursday&apos;s piece, Isabel told us all about <a href="/blog/elasticsearch-testing-qa-elasticsearch-continuous-integration">Elasticsearch Continuous Integration</a>. We&apos;ll be bringing you our final installment next Thursday, so stay tuned to this space!


	&quot;Works on my machine&quot; is a phrase that became famous for software projects lacking automated testing infrastructure. Even today, when most checks are done automatically on an integration test server, it&apos;s still crucial to be able to reproduce bugs and test features on your local box. Ideally, those should be the same tests that are being run by the CI environment (or some stripped down version thereof).

<h2>Testing Layers</h2>

	Elasticsearch tests check the code base from multiple perspectives. Traditional unit tests are the usual check that core algorithm implementations are correct and all methods behave the way they should.


	<img src="https://api.contentstack.io/v2/assets/575e4d9212798d8e382001b2/download?uid=blt080cc9ec8644886b?uid=blt080cc9ec8644886b" width="512" height="250">


	One level up, integration tests run against a locally running cluster, making sure all pieces of the application work nicely together and can be interacted with through the Java Client API. REST tests make sure all REST endpoints work according to their specification.


	Backwards compatibility tests are a special case that were introduced recently. Instead of running some test against a cluster containing nodes of only one Elasticsearch version, a previous release can be downloaded, installed and started. Tests then run against a mixed node cluster making sure that everything works as expected and is compatible between releases.

<h2>Testing the Elasticsearch Java Layer</h2>

	Elasticsearch attacks testing from a bunch of different angles. Java code is tested on more than on the unit test level; the Java Client API is also checked by integration tests that pull up complete Elasticsearch clusters to run requests against.


	Essentially, the goal for integration tests (based on the class ElasticsearchIntegrationTest) is to make sure Java API calls work against a full running cluster. It is cheap to pull up an example Elasticsearch cluster in terms of CPU power and memory needed, even on an ordinary laptop. When extending the above test class, it also becomes simple in terms of development overhead. The cluster is pulled up for you and reused between tests unless you specify something else in the test&apos;s ClusterScope annotation.


	Looking at an example integration test, let&apos;s walk through the most important annotations and features that makes writing Elasticsearch integration tests so trivial:

<pre class="prettyprint lang-js">01 // make sure all tests in the test suite run on a separate test cluster as we will modify the 
02 // cluster configuration 
03 @ElasticsearchIntegrationTest.ClusterScope(scope = ElasticsearchIntegrationTest.Scope.SUITE)
04 public class TemplateQueryTest extends ElasticsearchIntegrationTest {
05 
06 @Before
07 public void setup() throws IOException {
08 // create an index, make sure it is ready for indexing and add documents to it
09 createIndex(&quot;test&quot;);
10 ensureGreen(&quot;test&quot;);
11 
12 index(&quot;test&quot;, &quot;testtype&quot;, &quot;1&quot;, jsonBuilder().startObject().field(&quot;text&quot;, 
 &quot;value1&quot;).endObject());
13 index(&quot;test&quot;, &quot;testtype&quot;, &quot;2&quot;, jsonBuilder().startObject().field(&quot;text&quot;, 
 &quot;value2&quot;).endObject());
14 refresh();
15 }
16 
17 // for our test we want to make sure the config path of the cluster actually points
18 // to the test resources that we provide - this is the cluster modification referred
19 // to earlier
20 @Override
21 public Settings nodeSettings(int nodeOrdinal) {
22 return settingsBuilder().put(super.nodeSettings(nodeOrdinal))
23 .put(&quot;path.conf&quot;, this.getResource(&quot;config&quot;).getPath()).build();
24 }
25 
26 @Test
27 public void testTemplateInBody() throws IOException {
28 Map&lt;String, Object&gt; vars = new HashMap&lt;&gt;();
29 vars.put(&quot;template&quot;, &quot;all&quot;);
30 
31 TemplateQueryBuilder builder = new TemplateQueryBuilder(
32 &quot;{\&quot;match_{{template}}\&quot;: {}}\&quot;&quot;, vars);
33 
34 // the search client to use in the test comes pre-configured as part of the
35 // integration test
36 SearchResponse sr = client().prepareSearch().setQuery(builder)
37 .execute().actionGet();
38 
39 // specific assertions make checks very simple
40 assertHitCount(sr, 2);
41 }
</pre>

	In our example, line 03 defines the cluster scope of that test to be only for the test suite. This makes sense if you change the cluster configuration, e.g. hard setting a specific configuration option like we do here in line 22 for the configuration path of the cluster.


	Starting in line 06, the setup method simply sets up an index, makes sure it is green before modifying it and adds a couple of test documents to it.


	As shown in line 36, the client to use is available as part of the integration test, all pre-configured and ready to use. So are helper assertions like the one in line 40 that make it easy to check the state of results.


	In addition to regular integration tests, Elasticsearch also tests backwards compatibility for those versions that should be backwards compatible. Essentially this is achieved in a similar way to how integration tests work. A cluster is started for the test, and the release to check backwards compatibility against is downloaded and installed. Then, for each test, a random number of nodes from the comparison release is added to the test cluster and requests are then executed against this mixed node cluster. This way, we can automatically verify that changes do not break backwards compatibility where they shouldn&apos;t, and at the per commit level if needed.

<h2>Testing the REST Layer</h2>

	The Elasticsearch REST API is defined in <a href=" https://github.com/elasticsearch/elasticsearch/tree/master/rest-api-spec" target="_blank">its own API specification</a>. Based on this spec, tests can be defined declaratively in YAML.


	The snippet below defines, in a concise way, a test to check the templating based search query. Line 01 to 04 defines the query to execute. Based on the specification for search requests, this snippet can automatically be turned into a valid GET URL and request body.

<pre class="prettyprint lang-js">01 - do:
02 search_template:
03 body: { &quot;template&quot; : { &quot;query&quot;: { &quot;term&quot;: { &quot;text&quot;: { &quot;value&quot;: &quot;{{template}}&quot; } } } }, 
04 &quot;params&quot;: { &quot;template&quot;: &quot;value1&quot; } }
05
06 - match: { hits.total: 1 }
</pre>

	Line 06 then defines the expected result. In this case only the number of hits returned is being checked.

<h2>Code Checks Built into Each Compile Run</h2>

	Tests aren&apos;t the only way code quality is ensured within Elasticsearch. On each Maven build, we check for the usage of &quot;forbidden Java APIs.&quot; In case there are known faster versions of the same API functionality, the slower API call is detected in the code base, causing the build to fail. Another example would be calls that send output to STDOUT (like calls to System.out.println that are usually a sign of forgotten debug statements), instead of using the logging framework to communicate messages. Also there are certain functions that are downright dangerous to use if one cares about compatibility across systems (think using the default charset of the machine the code is running on instead of explicitly defining which charset is supposed to be used).


	For releases, checks are even more restrictive. During development broken tests can be disabled and marked with the special annotation &quot;AwaitsFix.&quot; In the case of cutting a release, the AwaitsFix broken tests will fail the build, telling the release manager that there is still functionality that is not yet working.


	In our final installment of this series, we&apos;ll cover Elasticsearch&apos;s randomized testing framework.
</body></html>

Elasticsearch Testing & QA: Testing Levels of Elasticsearch

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Isabel Drost-Fromm

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Isabel Drost-Fromm

The Great Query Refactoring: Thou shalt only parse once

Elasticsearch Testing & QA: Testing Levels of Elasticsearch

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards