<html><head></head><body>Quite some time ago, with the GA release of Elasticsearch 5.0 we released an <a href="/guide/en/elasticsearch/reference/current/indices-shrink-index.html">API</a> that allows to shrink an index into a new index with fewer shards than the original index. The reasoning behind adding this functionality was to provide a tool to keep the number of shards in a cluster at bay. Indices are commonly created with a large number of shards to maximise indexing throughput but later once theses indices are rolled over into a daily or hourly index, the number of shards should be reduced again to maximise resource utilization of the cluster. While this was a step in the right direction, we were still missing the ability to go in reverse: to increase the number of shards in an index if the index has become too small for the number of documents in it.
<h2>We had a story for this, we thought. </h2>For the entire history of Elasticsearch the answer to the question of how to split an index was to just add another index and grow your shard count that way. Under the hood Elasticsearch just operates on a set of shards which allows you to add indices after the fact and search across them. This sounds like a great plan for use-cases where documents are largely immutable like metrics and logging, but not as great when &#xA0;you have to update documents, as is the case for use cases like full-text search. For this you had to go and figure out in what index the document was added and update it there or you delete the document everywhere and add it to the latest index. All of these options have downsides and can cause temporary duplicates or missing documents which are not always acceptable.
After having some deep technical conversations with some users on a trip to Munich I was convinced we have to do something about this. But some of the options scared me a fair bit. For instance splitting an index or a shard while the index is actively indexing sounds very appealing in the first place but when you start thinking about the implications and how it would work under the hood, the approach doesn&#x2019;t sound all that great anymore. Aside from all the internal changes of how we do consistent hashing, how the number of shards are immutable in an index and how we can make this all work in a backwards compatible way, the performance implications of resharding while a cluster is rebalancing or loses a node had so many edge cases that this feature would likely take many engineering iterations to get right.
Luckily we already had implemented the shrink API that was basically doing the same thing in reverse with a lot of interesting properties. Shrink works by creating a copy of the index that recovers from the original index but uses fewer shards. Wait, a copy? That sounds very expensive copying all the data. To prevent this we make use of the properties lucene provides us by never touching a file more than once. Given that we can take point-in-time snapshots of the source index and use hard-links on the filesystem level, the shrink operation is almost instant and doesn&#x2019;t require any additional space. All properties of Elasticsearch indices (such as the fixed number of shards) are maintained during this time. If any issues occur during the shrink, the shrink can be aborted and the target index deleted.
This sounds great, let&#x2019;s see if we can do the same thing for splitting an index as well. And in fact it worked like a charm. The <a href="/guide/en/elasticsearch/reference/current/indices-split-index.html">split</a> API uses the same mechanics as the shrink API. Once an index is split, Elasticsearch allocates the shards of the target index alongside of the source index and recovers from the point-in-time snapshot of the underlying Lucene index. The only significant difference is that we need to do a single pass through the index on disk to mark all documents as deleted that don&#x2019;t belong in the target shard. This means we have an additional space requirement of 1 bit per document while all other data files are reused via hard-links. Just like the shrink process this is a reasonably fast operation that typically happens within a couple of seconds and can be monitored with the <a href="/guide/en/elasticsearch/reference/current/indices-recovery.html">recovery API</a>.
Unfortunately you can&#x2019;t rename an index in Elasticsearch today. To make efficient use of this API, the usage of <a href="/guide/en/elasticsearch/reference/current/indices-aliases.html">index aliases</a> is highly recommended. If an index needs to be resized, users should split or shrink the index, flip an alias to point to the new index and delete the old index in a single atomic step, using the <a href="/guide/en/elasticsearch/reference/6.2/indices-aliases.html">update-aliases API</a>.
<h2>It&#x2019;s all about trade-offs.</h2>That is likely the story of every software engineers life. In this case we made the trade-off that indexing has to stop for a short amount of time until the index is resized since source indices must be read-only. But as mentioned above these operations are considered fast, and given that this provides transactional behavior with the least amount of surprise to the user we are following our consistent strategy to go with the safe option rather than the fancy one. For details on split factors and how to pick the number of target shards, please see the official <a href="/guide/en/elasticsearch/reference/current/indices-split-index.html">documentation</a> for the specific version of Elasticsearch you are using.
</body></html>

Resizing Elasticsearch shards for fun and profit

Elasticsearch has a powerful <a href="/guide/en/elasticsearch/reference/current/search-search.html">_search API</a> that allows it to search against all indices on &nbsp;the local cluster. We recently released Elasticsearch 5.3.0 including a new functionality called <a href="/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html">Cross-Cluster Search</a> that allows users to compose searches not just across local indices but also across clusters. This means that one can search against data that belongs to other remote clusters. Historically, the Tribe Node was used when searches needed to span multiple clusters, yet it works very differently. In this blog post, we will cover why we implemented Cross-Cluster Search, how it works and compares to the Tribe Node, and why we are convinced it is a step in the right direction for the future of federated search using Elasticsearch.<h3>Keep it simple</h3>When we sat down to try and redesign the next-generation Tribe Node, we re-evaluated the problems that it was trying to solve. The goal was to make federated search possible without all the limitations that the Tribe Node provides, and without adding a specific API for it, so that maintaining such feature would also be easier. We realized that some of the features that the Tribe Node offers besides federated search are commodities. The Tribe Node supports many Elasticsearch APIs allowing, for instance, to retrieve the cluster state or nodes stats via a Tribe Node, which will return the information collected from all the remote clusters and merged into a single view. Merging information coming from different sources is nothing complicated though, and is easily performed on the client side by sending multiple requests to the different clusters. The hard problem that must be addressed on the server side is federated search. It involves a distributed scatter and gather to be executed on nodes belonging to multiple clusters as well as the result merging and reduction requiring internal knowledge. That is why we decided to focus on solving this specific problem in a sustainable and robust way by adding support for Cross-Cluster Search to the existing _search API.<h3>Search API Detour</h3>The _search API allows Elasticsearch to execute searches, queries, aggregations, suggestions, and more against multiple indices, each one composed by one or more shards. When a Client sends a search request to an Elasticsearch cluster, the node that receives the request acts as the coordinating node for the whole execution of the request. It identifies which indices, shards, and nodes the search has to be executed against. While executing, all data nodes holding a shard that is queried receive requests in parallel, then each node executes the query phase locally and sends the results back to the coordinating node. The coordinating node waits for all shards to respond in order to reduce the results down to the top-N hits that need to be fetched from the shards. A second execution phase then fetches the top-N documents from the shards in order to return the results back to the Client.<h3>How Cross-Cluster Search Works</h3>Since Elasticsearch 5.3.0, it is possible to register remote clusters through the <a href="/guide/en/elasticsearch/reference/current/cluster-update-settings.html">cluster update settings API</a> under the <code>search.remote</code> namespace. Each cluster is identified by a cluster alias and a list of seed nodes that are used to discover other nodes belonging to the remote cluster, as follows:<pre class="prettyprint">PUT _cluster/settings
 {
 "persistent": {
 "search": {
 "remote": {
 "cluster_one": {
 "seeds": ["remote_node_one:9300"]
 },
 "cluster_two": {
 "seeds": ["remote_node_two:9300"]
 }
 }
 }
 }
 }</pre>Once one or more remote clusters are registered, it is possible to execute search requests against their indices using the _search API of the cluster the remotes are configured on. In contrast to a so-called local index, remote indices must be disambiguated with their corresponding configured cluster alias separated by a colon (e.g. <code>cluster_two:index_test</code>).Whenever a search request expands to an index on a remote cluster, the coordinating node resolves the shards of these indices on the remote cluster by sending one _search_shards request per cluster. Once the shards and remote data nodes are fetched, searches are executed just like any other search on the local cluster explained above, using the exact same code-paths which improves testability and robustness dramatically. From a search execution perspective, there is no difference between local indices and indices that belong to remote clusters as long as the coordinating node can reach some nodes belonging to the remote clusters. Finally, the hits returned as part of the search response which belong to remote clusters have their index name prefixed with their cluster alias.When registering a remote cluster, Elasticsearch discovers by default up to 3 nodes per configured remote cluster through the seed nodes in the configuration. In contrast to nodes in the local cluster, where any node connects to any other node, cross cluster search nodes are only connected in an unidirectional fashion. Those are the nodes that the coordinating node will communicate with as part of a Cross-Cluster Search request. It is possible to control how many nodes are discovered when registering the remote clusters, as well as mark nodes belonging to remote clusters as gateways in order to control exactly which nodes will receive connections from the outside world. In case a node holds data that has to be accessed as part of a cross cluster search request, that node will not receive a direct connection from the remote coordinating node, but rather from another node marked as gateway in its own cluster, which will act as a proxy.Additionally, it is possible to control which nodes can act as coordinating nodes as part of a cross cluster search request through the <code>search.remote.connect</code> setting. This is useful to control which nodes in a cluster can send requests to remote clusters. If a node that is not allowed to connect to remote clusters receives a search request that involves remote clusters, an error will be returned.<h3>What about the Tribe Node?</h3>Searching against multiple clusters isn’t something new to Elasticsearch. In fact, users have been doing that up until now using a Tribe Node.The Tribe Node is a separate node whose main job is to sniff the cluster states of the remote clusters and merge them altogether. In order to do that, it joins all the remote clusters which makes it a very special node that doesn’t belong to a cluster of its own, yet it joins multiple clusters.When a Tribe Node receives a search request, it knows right away which nodes to forward it to as it holds the cluster state of all the registered remote clusters, it’s in fact a node in all of the clusters. That means that the additional “find out where the remote shards are and which node they belong to” step required with Cross-Cluster Search is not required with a Tribe Node. It is important to note that Cross-Cluster Search doesn’t require additional nodes, given that any node can act as the coordinating node for a search request, regardless of whether or not the request spans across multiple clusters. That also means that no additional nodes join the remote clusters, hence cluster state updates don’t have to be sent to those, which could potentially slow down the remote clusters when using a Tribe Node. That is because the Tribe Node receives and has to acknowledge every single cluster state update from every remote cluster.Furthermore, when merging cluster states from the remote clusters, the Tribe Node cannot keep indices that have the same name in its own cluster state, although they belong to multiple clusters. This is quite a big limitation which Cross-Cluster Search addresses, as well as being able to dynamically register, update, and remove remote clusters, which requires a node restart when using the Tribe Node.Also, being the tribe node such a special node, it turned out very hard to maintain code-wise over time, as it is the exception to the main assumption for an Elasticsearch node, that a node must belong to one and only one cluster.<h3>Further Search Improvements</h3>Elasticsearch’s retrieval capabilities are outstanding given that even with significant cluster size users can search across as many shards as they feel like. But wait, that’s not necessarily the case — at least in the current state of affairs. Elasticsearch comes with a limitation in the form of a soft-limit <code>action.search.shard_count.limit</code> which rejects search requests that expand to more than 1000 shards. Why would we want to limit this? Well, the reason here is obvious given the implementation detail mentioned in the search detour section above. We fan out to all shards and maintain all shards results in a dense data structure until all shards responded. Imagine you are running large aggregations, the coordinating node has to maintain a non-trivial amount of memory per search request to the entire duration of the initial phase.Now with the addition of cross cluster search where we are emphasizing searches across many, many shards, having such a soft limit isn’t providing a good user experience after all. In order to eliminate the impact of querying a large number of shards, the coordinating node now reduces aggregations in batches of 512 results down to a single aggregation object while it’s waiting for more results to arrive. <a href="https://github.com/elastic/elasticsearch/pull/23946">This upcoming improvement</a> was the initial step to eventually remove the soft limit altogether. The upcoming 5.4.0 release will also allow to reduce the top-N documents in batches. With these two major memory consumers under control, we now default the <code>action.search.shard_count.limit</code> setting to <a href="https://github.com/elastic/elasticsearch/pull/24012">unlimited</a>. This allows users to still limit and protect their searches in the number of shards while &nbsp;providing a good user experience for other users.<h3>Conclusions</h3>Cross-Cluster Search is the new way of executing federated search, which will replace the Tribe Node in the future. It works very differently compared to the Tribe Node and is not subject to most of the drawbacks that the Tribe Node comes with. We invite you to try it out by <a href="/downloads/elasticsearch">downloading Elasticsearch 5.3.x</a> (or <a href="/cloud/as-a-service">deploying on Elastic Cloud</a>) and give us feedback.

Tribe Nodes & Cross-Cluster Search: The Future of Federated Search in Elasticsearch

<html><head></head><body>This is the first post in a three-part series about Instant Aggregations. See how the story progresses in <a href="https://www.elastic.co/blog/the-great-query-refactoring-thou-shalt-only-parse-once">The Great Query Refactoring: Thou Shalt Only Parse Once</a> and wraps up in <a href="https://www.elastic.co/blog/instant-aggregations-rewriting-queries-for-fun-and-profit">Instant Aggregations: Rewriting Queries for Fun and Profit</a>. Enjoy the trilogy! It&#x2019;s early 2013 and an unusually sunny day in Amsterdam and a group of people are meeting around table soccer and ping pong tables for what we call a company all-hands. Just recently Rashid Khan, one of the big characters behind Kibana, joined Elastic and we are still just a handful of engineers. I&#x2019;m hacking around trying to get checksums to work for recovery, listening to a conversation between Shay and Rashid. It&#x2019;s Kibana&#x2019;s initial dashboard slowness that causes this intense conversation. Even though Kibana fires up almost identical searches each time you open the home page, elasticsearch has to recompute everything from scratch. Someone might ask, no caching eh? True!
A closer look under the hood shows that searches are almost the same, but are subject to this annoying property of time: it never stands still. If you have used Kibana yourself you might have realized that a default filter is always based on the current time (NOW) going backwards for a defined time range. In other words you never fire the same query more than once a millisecond.
Then the discussion got serious: Rashid and Shay started talking about caching and adding REST level primitives to control the cache key. Time to stop working on checksums; I gotta get involved! If you try to solve one of the hardest problems in computer science and the discussion is heading towards allowing the user to control it, you are either a really brave engineer or all other options would require you to be a hell of a brave engineer! The discussion continued for a while and ideas basically went through the roof. &#xA0;You might have experienced this in your day to day job before. Luckily, we had so many other problems to solve at that time that we just dropped the ball on it for a while.
Fast forward: it&#x2019;s October 1st and my calendar says &#x201C;The Dudes are in Berlin&#x201D; meaning that Shay and a bunch of other team leads were coming into town for some planning sessions. That&#x2019;s usually an intensive time in a distributed company like Elastic since we don&#x2019;t meet in person more than twice per year. After 3 days of discussions, brainstorming and arguing Shay and I went out for Schnitzel to this awesome Austrian place near my house. Honestly, neither Shay nor I were really up for any more discussions but suddenly the caching thing came up again. I don&#x2019;t blame anybody; I&#x2019;m not sure who opened that particular can of worms.
Anyway, this time we came up with a plan! Admittedly, not low hanging fruit, but something that could actually work well, is fully transparent, easy to test and can be disabled if it&#x2019;s not working. You noticed that escape hatch, did you? Caching is hard but let me explain what we had in mind. Bear with me, I&#x2019;m going to take a big swing:
Elasticsearch is based on Apache Lucene&#x2122; which works based on point-in-time view of an index. Such a snapshot is basically a set of write once index segments, each holding a subset of the indexed documents. The software construct we use to represent such a snapshot is what we call a &#x201C;top-level&#x201D; indexreader. We know that, unless the top-level reader changes, queries are idempotent or in other words, cacheable. In Elasticsearch there is exactly one Lucene index per shard, so we can simplify things to use one top-level reader per shard. Now, if we can identify the outer bounds of an index for any date field we could also make much better decisions if for instance all or even no documents at all would match a certain filter and therefore could rewrite the query to <a href="/guide/en/elasticsearch/reference/current/query-dsl-match-all-query.html">match-all</a> &#xA0;or match-no-docs respectively. If we could manage to do that then we could put queries that appear to be un-cachable into the<a href="https://github.com/elastic/elasticsearch/pull/7161"> request cache</a> we added basically just before that Schnitzel brainstorming session. The request cache utilizes Lucene&#x2019;s top-level reader as well as the search request&#x2019;s binary representation (the plain unmodified JSON, YAML or CBOR bytes) as a combined cache key. This allowed us to minimize the space requirement at the same time as minimizing the number of objects to represent a cache entry.
Back to the idea, the main driver of it was a <a href="https://issues.apache.org/jira/browse/LUCENE-5610">new utility</a> in Lucene that allows us to fetch the upper and lower bounds of a field from a top-level reader. With this information we could do anything you could possibly imagine with a query that has time properties. You can imagine when two engineers get excited over Schnitzel and caching is involved it ain&#x2019;t gonna end well.
With all that excitement I went and asked <a href="http://blog.mikemccandless.com">Mike Mccandless </a>to give the idea a shot. Mike pulled off a <a href="https://github.com/mikemccand/elasticsearch/commits/rewrite_query_cache_key">prototype</a> quite quickly. As usual after all that excitement we had to face reality, but the basic idea worked, YAY! Well, reality was that the prototype worked but it was far from anything that could go into production any time soon. We had to realize that to effectively modify queries in a generic and safe way (both are good properties to have if you want to use something as a cache key) we need an intermediate representation for our entire request constructs.
At this point we had no way to parse the request, modify it and write it back out into a byte representation that can be used as a cache key. This was kind of a downer for all of us since we had to face the fact that each of these ~70 queries, aggregations, suggest, highlight, sort, etc. classes had to be refactored in order read, normalize, modify and write back its values. What could possibly go wrong.
<h3>Is this worth the trouble?</h3>After that prototype and the reality check that our code wasn&#x2019;t ready we needed to talk about the exciting possibilities again. When you look at your code and you realize you have to basically invest 6 month worth of engineering time to make the first step towards a new feature you think twice about whether it&#x2019;s worth it.
That said, with the immense growth of time-based data and how the data is structured on a macro level it became obvious why it would make sense to invest in solutions that are done the right way. A typical installation of logging data might have daily indices spanning period of a week or a month. Searches typically span all indices in that time range but, except for the current daily index, all other indices are static: they don&#x2019;t receive any changes anymore and therefore they maintain the same point-in-time snapshot. This means we&#x2019;d get 100% cache hits if queries included the entire day for each shard. In other words we could reduce the search workload for these machines dramatically and kibana dashboards would appear instantly.
Well, these were convincing arguments, so now it was time to get things started. In March 2015 we started prototyping how we could represent queries and write them back out. Nobody knew that it would take another 12 months for all the needed parts to come together and yet another 6 months for this feature to be released in an alpha release. I won&#x2019;t tell the rest of the story because I want to leave that to the hard working folks who implemented all these changes. So, stay tuned for the upcoming articles in this series.
</body></html>

a-tale-of-caching-and-why-it-matters-ping-pong-elastic-elasticsearch-full-bleed.jpg

a-tale-of-caching-and-why-it-matters-elasticsearch-ping-pong-thumbnail.jpg

The tale of caching and why it matters…

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-simon-willnauer-300x300.jpg

Founder & Elasticsearch Tech Lead

Simon Willnauer

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Simon Willnauer

Founder & Elasticsearch Tech Lead, Elastic

Resizing Elasticsearch shards for fun and profit

Tribe Nodes & Cross-Cluster Search: The Future of Federated Search in Elasticsearch

The tale of caching and why it matters…

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards