Avoiding and Correcting Hotspots: How Elasticsearch Serverless Balances Shards

Free yourself from operations with Elastic Cloud Serverless. Scale automatically, handle load spikes, and focus on building—start a 14-day free trial to test it out yourself!

You can follow these guides to build an AI-Powered search experience or search across business systems and software.

The Elasticsearch Serverless Balancer addresses write load hotspots, prevents data node out-of-memory (OOM) events and avoids index-level hotspots in Elasticsearch Serverless clusters: these are workload edge cases that in non-Serverless require manual intervention and custom tuning of cluster settings. Serverless shard balancing focuses on staying within the bounds of node-level resource constraints. Rebalancing moves are explainable, where moves are made explicitly to either avoid performance degradation or correct hotspots when they develop. Shard movements are generally found to be fewer, as well.

How Elasticsearch Shard Balancing Works

Elasticsearch uses a weights-based algorithm to create a Desired Balance, an assignment of shards to data nodes. The Balancer determines the target allocation of shards across a cluster of nodes using four key metrics weighted in a linear algorithm. A total weight is calculated per node, and the shard balancer aims to equalize the total weights across cluster nodes. A final Desired Balance shard allocation is precomputed based on the latest cluster state information, and then the elected master node initiates incremental shard moves to reach the desired shard allocation.

The four metrics are:

• Write Load: the total write threadpool activity per node, using the sum of threadpool indexing activity per data-stream shard.

• Disk Usage: the total disk usage of shards per node, using the sum of disk space used per shard.

• Shard Count: the total number of shards assigned to a node.

• Index Balance (shard anti-affinity): per index, how many shards in the index are assigned to the node.

The total weight of a node is calculated using a linear algorithm that finds the deviation from the node-level cluster average for each individual metric, applies a different weight factor multiplier to each, and then takes the sum of all resultant values. The weight factor multipliers attempt to equalize the relative magnitude of each metric so that metrics with large values do not eclipse metrics with naturally small values. Write load tends to be a small value, related to thread usage, and thus gets multiplied by a relatively larger weight factor of 10; whereas disk usage in bytes is a very large number and therefore gets multiplied by a tiny weight factor of 2e-11.

The following are the cluster settings with default values, representing the different weight factors:

cluster.routing.allocation.balance.shard: 0.45

cluster.routing.allocation.balance.index: 0.55

cluster.routing.allocation.balance.disk_usage: 2e-11

cluster.routing.allocation.balance.write_load: 10.0

The linear algorithm looks something like this:

final float shardWeightFactor =
    settingValue("cluster.routing.allocation.balance.shard");
final float writeLoadWeightFactor = 
    settingValue("cluster.routing.allocation.balance.write_load");
final float diskUsageWeightFactor = 
    settingValue("cluster.routing.allocation.balance.disk_usage");
final float indexWeightFactor = 
    settingValue("cluster.routing.allocation.balance.index");

final float shardCountDeviation = numShardsOnNode - averageShardsPerNode;
final float writeLoadDeviation = totalWriteLoadOnNode - averageWriteLoadPerNode;
final float diskUsageDeviation = totalShardDiskUsageOnNode - averageShardDiskUsagePerNode;
final float indexDeviation = numIndexShardsOnNode - averageNumIndexShardsPerNode;

return shardCountDeviation * shardWeightFactor
    + writeLoadDeviation * writeLoadWeightFactor
    + diskUsageDeviation * diskUsageWeightFactor
    + indexDeviation * indexWeightFactor;

Shard movements are triggered to ensure that the difference in total node weight across cluster nodes remains below the cluster.routing.allocation.balance.threshold with a default value of 1: whenever the threshold is exceeded, shards are moved from the most heavily weighted nodes to the least heavily weighted nodes until the difference between the most heavily weighted and least heavily weighted node is at or below the threshold. Whenever cluster activity occurs that changes shard allocation (e.g., create/delete index, add/remove node, or the disk usage grows), the Balancer rechecks the weights across nodes and triggers shard rebalancing if the delta between the most and least heavily weighted nodes exceeds the configured threshold. The threshold-based approach attempts to balance the trade-off between keeping the cluster perfectly balanced and minimizing shard movements. Large Elasticsearch deployments that use nodes with greater resources typically benefit from raising the threshold setting: a larger weight delta between nodes reduces shard rebalancing.

Shard movement is also constrained by strict shard assignment rules that prohibit certain node assignments according to cluster and index level settings. Examples include: not assigning copies of the same shard to the same node or host; not allowing further assignment of shards to a node that does not have spare disk space; and excluding node(s) as host for a particular index. More on this below.

How a Balanced Cluster Looks (Based on Weights)

Using the linear algorithm and cluster setting defaults previously described, the following is an example of what the Balancer considers balanced. Notably, it can sometimes allow considerable deviation across nodes in any one particular metric. For simplicity, index balance is not included.

Weight Node1 = 0.45 (5 - 6) + 10 (0.3 - 0.33) + 2e-11 (1e+11 - 8e+10) =   - 0.35

Weight Node2 = 0.45 (7 - 6) + 10 (0.4 - 0.33) + 2e-11 (4e+10 - 8e+10) =   0.35

Weight Node3 = 0.45 (6 - 6) + 10 (0.3 - 0.33) + 2e-11 (1e+11 - 8e+10) =   0.10

Limitations of Weights-Based Shard Allocation

Elasticsearch Serverless deployments are managed by Elastic and could run into many edge cases that, without additional configuration, the weights-based shard allocation handles poorly. In self-managed Elasticsearch deployments, it is possible to work around many of these issues by configuring the cluster to suit the user’s workload. However, Elasticsearch Serverless is configured once and must work across all customer use cases. Issues experienced by some Elasticsearch customers (early adopters of Serverless among them) include:

• Continuous rebalancing background noise in active clusters. This could be because the threshold setting needs tuning or because the cluster is very busy.

• The Balancer’s behavior cannot be tuned in a predictable manner. Adjusting the Balancer settings (individual weight factors) can lead to unpredictable outcomes due to the linear algorithm. For example, decreasing the shard count weight factor relative to the other weight factors can lead to data node OOM events when shard count balancing is deprioritized and too many shards pile up on a single node.

• No explanation of why the Balancer is making shard moves. The linear algorithm is difficult to understand without relevant node metrics.

• The linear algorithm allows a high value in one metric to cancel out a low value in another metric. For example, a node can have a higher than average (across cluster nodes) write load, but counterbalance with a lower than average shard count (or vice versa), and the linear algorithm cancels out the spikes: no shards are moved to address the write load hotspot.

• Index-level hotspots can occur when a disproportionate number of index shards are assigned to the same node, rather than spreading out across nodes, despite the index balance weight in the linear algorithm. Index balance weight can, in some situations, be little compared to the other weight factors. It can also get skewed and counterbalanced by another non-average individual weight in the linear algorithm, as described in a previous bullet.

• No search load balancing.

• Regular indices do not have write load estimate support, leaving some write load hotspots unaddressed. Only data stream indices have write load estimates.

• Write load hotspots can be missed. Write load estimates are only refreshed at rollover time, which can be infrequent in some configurations, causing new load to be ignored for some time. The write load is also the average write load activity over a potentially large window of time between index rollover events, so temporary write load increases can disappear when averaged with inactive write periods.

The above issues persist in some Elasticsearch deployments and require monitoring and workload tuning to manage when they do occur. Shard allocation balancing in Elasticsearch Serverless aims to address these issues and avoid any manual intervention requirements using a new approach that is explained in subsequent sections of this article.

Elasticsearch Serverless Shard Allocation 

Elasticsearch Serverless considers node resources individually: shards are rebalanced away from a node when any resource usage on that node grows to threaten performance, and shard movements to a node are declined when the assignment could threaten that node’s performance.

The Elasticsearch single combined score per node is replaced in Elasticsearch Serverless with independent per-resource decisions:

The Elasticsearch Balancer has three phases, in order of priority, for shard movement decisions. The first phase is to assign unassigned shards. Assignment of unassigned shards is the top priority for data availability reasons. The second phase is to move shards that can no longer remain where they are assigned due to cluster configuration changes. Internally, AllocationDecider implementations enforce cluster settings, like index-level shard allocation filtering, shard allocation awareness, disk usage thresholds, or moving shards off of a node before shutdown. The third phase rebalances shards when the cluster.routing.allocation.balance.threshold is exceeded, using the previously described weights algorithm.

The new Serverless balancing approach adds additional logic to the Balancer’s first and second phases, leveraging the existing AllocationDecider logic, and eliminates the third phase. Previously, each AllocationDecider had simple responses of YES and NO. Now, the decision type of NOT_PREFERRED has been added, along with several new AllocationDecider implementations. An AllocationDecider will return NOT_PREFERRED when it observes that performance might suffer from a shard’s assignment to a particular cluster node. The Serverless Balancer will prefer a node assignment for the shard where all AllocationDecider implementations reply YES.

A NOT_PREFERRED shard allocation may be left uncorrected if all other node assignments return NO or NOT_PREFERRED. Such responses mean that the shard cannot be assigned elsewhere without either violating a cluster/index rule or potentially degrading the performance of another cluster node. Serverless Autoscaling activates before all cluster nodes hotspot: even one unaddressable hotspot leads to a scale-up event. New AllocationDecider implementations have also been added for important finite resources, like available heap memory (further discussion below), using only the original YES and NO decisions: exceeding certain categories of resources can lead to node unavailability.

The individual weight metrics in the Balancer’s linear algorithm have been replaced by resource-aware AllocationDecider implementations, and new AllocationDecider implementations are being built for additional resources: Serverless Search Tier load-balancing improvements are currently in development. Each shard migration will have a clear purpose to address a potential resource usage bottleneck.

Internal stats have shown far fewer shard movements in general, without any noticeable accompanying node performance degradations – one workload showed a 50% reduction in shard movements with the same write throughput. Fewer shard movements has the benefit of: avoiding momentary read/write latencies from warming up local caches; and saving on cloud infrastructure costs moving data between servers.

Serverless IndexBalanceDecider: Avoid Colocation of Index Shards

The IndexBalanceDecider ensures index shard anti-affinity much more strictly than the original weights-based linear algorithm could achieve. Colocation of index shards in excess of the index’s average shards per available node is avoided, except in the case of a strict NO assignment (essentially non-existent right now in Serverless except for shutting down nodes and rolling upgrade incompatible version checks) or NOT_PREFERRED assignment due to temporary node hotspotting.

The IndexBalanceDecider is a very effective means of pre-balancing both write load and search load before user workloads begin to generate load statistics: each index begins life with its shards distributed across as many nodes as possible.

IndexBalanceDecider Results: Even Write Load Distribution Across Cluster Nodes

Write load across data nodes became much more evenly distributed after the IndexBalanceDecider was enabled in the Serverless Production environment. Projects fleet-wide generally show even ingest load (counted in saturated WRITE threadpool threads), combining the release of the IndexBalanceDecider and many other prior improvements:

A reproducible workload demonstrates a clear before and after view of the impact of the new IndexBalanceDecider when an ingest workload was run with and without it enabled:

The IndexBalanceDecider also serves in the Serverless Search Tier to distribute shards of the same index as much as allowed, similarly limited only by the tier’s node count and the number of shards in each index.

Serverless HeapUsageDecider: Assign Shards by Available Heap

The HeapUsageDecider limits shard count on a node based on available heap to hold in-memory shard metadata and run associated write/read operations, removing the dependency on shard count limits per node. The HeapUsageDecider returns a strict YES or NO decision, rather than using the new NOT_PREFERRED decision type, because a data node risks an OOM event if the estimated available heap memory is exceeded.

HeapUsageDecider Results: Reduced Data Node OOMs

Data node OOMs in the serverless index tier decreased significantly as the HeapUsageDecider rolled out to the Serverless production environment.

Index tier OOM errors still occur from time to time in the Serverless Index Tier, though at a much reduced rate, as miscellaneous runaway memory usage edge cases are surfaced. The remaining OOM errors are being progressively resolved as they are identified, through a combination of memory usage improvements in the code, adding component level limits, and updating the internal Elasticsearch Serverless memory model service to more completely account for memory usage.

The HeapUsageDecider is not yet turned on in the Serverless Search Tier, due to the need for additional and different metrics, but that work is in active development.

Serverless WriteLoadDecider: Prevent and Correct Write Load Hotspots

The WriteLoadDecider receives periodically refreshed (every 30 seconds by default) per shard and per node write load stats and uses the data to correct and avoid write load hotspots. The master node retrieves stats directly from each data node’s write threadpool: an Elasticsearch node tracks the total time that its WRITE threadpools is in use, and each individual Elasticsearch shard instance tracks how much time it spent using its node’s WRITE threadpool.

A write load hotspot is identified at the node level. The criteria for a hotspot is the presence of WRITE threadpool queue latency above a configured threshold and sufficiently high, and sustained, WRITE threadpool thread saturation. Once that situation is detected, the Balancer is signaled to select shards to move away from a hotspotting node, until fresh non-hotspotting write load stats are received from the node. The Balancer will do nothing if all nodes are hotspotting at once, expecting the Autoscaler to solve the problem by introducing more, or bigger, data nodes to the cluster.

The WriteLoadDecider uses a heuristic to choose shards to move away from a hotspotting node that aims to minimize ingest disruptions while still effectively reducing a node’s write load. A shard write load threshold is identified on a hotspotting node: the threshold is currently calculated as ½ the ingest load of the hottest shard on that node. Shards that can be moved are then prioritized in the following order:

threshold = ½ * maxWriteLoadShardOnNode

1. Shards with write load in the range [threshold, maxWriteLoadShardOnNode), the shard at or closest to threshold preferred.

2. Shards with write load in the range (threshold, 0], the shard closest to threshold preferred.

3. Shards with write load equal to maxWriteLoadShardOnNode.

4. Shards with zero write load.

The heuristic prefers to avoid disruption to the highest ingest shards and instead chooses middlingly loaded shards. Movement of the hottest shard will cause the most latency disruption; and movement of the coldest shards will be the least effective in resolving a hotspot.

The Balancer limits write load hotspot correction shard moves to one move per hotspotting node per stats refresh period, in order to see the effect of a move in real-time node-level write load, before attempting any further corrections. This was a simple initial design that proved effective. Furthermore, the Balancer will not move a shard whose write load alone is sufficient to meet the node-level hotspot criteria: this would just relocate a hotspot to another data node, not actually resolve the hotspot. The Serverless Autoscaler and Serverless Autosharding components are relied upon to resolve hotspots that reallocation of shards cannot.

The WriteLoadDecider returns NOT_PREFERRED when acceptance of a shard could cause a node to start experiencing WRITE threadpool queue latency and create a hotspot. A shard will still be relocated to a NOT_PREFERRED node, however, and risk some performance degradation, as a better option than, say, risking a data node OOM from keeping a shard on a data node where the HeapUsageDecider returns NO.

WriteLoadDecider Results: Hotspots are Quickly Corrected 

Hotspot stats showed general improvement as the WriteLoadDecider was rolled out to Serverless production, in particular the fleet-wide time to correct a hotspot decreased greatly:

Since these graphs were collected, additional work has been released incrementally to better prevent and correct hotspots, and improvements are still in progress.

Serverless Autoscaling, Autobalancing, and Autosharding

Elasticsearch Serverless relies on both new autobalancing logic and new autoscaling logic. The Serverless Balancer must sufficiently distribute shard resource usage across nodes in order to fully saturate the cluster’s resources. The Serverless Autoscaler will trigger a scale-up event when it receives a report that a certain percentage of the total cluster resources are in use and more resources are needed. The Autoscaler will not scale up the cluster if one node is hotspotting and another node has an excess of available resources because the resources are summed across nodes. Therefore, the Balancer must first do a good job on load distribution, and then the Autoscaler will activate as needed.

Autosharding based on write load is also in progress and coming soon to address shard hotspots. Elasticsearch Serverless projects have a default number of shards per index based on the project type. These defaults generally work, but do not account for all possible workloads. Hotspots can occur when an index has too few shards, as well as too many. Too few index shards leads to the Balancer being unable to further distribute an index’s write load across available data nodes, and then the Autoscaler will not see a problem because the cluster-level resources are not fully consumed. Conversely, indices cannot by default have too many shards, since that could degrade search performance for small indices and potentially strain cluster metadata operations if the total number of shards in a cluster grew too large.

Production Example: 708 TB Data Set, 37 Index Tier Nodes (not counting Search Tier), 4,100 Indices, 30,000 Shards

The following graphs cover a period when the Index Tier, in an Elasticsearch Serverless project, scales up from 10 to 37 indexing nodes and then back down to 10 after a write load spike dissipated.

Graph of the Ingest Load Per Index Node

This graph shows fairly even distribution of load, though a little less even temporarily during scale-up. There are nearly 250 fully saturated write threads at peak load. 

Graph of CPU Saturation Per Index Node

CPU usage remains within safe bounds. Usage is mostly below 60%, except for momentary outliers that reach into the 90% range as write load rises before nodes are added to the cluster.

Graph of WRITE Threadpool Queue Latency Per Node 

When a node’s WRITE threadpool is fully saturated, tasks are placed in the threadpool’s queue. Queuing can happen with few tasks, if active write tasks are long-running, or there may simply be a lot of tasks.

This graph’s time window is zoomed in further than the others. One node reaches 75 seconds of queue latency during the scale-up spike. There are 29 nodes when the queue latency spike occurs at 19h25m, before autoscaling calls for 37 nodes at 19h28m.

Graphs of Total Cluster Ingest per Second, in Documents and MBs

Ingest rate peaks at 190,000 documents / second and 54.40MB / second. The document ingest rate is respectable at 4000-5000 docs/sec per node. The MBs ingest rate, however, is very low in this case: this can happen when indexing operations involve heavy computation. Document ingestion rate can also vary depending on the size of the documents.

What’s Next for Elasticsearch Serverless Balancing

The team is currently working on shard balancing improvements for the Serverless Search Tier, focusing on creating metrics and AllocationDecider implementations for search performance. The team is excited to share these improvements soon!