How do I rebalance the uneven shard distribution in my Amazon OpenSearch Service cluster?

4 minute read
-1

The disk space in my Amazon OpenSearch Service domain is unevenly distributed across the nodes. As a result, the disk usage is heavily skewed.

Short description

Disk usage can be heavily skewed because of the following reasons:

  • Uneven shard sizes in a cluster. Although OpenSearch Service evenly distributes the number of shards across nodes, varying shard sizes require different amounts of disk space.
  • Available disk space on a node. For more information, see Disk-based shard allocation on the Elasticsearch website.
  • Incorrect shard allocation strategy. For more information, see Demystifying OpenSearch Service shard allocation.

To rebalance the shard allocation in your OpenSearch Service cluster, consider the following approaches:

  • Check the shard allocation, shard sizes, and index sharding strategy.
  • Be sure that shards are of equal size across the indices.
  • Keep shard sizes between 10 GB to 50 GB for better performance.
  • Add more data nodes to your OpenSearch Service cluster.
  • Update your sharding strategy.
  • Delete the old or unused indices to free up disk space.

Resolution

Check the shard allocation, shard sizes, and index sharding strategy

To check the number of shards allocated to each node and the amount of disk space used on each node, use the following API:

GET _cat/allocation?v

To check the shards allocated to each node and the size of each shard, use the following API:

GET _cat/shards?v

Note: This API shows that the size of shards can vary for different indices.

The uneven sharding strategy for indices can cause data skewness. In this case, shards of bigger indices reside on only a few nodes. To check the sharding strategy for indices. use the following API:

GET _cat/indices?v

Be sure that shards are of equal size across the indices

If the index size varies significantly, then use the rollover index API to create a new index when certain index sizes are reached. Or, you can use the Index State Management (ISM) to create a new index for OpenSearch Service versions 7.1 and later. For more information about rolling an alias using ISM, see rollover on the Open Distro website.

Keep shard sizes between 10 GB to 50 GB for better performance

If you have a large class of instances, then use the Petabyte scale for Amazon OpenSearch Service to determine shard sizes. For example, an OpenSearch Service domain with several i3.16xlarge.search instances can support shard sizes of up to 100 GB because there are more resources available. For more information about sharding strategy, see Choosing the number of shards.

Add more data nodes to your OpenSearch Service cluster

If your OpenSearch Service cluster has reached high disk usage levels, then add more data nodes to your cluster. The addition of data nodes also adds more resources to improve cluster performance.

Note: OpenSearch Service doesn't automatically rebalance the cluster when there's a lack of available storage space. As a result, if a data node runs out of unused storage space, then the cluster blocks any writes. For more information about disk space management, see How do I troubleshoot low storage space in my Amazon OpenSearch Service domain?

Update your sharding strategy

By default, Amazon OpenSearch Service has a sharding strategy of 5:1, where each index is divided into five primary shards. Within each index, each primary shard also has its own replica. OpenSearch Service automatically assigns primary shards and replica shards to separate data nodes, and makes sure that there's a backup in case of failure.

To modify OpenSearch Service default behavior, design your indices so that shards are distributed equally by size:

  • For existing indices, use the reindex API to change the number of primary shards. The _reindex API can be used to merge smaller indices into a bigger index, or it can be used to split up the bigger index. When the bigger index is split into more primary shards, the shard sizes are decreased.
  • For new indices, use the index template API to define the number of primary and replica shards.

Then, update the indices settings for your shards. For more information, see Update index settings API on the Elasticsearch website.

Delete the old or unused indices to free up disk space

OpenSearch Service or Elasticsearch version 6.8 or later support ISM. With ISM, you can define custom management policies so that old or unused indices are deleted after an established duration.


Related information

Calculating storage requirements

Get started with Amazon OpenSearch Service: How many shards do I need?

AWS OFFICIAL
AWS OFFICIALUpdated a year ago