How do I troubleshoot low storage space in my OpenSearch Service domain?

6 minute read
1

My Amazon OpenSearch Service domain is out of storage space, and I received an error.

Short description

Your OpenSearch Service domain has storage space requirements for processing workloads. The ClusterBlockException error might occur because your OpenSearch Service domain ran out of storage space. For example, when you allocate storage to a cluster node, up to 20% of that space (20 GB) is reserved space. Also, operating systems such as Linux reserve 5% of the file system to support any critical processes performed by the root user. For more information about OpenSearch Service storage space requirements, see Calculating storage requirements.

To resolve low storage space issues, complete the following steps:

  • Update your cluster sharding strategy so that shards are evenly distributed across all nodes.
  • Increase the size of the domain's Amazon Elastic Block Store (Amazon EBS) volumes. You can also add more cluster nodes to the OpenSearch Service domain.
  • Reduce the amount of data that's stored in your domain. You can delete unnecessary files, optimize old indices, or reduce the domain's replica count. Be sure to take a manual snapshot first.
    Note: Reducing the domain's replica count can reduce fault tolerance. It's a best practice to configure at least one replica for each index.
  • Use Index State Management (ISM) to manage low storage space.
  • Use Amazon CloudWatch metrics to monitor the amount of available storage in your cluster.

Resolution

Updating your storage configuration settings

Important: Before you update your configuration settings, verify the configuration changes that can initiate a blue/green deployment. Also, make sure that your dedicated primary node type is the recommended node type.

Use the cat allocation (from the Elastic website) command to check how much storage space is available for each node in your cluster:

curl -XGET "es_endpoint/_cat/allocation?v"

To update your OpenSearch Service domain configuration settings, complete the following steps:

1.     Open the OpenSearch Service console.

2.     From the navigation pane, under Managed clusters, choose Domains.

3.     Choose the domain that you want to add storage to.

4.    Update your domain settings.

Note: If you're using an Amazon EBS volume for storage, then update your Storage configuration settings. Or you can update the number of data nodes.

5.    Choose Submit.

If your domain uses EBS volumes for data storage

If the domain uses EBS volumes for storage, take one of the following actions:

  • Increase the size of the EBS volumes. The maximum volume size depends on the node's instance type or Elasticsearch version type. For example, 512 GiB is the maximum volume size for Elasticsearch version 1.5.
  • If you can't increase the size of the EBS volumes, then add nodes or scale up your domain and choose a new EBS limit. For more information about EBS volumes, see EBS volume size quotas.
  • Create a backup of unwanted indices to your Amazon Simple Storage Service (Amazon S3) bucket. Then, delete the indices from your OpenSearch Service cluster to free up disk space. Or, you can use Curator to rotate data in OpenSearch Service (from the Python Package Index website). You can also use ISM to rotate indices in OpenSearch Service.
    Note: You must have a manual snapshot repository set up to create any backups.

If your domain uses Amazon Elastic Compute Cloud (Amazon EC2) I3 instances for data storage

If you're using Amazon EC2 I3 instances for data storage, then take one of the following actions:

  • Add more nodes to your OpenSearch Service cluster.
  • Scale up the instance type to obtain more storage space.
  • Delete any unwanted indices from your OpenSearch Service cluster.

Delete unused or old indices

You can check the indices creation date with the following command:

GET _cat/indices?h=h,s,i,id,p,r,dc,dd,ss,creation.date.string&s=creation.date.string:desc

To delete single indices, run the following command:

DELETE <index-name>

To delete multiple indices, run the following command:

DELETE cwl-index-*

Use ISM to manage low storage space

ISM allows you to automate routine tasks and then apply them to indices and index patterns in OpenSearch Service. With ISM, you can define custom management policies that help you maintain issues, such as low disk space. For example, you can use a rollover operation and an ISM policy to automate deletion of old indices based on conditions, such as index size. The rollover operation rolls over a target to a new index when an existing index meets the defined condition.

The following example ISM policy deletes indices after 50 minutes:

PUT _plugins/_ism/policies/delete_ism_policy
{
    "policy": {
        "policy_id": "delete_ism_policy",
        "description": "A simple default policy that deletes old unused indices“,
        "last_updated_time": 1658834661281,
        "schema_version": 13,
        "error_notification": null,
        "default_state": "example_hot_state",        #Default state
        "states": [
            {
                "name": "example_hot_state",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": “50m”   #Note that after 50 minutes index will transit from hot_state to delete state
                        }
                    }
                ]
            },
            {
                "name": "delete",                    #Indices moved to delete state
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [                  #This ISM policy attaches to all indices for the index pattern and you can mention your index pattern here
                    "sample*"
                ],
                "priority": 100,                     #Priority can be set from 0 to 100
                "last_updated_time": 1658834436349
            }
        ]
    }
}

Use the following API call to manually attach the ISM policy to the index:

POST _plugins/_ism/add/your-index-*
{
     "policy_id": "<policy_id>"
}

For more information, see How do I use ISM to manage low storage space in Amazon OpenSearch Service?

Use CloudWatch alarms to monitor storage

You can use the CloudWatch FreeStorageSpace metric to monitor the amount of available storage in your cluster. To receive notifications whenever storage space is low, configure your CloudWatch alarms.

For more information, see Monitoring OpenSearch cluster metrics with Amazon CloudWatch.

Related information

Operational best practices for Amazon OpenSearch Service

How do I make my Amazon OpenSearch Service domain more fault tolerant?

Why is my Amazon OpenSearch Service cluster in red or yellow status?

AWS OFFICIAL
AWS OFFICIALUpdated a year ago