Why is my AWS DMS replication DB instance in the storage-full status?

4 minute read
0

I want to know why my AWS Database Migration Service (AWS DMS) replication DB instance is in the storage-full status.

Short description

For AWS DMS replication DB instances, you can specify the allocated storage when you create a replication DB instance. By default, the volume sizes of dms.t2.* and dms.r4.* replication DB instance types are 50 GiB. The volumes sizes of dms.c4.* replication DB instance types are 100 GiB. To monitor the free storage space of the replication DB instance, review the FreeStorageSpace Amazon CloudWatch metric. Replication DB instance storage is used mostly for task logs and cached changes that are collected for ongoing replication. The reason why your AWS DMS replication DB instance is in the storage-full status might depend on the migration type.

Migrations with a full load

Task logs can cause storage-full issues if there are many task logs. For example, if you turn on detailed debug logging for a task, then your DB instance has more task logs.

Migrations with ongoing replication

Cached events can cause storage issues during a full load phase, especially if you configure your migration to use the StopTaskCachedChangesNotApplied task setting. If the storage size is too small to store the cached changes, then your replication DB instance has a storage-full status. Swapped events during the change data capture (CDC) phase can cause storage issues when the changes from the source can't be applied to the target. AWS DMS first stores the cached events in memory. When the values defined by the MemoryKeepTime or MemoryLimitTotal parameters are exceeded, AWS DMS swaps the cached events to storage. If there isn't enough storage for the swapped data, then the replication DB instance has a storage-full status. For more information, see Change processing tuning settings.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Delete the task logs

If the replication DB instance is in an available status, then delete the task logs:

  1. Open the AWS DMS console.
  2. From the navigation pane, choose Replication instances.
  3. Choose the name of the replication DB instance. From the Log management section, you see all the tasks and the Log size of each task.
  4. Select the tasks that you want to delete logs for, and then choose Delete.

For more information, see Viewing and managing AWS DMS task logs.

Modify the task setting

To use the AWS CLI to delete the DMS task logs, modify the task setting. Run the following command to create a tasksetting.json JSON file:

{
  "Logging": {
    "DeleteTaskLogs":true
  }
}

After you have created the JSON file, run the command below to modify the task setting.

Note: To modify the task, the task must have a status of Stopped or Failed.

aws dms modify-replication-task --replication-task-arn  --replication-task-settings file://tasksetting.json

To modify the task setting, the preceding command changes the parameter "DeleteTaskLogs": true. After you modify the parameter, all of the task logs that are present are deleted from the replication DB instance. Then, the task setting removes the "DeleteTaskLogs": true parameter. Confirm that the logs are deleted from the replication DB instance and that the task setting removed the parameter "DeleteTaskLogs": true.

To delete the logs periodically, you must modify the task setting with the "DeleteTaskLogs": true parameter every time. You can use cronjob or Lambda scheduler to schedule this command to run periodically. For more information, see modify-replication-task.

Increase the storage size of the replication DB instance

If the replication DB instance is in a storage-full status, you can't delete logs. You must increase the allocated storage size, or delete the replication task. To increase the storage size of a replication DB instance:

  1. Open the AWS DMS console.
  2. From the navigation pane, choose Replication instances.
  3. Choose the name of the replication DB instance that you want to modify the volume size for.
  4. Choose Modify.
  5. For the Allocated storage (GB) field, enter a new value.
  6. Select Apply changes immediately, and choose Modify.

Important: The allocated storage size can't be decreased after it has been increased.

Related information

Best practices for AWS Database Migration Service

AWS OFFICIAL
AWS OFFICIALUpdated 5 months ago