How does AWS DMS use memory for migration?

Short description

An AWS DMS replication instance uses memory to run the replication engine. This engine is responsible for running SELECT statements on the source engine during the full load phase. Also, the replication engine reads from the source engine's transaction log during the change data capture (CDC) phase. These records are migrated to the target, and then compared against the corresponding records on the target database as part of the validation process. This is how generic migration flow works in AWS DMS.

AWS DMS also uses memory for task configuration and for the flow of data from source to target.

Resolution

Tasks with limited LOB settings

When you migrate data using an AWS DMS task with limited LOB settings, memory is allocated in advance based on the LobMaxSize for each LOB column. If you set this value too high, then your task might fail. This fail happens due to an Out of Memory (OOM) error, depending on the number of records that you are migrating and the CommitRate.

So, if you configure your task with high values, make sure that the AWS DMS instance has enough memory.

{
  "TargetMetadata": {
  "SupportLobs": true,
  "FullLobMode": false,
  "LobChunkSize": 0,
  "LimitedSizeLobMode": true,
  "LobMaxSize": 63,
  "InlineLobMaxSize": 0,
  }

For more information, see Setting LOB support for source databases in an AWS DMS task.

Tasks with ValidationEnabled

When you migrate using an AWS DMS task that has ValidationEnabled=true, you might see additional memory usage. This happens because AWS DMS retrieves ThreadCount * PartitionSize records from both the source and target databases. It then compares the corresponding data on the replication instance. So, you observe additional memory usage on the replication instance, source database, and target database during migration.

To limit the amount of memory in use, ignore LOB columns by using SkipLobColums. You can also perform validation separately from the migration task by using a separate replication instance or AWS DMS task. To do this, use the ValidationOnly setting:

"ValidationSettings": {
  "EnableValidation": true,
  "ThreadCount": 5,
  "PartitionSize": 10000,
  "ValidationOnly": false,
  "SkipLobColumns": false,
  },

For more information, see AWS DMS data validation.

Tasks with parallel threads in full load and CDC phases

When you use a non-RDBMS target, then ParallelLoadThreads * ParallelLoadBufferSize determines the number of threads and the size of data transfer to the target. Similarly, ParallelApplyThreads * ParallelApplyBufferSize determines the number of threads and the size of data transfer during the CDC phase. AWS DMS holds the data that's pulled from the source in ParallelLoadQueuesPerThread and ParallelApplyQueuesPerThread. When tuning these settings, make sure that the AWS DMS instance and target have the capacity to handle the workload.

{
  "TargetMetadata": {
    "ParallelLoadThreads": 0,
    "ParallelLoadBufferSize": 0,
    "ParallelLoadQueuesPerThread": 0,
    "ParallelApplyThreads": 0,
    "ParallelApplyBufferSize": 0,
    "ParallelApplyQueuesPerThread": 0
  },

For more information on these settings, see Target metadata task settings.

Tasks with batch apply settings

When you use an AWS DMS task with batch apply settings, use these best practices:

• The default batch configuration is always enough to handle the normal workload. In batch process, the size of the batch and frequency at which it's applied on the target are determined by the BatchApplyTimeoutMin, BatchApplyTimeoutMax, and BatchApplyMemoryLimit settings. These settings work together to apply changes in batch. If you need to tune these settings because of heavy workload on the source, be sure that the AWS DMS instance has enough memory. Otherwise, an OOM error might occur.
• Don't set BatchApplyMemoryLimit to more than the size of the replication instance memory or an OOM error might occur. Be aware of other tasks running simultaneously with the AWS DMS task that you're using for migration when you set BatchApplyMemoryLimit.
• Long-running transactions are retained in memory if BatchApplyPreserveTransaction = true across multiple batches. This can also cause OOM errors, depending on the next section's memory settings.
• Use the BatchSplitSize setting to set the number of changes to include in every batch, and to limit memory consumption:

{
  "TargetMetadata": {
    "BatchApplyEnabled": false,
  },
},
  "ChangeProcessingTuning": {
    "BatchApplyPreserveTransaction": true,
    "BatchApplyTimeoutMin": 1,
    "BatchApplyTimeoutMax": 30,
    "BatchApplyMemoryLimit": 500,
    "BatchSplitSize": 0,
  },

For more information on using batch apply mode, see Change processing tuning settings.

Other memory-related task settings

• During CDC, MinTransactionSize determines how many changes happen in each transaction. The size of transactions on the replication instance is controlled by MemorylimitTotal. Use this setting when you run multiple CDC tasks that need a lot of memory. Be sure to apportion this setting based on each task's transactional workload.
• Set MemoryKeepTime to limit the memory that is consumed by long-running transactions on the source. Or, if large batch of INSERT or UPDATE statements are running on the source, then increase this time. Increase this time to retain the changes from processing in the net changes table.
• Set StatementCacheSize to control the number of prepared statements that are stored on the replication instance.
• If your AWS DMS replication instance contains a large volume of free memory, then tune the settings in this example. This means that AWS DMS handles the workload in memory itself, rather flushing frequently to AWS DMS storage.

"ChangeProcessingTuning": {
    "MinTransactionSize": 1000,
    "CommitTimeout": 1,
    "MemoryLimitTotal": 1024,
    "MemoryKeepTime":
  60,
    "StatementCacheSize": 50
  },

For more information on these settings, see Change processing tuning settings.

Monitor the memory usage of your replication instance

There are several ways to monitor the memory usage of your replication instance. To isolate the single task that is consuming the most memory, sort your tasks by MemoryUsage. To learn why the task is holding memory, compare CDCChangesMemorySource and CDCChangesMemoryTarget, and then troubleshoot the respective endpoint.

The replication instance itself uses minimal memory to run the replication engine. To check if additional AWS DMS tasks can run on the replication instance, review the AvailableMemory metric in Amazon CloudWatch. Then, create a new task to use the amount of FreeMemory available. When you run the AWS DMS task, monitor FreeMemory and SwapUsage to see if resource contention is a concern. For more information, see Replication instance metrics.

Avoid memory issues

To gauge how much memory that your AWS DMS task is using, test an instance with the same configuration in a lower environment (dev and staging).

Also, perform a proof of concept migration before working with production data.

---

Related information

Choosing the right AWS DMS replication instance for your migration

Choosing the best size for a replication instance