Replicate Data within and between AWS Regions Using Amazon S3 Replication

1.1 — Sign in to the Amazon S3 console 

• If you have not already done so, create an AWS account. 
• Log in to the AWS Management Console using your account information.
• From the AWS console services search bar, enter S3. Under the services search results section, select S3.
  

1.2 — Create an S3 bucket 

• Choose Buckets from the Amazon S3 menu in the left navigation pane and then choose the Create bucket button.
  

1.3

• Enter a descriptive, globally unique name for your bucket. Select which AWS Region you would like your bucket created in. For this example, the EU (Frankfurt) eu-central-1 Region is selected. S3 Replication requires Bucket Versioning to be enabled for both source and destination S3 buckets. For more information about versioning, see Using versioning in S3 buckets.
• You can leave the remaining options as defaults. Navigate to the bottom of the page and choose Create bucket.

1.4

• Repeat the above steps to create another S3 bucket to serve as the destination bucket for replicating objects. Make sure to enable Bucket Versioning for the destination S3 bucket as well. 

2.1 – Select source S3 bucket

• From your list of S3 buckets, choose the S3 bucket that you want to configure as your source for replication.
  

2.2 

Once you select the source S3 bucket, the console takes you to the S3 bucket landing page, as shown in the following screenshot. Here, you can review the Objects, Properties, Permissions, Metrics, Management, and Access Points for the selected S3 bucket.

2.3 – Create an S3 Replication rule for the selected S3 bucket

• Choose the Management tab of the replication source bucket. Under Management, you will see Replication rules. Select Create replication rule.
  

3.1 – Enable replication rule

• Provide a Replication rule name and enable the replication rule by selecting Enabled under the Status section. If the replication rule is disabled, it will not run.
  

Priority indicates which rule has precedence whenever two or more replication rules conflict. You will have the option to edit the priority of each replication rule on the replication configuration page. Amazon S3 attempts to replicate objects according to all replication rules. However, if there are two or more rules with the same destination bucket, then objects are replicated according to the rule with the highest priority. A rule with priority 1 is executed before a rule with priority 2. The lower the number, the higher the priority. For example, say you have a replication rule to replicate all objects with tag foo1 and another replication rule to replicate all objects with tag foo2. If you have one object tagged with foo1 and foo2, it will only get replicated with the replication rule with the higher priority. When you have only one replication rule going to one destination bucket, priority is not considered. 

3.2 – Choose what to replicate

• Narrow the scope of replication by defining a Filter type (Prefix or Tags), or choose to replicate the entire bucket. For example, if you want to only replicate objects that are in the Prefix “Finance”, specify that scope. For more information on filtering objects for replication, visit the documentation on specifying a filter in the S3 User Guide.

3.3 – Choose destination S3 bucket

• Choose the destination bucket by selecting the Browse S3 button. You can replicate to a destination bucket in the same or different AWS Region, and in the same or different AWS account. Note that you will need two different S3 buckets to configure replication, and both buckets (source and destination) must have S3 Versioning enabled. The S3 console does not provide you with a way to create a new S3 bucket in the replication setup process. In this example, we chose the destination bucket to be “aws-s3-replication-tutorial-destination-bucket.”

3.4 - IAM considerations

• When creating new replication rules from the same source bucket, make sure that the IAM role associated with this configuration has sufficient permissions to write new objects in the new destination bucket. You can choose to create a new IAM role or select an existing IAM role with the right set of permissions. For more information, see the documentation on setting up permissions for S3 Replication.

3.5 - Encryption options (Skip this step if your objects are not encrypted)

• If your objects are encrypted with Amazon S3-managed encryption keys (SSE-S3) or AWS Key Management Service (AWS KMS), you will need to specify the encryption options while setting up replication. S3 Replication supports SSE-S3 (default encryption) and AWS KMS server-side encryption. If you choose AWS KMS encryption, you will need to provide the AWS KMS keys to decrypt in source and re-encrypt in destination. To save on AWS KMS costs, you also have the option to enable Amazon S3 Bucket Keys. 

3.6 - Select the S3 storage class for your destination

• Next, you will have the option to choose a different S3 storage class for your replicated objects at the destination bucket. Consider choosing lower cost storage classes as appropriate for your workloads. For example, you can choose the S3 Glacier Instant Retrieval storage class if your replicated objects will be infrequently accessed but need to be retrieved in milliseconds, S3 Glacier Deep Archive to archive data that rarely needs to be accessed, and S3 Intelligent-Tiering to optimize storage costs for data with unpredictable or changing access patterns. For more information, see the documentation on using Amazon S3 storage classes.

3.7 – Additional replication options

• Choose Additional replication options, such as enabling S3 Replication Time Control (S3 RTC), Replication metrics and notifications, Delete marker replication, and Replica modifications sync. S3 RTC helps you meet compliance and business requirements as it provides an SLA of 15 minutes to replicate 99.99% of your objects. RTC can be enabled along with S3 Cross-Region Replication (S3 CRR) and S3 Same-Region Replication (S3 SRR) and has replication metrics and notifications enabled, by default. For non-RTC rules, you have the option to select Replication metrics and notifications, which provide detailed metrics to track minute-by-minute progress of bytes pending, operations pending, and replication latency for the replication rule. Selecting Delete marker replication means deletes on the source bucket will be replicated to the destination bucket. This should be enabled if you want to keep the source and destination buckets in sync, but not if the goal is to protect against accidental or malicious deletes. To establish two-way replication between two S3 buckets, create bidirectional replication rules (A to B, and B to A) and enable Replica modification sync for both of the replication rules in the source and destination S3 buckets. This will help you to keep object metadata such as tags, ACLs, and Object Lock settings in sync between replicas and source objects.
• Review the replication configuration, and choose Save.

• Repeat the previous steps to create another S3 Replication rule from the same source S3 bucket to another destination S3 bucket. Provide a Replication rule name and enable the replication rule by selecting Enabled under the Status section. Choose what to replicate by choosing the scope of the replication rule.
  

• Choose the destination bucket by selecting the Browse S3 button. In this example, we chose the destination bucket to be “ack-test-bucket-us-east-1”.
  

• Select the Destination S3 storage class. In this example, we chose to replicate to the S3 Standard storage class in the destination bucket.
  

• Choose Additional replication options for the replication rule. In this example, we chose to enable S3 Replication Time Control (RTC).
  

Once you save the replication rule, you are back on the S3 Replication landing page, as shown in the following screenshot. Here, you can review the replication configuration with all the different replication rules, and the rule priorities and the additional options, such as encryption and RTC. 

Next, upload a new object to the replication source bucket to test the newly added replication configuration. Confirm that you see that object replicated to the new destination bucket. Replication metrics can take a few minutes to show up in the S3 console.

6.1 

Now that you've configured replication for this bucket, you can track per-destination metrics and notifications. Open the Metrics tab for the source bucket.

6.2 

Navigate down to view Replication metrics and select one or more Replication rules to monitor. Choose Display charts to see Operations pending replication, Bytes pending replication, and Replication latency for all replication rules.

Additionally, you can use the View in CloudWatch link to view the Replication metrics on Amazon CloudWatch. Here you can get a comprehensive view of the replication metrics for each replication rule, source bucket, and destination bucket in one place. Additionally, you can gather actionable insights and set alarms to monitor the metrics. For more information, see Using Amazon CloudWatch alarms.

You can use Amazon S3 Inventory to audit and report on the replication status of your objects for business, compliance, and regulatory needs. Replication status can help you determine the current state of an object being replicated. The replication status of a source object will return either Pending, Completed, or Failed. The replication status of a replica will return Replica. For more details on replication status, see Getting replication status information. For more information on configuring the Amazon S3 Inventory report, see the documentation on managing and analyzing your data at scale using Amazon S3 Inventory and Amazon Athena.

8.1 — Delete test object

• If you have logged out of your AWS Management Console session, log back in. Navigate to the S3 console and select the Buckets menu option. First, you will need to delete the test object from your test bucket. Select the name of the bucket you have been working with for this tutorial. 

• Put a check mark in the check box to the left of your test object name, then choose the Delete button.

• On the Delete objects page, verify that you have selected the proper object to delete and enter delete into the Permanently delete objects confirmation box. Then, choose the Delete object button to continue. 

• Next, you will be presented with a banner indicating if the deletion has been successful.
  

8.2 — Delete test bucket

• Finally, you need to delete the test buckets you have created. Return to the list of buckets in your account. Select the radio button to the left of the source bucket you created for this tutorial, and then choose the Delete button. 
  

• Review the warning message. If you desire to continue deletion of this bucket, enter the bucket name into the Delete bucket confirmation box, and choose Delete bucket.
  

• Repeat the previous steps to delete the destination bucket created as part of this tutorial as well. Return to the list of buckets in your account. Select the radio button to the left of the source bucket you created for this tutorial, and then choose the Delete button.
  

• Review the warning message. If you desire to continue deletion of this bucket, enter the bucket name into the Delete bucket confirmation box, and choose Delete bucket.