How do I stream SES notifications across different Kinesis Data Streams accounts?

6 minute read
0

I want to set up Amazon Simple Email Service (Amazon SES) event notifications to send data to an Amazon Kinesis data stream in another AWS account.

Short description

To set up Amazon SES event notifications to an Amazon Kinesis data stream in another account, complete the following steps:

  1. Configure the SES source. Update your configuration settings in SES to push to an Amazon Simple Notification Service (Amazon SNS) topic. The updated setting configures AWS Lambda to subscribe to the topic as a trigger.
  2. Subscribe to SNS.
  3. Configure cross-account access.
  4. Configure Lambda to send data to Amazon Kinesis Data Streams. Lambda then uses the code to get the full event from the Amazon SNS topic. Lambda also assumes a role in another account to put the records into the data stream.
  5. Use the Amazon Kinesis Client Library (KCL) to process records in the stream.

Resolution

Configure the SES source

Create an SES environment, and add your email account to the list. Verify that the email account can send emails through the service to your own email address. Create a configuration set for the emails to track all the possible event types that can occur when you perform a test. In the configuration set, use the default tracking domain to set the destination to Simple Notification Service (SNS).

Note: Create the topic either when you create your configuration set or before you create the set. For more information about configuration sets, see Using configuration sets in Amazon SES.

Subscribe to SNS

To test the notification topic that you created, temporarily subscribe your own email address in the SNS topic. The subscription lets you verify that you can receive notifications of the messages that Amazon SES sends. When you configure Lambda, reference the subscription list.

Important: When you test, the open or click events are tracked only in emails that are in HTML format. AWS embeds an invisible GIF image that contains a tracking token. This tracking token tracks any notification emails that are opened and submits feedback for the event back to the topic.

After you integrate SES and SNS, use the Amazon SES console to verify whether the integration was successful. In the Amazon SES console, choose Email Addresses. Then, choose Send a Test Email and select raw as the email format.

Note: If you use the formatted option, then the email isn't sent in HTML. As a result, feedback for the open and click email events are also not sent.

In the console, use the following template:

Subject: Amazon SES Raw Email TestMIME-Version: 1.0
Content-Type: text/html
X-SES-CONFIGURATION-SET: SESxLMxKIN
<!DOCTYPE html>
<html>
<body>
<h1>This text should be large, because it is formatted as a header in HTML.</h1>
<p>Here is a formatted link: <a href="https://docs.aws.amazon.com/ses/latest/DeveloperGuide/Welcome.html">Amazon Simple Email Service Developer Guide</a>.</p>
</body>
</html>

In the preceding example template, the configuration set name is included in the headers to specify the configuration for the events. The configuration set sends you an email and notifications for all the actions that are taken with the email in your inbox. For more information, see Setting up Amazon SNS notifications.

Configure cross-account access

Configure the setup to send the messages to another account. The cross-account setup requires roles and policies to allow you to assume roles in the destination account.

Note: You must create the role for Lambda and the role that you're assuming. The role that you're assuming has the permissions to interact with the necessary services.

The following is an example template for the source account that you must attach to the Lambda role:

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::222222222222:role/role-on-source-account"
    }
}

The following is an example template for the destination account that you attach to the new role that Lambda assumes:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::111111111111:role/my-lambda-executionrole-source-account"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Note: The preceding template defines the trust relationship of the Lambda role. The Lambda role assumes the role that delivers data to the data stream. The assumed role must have permissions to put records into a data stream.

Configure Lambda to send data to Kinesis Data Streams

To configure the Lambda role to send data to Data Streams, use a template similar to the following one:

from __future__ import print_functionimport base64 #Kinesis only accepts base64 encoded payloads. 
import boto3  #Allow programmatic access to AWS services.
import json
print('Loading function')
def lambda_handler(event, context):
    # Setting up the assumeRole for cross-account
    sts_connection = boto3.client('sts')
    acct_b = sts_connection.assume_role(
        RoleArn="arn:aws:iam::[ADD_ACCOUNT_ID]:role/AssumeLambdaCrossAccount", #TODO: Add relevant role ARN
        RoleSessionName="cross_acct_lambda"
    )
    print(acct_b)

    ACCESS_KEY = acct_b['Credentials']['AccessKeyId']
    SECRET_KEY = acct_b['Credentials']['SecretAccessKey']
    SESSION_TOKEN = acct_b['Credentials']['SessionToken']

    #Gets the SNS message that was posted (cliks, opens, sends, deliveries)
    message = event['Records'][0]['Sns']['Message'] #Fetching the message in the JSON structure received.
    print("From SNS: " + message) #Checking the message received from SNS.
   
    #TODO:Generate a random partitionKey for kinesis.
    #If a random partitionKey is not used in a multi shard stream it will lead to performance problems
    #PartitionKey = RandomKey

    print("Add data to cross-account stream")
    kinesis_client = boto3.client('kinesis',region_name="us-east-1",aws_access_key_id=ACCESS_KEY,aws_secret_access_key=SECRET_KEY,aws_session_token=SESSION_TOKEN) #Initiates the kinesis service with cross-account credentials..
    kinesis_client.put_record(StreamName="CrossAccountTest",Data=json.dumps(message),PartitionKey="partitionKey")

In the preceding example, replace RoleArn with your stream name and AWS Region with your Region. This function takes the SNS message that SES receives and assumes the role from the destination account. The assumed role then puts messages into the destination stream.

To configure Lambda as a subscriber to the SNS topic, complete the following steps:

  1. Sign in to the Amazon SNS console.
  2. On the navigation pane, choose Topics.
  3. On the Topics page, select a topic.
  4. Under Subscriptions, choose Create subscription.
  5. On the Create subscription page, verify the Topic ARN.
  6. For protocol type, choose AWS Lambda.
  7. Enter the ARN of your function to establish the endpoint.
  8. Choose Create subscription.

For more information, see Fanout to Lambda functions.

If a message is published to an SNS topic that Lambda subscribes to, then Lambda is invoked with the payload of the published message. For more information, see Invoking AWS Lambda functions via Amazon SNS.

Note: If you don't want to set up cross-account access, then remove the Credentials sections and write the sections to a local data stream. For more information, see Kinesis on the AWS Boto3 website.

Use the KCL to process records

Use the KCL to process records in the data stream. After the data is in the data stream, build custom applications and implement Kinesis Data Analytics for analysis and visualization.

AWS OFFICIAL
AWS OFFICIALUpdated 8 days ago