Amazon DynamoDB

Prerequisites

• An active Amazon Web Services (AWS) account is available.

• Streams are enabled on the DynamoDB tables to be replicated and the value of the StreamViewType parameter is set to New and old images (NEW_AND_OLD_IMAGES in the CLI). Without this configuration, while the historical data is successfully ingested, the incremental data ingestion fails.

• An AWS IAM Policy is created with the required permissions for the DynamoDB user to ingest data from the DynamoDB database (if using Amazon Kinesis Data Streams).

  Note: Hevo does not modify any data in the Source tables. The permissions are used solely to store the last processed state for a table by the KCL.

• A valid Access Key and Secret Key are available in AWS.

• A valid AWS region is used to run the DynamoDB instance.

---

Perform the following steps to configure your Amazon DynamoDB Source:

Enable Streams

You need to enable Streams on all DynamoDB tables you want to sync through Hevo. To do this:

1. Sign in to the AWS Management Console and select the DynamoDB service.

2. In the left navigation bar of the DynamoDB console, under Dashboard, select Tables, and then, select the table for which you want to enable streams. For example, customer in the image below.

   

3. In the Exports and streams tab, scroll down to the DynamoDB stream details section and click Enable.

   

4. In the Enable DynamoDB stream page, select New and old images and click Enable stream.

   

5. Repeat Steps 2-4 for all the tables you want to synchronize.

---

Create an IAM Policy

Note: An IAM policy is needed for KCL (Kinesis Data Streams) only.

The policy is an object in AWS, which, when associated with an identity or resource, defines their permissions. Therefore, when Hevo makes a request to access the data in your DynamoDB account, the policy is applied to the related API. AWS evaluates the permissions in the policy to determine whether the request is allowed or denied. Most policies are stored in AWS as JSON documents.

Perform the following steps to create the IAM policy:

1. Log in to the Amazon IAM Console.

2. Click Policies in the left navigation bar.

   

3. Click Create policy in the right pane.

   

4. Click the JSON tab and paste the following policy into the editor. The JSON statements list the permissions the policy would assign to Hevo.

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "dynamodb:DescribeStream",
              "dynamodb:DescribeTable",
              "dynamodb:GetItem",
              "dynamodb:GetRecords",
              "dynamodb:GetShardIterator",
              "dynamodb:ListStreams",
              "dynamodb:ListTables",
              "dynamodb:Scan",
              "dynamodb:CreateTable",
              "dynamodb:PutItem",
              "dynamodb:GetItem",
              "dynamodb:UpdateItem",
              "dynamodb:DeleteItem"
            ],
            "Resource": [
              "*"
            ]
          }
        ]
        }
   

   Note: Hevo does not modify any data in the Source tables. The permissions are used solely to store the last processed state for a table by the KCL.

   

5. Click Review policy.

6. In the Review policy page, provide a Name for the policy. For example, Hevo-access.

7. (Optional) Provide a Description.

   

8. Click Create policy. You can view the new policy in the list.

   

---

Create the AWS Access Key and the AWS Secret Key

The AWS Access Key and the AWS Secret Key allow Hevo to establish authentication and replicate your Amazon DynamoDB data into your desired Destination system. You need to specify these while configuring Amazon DynamoDB as a Source in Hevo.

To retrieve these:

1. Log in to the Amazon IAM Console.

2. In the left navigation pane, under Groups, click Users, and then, click the User name for which you want to create an access key.

   

3. In the Summary page, click the Security credentials tab, and then, click Create access key.

   

4. In the Create access key dialog box:

   1. Locate the AWS Access Key under Access key ID.

   2. Click Show under Secret access key to view the AWS Secret Key.

      Note: Once you exit this dialog box, you cannot access the same AWS Access Key and the AWS Secret key. However, you can create a new key and secret by repeating these steps.

   3. Optionally, click Download .csv file to download and save the AWS Access Key and the AWS Secret key in your local machine.

      

---

Retrieve the AWS Region

To configure Amazon DynamoDB as a Source in Hevo you need to provide the AWS region where your DynamoDB instance is running.

To know your AWS region:

1. Log in to the Amazon DynamoDB Console.

2. On the top-right, locate the AWS region.

   

---