Amazon DynamoDB

Amazon DynamoDb is a fully managed, multi-master, a multi-region non-relational database that offers built-in in-memory caching to deliver reliable performance at any scale.

Hevo provides a way to load data from the DynamoDB tables into your data warehouse. Here we’ll discuss how to connect your DynamoDB to Hevo.

Please note, All data in DynamoDB Streams are subject to a 24 hour lifetime. The data older than 24 hours are susceptible to trimming (removal) at any moment.

If you disable a stream on a table, the data in the stream will continue to be readable for 24 hours. After this time, the data expires and the stream records are automatically deleted. There is no mechanism for manually deleting an existing stream. We suggest you keep the frequency accordingly.

Pre-configuration requirements

You’ll need to enable Streams on all DynamoDB tables you want to sync through Hevo. Follow the below steps to enable Streams for the tables.

  1. On your AWS console, select DynamoDB, select Tables on the left pane.

  2. Select a table. Stream details will open up with the current details on the table.

  3. Click on Manage Stream option.

  4. Select New and old images - both the new and the old images of the item and click Enable button.

  5. You need to do this for all the tables you want to sync.

Setup Instructions

Go to your Hevo dashboard and follow below steps to add DynamoDB as a source in a Pipeline.

  1. Select Pipelines in the left pane and click Create Pipeline to create a pipeline.

  2. Select DynamoDB as source type to continue.

  3. Configure AWS Access Key, AWS Secret Key, and AWS Region using which Hevo will try to connect to the DynamoDB. Click the Continue button to set up the job settings.
  4. You’ll get a list of the tables available to replicate. Note that Hevo only lists the tables for which DynamoDB Streams is enabled. Deselect the tables you don’t want to replicate. Click the Continue Button to configure the destination.

  5. Select the destination where you want to replicate DynamoDB tables or click on New Destination to create a new Destination. Check out How to add Destination tutorial for the detailed walkthrough on steps needed for adding new Destination.

Schema and Type mapping

Hevo will replicate the exact schema of the tables from the source DynamoDB to your data warehouse. In rare cases, we skip some columns with unsupported source data type while transforming and mapping.

The following table shows how your DynamoDB data types get transformed to a warehouse type.

DynamoDB Type Warehouse Type
String VARCHAR
Binary Bytes
Number Decimal/Long
STRINGSET JSON
NUMBERSET JSON
BINARYSET JSON
Map JSON
List JSON
Boolean Boolean
NULL -

How does it work?

Hevo uses Amazon DynamoDB operations API to fetch the items present in the DynamoDB tables.