Iceberg Data Lake

Prerequisites

• You have an active AWS account and an IAM user in the account with permission to:

  • Create an IAM policy.

  • Add an IAM role to generate the IAM role-based credentials.

• An Amazon S3 bucket is available. Refer to the Create an Amazon S3 bucket section for the steps if you do not have one.

• An AWS Glue database is available. Refer to the Create a Glue Database section for the steps if you do not have one.

• The IAM role-based credentials are available to enable Hevo to connect to the S3 bucket and Glue database.

Note: Your Glue database and Amazon S3 bucket must exist in the same AWS region.

---

Create an AWS Glue Database (Optional)

---

Create an IAM Policy for the S3 bucket and Glue Database

To allow Hevo to access your S3 bucket and Glue database, you must create an IAM policy with the following permissions:

Permission Name	Allows Hevo to
s3:GetObject	Retrieve the Iceberg metadata for a table from S3
s3:PutObject	Write the Iceberg metadata and table data into S3.
glue:CreateTable	Create Iceberg tables when loading to a new table.
glue:DeleteTable	Delete Iceberg tables during truncate and load.
glue:GetTable	Append data to an existing table.
glue:GetTables	List all tables.
glue:UpdateTable	Alter a table during a schema change.
glue:GetDatabase	Retrieve the details, such as name and catalog ID, of the specified database.

Perform the following steps to create the IAM policy:

1. Log in to the AWS IAM Console.

2. In the left navigation pane, under Access management, click Policies.

   

3. On the Policies page, click Create policy.

   

4. On the Specify permissions page, click JSON.

   

5. Paste the following JSON statements in the Policy editor:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "GlueTable",
               "Effect": "Allow",
               "Action": [
                   "glue:CreateTable",
                   "glue:DeleteTable",
                   "glue:GetTable",
                   "glue:GetTables",
                   "glue:UpdateTable"
               ],
               "Resource": [
                   "arn:aws:glue:<aws_region>:<your_aws_account_id>:table/<your_glue_database>/*",
                   "arn:aws:glue:<aws_region>:<your_aws_account_id>:catalog/<your_glue_database>",
                   "arn:aws:glue:<aws_region>:<your_aws_account_id>:database/<your_glue_database>",
                   "arn:aws:glue:<aws_region>:<your_aws_account_id>:catalog"
               ]
           },
           {
               "Sid": "GlueDatabase",
               "Effect": "Allow",
               "Action": "glue:GetDatabase",
               "Resource": [
                   "arn:aws:glue:<aws_region>:<your_aws_account_id>:catalog/<your_glue_database>",
                   "arn:aws:glue:<aws_region>:<your_aws_account_id>:database/<your_glue_database>",
                   "arn:aws:glue:<aws_region>:<your_aws_account_id>:catalog"
               ]
           },
           {
               "Sid": "S3Object",
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<your_bucket_name>/*"
               ]
           }
       ]
   }
   

   Note: Replace the placeholder values in the commands above with your own. For example, <your_glue_database> with gluecatalog_for_iceberg.

6. At the bottom of the page, click Next.

   

7. On the Review and create page, specify the Policy name and then click Create policy at the bottom of the page.

   

You must assign this policy to the IAM role that you create for Hevo so that it can access the required AWS resources.

---

Obtain the External ID for Iceberg Destinations

Hevo Edge auto-assigns an external ID to the Iceberg Destination type that remains unchanged for your team. You need to obtain the value for this ID from the Iceberg Destination configuration screen.

1. Log in to your Hevo account and select DESTINATIONS in the Navigation Bar.

2. Click the Edge tab in the Destinations List View and click + CREATE EDGE DESTINATION.

3. On the Create Destination page, click Iceberg Data Lake.

4. In the Connect to your Iceberg section of the displayed screen, click the copy ( ) icon next to the value in the External ID field and save it securely.

   

You must add this external ID to the trust policy of the IAM role that you create for Hevo.

---

Obtain IAM Role-Based Credentials

Hevo connects to the S3 bucket and Glue database specified in your Iceberg Data Lake Destination using IAM role-based credentials. For this, you need to:

1. Add an IAM role and assign to it the IAM policy created in Step 2.

2. Obtain the Amazon Resource Name (ARN) of your IAM role.

1. Create an IAM role and assign the IAM policy

1. Log in to the AWS IAM Console.

2. In the left navigation pane, under Access management, click Roles.

   

3. On the Roles page, click Create role.

   

4. On the Select trusted entity page, do the following:

   1. In the Trusted entity type section, select Custom trust policy.

      

   2. In the Custom trust policy section:

      

      1. Copy the following JSON statements and paste them into the editor window:

         {
             "Version": "2012-10-17",
             "Statement": [
                 {
                     "Effect": "Allow",
                     "Principal": {
                         "AWS": "arn:aws:iam::393309748692:role/customer-aws_integration"
                     },
                     "Action": "sts:AssumeRole",
                     "Condition": {
                         "StringEquals": {
                             "sts:ExternalID": "External ID assigned by Hevo"
                         }
                     }
                 }
             ]
         }
         

      2. Replace the External ID assigned by Hevo (in line 12) with the value that you obtained in Step 3, and then click Next.

5. On the Add permissions page, in the Permissions policies section, search for and select the policy you created in Step 2, and then click Next at the bottom of the page.

   

6. On the Name, review, and create page, specify a Role name and a Description, and then click Create role at the bottom of the page.

   

Once the role is created, you are redirected to the Roles page.

2. Obtain the ARN

1. On the Roles page of your IAM console, search for and click the role that you created above.

2. On the <Role name> page, in the Summary section, click the copy icon below the ARN field and save it securely like any other password.

   

You can specify this ARN while configuring Iceberg Data Lake as a Destination in Edge.

---