Activity Logs - CloudWatch Sync

Last updated on Oct 25, 2023

Amazon CloudWatch Logs is a monitoring and management service provided by Amazon Web Services (AWS). It allows you to collect and track metrics, collect and monitor log files, and set alarms.

You can push the activity logs corresponding to actions, status updates, and failures for any Hevo assets, such as Pipelines, Models, and Workflows, to your CloudWatch Logs account.

Note: Starting Release 2.17, the Workflows feature and its activity logs are available only for existing users. Users signing up from Release 2.17 onwards cannot create Workflows. For more information, please contact Hevo Support or your account executive.

The following image illustrates the key steps that you need to complete to configure Amazon CloudWatch Logs in Hevo:

Amazon CloudWatch Logs


Prerequisites


Obtain Amazon CloudWatch Logs Credentials

You must either Create access credentials or Generate IAM role-based credentials to allow Hevo to connect to your Amazon CloudWatch account and sync logs with it.

Create access credentials

You need the access key and secret access key from your Amazon CloudWatch account to allow Hevo to access the data from it.

  1. Log in to the Amazon IAM Console.

  2. In the left navigation bar, under Access management, click Users, and then, in the right pane, click the User name for which you want to create an access key.

    click-role

  3. In the Summary page, click the Security credentials tab, and then, click Create access key.

    security-credentials

  4. In the Access key best practices & alternatives page, do the following:

    Select

    1. Select one of the recommended methods. For example, in the image above, Command Line Interface (CLI) is selected.

    2. Select the I understand the above recommendation… check box, to proceed with creating the access key.

    3. Click Next.

  5. In the Set description tag - optional page, do the following:

    description

    1. (Optional) Specify a description for the access key in the Description tag value field.

    2. Click Create access key.

  6. In the Retrieve access keys page, do the following:

    copy

    • Click the copy icon in the Access key field and save the key securely like any other password.

    • Click the copy icon in the Secret access key field and save the key securely like any other password.

      Note: Once you exit this page, you cannot access these keys again.

    • (Optional) Click Download .csv file to save the access key and the secret access key on your local machine.

Generate IAM role-based credentials

To generate your IAM role-based credentials, you need to:

  1. Create an IAM policy with the permission to allow Hevo to sync data with your Amazon CloudWatch logs.

  2. Create an IAM role for Hevo, as the Amazon Resource Name (ARN) and the external ID from this role are required to configure Amazon CloudWatch Logs in Hevo.

These steps are explained in details below:

1. Create an IAM policy

  1. Log in to the AWS IAM Console.

  2. In the left navigation bar, under Access management, click Policies.

    policies

  3. In the Policies page, click Create policy.

    create-policies

  4. In the Specify permissions page, click the JSON tab and paste the following policy in the editor. The JSON statements list the permissions the policy would assign to Hevo.

    Note: Replace the placeholder values in the commands below with your own. For example, <aws_region> with us-east-1.

     {
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Sid": "VisualEditor0",
                 "Effect": "Allow",
                 "Action": "logs:CreateLogStream",
                 "Resource": "arn:aws:logs:<aws_region>:<account-id>:log-group:<log-group-name>"
             },
             {
                 "Sid": "VisualEditor1",
                 "Effect": "Allow",
                 "Action": [
                     "logs:DeleteLogStream",
                     "logs:PutLogEvents"
                 ],
                 "Resource": "arn:aws:logs:<aws_region>:<account-id>:log-group:<log-group-name>"
             }
         ]
     }                 
    

    JSON

  5. Click Next.

  6. In the Review and policy page, specify a Policy name and Description for your policy and click Create policy.

    review-policy

You are redirected to the Policies page, where you can see the policy that you created.

2. Create an IAM role and obtain its ARN and external ID

Perform the following steps to create an IAM role:

  1. Log in to the AWS IAM Console.

  2. In the left navigation bar, under Access management, click Roles.

    roles

  3. In the Roles page, click Create role.

    create roles

  4. In the Select trusted entity page:

    • In the Trusted entity type section, select AWS account.

      aws-cli

    • In the An AWS account section, do the following:

      external-id

      1. Select the Another AWS account option, and specify Hevo’s Account ID, 393309748692.

      2. In the Options section, select the Require external ID… check box, and specify an External ID of your choice.

        Note: You must save this external ID in a secure location like any other password. This is required while setting up a Pipeline in Hevo.

  5. Click Next.

  6. In the Permissions policies section, select the policy that you created in Create an IAM policy above and click Next at the bottom of the page.

    add permission

  7. In the Name, review, and create page, specify the Role name and click Create role.

    role-name

  8. In the Roles page, select the role that you created above.

    roles

  9. In the <Role name> page, the Summary section, click the copy icon to copy the ARN and save it securely like any other password. Use this ARN while configuring Amazon CloudWatch logs in Hevo.

    copy arn


Create a Log Group for Amazon CloudWatch

  1. Log in to the AWS CloudWatch Console.

  2. In the left navigation bar, under Logs, click Log groups.

    log-groups

  3. In the Log groups page, click Create log group.

    create

  4. In the Create log group page, specify the following:

    log-groups

    • Log group name: A unique name for your log.

    • Retention setting: The duration for which your organization wants to retain information.

    • Click Create.

  5. In the Log groups page, click the log group that you created above.

    select

  6. In the Log group details section, copy the ARN and save it securely like any other password. Use this ARN while configuring your Amazon CloudWatch Logs in Hevo.

    copy-arn


Retrieve the AWS Region

To configure Amazon CloudWatch in Hevo you need to provide the AWS region where your logs are available.

To retrieve your AWS region:


Configure your Amazon CloudWatch Settings

Perform the following steps to configure Amazon CloudWatch:

  1. Log in to your Hevo account.

  2. In the User Information Panel, click the drop-down next to your username and click Team.

    select-teams

  3. In the left navigation bar, click Activity Logs and click ENABLE, to enable CloudWatch Sync for your activity logs.

    activity-logs

  4. In the CloudWatch Sync Settings window, select one of the methods to sync your activity logs to your CloudWatch Logs account:

    • Connect using IAM Role:

      IAM

      • IAM Role ARN: The Amazon Resource Name (ARN) for your Amazon CloudWatch that you retrieved above.

      • External Id: The external ID that you specified above.

      • Region: The AWS region where your CloudWatch log group is created.

      • Log Group: The log group that you created in Step 2.

    • Connect using Access Credentials:

      access-credentials

      • AWS Key: The AWS access key that you retrieved in the Create access credentials section above.

      • AWS Secret: The AWS secret access key for the access key ID that you retrieved in the Create access credentials section above.

      • Region: The AWS region where your CloudWatch log group is created.

      • Log Group: The log group that you created in Step 3.

  5. Click TEST CONNECTION to check connectivity with your Amazon CloudWatch Logs account and click ENABLE.


Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Apr-25-2023 2.12 Added sections:
- Obtain Amazon CloudWatch Logs Credentials
- Create a Log Group for Amazon CloudWatch.

Tell us what went wrong