Azure Blob Storage

Prerequisites

• An active Azure Blob Storage account.

• A storage account is created under your Azure Blob Storage account.

  Note: An Azure Blob Storage account can contain multiple storage accounts in it.

• A container is created under your storage account, with the folders and files that you want to ingest.

  Note: An storage account can contain multiple containers in it.

• A connection string is created with the required permissions.

• You are assigned the Team Administrator, Team Collaborator, or Pipeline Administrator role in Hevo, to create the Pipeline.

Creating a Connection String

Hevo uses a connection string for authentication on your Azure Blob Storage account. You must create this string to ingest data from your Azure Blob Storage container and load it into your desired Destination. To do so:

1. Log in to your Azure Blob Storage account.

2. Under the Azure services section, click All resources.

   

3. Click the storage account that contains the container that you want to use. For example, Azure Hevo Resource in the image below.

   

4. In your respective Storage account page, left navigation bar, under Security + Networking, click Shared access signature.

   

5. In the Shared access signature page, Hevo recommends that you do the following:

   

   1. Under the Allowed services section, ensure that at least the Blob check box is selected.

   2. Under the Allowed resource types section, select the Service, Container, and Object check boxes.

   3. Under the Allowed permissions section, ensure that at least the Read and List check boxes are selected.

   4. Under the Blob versioning permissions section, ensure that the Enables deletion of versions check box is deselected.

   5. Under the Allowed blob index permissions section, ensure that the Read/Write and Filter check boxes are deselected.

   6. Specify the Start and expiry date/time.

      Note: You must specify the expiry date/time to a later date to prevent reauthorization of your Hevo Pipeline with Azure Blob Storage as the Source.

   7. Click Generate SAS and connection string.

6. Copy the connection string. This string is visible only once, so you must copy and store it in a secure location like any other password.

   

You have successfully created your connection string. Use this string for creating Hevo Pipelines.

Obtain the Container Name

Perform the following steps to retrieve the container name:

1. Log in to your Azure Blob Storage account, and in the Search bar, type storage account.

2. From the search results, under Services, click Storage accounts.

   

3. In the Storage accounts page, click on your storage account name. For example, hevodocstorage1.

   

4. In the left navigation pane, under Data Storage, click Containers.

   

5. In the Containers page, from the Name column, copy the required value. For example, hevodocfs. Use this value while configuring your Hevo Pipeline.

   

Configuring Azure Blob Storage as a Source

Perform the following steps to configure Azure Blob Storage as the Source in your Pipeline:

1. Click PIPELINES in the Navigation Bar.

2. Click + CREATE PIPELINE in the Pipelines List View.

3. In the Select Source Type page, select Azure Blob Storage.

4. In the Configure your Azure Blob Storage Source page, specify the following:

   

   • Pipeline Name: A unique name for your Pipeline, not exceeding 255 characters.

   • Source Setup: The credentials needed to allow Hevo to access your data.

     Perform the following steps to connect to your Azure Blob Storage account:

     1. Specify the following:

        • Connection String: The connection string that you created in your Azure Blob Storage account.

        • Container Name: The container from which you want to ingest files and folders.

     2. Click TEST & CONTINUE.

   • Data Root: The directory path of your Azure Blob Storage container that you specified above.

     Perform the following steps to select the folder(s) and the data format which you want to ingest using Hevo:

     

     1. Select the folders to be ingested: The folder(s) from which you want to ingest data. By default, the folder(s) are listed from the root of the container. You can either select the Root Folder to ingest data for all folder(s) and files in your specified container, or the specific folder(s) to ingest data from those folder(s). On selecting a folder, Hevo automatically selects and ingests data for all its subfolder(s) and files.

     2. File Format: The format of the data file in the Source. Hevo supports the AVRO, CSV, JSON, TSV, and XML file formats to ingest data.

        Note: You can select only one file format at a time. If your Source data is in a different format, you can export the data to either of the supported formats, and then ingest the files.

        Based on the format you select, you must specify some additional settings:

        • CSV:

          • Field Delimiter: The character on which fields in each line are separated. For example, \t, or ,.

          • Treat First Row As Column Headers: If enabled, Hevo identifies the first row in your CSV file and uses it as a column header rather than an Event. If disabled, Hevo automatically creates the column headers during ingestion. Default setting: Enabled. Refer to section, Example.

          • Create Event Types from folders: If enabled, Hevo ingests each subfolder as a separate Event Type. If disabled, Hevo merges subfolders into their parent folders, and ingests them as one Event Type. Default setting: Disabled.

          • Convert date/time format fields to timestamp: Enable this option if you want to convert the date/time format within the files of selected folders to timestamp. For example, the date/time format 07/11/2022, 12:39:23 converts to timestamp 1667804963.

        • JSON, TSV, XML:

          • Create Event Types from folders: If enabled, Hevo ingests each subfolder as a separate Event Type. If disabled, Hevo merges subfolders into their parent folders, and ingests them as one Event Type. Default setting: Disabled.

          • Convert date/time format fields to timestamp: Enable this option if you want to convert the date/time format within the files of selected folders to timestamp. For example, the date/time format 07/11/2022, 12:39:23 converts to timestamp 1667804963.

     3. Click CONFIGURE SOURCE.

5. Proceed to configuring the data ingestion and setting up the Destination.