Azure Blob Storage

Azure Blob Storage is a Platform-as-a-Service (PasS) storage platform that is scalable, secure, cost-effective, and allows you to build apps, create data lakes, and analyze the data in them. It provides a tiered storage to optimally store large amounts of data. Hevo supports data replication of structured and semi-structured data for the following file formats: CSV, TSV, JSON, XML, AVRO.

Hevo uses the Microsoft Azure Client Library For Blob Storage version 12.17.0 for connecting to the Source and ingesting the Binary Large Objects (BLOBs or blobs) from it.


Prerequisites


Creating a Connection String

Hevo uses a connection string for authentication on your Azure Blob Storage account. You must create this string to ingest data from your Azure Blob Storage container and load it into your desired Destination. To do so:

  1. Log in to your Azure Blob Storage account.

  2. Under the Azure services section, click All resources.

    Azure Home Page

  3. Click the storage account that contains the container that you want to use. For example, Azure Hevo Resource in the image below.

    Storage account

  4. In your respective Storage account page, left navigation bar, under Security + Networking, click Shared access signature.

    Azure Blob Container

  5. In the Shared access signature page, Hevo recommends that you do the following:

    Generate string

    1. Under the Allowed services section, ensure that at least the Blob check box is selected.

    2. Under the Allowed resource types section, select the Service, Container, and Object check boxes.

    3. Under the Allowed permissions section, ensure that at least the Read and List check boxes are selected.

    4. Under the Blob versioning permissions section, ensure that the Enables deletion of versions check box is deselected.

    5. Under the Allowed blob index permissions section, ensure that the Read/Write and Filter check boxes are deselected.

    6. Specify the Start and expiry date/time.

      Note: You must specify the expiry date/time to a later date to prevent reauthorization of your Hevo Pipeline with Azure Blob Storage as the Source.

    7. Click Generate SAS and connection string.

  6. Copy the connection string. This string is visible only once, so you must copy and store it in a secure location like any other password.

    Generated string

You have successfully created your connection string. Use this string for creating Hevo Pipelines.


Configuring Azure Blob Storage as a Source

Perform the following steps to configure Azure Blob Storage as the Source in your Pipeline:

  1. Click PIPELINES in the Asset Palette.

  2. Click + CREATE in the Pipelines List View.

  3. In the Select Source Type page, select Azure Blob Storage.

  4. In the Configure your Azure Blob Storage Source page, specify the following:

    Azure Blob Connection Settings

    • Pipeline Name: A unique name for your Pipeline, not exceeding 255 characters.

    • Source Setup: The credentials needed to allow Hevo to access your data.

      Perform the following steps to connect to your Azure Blob Storage account:

      1. Specify the following:

        • Connection String: The connection string that you created in your Azure Blob Storage account.

        • Container Name: The container from which you want to ingest files and folders.

      2. Click TEST & CONTINUE.

    • Data Root: The directory path of your Azure Blob Storage container that you specified above.

      Perform the following steps to select the folder(s) and the data format which you want to ingest using Hevo:

      Azure Blob Data Root

      1. Select the folders to be ingested: The folder(s) from which you want to ingest data. By default, the folder(s) are listed from the root of the container. You can either select the Root Folder to ingest data for all folder(s) and files in your specified container, or the specific folder(s) to ingest data from those folder(s). On selecting a folder, Hevo automatically selects and ingests data for all its subfolder(s) and files.

      2. File Format: The format of the data file in the Source. Hevo supports the AVRO, CSV, JSON, TSV, and XML file formats to ingest data.

        Note: You can select only one file format at a time. If your Source data is in a different format, you can export the data to either of the supported formats, and then ingest the files.

        Based on the format you select, you must specify some additional settings:

        • CSV:

          • Field Delimiter: The character on which fields in each line are separated. For example, \t, or ,.

          • Treat First Row As Column Headers: If enabled, Hevo identifies the first row in your CSV file and uses it as a column header rather than an Event. If disabled, Hevo automatically creates the column headers during ingestion. Default setting: Enabled. Refer to section, Example.

          • Create Event Types from folders: If enabled, Hevo ingests each subfolder as a separate Event Type. If disabled, Hevo merges subfolders into their parent folders, and ingests them as one Event Type. Default setting: Disabled.

        • JSON, TSV, XML:

          • Create Event Types from folders: If enabled, Hevo ingests each subfolder as a separate Event Type. If disabled, Hevo merges subfolders into their parent folders, and ingests them as one Event Type. Default setting: Disabled.
      3. Click CONFIGURE SOURCE.

  5. Proceed to configuring the data ingestion and setting up the Destination.


Example: Automatic Column Header Creation for CSV Tables

Consider the following data in CSV format, which has no column headers.

CLAY COUNTY,32003,11973623
CLAY COUNTY,32003,46448094
CLAY COUNTY,32003,55206893
CLAY COUNTY,32003,15333743
SUWANNEE COUNTY,32060,85751490
SUWANNEE COUNTY,32062,50972562
ST JOHNS COUNTY,846636,32033
NASSAU COUNTY,32025,88310177
NASSAU COUNTY,32041,34865452

If you disable the Treat First Row As Column Headers option, Hevo auto-generates the column headers, as shown in the schema mapper below:

Column headers generated by Hevo for CSV data

The record in the Destination appears as follows:

Destination record with auto-generated column headers


Data Replication

Default Pipeline Frequency Minimum Pipeline Frequency Maximum Pipeline Frequency Custom Frequency Range (Hrs)
1 Hr 5 Mins 24 Hrs 1-24

Note: You must set the custom frequency in hours as an integer value. For example, 1, 2, 3 but not 1.5 or 1.75.


Limitations

  • Hevo currently does not support deletes. Therefore, any data deleted in the Source may continue to exist in the Destination.

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Sep-21-2022 NA Updated a note in section, Configuring Azure Blob Storage as a Source.
Aug-24-2022 1.96 New document.
Last updated on 21 Sep 2022

Tell us what went wrong