Google Cloud Storage (GCS)

Prerequisites

• An active GCS account from which data is to be ingested exists.

• The GCS bucket name is available to allow Hevo to ingest data from it.

• You are assigned the Team Administrator, Team Collaborator, or Pipeline Administrator role in Hevo, to create the Pipeline.

Obtaining the GCS Bucket Name and Folder Path

To obtain the GCS bucket name and folder path:

1. Log in to your Google Cloud console.

2. In the top navigation bar, click the Select a project button.

   

3. In the Select a Resource window, search and select the project which contains your storage bucket. For example, Hevo Doc.

   

4. In the left navigation menu, hover over Cloud Storage and click Buckets.

   

5. On the Buckets page, copy the name of the bucket from which you want to ingest data. For example, hevo-sales-bucket as shown in the image below.

   

6. (Optional) Obtain the path to a specific folder or subfolder within a bucket if you want to ingest data only from that folder. To do so:

   1. Click the bucket which contains the folder or subfolder. For example, hevo-sales-bucket.

   2. On the Bucket details page, click the folder or subfolder from which you want to ingest data, and click the Copy icon to copy its path.

      

Use these details while creating your Hevo Pipeline.

Configuring Google Cloud Storage as a Source

To configure Google Cloud Storage as a Source in Hevo:

1. Click PIPELINES in the Navigation Bar.

2. Click + CREATE PIPELINE in the Pipelines List View.

3. On the Select Source Type page, select GCS.

4. On the Configure your GCS account page, select the authentication method for connecting to your GCS Source:

   • To connect with a User Account, do one of the following:

     • Select a previously configured account and click CONTINUE.

     • Click + ADD GCS ACCOUNT and perform the following steps to configure an account:

       

       1. Select your Google account that has access to the GCS bucket you intend to connect.

       2. Click Allow to authorize Hevo to read data from the bucket.

          

   • To connect with a Service Account, do one of the following:

     Note: The service account must be granted the Storage Object Viewer role on the bucket from which you want to ingest data.

     • Select a previously configured account and click CONTINUE.

     • Click the attach () icon to upload the Service Account Key and click CONFIGURE GCS ACCOUNT.

       

       Note: Hevo supports only JSON format for the service account key file.

5. On the Configure your GCS Source page, specify the following:

   

   • Pipeline Name: A unique name for your Pipeline, not exceeding 255 characters.

   • Bucket Name: The name of the bucket from which you want to ingest data.

   • Path Prefix: The prefix of the path for the directory that contains your data. By default, Hevo ingests data from the root of the bucket. To ingest data only from a specific folder or subfolder within the bucket, specify the folder path that you obtained from your Google Cloud console in the following format: <folder name>/<subfolder name>/.

     For example, if the path you obtained is hevo-sales-bucket/HevoSalesData/SalesReports, then specify HevoSalesData/SalesReports/.

   • File Format: The format of the data file in the Source. Hevo supports the AVRO, CSV, JSON, and XML file formats to ingest data.

     Note: You can select only one file format at a time. If your Source data is in a different format, you can export the data to either of the supported formats, and then ingest the files.

     Based on the format you select, you must specify some additional settings:

     • CSV:

       • Specify the Field Delimiter. This is the character on which fields in each line are separated. For example, `\t`, or `,`.

       • Disable the Treat First Row As Column Headers option if the Source data file does not contain column headers. Hevo, then, automatically creates these during ingestion. Default setting: Enabled. Refer to section, Example.

       • Enable the Create Event Types from folders option if the selected folder(s) have subfolders containing files in different formats. Hevo reads each subfolder as a separate Event Type and creates a separate table in the Destination for each of your selected folders.

       Note: Files lying at the prefix path (and not in a subdirectory) are ignored.

     • JSON:

       • Enable the Create Event Types from folders option if the selected folder(s) have subfolders containing files in different formats. Hevo reads each subfolder as a separate Event Type and creates a separate table in the Destination for each of your selected folders.

         Note: Files lying at the prefix path (and not in a subdirectory) are ignored.

     • XML:

       • Enable the Create Event Types from folders option if the selected folder(s) have subfolders containing files in different formats. Hevo reads each subfolder as a separate Event Type and creates a separate table in the Destination for each of your selected folders.

         Note: Files lying at the prefix path (and not in a subdirectory) are ignored.

   • Include compressed files: Enable this option if you want to ingest compressed files of the selected file format from the folders. Hevo currently supports the tar.gz and zip compression types only. Hevo automatically unzips these compressed files on ingestion. Further, Hevo re-ingests these files if they are updated in your GCS account, as it is not possible to identify individual changes.

   • Convert date/time format fields to timestamp: Enable this option if you want to convert the date/time format within the files of selected folders to timestamp. For example, the date/time format 07/11/2022, 12:39:23 is converted to timestamp, 1667804963.

6. Click TEST & CONTINUE.

7. Proceed to configuring the data ingestion and setting up the Destination.