Google BigQuery

Prerequisites

• Access to a BigQuery project with one or more datasets containing at least one table.

• An active billing account is linked to your GCP project, as Hevo writes incremental data to temporary tables using streaming inserts.

• You are granted the BigQuery Data Editor, Data Viewer, and Job User permissions at the dataset level.

• Access to an existing GCS bucket in the BigQuery location as your datasets. You can specify this when creating your Pipeline to enable Hevo to use the GCS Export strategy for partitioned tables.

• You are assigned the Team Administrator, Team Collaborator, or Pipeline Administrator role in Hevo to create the Pipeline.

Note: You can select only one project per Pipeline.

Configuring Google BigQuery as a Source

Perform the following steps to configure BigQuery as the Source in your Pipeline:

1. Click PIPELINES in the Navigation Bar.

2. Click + CREATE PIPELINE in the Pipelines List View.

3. In the Select Source Type page, select Google BigQuery.

4. In the Configure your BigQuery Account page, connect to your BigQuery data warehouse using one of the following ways:

   Note: You cannot switch from a service account to a user account and vice-versa once you create the Pipeline.

   • To connect with a User Account, do one of the following:

     • Select a previously configured account and click CONTINUE.

     • Click + ADD BIGQUERY ACCOUNT and perform the following steps to configure an account:

       

       1. Sign in to your account.

       2. Click Allow to authorize Hevo to access your data.

          

   • To connect with a Service Account, do one of the following:

     • Select a previously configured account and click CONTINUE.

     • Attach the Service Account Key JSON file that you created in Google Cloud Platform (GCP) and click CONFIGURE BIGQUERY ACCOUNT.

       

5. In the Configure your BigQuery Source page, specify the following:

   

   • Pipeline Name: A unique name for your Pipeline, not exceeding 255 characters.

   • Authorized User/Service Account (Non-editable): The email address that you selected when connecting to your Google BigQuery account. This value is pre-filled.

   • Project ID: The project ID for which you want to create the Pipeline. Select the required ID from the drop-down.

   • GCS Bucket (Optional): The name of an existing container in your Google Cloud Storage, into which Hevo exports the ingested data before loading it to the Destination.

   • Select Dataset ID: Select one or more datasets that contain the data tables. You can select the tables you want to replicate from these datasets in subsequent Pipeline configuration steps.

   • Advanced Settings:

     • Include New Tables in the Pipeline: If enabled, Hevo automatically ingests data from tables created after the Pipeline has been built. If disabled, the new tables are listed in the Pipeline Detailed View in Skipped state, and you can manually include the ones you want and load their historical data.

       You can change this setting later.

6. Click TEST & CONTINUE.

7. Proceed to configuring the data ingestion and setting up the Destination.