On This Page
Google BigQuery is a fully-managed, server-less data warehouse that enables scalable analysis over huge sizes of data. Hevo allows users to migrate multiple datasets and tables within a BigQuery project to any other data warehouse of their choice.
Organization of Data in BigQuery
Google BigQuery uses Projects to store data. An organization can have multiple projects associated with it. However, each Pipeline can be associated with only one BigQuery project.
Within a project, the data tables are organized into units called datasets.
- Access to a BigQuery Project with one or more datasets containing at least one table.
You can select only one project per Pipeline.
Configuring Google BigQuery as a Source
Perform the following steps to configure BigQuery as the Source in your Pipeline:
Click PIPELINES in the Asset Palette.
Click + CREATE in the Pipelines List View.
In the Select Source Type page, select Google BigQuery.
In the Configure your BigQuery Account page, click + ADD BIGQUERY ACCOUNT.
Select your Google account that is linked with BigQuery, and click Allow to provide Hevo READ access to your analytics data.
In the Configure your BigQuery Source page, specify the following:
- Pipeline Name: A unique name for your Pipeline.
- Project ID: Select the project ID for which you want to create the Pipeline.
- Select Dataset ID: Select one or more datasets that contain the data tables. You can select the tables you want to replicate from these datasets in subsequent Pipeline configuration steps.
Include New Tables in the Pipeline: Applicable for all Pipeline modes except Custom SQL. If enabled, Hevo automatically ingests data from tables created after the Pipeline has been built. If disabled, the new tables are listed in the Pipeline Detailed View in Skipped state, and you can manually include the ones you want and load their historical data.
You can change this setting later.
Click TEST & CONTINUE.
Selecting Source Objects for Ingestion
By default, all datasets and the tables within these are selected to be replicated. You can change this setting by selecting specific dataset IDs while configuring your Pipeline. In the subsequent configuration pages, you can select the tables (Source objects) of these datasets that you want to ingest.
You can edit these settings at a later time through the Overview tab of the Pipeline Detailed View page as follows:
Note: Re-run the Pipeline for changes to take effect.
To include/skip a table:
Click the Kebab menu of the object
Click Include Object or Skip Object. Excluded tables have the status, SKIPPED.
To add or remove a dataset:
Click the Settings icon next to the Source name.
Click the Edit icon in the pop-up dialog and re-configure the integration.
Datasets that do not have tables are not included in the Pipeline.
Hevo automatically loads the historical data for the newly added tables. If you are creating a table in a dataset that is included in the Pipeline, Hevo automatically starts ingesting its Events in the Pipeline. If you exclude a table, its status is updated to SKIPPED in the Pipeline Overview section.
Historical Data: Hevo ingests the entire historical data for all the objects on the basis of the historical sync duration selected at the time of creating the Pipeline.
Incremental Data: Once the historical data is ingested, each subsequent run of the Pipeline fetches new data and appends any new or updated rows in the destinations.
Hevo does not track deletes or support Change Data Capture for Big Query.
Schema and Primary Keys
The Schema is derived based on the data in your BigQuery Source tables.
Updates in the BigQuery Source data are appended as new rows in the Destination. The existing rows are not modified. Therefore, both old and new entries exist in the Destination.
Deleted data is not marked or removed in the Destination.
Refer to the following table for the list of key updates made to this page:
|Date||Release||Description of Change|
|Jul-12-2021||1.67||Added the field Include New Tables in the Pipeline under Source configuration settings.|