Google BigQuery

1. Access the Google Cloud Platform in one of the following ways

• Sign in using your work email address if your organization has a Google Workspace or Cloud Identity account.

  

• Sign up for the Google Cloud Free Program, which offers a 90-day, $300 Free Trial.

  Note: A free trial allows you to evaluate Google Cloud products, such as BigQuery and Cloud Storage. However, to continue using Google Cloud, you must upgrade to a paid Cloud Billing account.

  1. Go to cloud.google.com, and click Get started for free.

  2. On the Sign in page, do one of the following:

     • Sign in using an existing Google account.

       

       Note: You should not have signed up for a free trial with this Google account earlier.

     • Click Create account, select a reason from the drop-down, and click Next. Here, we shall select For work or my business.

       

       Click Get a Gmail address and complete the required steps.

       

       You can specify an existing email address or set up a new one at this time.

       Note: You should not have signed up for a Google account with this email address.

  3. On the Account Information page, complete the form, and then click AGREE & CONTINUE.

     

  4. On the Payment Information Verification page, create your payment profile and click START FREE.

     Google verifies the payment method through a one-time transaction to it. You should create a profile with the Organization account type if you want to allow multiple users to log in.

     

2. Create a Google Cloud project

Note: To create a project, you must have the resourcemanager.projects.create permission. By default, all organization resource users and free trial users are granted this permission.

1. Sign in to the Google Cloud console. You can also use the account you created in Step 1.

2. Go to the Manage Resources page, and click on + CREATE PROJECT.

   

3. In the New Project page, specify the following, and click CREATE:

   

   • Project Name: A name for your project that helps you identify it.

   • Project ID: A string that uniquely identifies your project. You can keep the identifier assigned by Google Cloud or click EDIT to change it.

     Note: You cannot change the project ID once the project is created.

   • Organization: Select the name of the organization resource in which you want to create the project from the drop-down list.

     Note: This field is not visible if you are a free trial user.

   • Location: The parent organization or folder resource where you want to create the project. Click BROWSE to view and then select a value from the list.

You can select the created project from the drop-down while configuring BigQuery as a Source, a Destination, or an Activate Warehouse.

Note: You must enable billing on the project if you want to use this project in a Destination with streaming inserts enabled.

3. Enable billing for your Google Cloud project

Note: You can link a billing account to a Google Cloud project if you have the Organization Administrator or the Billing Account Administrator role. Skip this step if you are a free trial user as billing is enabled during the sign-up process itself.

1. Sign in to the Google Cloud console and go to the Manage Resources page.

   

2. In the Manage Resources page, click the More () icon next to the project, and select Billing.

   

3. In the Billing page, click on LINK A BILLING ACCOUNT. If there are no active billing accounts associated with your organization, you need to create one. Read Create a new Cloud Billing account for the steps.

   

4. In the displayed pop-up window, select a billing account from the drop-down, and click SET ACCOUNT.

   

You now have an active billing account associated with your GCP project.

4. Create a dataset in your GCP project (Optional)

Note: You must have the BigQuery Admin role to create a dataset. You can create the dataset now or have Hevo create it for you while configuring BigQuery as a Destination. Hevo creates a dataset in the location of your existing GCS bucket or the region of your Hevo account if a GCS bucket does not exist.

1. Sign in to the Google Cloud console and go to the BigQuery page.

2. In the Explorer pane, select the project where you want to create the dataset.

3. Click the More () icon, and click Create dataset.

   

4. On the Create dataset page, specify the following:

   

   • Project ID: This field is pre-filled with the ID of the project that you selected earlier. Click CHANGE if you want to select a different project.

   • Dataset ID: A unique name for your dataset containing letters (uppercase or lowercase), numbers, and underscores, not exceeding 1024 characters. For example, bq_destination_363 in the image above.

   • Data location: The geographical location where you want to create the dataset to store your BigQuery data. For example, asia-southeast1 (Singapore) in the image above.

     Note: You cannot change the dataset’s location once the dataset is created.

   • Default table expiration: The number of days after which BigQuery deletes any table created in the dataset if you select the Enable table expiration check box. Default value: Never.

     Note: If the project containing the dataset is not associated with a billing account, BigQuery sets this value to 60 days.

You can select the created dataset from the drop-down while configuring BigQuery as a Source, a Destination, or an Activate Warehouse.

5. Create a Google Cloud Storage bucket (Optional)

Note: You can perform the following steps only if you have the Storage Admin role and your GCP project is associated with a billing account. You can create a GCS bucket now or have Hevo create it for you while configuring BigQuery as a Destination. Hevo creates a GCS bucket in the location of your existing dataset or the region of your Hevo account if a dataset does not exist.

1. Sign in to the Google Cloud console, go to the Dashboard page, and click the Select from () icon at the top of the page.

   

2. In the window that appears, select the project for which you want to create the storage bucket.

   

3. Go to the Cloud Storage Buckets page and click on + CREATE.

   

4. On the Create a bucket page, specify the details in each section and click CONTINUE to move to the next step:

   

   • Name your bucket: A globally unique name for your bucket, which follows Google’s Bucket naming guidelines.

   • Choose where to store your data: The location type and the geographical location in which the bucket’s data is stored.

     • Multi-region: A geographical area containing more than two regions, such as the United States or Asia. You should select this location type when you need higher availability of data, lower downtime because of geo-redundancy, or when your customers are spread over a large geographical area.

     • Dual-region: A geographical area containing a specific pair of regions, such as Singapore and Taiwan in Asia. You should select this location type for higher availability of data than a regional location. The charges incurred to store data are the highest for this location type.

     • Region: A specific geographical location, such as Mumbai. You should select this location type for storing data that needs to be available faster in your BigQuery Destination.

     Select the geographical location from the drop-down list displayed for your location type. This location must be the same as the data location of your dataset. Read Location considerations for further details on the requirements.

     Note: You cannot change the bucket’s location once the bucket is created.

   • Choose a default storage class for your data: A parameter that affects the bucket’s data availability. The storage class is assigned by default to all objects uploaded to the bucket and also affects the bucket’s monthly costs for data storage and retrieval. Read Data storage pricing for a comparison of the storage costs.

     • Standard: This storage class is most suitable for storing data that must be highly available and/or is stored for only short periods of time. For example, data accessed around the world, streaming videos, or data used in data-intensive computations.

     • Nearline: This storage class is suitable for storing data read or modified once a month or less. For example, data accumulated over a month and accessed once to generate monthly reports.

     • Coldline: This storage class is ideal for storing data accessed at least once in 90 days. For example, a backup of an organization’s essential data taken every quarter for disaster recovery.

     • Archive: This storage class is appropriate for storing data accessed once a year or less. For example, the digitalization of data stored in physical files or data stored for legal or regulatory reasons.

   • Choose how to control access to objects: An option to restrict the visibility and accessibility of your data:

     • Prevent public access: Select the Enforce public access prevention check box if you want to restrict public access to your data via the internet.

     • Access control: Controls the access and access level to your bucket and its data:

       • Uniform: This option ensures the same level of access to all the objects in the bucket. If you want to change this setting to fine-grained, you should do it within 90 days.

       • Fine-grained: This option enables access control on individual objects (object-level) in addition to bucket-level permissions. This option is disabled for all Cloud Storage resources in the bucket if you grant Uniform access control.

   • Choose how to protect object data: An option to provide additional protection and encryption to prevent the loss of data in your bucket. You can choose a Protection tool from one of the following:

     • Soft delete policy (For data recovery): This option allows you to retain deleted objects for a default duration of 7 days, during which you can restore them. You can customize this retention period or disable soft delete by setting the duration to 0. When disabled, deleted objects are permanently removed.

       Note: Hevo uses GCS buckets only to temporarily stage data before loading it into your BigQuery Destination, and enabling or disabling soft delete does not affect this action.

     • Object versioning (For version control): This option allows you to retain multiple versions of an object, enabling recovery of previous versions if needed. However, as BigQuery does not support Cloud Storage object versioning, do not select this option while creating a bucket for Hevo.

     • Retention (For compliance): This option allows you to set a retention period on the bucket, which does not allow an object to be deleted or replaced until the time specified by the retention period elapses. If this value is set too low, the data in the bucket may get deleted before it is loaded to the BigQuery Destination.

5. Click CREATE.

You can select the created GCS bucket from the drop-down while configuring BigQuery as a Destination.