Near Real-time Data Loading using Streaming

Last updated on Feb 02, 2024
On This Page

Starting Release 2.20, Hevo has stopped supporting the streaming inserts feature in Google BigQuery Destinations. As a result, you cannot configure a new Google BigQuery Destination with streaming inserts enabled. Any existing Destinations created with this feature will continue to load data in near real-time. Additionally, you can create new Pipelines to load data to your existing Destination.

Hevo offers you the option to load data into your BigQuery Destination in near real-time by streaming it. Using BigQuery’s streaming APIs, Hevo writes the data to your BigQuery Destinations with the insertAll method.

Streaming offers the advantage of making data available for querying without having to wait for a task to finish loading data into BigQuery. However, you must also consider the following limitations before choosing to stream the data:

  • Streaming writes incur additional cost for you. Read Streaming pricing.

  • BigQuery does not support using streaming APIs if billing is not enabled for the GCP project. Refer to the section, Enable billing for your Google Cloud project for the steps to do this.

  • Once enabled, this option cannot be disabled for the Destination.

  • If streaming is selected, deduplication of Events cannot be enabled.

  • If the same tables are referred to in a different Destination, uniqueness cannot be guaranteed.

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Feb-05-2023 2.20 Added a warning container at the top of the page to mention about removing support for streaming inserts.
Jun-19-2023 NA Added information that billing must be enabled on the project to use streaming inserts.
Feb-22-2021 1.57 New document.
Was this page helpful?

Tell us what went wrong