Snowflake (Edge)

Last updated on Sep 25, 2024

Edge Pipeline is currently available under Early Access. You can request access to evaluate and test its features.

Snowflake offers a cloud-based data storage and analytics service, generally termed as data warehouse-as-a-service. Companies can use it to store and analyze data using cloud-based hardware and software.

Snowflake provides you one data warehouse automatically when you create an account. Each data warehouse can further have one or more databases, although this is not mandatory.

The data from your Pipeline is staged in Hevo’s S3 bucket before being finally loaded to your Snowflake warehouse.

The Snowflake data warehouse may be hosted on any of the following cloud providers:

  • Amazon Web Services (AWS)

  • Google Cloud Platform (GCP)

  • Microsoft Azure (Azure)

To connect your Snowflake instance to Hevo, you can either use a private link which directly connects to your cloud provider through Virtual Private Cloud (VPC) or connect via a public network using your Snowflake account URL.

A private link enables communication and network traffic to remain exclusively within the cloud provider’s private network while maintaining direct and secure access across VPCs. It allows you to transfer data to Snowflake without going through the public internet or using proxies to connect Snowflake to your network. Note that even with a private link, the public endpoint is still accessible and Hevo uses that to connect to your database cluster.

Please reach out to Hevo Support to retrieve the private link for your cloud provider.

The following image illustrates the key steps that you need to complete to configure Snowflake as a Destination in Hevo:

Configure Snowflake as a Destination


Data Type Mapping

Hevo maps a Source data type internally to a unified data type, referred to as the Hevo Data Type in the table below. This data type is used to represent the Source data from all supported data types in a lossless manner. The Hevo data types are then mapped to the corresponding data types that are supported in each Destination.

Hevo Data Type Snowflake Data Type
ARRAY ARRAY
BOOLEAN BOOLEAN
BYTEARRAY BINARY
BYTE BYTEINT
DATE DATE
- DATETIME
- TIMESTAMP
TIMESTAMP_NTZ
DECIMAL NUMBER
DOUBLE DOUBLE
FLOAT FLOAT
INTEGER INTEGER
JSON VARIANT
LONG BIGINT
SHORT SMALLINT
TIME TIME
- TIMESTAMPTZ
- TIMETZ
- ZONEDDATETIME
TIMESTAMP_TZ
VARCHAR VARCHAR

Destination Considerations

  • Snowflake converts the Source table and column names to uppercase while mapping to the Destination table. For example, the Source table, Table_namE_05, is converted to TABLE_NAME_05. The same conventions apply to column names.

Limitations

  • Hevo replicates a maximum of 4096 columns to each Snowflake table, of which six are Hevo-reserved metadata columns used during data replication. Therefore, your Pipeline can replicate up to 4090 (4096-6) columns for each table.

Tell us what went wrong