- Introduction
- Getting Started
- Creating an Account in Hevo
- Subscribing to Hevo via AWS Marketplace
-
Connection Options
- Connecting Through SSH
- Connecting Through Reverse SSH Tunnel
- Connecting Through VPN
- Connecting Through Mongo PrivateLink
- Connecting Through AWS Transit Gateway
- Connecting Through AWS VPC Endpoint
- Connecting Through AWS VPC Peering
- Using Google Account Authentication
- How Hevo Authenticates Sources and Destinations using OAuth
- Reauthorizing an OAuth Account
- Familiarizing with the UI
- Creating your First Pipeline
- Data Loss Prevention and Recovery
- Data Ingestion
- Data Loading
- Loading Data in a Database Destination
- Loading Data to a Data Warehouse
- Optimizing Data Loading for a Destination Warehouse
- Deduplicating Data in a Data Warehouse Destination
- Manually Triggering the Loading of Events
- Scheduling Data Load for a Destination
- Loading Events in Batches
- Data Loading Statuses
- Data Spike Alerts
- Name Sanitization
- Table and Column Name Compression
- Parsing Nested JSON Fields in Events
- Pipelines
- Data Flow in a Pipeline
- Familiarizing with the Pipelines UI
- Working with Pipelines
- Managing Objects in Pipelines
- Pipeline Jobs
-
Transformations
-
Python Code-Based Transformations
- Supported Python Modules and Functions
-
Transformation Methods in the Event Class
- Create an Event
- Retrieve the Event Name
- Rename an Event
- Retrieve the Properties of an Event
- Modify the Properties for an Event
- Fetch the Primary Keys of an Event
- Modify the Primary Keys of an Event
- Fetch the Data Type of a Field
- Check if the Field is a String
- Check if the Field is a Number
- Check if the Field is Boolean
- Check if the Field is a Date
- Check if the Field is a Time Value
- Check if the Field is a Timestamp
-
TimeUtils
- Convert Date String to Required Format
- Convert Date to Required Format
- Convert Datetime String to Required Format
- Convert Epoch Time to a Date
- Convert Epoch Time to a Datetime
- Convert Epoch to Required Format
- Convert Epoch to a Time
- Get Time Difference
- Parse Date String to Date
- Parse Date String to Datetime Format
- Parse Date String to Time
- Utils
- Examples of Python Code-based Transformations
-
Drag and Drop Transformations
- Special Keywords
-
Transformation Blocks and Properties
- Add a Field
- Change Datetime Field Values
- Change Field Values
- Drop Events
- Drop Fields
- Find & Replace
- Flatten JSON
- Format Date to String
- Format Number to String
- Hash Fields
- If-Else
- Mask Fields
- Modify Text Casing
- Parse Date from String
- Parse JSON from String
- Parse Number from String
- Rename Events
- Rename Fields
- Round-off Decimal Fields
- Split Fields
- Examples of Drag and Drop Transformations
- Effect of Transformations on the Destination Table Structure
- Transformation Reference
- Transformation FAQs
-
Python Code-Based Transformations
-
Schema Mapper
- Using Schema Mapper
- Mapping Statuses
- Auto Mapping Event Types
- Manually Mapping Event Types
- Modifying Schema Mapping for Event Types
- Schema Mapper Actions
- Fixing Unmapped Fields
- Resolving Incompatible Schema Mappings
- Resizing String Columns in the Destination
- Schema Mapper Compatibility Table
- Limits on the Number of Destination Columns
- File Log
- Troubleshooting Failed Events in a Pipeline
- Mismatch in Events Count in Source and Destination
- Audit Tables
- Activity Log
-
Pipeline FAQs
- Can multiple Sources connect to one Destination?
- What happens if I re-create a deleted Pipeline?
- Why is there a delay in my Pipeline?
- Can I change the Destination post-Pipeline creation?
- Why is my billable Events high with Delta Timestamp mode?
- Can I drop multiple Destination tables in a Pipeline at once?
- How does Run Now affect scheduled ingestion frequency?
- Will pausing some objects increase the ingestion speed?
- Can I see the historical load progress?
- Why is my Historical Load Progress still at 0%?
- Why is historical data not getting ingested?
- How do I set a field as a primary key?
- How do I ensure that records are loaded only once?
- Events Usage
- Sources
- Free Sources
-
Databases and File Systems
- Data Warehouses
-
Databases
- Connecting to a Local Database
- Amazon DocumentDB
- Amazon DynamoDB
- Elasticsearch
-
MongoDB
- Generic MongoDB
- MongoDB Atlas
- Support for Multiple Data Types for the _id Field
- Example - Merge Collections Feature
-
Troubleshooting MongoDB
-
Errors During Pipeline Creation
- Error 1001 - Incorrect credentials
- Error 1005 - Connection timeout
- Error 1006 - Invalid database hostname
- Error 1007 - SSH connection failed
- Error 1008 - Database unreachable
- Error 1011 - Insufficient access
- Error 1028 - Primary/Master host needed for OpLog
- Error 1029 - Version not supported for Change Streams
- SSL 1009 - SSL Connection Failure
- Troubleshooting MongoDB Change Streams Connection
- Troubleshooting MongoDB OpLog Connection
-
Errors During Pipeline Creation
- SQL Server
-
MySQL
- Amazon Aurora MySQL
- Amazon RDS MySQL
- Azure MySQL
- Generic MySQL
- Google Cloud MySQL
- MariaDB MySQL
-
Troubleshooting MySQL
-
Errors During Pipeline Creation
- Error 1003 - Connection to host failed
- Error 1006 - Connection to host failed
- Error 1007 - SSH connection failed
- Error 1011 - Access denied
- Error 1012 - Replication access denied
- Error 1017 - Connection to host failed
- Error 1026 - Failed to connect to database
- Error 1027 - Unsupported BinLog format
- Failed to determine binlog filename/position
- Schema 'xyz' is not tracked via bin logs
- Errors Post-Pipeline Creation
-
Errors During Pipeline Creation
- MySQL FAQs
- Oracle
-
PostgreSQL
- Amazon Aurora PostgreSQL
- Amazon RDS PostgreSQL
- Azure PostgreSQL
- Generic PostgreSQL
- Google Cloud PostgreSQL
- Heroku PostgreSQL
-
Troubleshooting PostgreSQL
-
Errors during Pipeline creation
- Error 1003 - Authentication failure
- Error 1006 - Connection settings errors
- Error 1011 - Access role issue for logical replication
- Error 1012 - Access role issue for logical replication
- Error 1014 - Database does not exist
- Error 1017 - Connection settings errors
- Error 1023 - No pg_hba.conf entry
- Error 1024 - Number of requested standby connections
- Errors Post-Pipeline Creation
-
Errors during Pipeline creation
- PostgreSQL FAQs
- Troubleshooting Database Sources
- File Storage
- Engineering Analytics
- Finance & Accounting Analytics
-
Marketing Analytics
- ActiveCampaign
- AdRoll
- Amazon Ads
- Apple Search Ads
- AppsFlyer
- CleverTap
- Criteo
- Drip
- Facebook Ads
- Facebook Page Insights
- Firebase Analytics
- Freshsales
- Google Ads
- Google Analytics
- Google Analytics 4
- Google Analytics 360
- Google Play Console
- Google Search Console
- HubSpot
- Instagram Business
- Klaviyo v2
- Lemlist
- LinkedIn Ads
- Mailchimp
- Mailshake
- Marketo
- Microsoft Ads
- Onfleet
- Outbrain
- Pardot
- Pinterest Ads
- Pipedrive
- Recharge
- Segment
- SendGrid Webhook
- SendGrid
- Salesforce Marketing Cloud
- Snapchat Ads
- SurveyMonkey
- Taboola
- TikTok Ads
- Twitter Ads
- Typeform
- YouTube Analytics
- Product Analytics
- Sales & Support Analytics
- Source FAQs
- Destinations
- Familiarizing with the Destinations UI
- Cloud Storage-Based
- Databases
-
Data Warehouses
- Amazon Redshift
- Amazon Redshift Serverless
- Azure Synapse Analytics
- Databricks
- Firebolt
- Google BigQuery
- Hevo Managed Google BigQuery
- Snowflake
-
Destination FAQs
- Can I change the primary key in my Destination table?
- How do I change the data type of table columns?
- Can I change the Destination table name after creating the Pipeline?
- How can I change or delete the Destination table prefix?
- Why does my Destination have deleted Source records?
- How do I filter deleted Events from the Destination?
- Does a data load regenerate deleted Hevo metadata columns?
- How do I filter out specific fields before loading data?
- Transform
- Alerts
- Account Management
- Activate
- Glossary
Releases- Release 2.30 (Oct 21-Nov 18, 2024)
- Release 2.29 (Sep 30-Oct 22, 2024)
-
2024 Releases
- Release 2.28 (Sep 02-30, 2024)
- Release 2.27 (Aug 05-Sep 02, 2024)
- Release 2.26 (Jul 08-Aug 05, 2024)
- Release 2.25 (Jun 10-Jul 08, 2024)
- Release 2.24 (May 06-Jun 10, 2024)
- Release 2.23 (Apr 08-May 06, 2024)
- Release 2.22 (Mar 11-Apr 08, 2024)
- Release 2.21 (Feb 12-Mar 11, 2024)
- Release 2.20 (Jan 15-Feb 12, 2024)
-
2023 Releases
- Release 2.19 (Dec 04, 2023-Jan 15, 2024)
- Release Version 2.18
- Release Version 2.17
- Release Version 2.16 (with breaking changes)
- Release Version 2.15 (with breaking changes)
- Release Version 2.14
- Release Version 2.13
- Release Version 2.12
- Release Version 2.11
- Release Version 2.10
- Release Version 2.09
- Release Version 2.08
- Release Version 2.07
- Release Version 2.06
-
2022 Releases
- Release Version 2.05
- Release Version 2.04
- Release Version 2.03
- Release Version 2.02
- Release Version 2.01
- Release Version 2.00
- Release Version 1.99
- Release Version 1.98
- Release Version 1.97
- Release Version 1.96
- Release Version 1.95
- Release Version 1.93 & 1.94
- Release Version 1.92
- Release Version 1.91
- Release Version 1.90
- Release Version 1.89
- Release Version 1.88
- Release Version 1.87
- Release Version 1.86
- Release Version 1.84 & 1.85
- Release Version 1.83
- Release Version 1.82
- Release Version 1.81
- Release Version 1.80 (Jan-24-2022)
- Release Version 1.79 (Jan-03-2022)
-
2021 Releases
- Release Version 1.78 (Dec-20-2021)
- Release Version 1.77 (Dec-06-2021)
- Release Version 1.76 (Nov-22-2021)
- Release Version 1.75 (Nov-09-2021)
- Release Version 1.74 (Oct-25-2021)
- Release Version 1.73 (Oct-04-2021)
- Release Version 1.72 (Sep-20-2021)
- Release Version 1.71 (Sep-09-2021)
- Release Version 1.70 (Aug-23-2021)
- Release Version 1.69 (Aug-09-2021)
- Release Version 1.68 (Jul-26-2021)
- Release Version 1.67 (Jul-12-2021)
- Release Version 1.66 (Jun-28-2021)
- Release Version 1.65 (Jun-14-2021)
- Release Version 1.64 (Jun-01-2021)
- Release Version 1.63 (May-19-2021)
- Release Version 1.62 (May-05-2021)
- Release Version 1.61 (Apr-20-2021)
- Release Version 1.60 (Apr-06-2021)
- Release Version 1.59 (Mar-23-2021)
- Release Version 1.58 (Mar-09-2021)
- Release Version 1.57 (Feb-22-2021)
- Release Version 1.56 (Feb-09-2021)
- Release Version 1.55 (Jan-25-2021)
- Release Version 1.54 (Jan-12-2021)
-
2020 Releases
- Release Version 1.53 (Dec-22-2020)
- Release Version 1.52 (Dec-03-2020)
- Release Version 1.51 (Nov-10-2020)
- Release Version 1.50 (Oct-19-2020)
- Release Version 1.49 (Sep-28-2020)
- Release Version 1.48 (Sep-01-2020)
- Release Version 1.47 (Aug-06-2020)
- Release Version 1.46 (Jul-21-2020)
- Release Version 1.45 (Jul-02-2020)
- Release Version 1.44 (Jun-11-2020)
- Release Version 1.43 (May-15-2020)
- Release Version 1.42 (Apr-30-2020)
- Release Version 1.41 (Apr-2020)
- Release Version 1.40 (Mar-2020)
- Release Version 1.39 (Feb-2020)
- Release Version 1.38 (Jan-2020)
- Early Access New
- Upcoming Features
SQL Server
On This Page
Hevo can load data from any of your Pipelines into a SQL Server database.
Prerequisites
-
The SQL Server is running.
-
The SQL Server database host’s IP or DNS and port are available. If you do not have these details, you can obtain them from your SQL Server administrator.
-
The necessary privileges on the database are granted to your database user.
-
You are assigned the Team Collaborator, or any administrator role except the Billing Administrator role in Hevo to create the Destination.
Set up an SQL Server Instance (Optional)
This section explains the steps to install the SQL Server 2019 Evaluation Edition on a Microsoft Windows Server 2019. To continue using SQL Server post-trial expiry, you must upgrade to the SQL Server Enterprise or Standard Edition. Read SQL Server 2019: Hardware and software requirements for the hardware and software you need to install and set up SQL Server 2019.
Note: You must have administrative rights on the Windows server to set up the SQL Server instance.
1. Download SQL Server on Windows
-
Go to Microsoft Evaluation Center.
-
In the navigation bar, click SQL Server, and select SQL Server 2019 from the drop-down.
-
In the SQL Server 2019 page, click the Get Started for free tab, and then click Download the EXE.
-
In the Evaluate SQL Server 2019 page, specify the following and click Download now:
-
Provide your details such as First name, Last name, and Job role.
-
Provide your company details such as the name, its size, and country/region.
-
Provide your email address that you want to register for the free trial.
-
-
In the Please select your SQL Server 2019 download page, under EXE download, select 64-bit edition and save the server setup file.
-
Proceed to install the SQL Server.
2. Install the SQL Server on Windows
-
Go to the location where you saved the SQL Server setup file. For example, C:\MSSQL.
-
Launch the SQL Server setup file, SQL2019-SSEI-Eval.exe, and in the Select an installation type window, select Basic.
The Basic installation installs the SQL Server with the default configuration and starts the instance. Select the Custom installation type if you want to manually pick the components to install. You can select Download Media to download the installation file, to install the SQL Server later. Read the SQL Server installation guide for further guidance.
-
In the Microsoft SQL Server License Terms window, review and accept the license terms.
-
In the Specify SQL Server install location window, specify the INSTALL LOCATION, and click Install to start the installation process.
-
Click Connect Now if you want to connect to the installed SQL Server. This opens a command line window for testing the connection.
-
Click Install SSMS to install the SQL Server Management Studio (SSMS). You can use SSMS to connect to and manage your SQL Server instances and perform operations in it, such as changing the authentication method and creating login users and databases.
You have successfully installed the SQL Server and started the instance.
Allow Access to the SQL Server Instance (Optional)
To set up access and allow connections to your SQL Server instance from external machines such as Hevo, you must:
1. Enable the TCP/IP protocol for the SQL Server
-
Open the SQL Server Configuration Manager.
-
In the left navigation pane, click SQL Server Network Configuration, and select Protocols for <SQL Server Instance Name> from the drop-down. Default name: MSSQLSERVER.
-
In the right pane, if the TCP/IP Protocol is Disabled, then perform this step. Else, skip to Step 4.
-
Right-click on TCP/IP, and select Enable.
-
Click OK to acknowledge the warning.
-
-
Verify the TCP/IP Port:
-
In the right pane, double-click the TCP/IP protocol name.
-
In the TCP/IP Properties dialog, click the IP Addresses tab.
-
Scroll to the IPAII section, and ensure the TCP Port value is 1433, which is the default port for SQL Server. Read Configure a Server to Listen on a Specific TCP Port if you want to change this value.
-
Click OK, to exit the TCP/IP Properties dialog.
-
-
Restart the SQL Server instance if you enabled the TCP/IP Protocol in Step 3, else skip this step. To restart:
-
In the left navigation pane, click SQL Server Services.
-
In the right pane, right-click on SQL Server (<your_instance_name>), and select Restart.
-
You have successfully enabled TCP/IP for your SQL Server instance.
2. Create firewall rules
The Windows Firewall filters the incoming and outgoing traffic to and from your Windows server. You must create Inbound Rules for allowing Hevo to access your SQL Server database. In addition, you must open the MS SQL port if your SQL Server is installed on Windows Virtual Machine. Contact your Windows system administrator for this.
To configure the firewall, you can use the Microsoft Management Console (MMC), netsh, or PowerShell. Read Programs to configure the firewall for the steps to do this.
Here, we are using Windows PowerShell.
Run the following command in Windows PowerShell to create an inbound rule named SQLServer Default Instance:
New-NetFirewallRule -DisplayName "SQLServer Default Instance" -Direction Inbound -LocalPort 1433 -Protocol TCP -Action Allow
The output from this command is:
3. Create a login for the SQL Server
To enable non-Windows users, such as Hevo, to connect to your SQL Server, you need to:
1. Enable SQL authentication
-
Open the SQL Server Management Studio (SSMS).
-
Connect to your SQL Server using Windows Authentication. This is the default authentication method.
-
In the Object Explorer, right-click on your SQL server name, and select Properties.
-
In the Server Properties dialog, click on Security in the left pane.
-
In the Security page, under Server authentication, select SQL Server and Windows Authentication mode, and click OK.
-
In the Microsoft SQL Server Management Studio dialog, click OK to acknowledge the warning to restart the SQL Server.
-
In the Object Explorer, right-click on your SQL server name, and select Restart.
The SQL Server and the SQL Server Agent are restarted.
2. Create an SQL login with sysadmin privileges
-
In the Object Explorer, right-click on your SQL server name, and select New Query.
-
Enter the following commands in the Query Editor pane:
CREATE LOGIN [<master_user>] WITH PASSWORD=N'<strong password>', DEFAULT_DATABASE=[master], DEFAULT_LANGUAGE=[us_english], CHECK_EXPIRATION=OFF, CHECK_POLICY=ON GO ALTER LOGIN [<master_user>] ENABLE GO ALTER SERVER ROLE [sysadmin] ADD MEMBER [<master_user>] GO
Replace all placeholder values in the commands above with your own. For example, <master_user> with hevouser.
-
In the Quick Access Panel, click Execute to run the commands.
Create a Database in the SQL Server Instance (Optional)
Note: Replace the placeholder values in all the commands with your own. For example, <database_name> with MSSQLDestination.
1. Create a database in your SQL Server
-
Open the SQL Server Management Studio (SSMS).
-
Connect to your SQL Server with the SQL login created in Step 2 above.
-
In the Object Explorer, under your SQL server name, right-click on Databases, and select New Database.
-
In the New Database dialog, specify a Database name, and click OK, to create the database.
You can provide this database name while configuring your SQL Server Destination.
2. Create a schema in your database (Optional)
-
Log in to your SQL Server instance as a master user using an SQL client tool, such as sqlcmd. For example,
sqlcmd -U hevouser
. -
Enter the following commands to access your database and create a schema in it:
USE [<your_database_name>] GO CREATE SCHEMA [<schema_name>] GO
You can provide this schema name while configuring your SQL Server Destination.
Create a Database User and Grant Privileges
1. Create a database user
Perform the following steps to create a login user and map it to a database user to connect to your SQL Server database:
-
Log in to your SQL Server instance as a master user using an SQL client tool, such as sqlcmd. For example,
sqlcmd -U hevouser
. -
Enter the following command to access the database where you want to add the user:
USE <database_name> GO
-
(Optional) Enter the following command to create a login user in the selected database:
Note: Skip this step if you want to use an existing login user.
CREATE LOGIN [<login_name>] WITH PASSWORD=N'<strong_password>', DEFAULT_DATABASE=[<database_name>] GO
-
Create a database user for the login user:
Note: You can specify an existing login user or the one created in the step above.
CREATE USER [<username>] FOR LOGIN [<login_name>] GO
Note: Replace the placeholder values in all the commands above with your own. For example, <database_name> with MSSQLDestination.
2. Grant privileges to the database user
The following table lists the privileges that Hevo requires to connect to and load data into your SQL Server Destination:
Privilege Name | Privilege Description | Applies to |
---|---|---|
ALTER | Allows Hevo to create, alter, and drop objects from the schema. | SCHEMA |
INSERT | Allows Hevo to insert rows, synonyms, tables and columns, and views and columns into the schema or database. | - SCHEMA - DATABASE |
SELECT | Allows Hevo to select rows, synonyms, tables and columns, and views and columns from the schema or database. | - SCHEMA - DATABASE |
UPDATE | Allows Hevo to update rows, synonyms, tables and columns, and views and columns in the schema or database. | - SCHEMA - DATABASE |
CREATE TABLE | Allows Hevo to create tables in the database. | DATABASE |
Perform the following steps to grant the required privileges to the database user for creating the database objects needed to load and store your data:
-
Log in to your SQL Server instance as a login user using an SQL client tool, such as sqlcmd. For example,
sqlcmd -U hevouser
. -
Enter the following command to access the database in which you created the database user:
USE <database_name> GO
-
Enter the following commands to grant privileges to the database user:
GRANT ALTER, SELECT, INSERT, UPDATE ON SCHEMA::<schema_name> TO <username> GO GRANT CREATE TABLE, INSERT, SELECT, UPDATE ON DATABASE::<database_name> to <username> GO
Note: Replace the placeholder values in all the commands above with your own.
Configure your SQL Server Connection Settings
Perform the following steps to configure SQL Server as a Destination in Hevo:
-
Click DESTINATIONS in the Navigation Bar.
-
Click + CREATE DESTINATION in the Destinations List View.
-
On the Add Destination page, select SQL Server.
-
On the Configure your SQL Server Destination page, specify the following:
-
Destination Name: A unique name for your Destination, not exceeding 255 characters.
-
Database Host: The IP address or DNS of your SQL Server instance. This can be an IP address, such as 10.123.10.001, or an endpoint, such as mssql.westeros.inc.
For URL-based hostnames, exclude the http:// or https:// part. For example, if the hostname URL is https://mssql.database.windows.net, enter mssql.database.windows.net.
If you want to configure a local database as your Destination, read Connecting to a Local Database for the steps to do this.
-
Database Port: The port on which your SQL Server listens for connections. Default value: 1433
-
Database User: A login user with a non-administrative role in the SQL Server database. This can be the login user that you created in Step 4 above.
-
Database Password: The password for the login user.
-
Database Name: The name of the Destination database to which the data is loaded. This can be the database that you created in Step 3 above. The login user must have permission to access this database.
-
Schema Name (Optional): The name of the Destination database schema. This can be the database schema that you created in Step 3 above. Default value: dbo
-
Additional Settings:
-
Connect through SSH: Enable this option to connect to Hevo using an SSH tunnel. This provides an additional level of security to your database by not exposing your SQL Server setup to the public. Read Connecting Through SSH.
If this option is disabled, you must whitelist Hevo’s IP addresses.
-
Sanitize Table/Column Names: Enable this option to remove all non-alphanumeric characters and spaces in a table or column name, and replace them with an underscore (_). Read Name Sanitization.
-
-
-
Click TEST CONNECTION. This button is enabled once all the mandatory fields are specified.
-
Click SAVE & CONTINUE. This button is enabled once all the mandatory fields are specified.
Additional Information
Read the detailed Hevo documentation for the following related topics:
Destination Considerations
-
You must disable any foreign keys defined in the target tables. Foreign keys do not allow data to be loaded until the reference table has a corresponding key defined.
-
You can replicate data for only 1018 columns in a given SQL Server table. Read Limits on the Number of Columns.
Revision History
Refer to the following table for the list of key updates made to this page:
Date | Release | Description of Change |
---|---|---|
Oct-03-2023 | NA | Added a step to run the commands in the Create an SQL login with sysadmin privileges section. |
Apr-25-2023 | 2.12 | Updated section, Configure your SQL Server Connection Settings to add information that you must specify all fields to create a Pipeline. |
Mar-10-2023 | NA | Added the privileges table in the section, Grant privileges to the database user to explain the privileges required by Hevo. |
Dec-07-2022 | 2.03 | - Added sections, Set up an SQL Server Instance, Allow Access to the SQL Server Instance, and Create a Database in the SQL Server Instance to help set up a SQL Server Destination. - Modified the sections, Create a Database User and Grant Privileges and Configure your SQL Server Connection Settings to add more clarity. |
Sep-07-2022 | NA | - Updated the Prerequisites section and added the Create a Database User and Grant Privileges section. - Updated the Configure your SQL Server Connection Settings to add a link to the Connecting to a Local Database page. - Deleted the Connect to a Local Database section. |
Jul-26-2021 | 1.68 | Added section, Connect to a Local Database. |
Jul-12-2021 | NA | Updated the section, Destination Considerations. |