- About Hevo
- Hevo Features
- Hevo System Architecture
- Core Concepts
- Free Trials
- Regulatory Compliance
- Hevo Support
- Getting Started
- Creating an Account in Hevo
- Connection Options
- Familiarizing with the UI
- Creating your First Pipeline
- Data Loss Prevention and Recovery
- Activity Log
- Data Ingestion
- Ingestion Modes and Query Modes
- Ingestion Modes
- Types of Data Synchronization
- Ingestion and Loading Frequency
- Ingestion Frequency and Data Synchronization
- Data Ingestion Statuses
- Deferred Data Ingestion
- Query Modes for Ingesting Data
- Handling of Primary Keys
- Handling of Updates
- Handling of Deletes
- Hevo-generated Metadata
- Data Loading
- Loading Data in a Database Destination
- Loading Data to a Data Warehouse
- Optimizing Data Loading for a Destination Warehouse
- Manually Triggering the Loading of Events
- Scheduling Data Load for a Destination
- Loading Events in Batches
- Data Loading Statuses
- Name Sanitization
- Table and Column Name Compression
- Parsing Nested JSON Fields in Events
- Data Flow in a Pipeline
- Familiarizing with the Pipelines UI
- Pipeline Objects
Working with Pipelines
- Best Practices for Creating Database Pipelines
- Creating a Pipeline
- Scheduling a Pipeline
- Modifying a Pipeline
- Prioritizing a Pipeline
- Viewing Pipeline Progress
- Troubleshooting Data Replication Errors
- Pausing and Deleting a Pipeline
- Log-based Pipelines
- Python Code-Based Transformations
- Drag and Drop Transformations
- Effect of Transformations on the Destination Table Structure
- Transformation Reference
- Using Schema Mapper
- Mapping Statuses
- Auto Mapping Event Types
- Mapping a Source Event Type with a Destination Table
- Mapping a Source Event Type Field with a Destination Table Column
- Schema Mapper Actions
- Fixing Unmapped Fields
- Resolving Incompatible Schema Mappings
- Resizing String Columns in the Destination
- Schema Mapper Compatibility Table
- Troubleshooting Failed Events in a Pipeline
- Pipeline FAQs
- Events Usage
- Free Sources
Databases and File Systems
- Data Warehouses
- Connecting to a Local Database
- Amazon DocumentDB
- Amazon DynamoDB
- Generic MongoDB
- MongoDB Atlas
- Support for Multiple Data Types for the _id Field
- Example - Merge Collections Feature
- Errors During Pipeline Creation
- Troubleshooting MongoDB Change Streams Connection
- Troubleshooting MongoDB OpLog Connection
- SQL Server
- Amazon Aurora MySQL
- Amazon RDS MySQL
- Azure MySQL
- Google Cloud MySQL
- Generic MySQL
- MariaDB MySQL
Errors During Pipeline Creation
- Error 1003 - Connection to host failed
- Error 1006 - Connection to host failed
- Error 1007 - SSH connection failed
- Error 1011 - Access denied
- Error 1012 - Replication access denied
- Error 1017 - Connection to host failed
- Error 1026 - Failed to connect to database
- Error 1027 - Unsupported BinLog format
- Failed to determine binlog filename/position
- Schema 'xyz' is not tracked via bin logs
- Errors Post-Pipeline Creation
- Errors During Pipeline Creation
- MySQL FAQs
- Amazon Aurora PostgreSQL
- Amazon RDS PostgreSQL
- Azure PostgreSQL
- Google Cloud PostgreSQL
- Heroku PostgreSQL
- Generic PostgreSQL
Errors during Pipeline creation
- Error 1003 - Authentication failure
- Error 1006 - Connection settings errors
- Error 1011 - Access role issue for logical replication
- Error 1012 - Access role issue for logical replication
- Error 1014 - Database does not exist
- Error 1017 - Connection settings errors
- Error 1023 - No pg_hba.conf entry
- Error 1024 - Number of requested standby connections
- Errors Post-Pipeline Creation
- Errors during Pipeline creation
- PostgreSQL FAQs
- File Storage
- Engineering Analytics
- Finance & Accounting Analytics
- Apple Search Ads
- Facebook Ads
- Facebook Page Insights
- Google Campaign Manager
- Google Ads
- Google Analytics
- Google Analytics 4
- Google Analytics 360
- Google Play Console
- Google Search Console
- Instagram Business
- LinkedIn Ads
- Microsoft Advertising
- Pinterest Ads
- SendGrid Webhook
- Salesforce Marketing Cloud
- Snapchat Ads
- TikTok Ads
- Twitter Ads
- YouTube Analytics
- Product Analytics
- Sales & Support Analytics
- Source FAQs
- Familiarizing with the Destinations UI
- Amazon Redshift
- Hevo Managed Google BigQuery
- Google BigQuery
- Destination FAQs
- Activate Concepts
- Familiarizing with the Activate UI
- Working with Activate
- Activate Warehouses
- Activate Targets
- Account Management
- Personal Settings
- Team Settings
- Account Suspension and Restoration
- General FAQs
- Release Notes
- Release Version 1.98
- Release Version 1.97
- Release Version 1.96
- Release Version 1.95
- Release Version 1.93 & 1.94
- Release Version 1.92
- Release Version 1.91
- Release Version 1.90
- Release Version 1.89
- Release Version 1.88
- Release Version 1.87
- Release Version 1.86
- Release Version 1.84 & 1.85
- Release Version 1.83
- Release Version 1.82
- Release Version 1.81
- Release Version 1.80 (Jan-24-2022)
- Release Version 1.79 (Jan-03-2022)
- Release Version 1.78 (Dec-20-2021)
- Release Version 1.77 (Dec-06-2021)
- Release Version 1.76 (Nov-22-2021)
- Release Version 1.75 (Nov-09-2021)
- Release Version 1.74 (Oct-25-2021)
- Release Version 1.73 (Oct-04-2021)
- Release Version 1.72 (Sep-20-2021)
- Release Version 1.71 (Sep-09-2021)
- Release Version 1.70 (Aug-23-2021)
- Release Version 1.69 (Aug-09-2021)
- Release Version 1.68 (Jul-26-2021)
- Release Version 1.67 (Jul-12-2021)
- Release Version 1.66 (Jun-28-2021)
- Release Version 1.65 (Jun-14-2021)
- Release Version 1.64 (Jun-01-2021)
- Release Version 1.63 (May-19-2021)
- Release Version 1.62 (May-05-2021)
- Release Version 1.61 (Apr-20-2021)
- Release Version 1.60 (Apr-06-2021)
- Release Version 1.59 (Mar-23-2021)
- Release Version 1.58 (Mar-09-2021)
- Release Version 1.57 (Feb-22-2021)
- Release Version 1.56 (Feb-09-2021)
- Release Version 1.55 (Jan-25-2021)
- Release Version 1.54 (Jan-12-2021)
- Release Version 1.53 (Dec-22-2020)
- Release Version 1.52 (Dec-03-2020)
- Release Version 1.51 (Nov-10-2020)
- Release Version 1.50 (Oct-19-2020)
- Release Version 1.49 (Sep-28-2020)
- Release Version 1.48 (Sep-01-2020)
- Release Version 1.47 (Aug-06-2020)
- Release Version 1.46 (Jul-21-2020)
- Release Version 1.45 (Jul-02-2020)
- Release Version 1.44 (Jun-11-2020)
- Release Version 1.43 (May-15-2020)
- Release Version 1.42 (Apr-30-2020)
- Release Version 1.41 (Apr-2020)
- Release Version 1.40 (Mar-2020)
- Release Version 1.39 (Feb-2020)
- Release Version 1.38 (Jan-2020)
- Upcoming Features
Shopify uses the concept of a custom app to allow access to store data for a merchant. These custom apps function exclusively for your Shopify store unlike public apps, which are built to work with many stores. The app is configured with the requisite Admin API scopes to fetch the different types of data from the store using Shopify’s REST APIs. You must install this app to view the API token, which is then used to set up a Pipeline in Hevo with Shopify as the Source.
The transfer of data from your Shopify store to the Destination location, therefore, comprises the following one-time setups:
Creating an app in Shopify.
Assigning permissions to the app to read and transform the data using Shopify’s Rest API.
Creating a Pipeline in Hevo for transferring data from Shopify to the Destination database or data warehouse.
A Pipeline only transfers data to the specified Destination. You need to use appropriate tools for transforming the data for further analysis or for publishing it through your e-commerce portal. Read Models.
For creating Pipelines using this Source, Hevo provides you a fully managed BigQuery data warehouse as a possible Destination. This option remains available till the time you set up your first BigQuery Destination irrespective of any other Destinations that you may have. With the managed warehouse, you are only charged the cost that Hevo incurs for your project in Google BigQuery. The invoice is generated at the end of each month and payment is recovered as per the payment instrument you have set up. You can now create your Pipeline and directly start analyzing your Source data. Read Hevo Managed Google BigQuery.
A valid Shopify account.
Data types in Source and Destination are the same.
Perform the following steps to configure your Shopify Source:
Create an App in Shopify
As a first step of transferring data from your Shopify store to the desired Destination, you must create a custom app, using which, you can make API calls to your Shopify store and fetch data. To do this:
Log in to your Shopify Store Admin account.
In the left menu bar, click Apps, and then Develop apps.
Click Allow custom app development.
Click Allow custom app development again to allow Shopify to create custom apps.
In the App Development page, click Create an app.
In the Create an app dialog:
In the App name field, provide a name for the custom app.
In the App developer field, select the email address that Shopify should use for contacting you or your team member regarding your app. Read Update your developer contact details for steps to add or update the email address of your team member to the custom app.
Click Create app.
Click Configure Admin API scopes.
Refer to section, Configure API Permissions in Shopify for steps to configure the app to fetch all data from Shopify.
Configure API Permissions in Shopify
To configure the app to make API calls, you must grant the required permissions for the different scopes and the data within these scopes. Then, you must generate the API token to be used for setting up a Pipeline in Hevo with Shopify as the Source.
To grant the required permissions and generate the API token, perform the following steps:
Under the Admin API access scopes section, enable the
read_<scope_name>check box for the following scopes to get the read access:
Shopify Payment disputes
Shopify Payment payouts
Click Install app.
In the confirmation dialog box, click Install.
In the API credentials tab, under the Admin API access token, click Reveal token once.
This shows your Admin API password that you must use while setting up a Shopify Pipeline in Hevo. The token is visible only once, so you must copy the token and store it in a secure location.
Configure Shopify Connection Settings
Once you have set up the app and generated the API credentials, create a Pipeline to complete the framework for replicating the data from Shopify into the Destination.
To do this:
Click PIPELINES in the Asset Palette.
Click + CREATE in the Pipelines List View.
In the Select Source Type page, select Shopify.
In the Configure your Shopify Source page, specify the following:
Pipeline Name: A unique name for your Shopify Pipeline.
Shop Name: Your shop name. You can find this in the address bar when logged into your shop. For example, in the URL https://mynewshop.myshopify.com/admin, the shop name is “mynewshop”.
Admin API Password: The password that you retrieved in Step 2 above.
No additional settings are needed, as Hevo connects to Shopify using the app details that you provide.
Click TEST & CONTINUE.
Proceed to configuring the data ingestion and setting up the Destination.
Note:You can transfer data of all data types.
|Default Pipeline Frequency||Minimum Pipeline Frequency||Maximum Pipeline Frequency||Custom Frequency Range (Hrs)|
|10 Mins||5 Mins||24 Hrs||1-24|
Note: The custom frequency must be set in hours, as an integer value. For example, 1, 2, 3 but not 1.5 or 1.75.
Historical Data: In the first run of the Pipeline, the Destination database is empty, and legacy data from your Shopify app is replicated into it. By default, Hevo replicates object data for the past one year.
For existing Pipelines: By default, Hevo replicates object data for the past one year.
For Pipelines created after Release 1.80: You can select the Historical Sync Duration as per your requirement while creating a Pipeline. Default value: 3 Months.
If you want data older or recent than this, use the Change Position option after creating the Pipeline. Hevo reloads the historical data as per the new position. If the new position is earlier than the current one and some or all of the historical data has already loaded, the consumption cost for the reloaded Events is incurred twice.
Incremental Data: Once the legacy data is available in the Destination, subsequent runs of the Pipeline sync the objects between the Source and Destination only on an incremental basis.
Note: From Release 1.86, Hevo ingests only new and updated data for Full Load objects to optimize the quota consumption. This feature is currently available on request only. You need to contact Hevo Support to enable it for your team.
Handling of deletes
Hevo tracks deletes for only the Product object. For this, Hevo uses the events resource that records all the Events like creation, update, and deletion for other Shopify resources, such as the addition of a product or the fulfillment of an order. So, whenever Hevo encounters a delete event for the Product object, the value of the
__hevo__marked_deleted column for the record is set to True.
Schema and Primary Keys
Hevo uses the following schema to upload the records in the Destination database:
The following is the list of tables (objects) that are created at the Destination when you run the Pipeline.
The table names are written in small case, except for the Snowflake data warehouse tables which are written in uppercase.
Hevo tracks deletes for the Product object only. Refer to section, Handling of deletes.
|Object||Object Type||Primary Key||Description|
|Application Charge||Billing||ID||Lists one-time charges made on the shop. This type of charge is available for apps that aren’t billed on a recurring basis.|
|Application Credit||Billing||ID||Contains data relating to issue credits to merchants that are used towards future app purchases in Shopify.|
|Recurring Application Charge||Billing||ID||Contains fixed-value, 30-day recurring charges.|
|Usage Charge||Billing||ID||Contains variable usage fees to existing recurring application charge.|
|Customer||Customer||ID||Stores information about a shop’s customers, such as their contact details, their order history, and whether they’ve agreed to receive email marketing.|
|Customer Address||Customer||ID||Stores the addresses that a customer has entered. Each customer can have multiple addresses associated with them with only one address being having is_default set to true at any point of time.|
|Customer Saved Searches||Customer||ID||A search query that represents a group of customers defined by the shop owner.|
|Discount Allocation||Discounts||ORDER_ID, INDEX||Contains information about the discounts applied to products on specific orders.|
|Discount Code||Discounts||ID||Contains the list of discount codes that can be shared with customers.|
|Price Rule||Discounts||ID||Contains discount configurations associated to each discount code.|
|Inventory Item||Inventory||ID||Contains the list of physical good available to be shipped to a customer holding essential information about the goods including its SKU and whether its inventory is tracked.|
|Inventory Level||Inventory||ID||Represents the available quantity of an inventory item at a specific location.|
|Location||Inventory||ID||Contains information about the geographical location of the stores, pop-up stores, headquarters, and warehouses. This information can be used to track sales and manage inventory.|
|Marketing Event||Marketing||ID||Contains information about the actions taken by Shopify to market features such as products, collections, and discounts.|
|Draft Order Lines||Orders||ID||Shows line item data for Draft Order object.|
|Draft Order||Orders||ID||Contains orders created on Shopify on behalf of customers are listed in Draft Order object.|
|Order Risk||Orders||ID||Contains the results of fraud checks that have been completed on an order. They appear in the Fraud analysis section of an order’s details page in the Shopify admin.|
|Refund||Orders||ID||Contains records of money returned to the customer and line items included in the refund, along with restocking instructions.|
|Transaction||Orders||ID||Contains transaction-level data for all orders.|
|Order||Orders||ID||Contains all the order data with combined amounts.|
|Order Lines||Orders||ID||Contains line item data for each order.|
|Tax Lines||Orders||ID||Contains tax line data for each line item of an order or a draft order.|
|Balance||Payment||ID||Contains the current balance for a Shopify Payments account. This amounts is comprised of any Transaction that is not yet included.|
|Balance Transaction||Payment||ID||Records the movement of money in or out of the Shopify Payments account.
Balance Transactions make up the canonical “statement of account”, or ledger for a Shopify Payments account.
|Dispute||Payment||ID||Disputes occur when a buyer questions the legitimacy of a charge with their financial institution.|
|Payout||Payment||ID||Contains information about the movement of money between a Shopify Payments account balance and a connected bank account.|
|CustomCollection||Product||ID||Contains information about the product groupings created by the merchant.|
|Product||Product||ID||Contains the list of products in a merchants store.|
|Product Variant||Product||ID||Contains data for all possible versions of each product with several options.|
|Smart Collection||Product||ID||Contains the grouping of products defined by rules set by the merchant. Shopify automatically changes the contents of a smart collection based on the rules.|
|Collect||Product||ID||Maps relationships between products and custom collections.
For every product in a custom collection there is a collect object that tracks the ID of both the product and the custom collection.
|Product Image||Product||ID||Lists the web URLs of the images for a product.
A product may have up to 250 images, and the images can be in .png, .gif or .jpg format.
|CarrierService||Shipping||ID||Contains information about the shipping providers associated with the store.|
|Fulfillment||Shipping||ID||Contains detailed fulfillment/shipment data associated with an order.
Parent object: Order.
|Fulfillment Lines||Shipping||ID||Maps each line item of an order to its respective fulfilment status.
Parent Object: Order
|Country||Store||ID||Contains information about the tax rates applied to orders recieve from the countries where the store sells its products.|
|Metafield||Store||ID||Contains metadata information about the store resources.|
|Policy||Store||HEVO_ID||Contains information about the policies, such as refund and privacy policies, that the merchant has configured for the store.|
|Province||Store||ID||Contains information about the sales tax applied to recieve orders from the sub-regions of the countries, such as states, provinces, territories, and counties.|
|ShippingZone||Store||ID||Contains information about the shipping zones and their countries, provinces, and shipping rates.|
|Shop||Store||ID||Contains general settings and information about the store.|
Read the detailed Hevo documentation for the following related topics:
OAuth authentication is not supported in private apps.
Hevo captures deletes only for the Product object. Only deletes after Release 1.85 are captured.
Hevo does not capture cascading deletes of the Product object. In Shopify, a Product object can have child objects, Product Image, and Product Variant. When a product is deleted in Shopify, the associated images and variants are deleted as well. However, Hevo captures the information of the deleted product only, and not the images and variants associated with it.
Refer to the following table for the list of key updates made to this page:
|Date||Release||Description of Change|
|Jul-27-2022||NA||Updated Note in section, Data Replication.|
|May-23-2022||NA||Updated sections, Create an App in Shopify and Configure API Permissions in Shopify to include information about custom apps in Shopify.|
|Apr-11-2022||1.86||Added a note in section, Data Replication to inform about optimized quota consumption for Full Load objects.|
|Apr-11-2022||1.85||- Updated the section, Data Replication to add information about handling of deletes for the Product object.
- Added limitations about capturing deletes.
|Jan-24-2022||1.80||Added information about configurable historical sync duration in the Data Replication section.|
|Oct-25-2021||NA||Added the Pipeline frequency information in the Data Replication section.|
|Sep-09-2021||1.71||Updated the section, Data Model to mention the new objects that Hevo now ingests.|
|Jul-12-2021||1.67||Updated the Data Model section with additional objects that Hevo now supports and merged the Appendix into it.|
|Jun-14-2021||1.65||Updated the default historical load duration to one year in the Data Replication section and suggested the Change Position option to fetch Events beyond or more recent than one year.|