On This Page
Shopify uses the concept of an app to allow access to store data for a merchant. The app is configured with the requisite permissions to fetch the different types of data from the store using Shopify’s REST APIs. The Pipeline in Hevo then integrates with the Shopify app to access the data and loads it into the Destination.
The transfer of data from your Shopify store to the Destination location, therefore, comprises the following one-time setups:
Creating an app in Shopify.
Assigning permissions to the app to read and transform the data using Shopify’s Rest API.
Creating a Pipeline in Hevo for transferring data from Shopify to the Destination database or data warehouse.
A Pipeline only transfers data to the specified Destination. You need to use appropriate tools for transforming the data for further analysis or for publishing it through your e-commerce portal. See Models for further information.
For creating Pipelines using this Source, Hevo provides you a fully managed BigQuery data warehouse as a possible Destination. This option remains available till the time you set up your first BigQuery Destination irrespective of any other Destinations that you may have. With the managed warehouse, you are only charged the cost that Hevo incurs for your project in Google BigQuery. The invoice is generated at the end of each month and payment is recovered as per the payment instrument you have set up. You can now create your Pipeline and directly start analyzing your Source data. Read Hevo Managed Google BigQuery.
A valid Shopify account.
Data types in Source and Destination are the same.
Creating an App in Shopify
As a first step, you must create an app, using which, you can make API calls to your Shopify store and fetch data. To do this:
Log in to your Shopify Store Admin account here.
In the left menu bar, click Apps, and then, Manage private apps.
Click Create new private app.
Note: Private apps function exclusively for your Shopify store unlike public apps, which are built to work with many stores. Therefore, with private apps, you can access your store’s data directly using Shopify’s APIs.
In app details:
Provide a name for the private app.
In Emergency developer email, enter the email address that Shopify should use for contacting you regarding your app.
Scroll to the Admin API section and complete the steps listed in the Configuring API Permissions in Shopify task below to enable the app to fetch the data from Shopify.
Configuring API Permissions in Shopify
To enable the app to make API calls, you must configure it with the required permissions to the different resources and the data within them and generate the API credentials.
To do this:
Under Admin API, click Show inactive Admin API permissions to view the list of Shopify API resources and the permissions you can configure for them.
Specify Read Access permission for the following resources:
Click Save. Click Create app in the confirmation dialog box.
Copy the API Credentials that are displayed and store them at a secure location.
Go back to Apps > Manage private apps and click on the app you just created.
Copy the following details. You will enter them in Hevo when connecting to the Shopify data Source.
Password (located below the API key)
Configuring Shopify as a Source
Once you have set up the app and generated the API credentials, create a Pipeline to complete the framework for replicating the data from Shopify into the Destination.
To do this:
Click PIPELINES in the Asset Palette.
Click + CREATE in the Pipelines List View.
In the Select Source Type page, select Shopify.
In the Configure your Shopify Source page, specify the following:
Pipeline Name: A unique name for your Shopify Pipeline.
Shop Name: Your shop name. You can find this in the address bar when logged into your shop. For example, in the URL https://mynewshop.myshopify.com/admin, the shop name is “mynewshop”.
Admin API Password: Your app password. See Step 6 of the procedure above.
No additional settings are needed, as Hevo connects to Shopify using the app details that you provide.
Click TEST & CONTINUE.
Proceed to configuring the data ingestion and setting up the Destination.
Note:You can transfer data of all data types.
|Default Pipeline Frequency||Minimum Pipeline Frequency||Maximum Pipeline Frequency||Custom Frequency Range (Hrs)|
|10 Mins||5 Mins||24 Hrs||1-24|
Note: The custom frequency must be set in hours, as an integer value. For example, 1, 2, 3 but not 1.5 or 1.75.
Historical Data: In the first run of the Pipeline, the Destination database is empty, and legacy data from your Shopify app is replicated into it. By default, Hevo replicates object data for the past one year.
For existing Pipelines: By default, Hevo replicates object data for the past one year.
For Pipelines created after Release 1.80: You can select the Historical Sync Duration as per your requirement while creating a Pipeline. Default value: 3 Months.
If you want data older or recent than this, use the Change Position option after creating the Pipeline. Hevo reloads the historical data as per the new position. If the new position is earlier than the current one and some or all of the historical data has already loaded, the consumption cost for the reloaded Events is incurred twice.
Incremental Data: Once the legacy data is available in the Destination, subsequent runs of the Pipeline sync the objects between the Source and Destination only on an incremental basis.
Note: From Release 1.86, Hevo ingests only new and updated data for Full Load objects to optimize the quota consumption.
Handling of deletes
Hevo tracks deletes for only the Product object. For this, Hevo uses the events resource that records all the Events like creation, update, and deletion for other Shopify resources, such as the addition of a product or the fulfillment of an order. So, whenever Hevo encounters a delete event for the Product object, the value of the
__hevo__marked_deleted column for the record is set to True.
Schema and Primary Keys
Hevo uses the following schema to upload the records in the Destination database:
The following is the list of tables (objects) that are created at the Destination when you run the Pipeline.
The table names are written in small case, except for the Snowflake data warehouse tables which are written in uppercase.
Hevo tracks deletes for the Product object only. Refer to section, Handling of deletes.
|Object||Object Type||Primary Key||Description|
|Application Charge||Billing||ID||Lists one-time charges made on the shop. This type of charge is available for apps that aren’t billed on a recurring basis.|
|Application Credit||Billing||ID||Contains data relating to issue credits to merchants that are used towards future app purchases in Shopify.|
|Recurring Application Charge||Billing||ID||Contains fixed-value, 30-day recurring charges.|
|Usage Charge||Billing||ID||Contains variable usage fees to existing recurring application charge.|
|Customer||Customer||ID||Stores information about a shop’s customers, such as their contact details, their order history, and whether they’ve agreed to receive email marketing.|
|Customer Address||Customer||ID||Stores the addresses that a customer has entered. Each customer can have multiple addresses associated with them with only one address being having is_default set to true at any point of time.|
|Customer Saved Searches||Customer||ID||A search query that represents a group of customers defined by the shop owner.|
|Discount Allocation||Discounts||ORDER_ID, INDEX||Contains information about the discounts applied to products on specific orders.|
|Discount Code||Discounts||ID||Contains the list of discount codes that can be shared with customers.|
|Price Rule||Discounts||ID||Contains discount configurations associated to each discount code.|
|Inventory Item||Inventory||ID||Contains the list of physical good available to be shipped to a customer holding essential information about the goods including its SKU and whether its inventory is tracked.|
|Inventory Level||Inventory||ID||Represents the available quantity of an inventory item at a specific location.|
|Location||Inventory||ID||Contains information about the geographical location of the stores, pop-up stores, headquarters, and warehouses. This information can be used to track sales and manage inventory.|
|Marketing Event||Marketing||ID||Contains information about the actions taken by Shopify to market features such as products, collections, and discounts.|
|Draft Order Lines||Orders||ID||Shows line item data for Draft Order object.|
|Draft Order||Orders||ID||Contains orders created on Shopify on behalf of customers are listed in Draft Order object.|
|Order Risk||Orders||ID||Contains the results of fraud checks that have been completed on an order. They appear in the Fraud analysis section of an order’s details page in the Shopify admin.|
|Refund||Orders||ID||Contains records of money returned to the customer and line items included in the refund, along with restocking instructions.|
|Transaction||Orders||ID||Contains transaction-level data for all orders.|
|Order||Orders||ID||Contains all the order data with combined amounts.|
|Order Lines||Orders||ID||Contains line item data for each order.|
|Tax Lines||Orders||ID||Contains tax line data for each line item of an order or a draft order.|
|Balance||Payment||ID||Contains the current balance for a Shopify Payments account. This amounts is comprised of any Transaction that is not yet included.|
|Balance Transaction||Payment||ID||Records the movement of money in or out of the Shopify Payments account.
Balance Transactions make up the canonical “statement of account”, or ledger for a Shopify Payments account.
|Dispute||Payment||ID||Disputes occur when a buyer questions the legitimacy of a charge with their financial institution.|
|Payout||Payment||ID||Contains information about the movement of money between a Shopify Payments account balance and a connected bank account.|
|CustomCollection||Product||ID||Contains information about the product groupings created by the merchant.|
|Product||Product||ID||Contains the list of products in a merchants store.|
|Product Variant||Product||ID||Contains data for all possible versions of each product with several options.|
|Smart Collection||Product||ID||Contains the grouping of products defined by rules set by the merchant. Shopify automatically changes the contents of a smart collection based on the rules.|
|Collect||Product||ID||Maps relationships between products and custom collections.
For every product in a custom collection there is a collect object that tracks the ID of both the product and the custom collection.
|Product Image||Product||ID||Lists the web URLs of the images for a product.
A product may have up to 250 images, and the images can be in .png, .gif or .jpg format.
|CarrierService||Shipping||ID||Contains information about the shipping providers associated with the store.|
|Fulfillment||Shipping||ID||Contains detailed fulfillment/shipment data associated with an order.
Parent object: Order.
|Fulfillment Lines||Shipping||ID||Maps each line item of an order to its respective fulfilment status.
Parent Object: Order
|Country||Store||ID||Contains information about the tax rates applied to orders recieve from the countries where the store sells its products.|
|Metafield||Store||ID||Contains metadata information about the store resources.|
|Policy||Store||HEVO_ID||Contains information about the policies, such as refund and privacy policies, that the merchant has configured for the store.|
|Province||Store||ID||Contains information about the sales tax applied to recieve orders from the sub-regions of the countries, such as states, provinces, territories, and counties.|
|ShippingZone||Store||ID||Contains information about the shipping zones and their countries, provinces, and shipping rates.|
|Shop||Store||ID||Contains general settings and information about the store.|
OAuth authentication is not supported in private apps.
Hevo captures deletes only for the Product object. Only deletes after Release 1.85 are captured.
Hevo does not capture cascading deletes of the Product object. In Shopify, a Product object can have child objects, Product Image, and Product Variant. When a product is deleted in Shopify, the associated images and variants are deleted as well. However, Hevo captures the information of the deleted product only, and not the images and variants associated with it.
Refer to the following table for the list of key updates made to this page:
|Date||Release||Description of Change|
|Apr-11-2022||1.86||Added a note in section, Data Replication to inform about optimized quota consumption for Full Load objects.|
|Apr-11-2022||1.85||- Updated the section, Data Replication to add information about handling of deletes for the Product object.
- Added limitations about capturing deletes.
|Jan-24-2022||1.80||Added information about configurable historical sync duration in the Data Replication section.|
|Oct-25-2021||NA||Added the Pipeline frequency information in the Data Replication section.|
|Sep-09-2021||1.71||Updated the section, Data Model to mention the new objects that Hevo now ingests.|
|Jul-12-2021||1.67||Updated the Data Model section with additional objects that Hevo now supports and merged the Appendix into it.|
|Jun-14-2021||1.65||Updated the default historical load duration to one year in the Data Replication section and suggested the Change Position option to fetch Events beyond or more recent than one year.|