On This Page
Shopify uses the concept of an app to allow access to store data for a merchant. The app is configured with the requisite permissions to fetch the different types of data from the store using Shopify’s REST APIs. The Pipeline in Hevo then integrates with the Shopify app to access the data and loads it into the Destination.
The transfer of data from your Shopify store to the Destination location, therefore, comprises the following one-time setups:
Creating an app in Shopify.
Assigning permissions to the app to read and transform the data using Shopify’s Rest API.
Creating a Pipeline in Hevo for transferring data from Shopify to the Destination database or data warehouse.
A Pipeline only transfers data to the specified Destination. You need to use appropriate tools for transforming the data for further analysis or for publishing it through your e-commerce portal. See Models for further information.
For creating Pipelines using this Source, Hevo provides you a fully managed BigQuery data warehouse as a possible Destination. This option remains available till the time you set up your first BigQuery Destination irrespective of any other Destinations that you may have. With the managed warehouse, you are only charged the cost that Hevo incurs for your project in Google BigQuery. The invoice is generated at the end of each month and payment is recovered as per the payment instrument you have set up. You can now create your Pipeline and directly start analyzing your Source data. Read Hevo Managed Google BigQuery.
A valid Shopify account.
Data types in Source and Destination are the same.
Creating an App in Shopify
As a first step, you must create an app, using which, you can make API calls to your Shopify store and fetch data. To do this:
Log in to your Shopify Store Admin account here.
In the left menu bar, click Apps, and then, Manage private apps.
Click Create new private app.
Note: Private apps function exclusively for your Shopify store unlike public apps, which are built to work with many stores. Therefore, with private apps, you can access your store’s data directly using Shopify’s APIs.
In app details:
Provide a name for the private app.
In Emergency developer email, enter the email address that Shopify should use for contacting you regarding your app.
Scroll to the Admin API section and complete the steps listed in the Configuring API Permissions in Shopify task below to enable the app to fetch the data from Shopify.
Configuring API Permissions in Shopify
To enable the app to make API calls, you must configure it with the required permissions to the different resources and the data within them and generate the API credentials.
To do this:
Under Admin API, click Show inactive Admin API permissions to view the list of Shopify API resources and the permissions you can configure for them.
Specify Read Access permission for the following resources:
Click Save. Click Create app in the confirmation dialog box.
Copy the API Credentials that are displayed and store them at a secure location.
Go back to Apps > Manage private apps and click on the app you just created.
Copy the following details. You will enter them in Hevo when connecting to the Shopify data Source.
Password (located below the API key)
Configuring Shopify as a Source
Once you have set up the app and generated the API credentials, create a Pipeline to complete the framework for replicating the data from Shopify into the Destination.
To do this:
Click PIPELINES in the Asset Palette.
Click + CREATE in the Pipelines List View.
In the Select Source Type page, select Shopify.
In the Configure your Shopify Source page, specify the following:
Pipeline Name: A unique name for your Shopify Pipeline.
Shop Name: Your shop name. You can find this in the address bar when logged into your shop. For example, in the URL https://mynewshop.myshopify.com/admin, the shop name is “mynewshop”.
Admin API Password: Your app password. See Step 6 of the procedure above.
No additional settings are needed, as Hevo connects to Shopify using the app details that you provide.
Click TEST & CONTINUE.
Proceed to configuring the data ingestion and setting up the Destination.
Note: You can transfer data of all data types.
- Historical Data: In the first run of the Pipeline, the Destination database is empty, and legacy data from your Shopify app is replicated into it. By default, Hevo replicates object data for the past one year.
If you want data older or recent than this, use the Change Position option after creating the Pipeline. Hevo reloads the historical data as per the new position. If the new position is earlier than the current one and some or all of the historical data has already loaded, the consumption cost for the reloaded Events is incurred twice.
- Incremental Data: Once the legacy data is available in the Destination, subsequent runs of the Pipeline sync the objects between the Source and Destination only on an incremental basis. The default frequency of uploading the objects is 10 minutes. You can modify this value, with the minimum frequency being 5 minutes and maximum being 24 hours.
To do this:
Click the More icon on the top right corner of the pane, and then, Change Schedule.
Select from the available frequencies or specify a custom duration and click Schedule. The new frequency is displayed below the Source name in the Pipeline. For example, in the image above, the sh_25 Shopify Source is synchronized every three hours.
Schema and Primary Keys
Hevo uses the following schema to upload the records in the Destination database:
The following is the list of tables (objects) that are created at the Destination when you run the Pipeline.
If you have selected the AutoMapping option, Hevo creates the tables in the Destination by default. Else, you must create the appropriate tables and map them accordingly. See sample image here:
Note: The table names are written in small case, except for the Snowflake data warehouse tables which are written in uppercase.
|Object||Object Type||Primary Key||Description|
|Application Charge||Billing||ID||Lists one-time charges made on the shop. This type of charge is available for apps that aren’t billed on a recurring basis.|
|Application Credit||Billing||ID||Contains data relating to issue credits to merchants that are used towards future app purchases in Shopify.|
|Recurring Application Charge||Billing||ID||Contains fixed-value, 30-day recurring charges.|
|Usage Charge||Billing||ID||Contains variable usage fees to existing recurring application charge.|
|Customer||Customer||ID||Stores information about a shop’s customers, such as their contact details, their order history, and whether they’ve agreed to receive email marketing.|
|Customer Address||Customer||ID||Stores the addresses that a customer has entered. Each customer can have multiple addresses associated with them with only one address being having is_default set to true at any point of time.|
|Customer Saved Searches||Customer||ID||A search query that represents a group of customers defined by the shop owner.|
|Tax Exemption||Customer||ID||Currently included as an array field within Customer object (6).|
|Discount Code||Discounts||ID||Contains the list of discount codes that can be shared with customers.|
|Price Rule||Discounts||ID||Contains discount configurations associated to each discount code.|
|Inventory Item||Inventory||ID||Contains the list of physical good available to be shipped to a customer holding essential information about the goods including its SKU and whether its inventory is tracked.|
|Inventory Level||Inventory||ID||Represents the available quantity of an inventory item at a specific location.|
|Location||Inventory||ID||Representsepresents geographical locations where your stores, pop-up stores, headquarters, and warehouses exist.|
|Draft Order Lines||Orders||ID||Shows line item data for Draft Order object.|
|Abandoned Checkouts||Orders||ID||A checkout is considered abandoned when a customer leaves the checkout after the first page without completing their purchase.|
|Draft Order||Orders||ID||Contains orders created on Shopify on behalf of customers are listed in Draft Order object.|
|Order Risk||Orders||ID||Contains the results of fraud checks that have been completed on an order. They appear in the Fraud analysis section of an order’s details page in the Shopify admin.|
|Refund||Orders||ID||Contains records of money returned to the customer and line items included in the refund, along with restocking instructions.|
|Transaction||Orders||ID||Contains transaction-level data for all orders.|
|Order||Orders||ID||Contains all the order data with combined amounts.|
|Order Lines||Orders||ID||Contains line item data for each order.|
|Tax Lines||Orders||ID||Contains tax line data for each line item of an order or a draft order.|
|Balance||Payment||ID||Contains the current balance for a Shopify Payments account. This amounts is comprised of any Transaction that is not yet included.|
|Balance Transaction||Payment||ID||Records the movement of money in or out of the Shopify Payments account.
Balance Transactions make up the canonical “statement of account”, or ledger for a Shopify Payments account.
|Dispute||Payment||ID||Disputes occur when a buyer questions the legitimacy of a charge with their financial institution.|
|Payout||Payment||ID||Contains information about the movement of money between a Shopify Payments account balance and a connected bank account.|
|Collection||Product||ID||Contains groupings of products that merchants create to make their stores easier to browse.|
|Product||Product||ID||Contains the list of products in a merchants store.|
|Product Variant||Product||ID||Contains data for all possible versions of each product with several options.|
|Smart Collection||Product||ID||Contains the grouping of products defined by rules set by the merchant. Shopify automatically changes the contents of a smart collection based on the rules.|
|Collect||Product||ID||Maps relationships between products and custom collections.
For every product in a custom collection there is a collect object that tracks the ID of both the product and the custom collection.
|Product Image||Product||ID||Lists the web URLs of the images for a product.
A product may have up to 250 images, and the images can be in .png, .gif or .jpg format.
|Fulfillment||Shipping||ID||Contains detailed fulfillment/shipment data associated with an order.
Parent object: Order.
|Fulfillment Lines||Shipping||ID||Maps each line item of an order to its respective fulfilment status.
Parent Object: Order
- OAuth authentication is not supported in private apps.
Refer to the following table for the list of key updates made to this page:
|Date||Release||Description of Change|
|Jul-12-2021||1.67||Updated the Data Model section with additional objects that Hevo now supports and merged the Appendix into it.|
|Jun-14-2021||1.65||Updated the default historical load duration to one year in the Data Replication section and suggested the Change Position option to fetch Events beyond or more recent than one year.|