- Introduction
- Getting Started
- Data Ingestion
- Data Loading
- Loading Data in a Database Destination
- Loading Data to a Data Warehouse
- Optimizing Data Loading for a Destination Warehouse
- Manually Triggering the Loading of Events
- Scheduling Data Load for a Destination
- Loading Events in Batches
- Name Sanitization
- Table and Column Name Compression
- Parsing Nested JSON Fields in Events
- Pipelines
- Data Flow in a Pipeline
- Ingestion Modes
- Familiarizing with the Pipelines UI
- Pipeline Objects
- Working with Pipelines
- Transformations
-
Schema Mapper
- Using Schema Mapper
- Mapping Statuses
- Auto Mapping Event Types
- Mapping a Source Event Type with a Destination Table
- Mapping a Source Event Type Field with a Destination Table Column
- Schema Mapper Actions
- Fixing Unmapped Fields
- Resolving Incompatible Schema Mappings
- Resizing String Columns in the Destination
- Schema Mapper Compatibility Table
- Failed Events in a Pipeline
- Pipeline FAQs
- Events Usage
- Sources
- Free Sources
- Analytics
- Collaboration
- CRM
- Data Warehouses
- Databases
- E-Commerce
- File Storage
- Finance & Accounting
-
Marketing
- AdRoll
- Apple Search Ads
- AppsFlyer
- Criteo
- Facebook Ads
- Facebook Page Insights
- Google Ads
- Google Campaign Manager
- Google Play Console
- Google Search Console
- HubSpot
- Instagram Business
- Klaviyo
- LinkedIn Ads
- Mailchimp
- Marketo
- Microsoft Advertising
- Outbrain
- Pardot
- Pinterest Ads
- Segment
- SendGrid
- SendGrid Webhook
- Salesforce Marketing Cloud
- Snapchat Ads
- Taboola
- Twitter Ads
- Streaming
- Source FAQs
- Destinations
- Transform
- Activate
- Alerts
- Account Management
- Troubleshooting
-
Troubleshooting Sources
- Troubleshooting Amazon DynamoDB
- Troubleshooting MongoDB
- Troubleshooting MS SQL
- Troubleshooting MySQL
- Troubleshooting Oracle
-
Troubleshooting PostgreSQL
-
Errors during Pipeline creation
- Error 1003 - Authentication failure
- Error 1006 - Connection settings errors
- Error 1011 - Access role issue for logical replication
- Error 1012 - Access role issue for logical replication
- Error 1014 - Database does not exist
- Error 1017 - Connection settings errors
- Error 1023 - No pg_hba.conf entry
- Error 1024 - Number of requested standby connections
-
Errors during Pipeline creation
- Troubleshooting Salesforce
- Troubleshooting Destinations
-
Troubleshooting Sources
- Glossary
- Release Notes
- Release Version 1.88
- Release Version 1.87
- Release Version 1.86
- Release Version 1.84 & 1.85
- Release Version 1.83
- Release Version 1.82
- Release Version 1.81
- Release Version 1.80 (Jan-24-2022)
- Release Version 1.79 (Jan-03-2022)
- Release Version 1.78 (Dec-20-2021)
- Release Version 1.77 (Dec-06-2021)
- Release Version 1.76 (Nov-22-2021)
- Release Version 1.75 (Nov-09-2021)
- Release Version 1.74 (Oct-25-2021)
- Release Version 1.73 (Oct-04-2021)
- Release Version 1.72 (Sep-20-2021)
- Release Version 1.71 (Sep-09-2021)
- Release Version 1.70 (Aug-23-2021)
- Release Version 1.69 (Aug-09-2021)
- Release Version 1.68 (Jul-26-2021)
- Release Version 1.67 (Jul-12-2021)
- Release Version 1.66 (Jun-28-2021)
- Release Version 1.65 (Jun-14-2021)
- Release Version 1.64 (Jun-01-2021)
- Release Version 1.63 (May-19-2021)
- Release Version 1.62 (May-05-2021)
- Release Version 1.61 (Apr-20-2021)
- Release Version 1.60 (Apr-06-2021)
- Release Version 1.59 (Mar-23-2021)
- Release Version 1.58 (Mar-09-2021)
- Release Version 1.57 (Feb-22-2021)
- Release Version 1.56 (Feb-09-2021)
- Release Version 1.55 (Jan-25-2021)
- Release Version 1.54 (Jan-12-2021)
- Release Version 1.53 (Dec-22-2020)
- Release Version 1.52 (Dec-03-2020)
- Release Version 1.51 (Nov-10-2020)
- Release Version 1.50 (Oct-19-2020)
- Release Version 1.49 (Sep-28-2020)
- Release Version 1.48 (Sep-01-2020)
- Release Version 1.47 (Aug-06-2020)
- Release Version 1.46 (Jul-21-2020)
- Release Version 1.45 (Jul-02-2020)
- Release Version 1.44 (Jun-11-2020)
- Release Version 1.43 (May-15-2020)
- Release Version 1.42 (Apr-30-2020)
- Release Version 1.41 (Apr-2020)
- Release Version 1.40 (Mar-2020)
- Release Version 1.39 (Feb-2020)
- Release Version 1.38 (Jan-2020)
- Upcoming Features
Stripe
Stripe is a payment gateway that facilitates payments (in person or online) by transferring money between a merchant account and a payment processor. It can be incorporated into websites and mobile applications and provides technical, fraud prevention and banking infrastructure to facilitate online payments.
For creating Pipelines using this Source, Hevo provides you a fully managed BigQuery data warehouse as a possible Destination. This option remains available till the time you set up your first BigQuery Destination irrespective of any other Destinations that you may have. With the managed warehouse, you are only charged the cost that Hevo incurs for your project in Google BigQuery. The invoice is generated at the end of each month and payment is recovered as per the payment instrument you have set up. You can now create your Pipeline and directly start analyzing your Source data. Read Hevo Managed Google BigQuery.
Prerequisites
-
An active Stripe account with either Administrator or Developer role. Read Teams and user roles.
-
Either a Restricted API Key or a Secret Key, which allows Hevo to read your Stripe data. Read Creating the API Key.
Configuring Stripe as a Source
Perform the following steps to configure Stripe as a Source in Hevo:
-
Click PIPELINES in the Asset Palette.
-
Click + CREATE in the Pipeline List View page.
-
In the Select Source Type page, select Stripe.
-
In the Configure Your Stripe Source page, specify the following:
-
Pipeline Name: A unique Name for your Pipeline.
-
API Key: The API key that you created in Stripe to enable Hevo to read your Stripe data.
-
-
Click TEST & CONTINUE.
-
Proceed to configuring the data ingestion and setting up the Destination.
Creating the API Key
You may create a Restricted API Key or a Standard Secret API Key and share it with us. The restricted API key gives you more control as to which objects you want to give Hevo access to. If you have an Administrator or Developer role, your API keys are always available in the Stripe Dashboard. Read Stripe’s documentation on API keys.
To create a restricted API Key:
-
Go to the Developer Dashboard.
-
Disable the Viewing test data option if it is enabled, in order to view live data.
-
Scroll down to the Developers menu on the left and click API Keys.
-
Click Create restricted key.
-
Specify a unique name for this API Key. For example, Stripe connector for Hevo.
-
Select
READ
permission for each resource category, to apply it to all the resources within it:-
Core
-
Billing
-
Checkout
-
Order
-
Connect
-
Issuing
-
Webhook
-
Reporting
-
-
Once done, click Create API Keys at the bottom of the page.
-
On your dashboard under the Restricted Keys section, click Reveal key token for the API keys that you just created, and copy it. You will provide this key while setting up Stripe as a Source in your Hevo Pipeline.
Source Considerations
-
Stripe informs the user of a change in an object in their account through an event. For example, when a charge succeeds, it creates a
charge.succeeded
event. Stripe maintains a record of all the events that it created for the past 30 days in anevents
object. Hevo refers to thisevents
object while replicating the data to the Destination to make sure all the deletes and updates are duly accounted for.Make sure to select Stripe’s
events
object while creating your Pipeline, else, Hevo will not be able to capture any updates. To enable Hevo to capture deletes for your Pipeline, you need to contact Hevo Support. -
Pausing a Pipeline for more than 30 days can lead to a loss of data, as the
events
object tracks updates and deletes of the past 30 days only.
Data Replication
Default Pipeline Frequency | Minimum Pipeline Frequency | Maximum Pipeline Frequency | Custom Frequency Range (Hrs) |
---|---|---|---|
30 Mins | 15 Mins | 24 Hrs | 1-24 |
Note: The custom frequency must be set in hours, as an integer value. For example, 1, 2, 3 but not 1.5 or 1.75.
-
Historical Data: Hevo ingests your historical data in reverse order, starting from the latest to the earliest Event for all Pipelines. This enables you to have quicker access to your most recent data.
- For Pipelines created before Release 1.80: Hevo ingests historical data on the basis of the offset selected at the time of creating the Pipeline. By default, it fetches the entire past data associated with your Stripe account.
- For Pipelines created after Release 1.80: You can select the Historical Sync Duration as per your requirement while creating a Pipeline. Default value: 3 Months.
Note: The time taken to fetch the historical data depends on the number of days selected at the time of creating the Pipeline. Hevo defaults this period to 30 days. If required, you may customize it to any number or even select the starting date of your Stripe account.
-
Incremental Data: Once the historical load is completed, each run of the Pipeline fetches the next available new and modified data. Hevo also fetches the Events object from Stripe, which tracks the updates and deletes made in the last 30 days, in order to sync the related Events with your Destination.
Updated and Deleted Objects
Hevo replicates the updated and deleted records using the Stripe events
object. For the deleted records, Hevo sets the is_row_deleted
column to True and the deleted_timestamp
column to the timestamp when the record was deleted in Stripe, in the ingested data and the Destination.
Hevo identifies the updated
and the deleted
records from the Stripe events
object and replicates that information for the following objects:
Object Name | Event Type Tracked |
---|---|
ACCOUNT | updated |
BANK_ACCOUNT | deleted |
CARD | deleted |
CHARGE | updated |
CUSTOMER | - updated - deleted |
DISPUTE | updated |
INVOICE | - updated - deleted |
INVOICE_ITEM | deleted |
ORDER | updated |
PAYMENT_INTENT | updated |
PAYOUT | updated |
PERSON | deleted |
PRICE | deleted |
PRODUCT | - updated - deleted |
REFUND | updated |
SETUP_INTENT | updated |
SUBSCRIPTION | updated |
Note: As Hevo also replicates the deleted data now, you might see more Events in your Destination than in the Source.
Schema and Primary Keys
Hevo uses the following schema to upload the records in the Destination:
Data Model
The following is the list of tables (objects) that are created at the Destination when you run the Pipeline:
Table | Description |
---|---|
Account | Contains all the properties of a Stripe account, such as its current email address and whether the account is enabled to make live charges. |
Application Fee | Contains the transaction fee amount collected on top of a charge made for a user. |
Application Fee Refund | Contains the reference to the application fee collected from the user and the portion of it to be refunded in case of a transfer reversal. |
Authorization | Contains the information that must be approved for any purchase to be completed successfully from an issued card. |
Balance transaction | Contains the details of funds moving through an account. It is created for every in-flow and out-flow of funds to and from your Stripe account. Read Balance transaction types. |
Bank Account | Contains information of the bank accounts connected to the payment methods of the customers object. |
Card | Contains details such as cardholder, and the month and year of expiry of the physical or virtual cards that are issued to the cardholders. |
Cardholder | Contains information such as the name, email, and phone number of the individual or business entity who issues a card. |
Charge | Contains the details of a charge made on a credit or a debit card. You can retrieve and refund individual charges as well as list all charges. They are identified by a unique, random ID. |
Credit Note | Contains the adjusted or refunded amount to be added to an already finalized invoice instead of recreating the invoice. |
Credit Note line item | Contains the individual adjustments or refunds that you are adding to a credit note associated with an invoice. |
Customer | Contains the details of a customer, such as the name, email, default payment method and delinquency status. |
Customer Balance Transaction | Contains the amount by which a customer’s account balance needs to be incremented or decremented. |
Coupon | Contains the information about a percent-off or amount-off discount you might want to apply to a customer. It can be applied to invoices or orders. |
Discount | Contains the information such as the beginning and ending of the discount and the coupon object. It represents the actual application of a coupon to a particular customer. |
Dispute Balance Transaction | Contains the mapping between each dispute and transaction entities. It is a custom table generated by Hevo to maintain data sanity. |
Fee Detail | Contains the detailed breakdown of fees paid for each transaction present in the balance transaction table. |
File Link | Contains a URL that you can use to retrieve the contents of a file without any authentication. |
Invoice | Contains the statements of amounts owed by a customer. Invoices are either generated one-off or periodically from a subscription. |
Invoice Line Item | Contains the details of an invoice, such as amount, currency, period and proration. It is stored inside the invoice object. |
Payment Intent | Contains the guide for the process of collecting a payment from your customer during a particular session. Read How intents work to know more about the payment flow followed at Stripe. |
Payment Method | Contains the details of the payment method that a customer has used on one or more payment intent(s) along with details. Supported Payment Methods are Card (card/wallet), FPX, and ideal. |
Payment Method Details | Contains the detailed data for each payment method that is created on your Stripe Account, will be individually stored in your destination to maintain data sanity. Examples: - payment_method_card - payment_method_fpx - payment_method_ideal |
Payout | Contains the details of the received funds from Stripe. It is created when you initiate a payout to either a bank account or debit card of a connected Stripe account. |
Person | Contains the details of the person linked to a Stripe account. |
Plan (pricing) | Contains the details such as the base price, currency and billing cycle for recurring purchases of products. It helps you track the pricing. |
Price | Contains the details such as the unit cost, currency, and billing cycle for recurring and one-time purchases of the different products you offer to your customers. For example, a standard or premium version of your service. |
Product | Contains a list of all offers for the product. Each product can have multiple tiers of pricing plans. |
Promotion Code | Contains a customer-redeemable code that can be used to create single or multiple codes for a coupon. |
Refund | Contains the details of a charge that is to be refunded due to a dispute, return, or failure to deliver a service. |
Review | Contains human-added information to supplement automated fraud detection cases. |
Setup Intent | Contains the guide for the process of saving a customer’s payment credentials for future payments. |
Source | Contains the details of a customer’s payment methods that can be used with the Stripe API. |
Subscription | Contains the details of a recurring charge for a customer. Each subscription has multiple line items. |
Subscription Item | Contains line item data for each subscription. Each line item is associated with a plan. |
Subscription Schedule | Contains the attributes required to manage the lifecycle of customers’ subscriptions by predefining existing changes. |
Top-Up | Contains the amount by which customers wish to top up their Stripe account balance. |
Transaction | Contains details of funds entering or leaving a Stripe account. A transaction object is created whenever an issued card is used for making a purchase or a refund. |
Transfer | Contains the details to move funds between Stripe accounts such as amount, destination and so on. It is used when you need to directly move a fund from your customer to the seller who’s offering the service through your platform. |
Transfer Reversal | Contains the amount that is to be deducted from the seller’s account (who originally received it via a successful transfer) and added back to your platform. |
Usage Record | Contains a report of a customer’s usage and other metrics that are sent to Stripe. |
Limitations
- For new and existing Pipelines, Hevo replicates only the data that has been deleted after Release 1.86.
Revision History
Refer to the following table for the list of key updates made to this page:
Date | Release | Description of Change |
---|---|---|
Apr-11-2022 | 1.86 | - Updated the Data Replication and Source Considerations sections to add information about how Hevo handles deletes in Stripe objects. |
Mar-07-2022 | 1.83 | Added information about reverse historical load in the Data Replication section. |
Jan-24-2022 | 1.80 | Added information about configurable historical sync duration in the Data Replication section. |
Jan-03-2022 | 1.79 | Added objects to section, Data Model. |
Oct-25-2021 | NA | Added the Pipeline frequency information in the Data Replication section. |
Aug-23-2021 | 1.70 | Added the section, Source Considerations. |