Pipeline Mode

On this page, you will have 3 options for Ingestion mode. This will define how you want Hevo to read your data from MySQL source.

  • Table: In this mode, your tables will be polled individually at a fixed frequency. You can use this mode when you are looking to fetch data from multiple tables in your database and you would like to have control over ingestion for every table individually.
  • Custom SQL: If you are looking to fetch data in a different structure than how it is stored in your source tables, you can use this mode. It will allow you to write a custom query and data will be polled using that query at a fixed frequency.
  • BinLog: Data will be polled using MySQL’s BinLog in this mode. This mode is useful when you are looking to replicate the complete database as it is. This mode is very efficient in replicating but leaves you with less control and manageability over data ingestion. A few pointers on setting up the binary logs may be found here.

Connection Settings

Provide your MySQL database connection details on MySQL Connection Settings page. You will have the following options in the connection details block:

  • Source Name: A unique name for this source
  • MySQL Host: MySQL host’s IP address or DNS
  • MySQL Port: The port on which your MySQL server is listening for connections (default is 3306)
  • MySQL User: The read-only user that can read the tables in your database.
  • MySQL Password: Password for the read-only user
  • Database Name: The database that you wish to replicate
  • If you want to connect to Hevo using an SSH tunnel, check Connecting Through SSH. Else, you will have to whitelist Hevo’s IP addresses as listed here.

There are some advanced settings for Binary-Log ingestion mode; you can find those below:

  • Load Historical Data: Enable this if you wish to load historical data.
  • Load All Databases: Enable this if you wish to load data from all databases on the selected Host.
  • Specify Database Names: You can also specify a comma-separated list of database names you want to load data from. It only appears when the above setting Load All Databases is set to false.
  • Merge Tables: If enabled, Hevo will merge tables with the same name from different databases while loading the data to the destination. Hevo will load the database name with each record.

Job Settings

Job Settings may be required if Hevo needs your inputs to query data from your database. Each Job listed represents a table in your database. Each Job can be configured with a Query Mode. You can read more about Query Modes.

Last updated on 25 Aug 2020