On This Page
Hevo supports the following variations of PostgreSQL as a Source:
Click each variant above to know the prerequisites and pre-configuration setups specific to it. Once you complete the setups, follow the steps mentioned below to configure it as a Source in your Hevo Pipeline and start ingesting data.
Hevo recommends logical replication as the default mode to ingest incremental data from your PostgreSQL Source database, especially for high transaction volumes. However, you can also select a different ingestion mode, such as Table or Custom SQL.
Select the Source Type
To select PostgreSQL as the Source:
Click PIPELINES in the Navigation Bar.
Click + CREATE in the Pipelines List View.
In the Select Source Type page, select the PostgreSQL variant.
Specify PostgreSQL Connection Settings
In the Configure your PostgreSQL Source page, specify the connection settings for your PostgreSQL Source.
Read the documents in this section to know the detailed configuration steps for each MySQL variant.
Object settings must be configured if the Ingestion mode is Table.
To do this:
Once your respective Source connection settings are specified in Step 2 above, select the objects to be replicated in the Select the objects you want to replicate page, and then click CONTINUE.
Note: Each object represents a table in your database.
In the Configure Objects page, the default Query Mode is set to XMIN for each selected object from Release 2.13 onwards. If you want to specify a different query mode for an object, select that mode from the drop-down.
Pausing the Pipeline
For Pipelines created with Logical Replication ingestion mode, Hevo replicates the data using the log generated by the Source. Pausing a log-based Pipeline for more than 24 hours may lead to data loss, as a result of the log being deleted. The log can get deleted due to the expiry of its retention period or limited disk storage space in the case of large log files.
In case there is a loss of data after resuming a paused Pipeline, restart the historical load for all the objects to ingest the lost data. To do so, in the Pipeline Overview page:
Select the Objects check box to select all the objects in the Pipeline. You can also select specific objects by selecting the check box next to their names.
Select the Restart option from the MORE drop-down to start the historical data ingestion.
The historical load starts immediately. The re-ingested data does not count towards your quota consumption and is not billed.
Deleting the Pipeline
For Pipelines with Logical Replication as the Pipeline mode, Hevo creates a replication slot in the Source to record any changes. If you delete the Pipeline, Hevo automatically drops this replication slot. However, sometimes, due to issues at the PostgreSQL Source end, the slot might not get dropped automatically. If that happens, use the following command to manually drop the slot:
For example, to drop a slot named
test_postgreSQL_slot, use the following query:
Refer to the following table for the list of key updates made to this page:
|Date||Release||Description of Change|
|May-30-2023||2.13||Updated section, Object Settings to add information about the default query mode.|
|Feb-07-2023||2.07||Updated the page for clarity.|
|Apr-21-2022||1.86||- Updated sections, Specify PostgreSQL Connection Settings and Object Settings.
- Removed section, Select the Pipeline Mode.
|Mar-07-2022||1.83||Updated the section, Select the Pipeline Mode with a note about the logical replication.|
|Jan-24-2022||1.80||Added information about the impact of pausing a Pipeline in the Source Considerations section.|
|Jan-03-2022||1.79||Updated the description of the Include New Tables in the Pipeline advance setting in the Specify PostgreSQL Connection Settings section.|
|Dec-06-2021||1.77||Added section, Source Considerations.|
|Jul-26-2021||1.68||Added a note for the Database Host field.|
|Jul-12-2021||1.67||Added the field Include New Tables in the Pipeline under Source configuration settings.|