PostgreSQL (Edge)
On This Page
Hevo Edge supports the following variations of PostgreSQL as a Source:
Handling Toast Data in Hevo Edge
PostgreSQL allows for columns to store large amounts of data. For this, it uses The Oversized-Attribute Storage Technique (TOAST), where if the column value exceeds a certain threshold (around 8 KB, the default page size in PostgreSQL), it is compacted and may also be stored in a separate TOAST table. Such a column is referred to as a TOASTed column.
Hevo Edge identifies the toasted columns in the ingested data and replicates data from them to your Destination tables using the merge operation.
Note: Hevo Edge does not replicate data from toasted columns if your Pipeline loads data in the Append mode.
Resolving Data Loss in Paused Pipelines
For log-based Edge Pipelines created with any variant of the PostgreSQL Source, the data to be replicated is identified from the write-ahead logs (WAL) by the publications created on the database tables. Hence, disabling the log-based Pipeline may lead to data loss, as the corresponding WAL segment may have been deleted. The log can get deleted due to the expiry of its retention period or limited storage space in the case of large log files.
If you notice a data loss in your Edge Pipeline after enabling it, resync the Pipeline. The Resync Pipeline action restarts the historical load for all the active objects in your Pipeline, thus recovering any lost data.
Note: The re-ingested data does not count towards your quota consumption and is not billed.