Auto Mapping Event Types
On This Page
Hevo provides you an option to automatically map Event Types and fields from your Source to the corresponding tables and columns in the Destination, thereby eliminating the need for any human intervention. You can enable Auto Mapping at the Pipeline or the Event Type level. For example, you can disable Auto Mapping for the entire Pipeline but enable it for just one Event Type or vice versa.
Once you configure the Source and Destination of a Pipeline and select Auto Mapping, Hevo replicates the Source schemas as is to the Destination table. It also handles any future schema changes that may occur in the Source data.
As part of Auto Mapping, Hevo:
Creates the respective tables and fields in the Destination with the appropriate schema if these are not present already.
Maps all the Source Event Types automatically to the tables in the Destination.
If multiple Pipelines are configured to ingest data from objects with the same name in the Source(s) to a common Destination, then, once a corresponding table is created or mapped in the Destination for the first Pipeline, Auto Mapping picks up the same Destination table for the remaining Pipelines too. This can cause the data in the Destination table to be overwritten every time any of these Pipelines run. In such a scenario, you can do the following:
Provide a different Destination prefix during each Pipeline setup so that a different table is created each time in the Destination.
Disable Auto Mapping while creating the Pipelines, and manually map the desired table in the Destination. Subsequently, you can enable Auto Mapping for the Pipeline or the object.
The following table describes how Auto Mapping handles different data scenarios:
|Sr. No.||Data Scenario||Auto-Mapping On||Auto-Mapping Off|
|1||New Event Types are added in the Source.||No user intervention is required. Hevo creates and maps the respective Destination tables with the selected Event Types, and initiates the data replication process.||You need to create and map the respective tables manually to the selected Event Types before being able to sync the data with the Destination.|
|2||Skipped Event Types are included for data replication to the Destination.||No user intervention is required. Hevo creates and maps the respective Destination tables with the selected Event Types, and initiates the data replication process.||You need to create and map the respective tables manually to the selected Event Types before being able to sync the data with the Destination.|
|3||New fields are added to the existing Event Types in the Source.||No user intervention is required. Hevo creates the respective columns in the Destination.||You need to map the respective columns manually to the Event Types field before being able to load the data to the Destination.|
|4||Fields are deleted from the existing Event Types in the Source.||No user intervention is required. Hevo does not load any data further in the respective columns.||No user intervention is required. Hevo does not load any data further in the respective columns.|
|5||Fields data types are changed to compatible data types in the Source.||No user intervention is required if data type promotion is supported for the Destination, else Hevo sidelines the affected Events. In such a case, you can use Transformations, or you can create and map the table again.||Hevo sidelines the affected Events until mappings are resolved. You can use Transformations, or you can create and map the table again.|
|6||Fields data types are changed to incompatible data types in the Source.||Hevo sidelines the affected Events until mappings are resolved. You can use Transformations if possible, else you need to create and map the table again.||Hevo sidelines the affected Events until mappings are resolved. You can use Transformations if possible, else you need to create and map the table again.|
|7||Tables are deleted in the Destination.||No user intervention is required. Hevo recreates the deleted tables and loads the Events in the Destination.||You need to re-create and map the deleted tables manually before being able to sync the data with the Destination.|
|8||Columns are deleted in the Destination.||No user intervention is required. Hevo recreates the deleted columns and loads the Events in the Destination.||You need to re-map the deleted columns manually before being able to sync the data with the Destination.|
When you disable Auto Mapping for an Event Type, its current mapping is retained, but its status changes from Auto Mapped to Mapped. Any subsequent change made to its schema in the Source, for example, change in the data type of a field or addition or deletion of columns, is not automatically made in the Destination table moving forward. Read Modifying Schema Mapping for Event Types to know how you can modify or delete the mapping for such Event Types.
Changes to the Source data or its mapping are reflected in the Destination only when the changed Events are next ingested. For example, suppose a field is created in an Event Type in the Source. Irrespective of whether Auto Mapping is enabled or not, the corresponding column is created in the Destination automatically only when Hevo ingests an Event that carries the value for the new field.
Click the VIEW SCHEMA button below the Pipeline Summary Bar to see the Source schema details such as the list of objects synced and data types of the fields pertaining to each object.
When tables are created in the Destination for the first time based on the Source schema, any primary keys defined in the Source are retained in the Destination tables during Auto Mapping. In addition, for the Amazon Redshift data warehouse, sort keys are automatically created on the primary keys defined for the respective tables. This is similar to the functionality offered in manual schema mapping.
As of Release 1.69, for existing tables in the Destination, the primary keys from the Source schema are mapped if a primary key is not already defined for the table. If the Destination table already contains duplicate Events, those are not deleted as part of this mapping; only the new Events are de-duplicated.
Note: A sort key defines the column by which data is sorted in a table. Availability of sorted data improves the query efficiency during loading of Events as number of scans required to identify the location of update are substantially reduced. Consequently, job performance is improved.
Refer to the following table for the list of key updates made to this page:
|Date||Release||Description of Change|
|Jan-10-2023||2.05||Reorganized content for better clarity.|
|Dec-16-2022||NA||Added information about dealing with multiple Pipelines ingesting data from the same Source table.|
|Apr-11-2022||1.86||Updated the table in the Overview section to reflect handling of previously skipped Event Types.|
|Dec-10-2021||NA||Updated the screenshots to reflect the latest UI.|
|Aug-09-2021||1.69||- Created a table to illustrate how Auto Mapping handles different data scenarios.
- Revised the content to highlight how Hevo handles deduplication if primary keys are not present in the Destination tables.