Resolving Incompatible Schema Mappings
ON THIS PAGE
The Event Types and Events in the Source must be correctly mapped to the Destination tables and columns for the Pipeline to successfully load the Events from the Source into the selected Destination. For example, if a Source field of type String is mapped to a Destination field of type Int. the value of the Source field is not guaranteed to be compatible with an Int in all instances. For instance, a value like like “5e9” cannot be transformed to a number and will fail. See Schema Mapper Compatibility Table for the Source and Destination data types supported by Hevo.
If Hevo encounters any incompatibilities in the mappings between Source Events and the Destination tables and fields, it informs you about it in a number of ways.
Identifying Incompatible Schema Mapping
The count of the Event Types that have incompatible mapping is indicated on the Schema Mapper icon. In addition, a Warning icon is displayed next to the incorrectly mapped Event Types to inform users that there are unresolved issues in the schema mapping.
Click on any Event Type with a warning symbol to see the list of incompatible fields within it, in the right pane. The incompatibly mapped fields also have the Warning icon next to their name.
In the image below, the Source field is of type varchar (512) while the mapped field in the Destination is of type int, and therefore, a Warning icon is displayed.
You can hover over the Warning icon to view the details of the incompatibility and the courses of action available to you.
Resolving Mapping Errors
The warnings in Schema Mapper are meant to preemptively inform you about potentially incompatible mappings and help you identify and resolve them. If you are using an existing Destination table, you can do one of the following:
Proceed with these inconsistencies in the data as they are. For this, click APPLY CHANGES. Hevo displays a dialog box to confirm your action. Click CONFIRM to proceed.
- Skip the Event that has warnings. Again, a warning is displayed that the data for the field would not be replicated to the Destination table and would be lost. Note: Hevo does not retain the data for the skipped field anywhere.
- Select a compatible Destination field and type to fix the issue, and then click APPLY CHANGES.
Resolving Schema Mapping Incompatibilities with a New Table
If you choose to create a new Destination table during schema mapping, Hevo by default, creates the table columns of the correct and compatible field type. Unless you subsequently, manually select an incompatible field, no incompatibilities are expected in this scenario.
However, if any fields in the Source are reserved fields in Hevo, these cannot be mapped to the Destination table with the same name. For example, the field __hevo__ingested_at is a reserved field in Hevo. During table creation, this may get included as is in the Destination table creation and generate an error when you apply the mapping.
You need to modify the name in the mapped Destination field to something different, for example, __hevo__ingestion_time to resolve this issue. Since this is a manual step in an otherwise automated action, you need to explicitly confirm the change before it can be applied.
Handling Different Data Types in Source Data
During schema mapping, Hevo automatically promotes the data type of a Destination column so that it can accommodate maximum variations in the Source data type. This helps to losslessly hold existing and new data and avoid the sidelining of Events for data type mismatches.
For example, assume that initially a few Source Events are ingested with datatype as
long for a field. Based on that, Hevo creates a
long data type for the Destination column. Now, the next few values come in as
decimal for the same field. To accommodate these fields, Hevo alters the data type of the Destination column to a more relaxed one, which is
decimal, as now both
decimal values can be loaded.
Since this operation incurs cost and time, this is done only for tables smaller than 50 GB (subject to change) in size. Also, this alteration of data type is not done if a field is a primary key.
This feature is currently available for:
- New teams created in or after Hevo Release 1.56
- The BigQuery Destination
- Pipelines where AutoMapping is enabled
- Tables with size less than 50 GB
If the Destination table size increases beyond 50GB after such an alteration has been performed, as the Destination table has the more relaxed data type now, any data type changes that are already factored in are easily accommodated.
- The column order might change when we convert column data types.
- Any Models and Views that you may have created may get impacted if the Source data has multiple data types.