How do I filter out specific fields before loading data to the Destination?

Last updated on Jun 12, 2023

You can filter specific fields of an Event before the Destination table gets created for the corresponding Event Type, by using any one of the following:

Note: Event Types from the Source are mapped to Tables in your Destination, while fields are mapped to the table columns.

Prerequisites

To filter columns using Python-based Transformations:

  1. Create the Pipeline with Auto Mapping disabled.

  2. Click Transformations to access the Python-based Transformation interface.

    Python based Transformation interface

  3. Write the following code in the CODE console of the Transformation:

     from io.hevo.api import Event
    
     def transform(event):
    
     properties = event.getProperties()
     #list of columns to be deleted, below is the sample for same
     del_columns=["item","qyt"]
     for i in properties.keys():
         if i in del_columns:
             del properties[i]
    
     return event
    
    
  4. Click GET SAMPLE and then click TEST to test the transformation.

  5. Click DEPLOY.

  6. Once the Transformation is deployed, enable Auto Mapping for the Event Type from the Schema Mapper. Auto Mapping automatically maps Event Types and fields from your Source to the corresponding tables and columns in the Destination excluding the filtered columns.

Read more at Python Code-Based Transformations.

To filter columns using Drag and Drop Transformations:

  1. Create the Pipeline with Auto Mapping disabled.

  2. Click Transformations to access the Python-based Transformation interface.

  3. In the Python-based Transformation interface, click ENABLE DRAG-AND-DROP INTERFACE.

    Drag-and-Drop Interface

  4. Drag the Drop Fields Transformation block to the canvas.

    Select Drop Fields

  5. Specify the Event Type from which you want to drop the fields. Alternatively, skip this filter to apply the transformation to all Event Types.

  6. Select the Field filters to specify the fields to be dropped.

  7. Click DEPLOY.

  8. Once the Transformation is deployed, enable Auto Mapping for the Event Type from the Schema Mapper. Auto Mapping automatically maps Event Types and fields from your Source to the corresponding tables and columns in the Destination excluding the filtered columns.

Read more at Drag and Drop Transformations.

To filter columns using Schema Mapper:

  1. Once the Pipeline gets created, in the Schema Mapper, select the required Event Type.

  2. Click CREATE TABLE & MAP.

    Create table and map option

  3. Deselect the columns that you want to exclude from the Destination table.

    Deselecting the columns to filter

  4. Specify the Destination Table Name and click CREATE TABLE & MAP.

  5. Once the table is created, enable Auto Mapping for the Event Type from the Schema Mapper. Auto Mapping automatically maps Event Types and fields from your Source to the corresponding tables and columns in the Destination excluding the filtered columns.

Read more at Schema Mapper.

The Destination table is created with the remaining columns, and the data is replicated into this table as per the Pipeline schedule.


Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Nov-07-2022 NA Created as a new document.

Tell us what went wrong