Transformations

Last updated on Oct 06, 2023

The data received from the Source system might, at times, not be suitable to be replicated to the Destination database or data warehouse as is. Using Transformations in Hevo, you can prepare the data in multiple ways before loading it to the Destination. Read Data Transformation to know more about scenarios where you may need to transform your data.

Note: For any Transformation that is applied on an Event, a maximum of 10,000 child Events can be generated.


Prerequisites


Writing the Transformation

You can design and apply Transformations on Events and Event Types in two ways:

To change the Transformation method, click the More (More) icon in the Transformation canvas.

Transformations UI

Note: You cannot use both methods in the same Transformation. Also, Transformation of one type cannot be converted to the other automatically. If you switch from one method to the other, you need to rewrite the Transformation completely.


Testing the Transformation

Hevo retains the first few Events ingested from your Source as sample data, which enables you to test your Transformation code. Once you write or modify your Transformations, you can test them against these real sample Events.

To test your Transformation script:

  1. In the Transformations UI, click the Select Event Type drop-down, and select the Source Event Type on which you want to test the Transformation.

    Select Event Type

  2. Click GET SAMPLE. You receive an Event from the sample data that Hevo stores. No changes are applied till you deploy the Transformation.

  3. Click TEST to test the Transformation and view the results.

    Sample Event

Sometimes, if the data in the Source changes, you may not see sample Events for it immediately. Or, you may want to test an edge-case scenario but do not have a sample Event for it. In that case, you can paste a sample Event containing the changes and test your Transformation.

You can also test the Transformation on a random sample Event not specific to any Event Type.

Testing Transformations on failed Events

You can test any Transformation you have written to resolve failed Events directly within the Transformation page.

To do this:

  1. Click the Select Event Type drop-down.

  2. Enable the Events not Loaded option to view only the Event Types with failed Events.

  3. Select the required Event Type and click GET SAMPLE.

  4. Click TEST.

View sample failed Events


Deploying the Transformation

Once you have tested the Transformation, click DEPLOY. The Transformation is applied to all applicable incoming Events.

Deploy transformation

You can click RESET to return to the last deployed stage.

We would advise you to monitor the Failed Events for any failures due to the newly deployed code.


Transformation Version History

Click the More (More) icon to view the history of all deployed Transformations.

You can restore any past deployed Transformation by selecting it and clicking RESTORE THIS VERSION. The version is restored in Draft state. You must click DEPLOY to finally deploy it.

It is advised that you test the restored Transformation before you deploy it. The system warns you if you try to directly deploy it.




Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Oct-25-2022 NA Updated section, Testing the Transformation for clarity on changed Source data and sample Events.
Apr-25-2022 NA Updated the screenshots to reflect the latest UI.
Jan-24-2022 NA Updated the screen shots related to the Python version.
Sep-20-2021 NA Added a note in the overview section about the maximum number of child Events that can be created from an Event using a Transformation.
Mar-23-2021 1.59 - Added the section Testing Transformations on Failed Events.
- Added the bullet point Successful ingestion of Failed Events in the page overview.

Tell us what went wrong