Metadata Column __hevo_source_modified_at
On This Page
For Pipelines created with some Sources, MongoDB (OpLog and Change Streams mode) and MySQL (BinLog mode), Hevo captures the time when an Event was created, modified, or deleted, using the
__hevo_source_modified_at metadata field. This is a timestamp value extracted from the Source. You can use this information to filter Events for loading or applying transformations. The Sources for which this feature is currently available include:
MongoDB (OpLog and Change Streams Pipeline mode)
MySQL (BinLog Pipeline mode)
Amazon DynamoDB (DynamoDB Streams and Kinesis Data Streams Pipeline modes).
_hevo_source_modified_at field contains the timestamp of the update in the form of number of milliseconds since January 1, 1970, 00:00:00 GMT.
This field is not available for mapping in the Schema Mapper. However, you can access it in Transformations using the
_hevo_source_modified_at Timestamp to Transform Events
from io.hevo.api import Event from datetime import datetime def transform(event): modifiedTs = event.getSourceModifiedAt() properties = event.getProperties() # get particular time in ms format startDate = datetime(2021, 1, 1) epoch = datetime.utcfromtimestamp(0) startTsInMillis = (startDate - epoch).total_seconds() * 1000.0 # Add a new field to the event based on modifiedTs if modifiedTs is None : properties['employeeInfoUpdate'] = 'Not Specified' elif modifiedTs >= startTsInMillis : properties['employeeInfoUpdate'] = 'New' else : properties['employeeInfoUpdate'] = 'Old' return event
The above transformation uses the
event.getSourceModifiedAt() method to retrieve the time when the Event was modified in the Source. Next, it compares this value against the
startTsInMillis field, which is the date 1 Jan, 2021 in Millisecond format. Based on the result of the comparison, the appropriate value is assigned to the new field
modifiedTs is not present, then the new field gets the value Not Specified. Else, it is assigned the value Old or New if it is earlier or later than 1 Jan, 2021, respectively.
Refer to the following table for the list of key updates made to this page:
|Date||Release||Description of Change|
|Jun-01-2021||1.64||Updated the overview text to indicate this metadata field is now available for Amazon DynamoDB Source also.|
|May-18-2021||1.63||Updated the overview text to indicate this metadata field is now available for MongoDB Sources also.|