Metadata Column __hevo_source_modified_at
On This Page
For Pipelines created with some Sources, Hevo captures the time when an Event is created, modified, or deleted, using the __hevo_source_modified_at
metadata field. This is a timestamp value extracted from the Source. You can use this information to filter Events for loading or applying transformations. The Sources for which this feature is currently available include:
-
Amazon DynamoDB (DynamoDB Streams and Kinesis Data Streams Pipeline modes).
-
MongoDB (OpLog and Change Streams Pipeline Mode)
-
MySQL (BinLog Pipeline Mode)
-
File-based Sources: Drive, Amazon S3, GCS and FTP/SFTP. Prior to Release 1.66,
__hevo_source_modified_at
was available as a field that could be mapped via the Schema Mapper.
The _hevo_source_modified_at
field contains the timestamp of the update in the form of number of milliseconds since January 1, 1970, 00:00:00 GMT.
This field is not available for mapping in the Schema Mapper. However, you can access it in Transformations using the event.getSourceModifiedAt()
method.
Example: Using __hevo_source_modified_at Timestamp to Transform Events
from io.hevo.api import Event
from datetime import datetime
def transform(event):
modifiedTs = event.getSourceModifiedAt()
properties = event.getProperties()
# get particular time in ms format
startDate = datetime(2021, 1, 1)
epoch = datetime.utcfromtimestamp(0)
startTsInMillis = (startDate - epoch).total_seconds() * 1000.0
# Add a new field to the event based on modifiedTs
if modifiedTs is None :
properties['employeeInfoUpdate'] = 'Not Specified'
elif modifiedTs >= startTsInMillis :
properties['employeeInfoUpdate'] = 'New'
else :
properties['employeeInfoUpdate'] = 'Old'
return event
The above transformation uses the event.getSourceModifiedAt()
method to retrieve the time when the Event was modified in the Source. Next, it compares this value against the startTsInMillis
field, which is the date 1 Jan, 2021 in Millisecond format. Based on the result of the comparison, the appropriate value is assigned to the new field employeeInfoUpdate
. If modifiedTs
is not present, then the new field gets the value Not Specified. Else, it is assigned the value Old or New if it is earlier or later than 1 Jan, 2021, respectively.
Result:
Revision History
Refer to the following table for the list of key updates made to this page:
Date | Release | Description of Change |
---|---|---|
Jun-28-2021 | 1.66 | Added File-based Sources to the bullet list in the page overview. |
Jun-01-2021 | 1.64 | Updated the overview text to indicate this metadata field is now available for Amazon DynamoDB Source also. |
May-18-2021 | 1.63 | Updated the overview text to indicate this metadata field is now available for MongoDB Sources also. |
May-05-2021 | 1.62 | New document. |