Metadata Column __hevo_source_modified_at

Last updated on Jun 12, 2023

For Pipelines created with some Sources, Hevo captures the time when an Event is created, modified, or deleted, using the __hevo_source_modified_at metadata field. This is a timestamp value extracted from the Source. You can use this information to filter Events for loading or applying transformations. The Sources for which this feature is currently available include:

  • Amazon DynamoDB (DynamoDB Streams and Kinesis Data Streams ingestion modes).

  • MongoDB (OpLog and Change Streams ingestion modes)

  • MySQL (BinLog ingestion mode)

  • File-based Sources: Drive, Amazon S3, GCS and FTP/SFTP. Prior to Release 1.66, __hevo_source_modified_at was available as a field that could be mapped via the Schema Mapper.

The _hevo_source_modified_at field contains the timestamp of the update in the form of number of milliseconds since January 1, 1970, 00:00:00 GMT.

This field is not available for mapping in the Schema Mapper. However, you can access it in Transformations using the event.getSourceModifiedAt() method.


Example: Using __hevo_source_modified_at Timestamp to Transform Events

from io.hevo.api import Event
from datetime import datetime
def transform(event):

    modifiedTs = event.getSourceModifiedAt()
    properties = event.getProperties()

    # get particular time in ms format
    startDate = datetime(2021, 1, 1)
    epoch = datetime.utcfromtimestamp(0)
    startTsInMillis = (startDate - epoch).total_seconds() * 1000.0


    # Add a new field to the event based on modifiedTs
    if modifiedTs is None :
        properties['employeeInfoUpdate'] = 'Not Specified'
    elif modifiedTs >= startTsInMillis :
        properties['employeeInfoUpdate'] = 'New'
    else :
        properties['employeeInfoUpdate'] = 'Old'

    return event

The above transformation uses the event.getSourceModifiedAt() method to retrieve the time when the Event was modified in the Source. Next, it compares this value against the startTsInMillis field, which is the date 1 Jan, 2021 in Millisecond format. Based on the result of the comparison, the appropriate value is assigned to the new field employeeInfoUpdate. If modifiedTs is not present, then the new field gets the value Not Specified. Else, it is assigned the value Old or New if it is earlier or later than 1 Jan, 2021, respectively.

Result:



Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Jun-28-2021 1.66 Added File-based Sources to the bullet list in the page overview.
Jun-01-2021 1.64 Updated the overview text to indicate this metadata field is now available for Amazon DynamoDB Source also.
May-18-2021 1.63 Updated the overview text to indicate this metadata field is now available for MongoDB Sources also.
May-05-2021 1.62 New document.

Tell us what went wrong