Apify

Last updated on Sep 03, 2024

Apify is an online platform that provides custom solutions, such as automation of data extraction from websites and its related processes, and create bots that can perform recurring actions for you. It also allows you to integrate multiple platforms, such as Google Sheets, Github, Slack, and many other APIs and webhooks.

You can replicate the data from your Apify account to a Destination database or data warehouse using Hevo Pipelines. Apify uses an API token to identify Hevo and authorize the request for accessing account data. Hevo ingests the data in Full Load mode. Refer to section, Data Model for the list of supported objects.


Source Considerations

  • Pagination: Each API response for an Apify object fetches one page with up to 1000 records except the Actor Versions object, for which fewer number of records may be fetched.

  • Rate Limit: Apify imposes a limit of 30 API calls per second. If the limit is exceeded, Hevo defers the ingestion till the limits reset. Read API Rate Limits to configure a suitable ingestion frequency for your Pipeline.


Limitations

  • Hevo currently does not support deletes. Therefore, any data deleted in the Source may continue to exist in the Destination.

  • You cannot specify a duration for loading the historical data. Hevo loads the entire data present in your Apify account.


Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Mar-05-2024 2.21 Updated the ingestion frequency table in the Data Replication section.
Sep-22-2023 NA Updated the page contents to reflect the latest Apify user interface (UI).
Jan-23-2023 2.06 New document.

Tell us what went wrong