Why is the historical data not getting ingested?

In case of API-based Sources, the historical data ingestion may sometimes halt if the API rate limit for fetching data from the Source is exceeded. The rate limit defines the number of requests handled by the Source’s API (Application Program Interface) in a defined time period. It is calculated across all the Pipelines created with that Source for an account.

The issue is observed mostly with Sources that have low API rate limits, and it automatically gets resolved the next day, as most rate limits are defined for a 24 hour period. You can also try reducing the ingestion frequency to reduce the probability of such issues. For example, if the Pipeline is set to ingest data every 15 minutes, the API is used every 15 minutes and reaches its limit sooner than if the ingestion were to happen every 3 hours.

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Nov-07-2022 NA Created as a new document.
Last updated on 04 Nov 2022

Tell us what went wrong