Best Practices to Avoid Reaching Source API Rate Limits (Edge)

Pipelines created for SaaS Sources use the APIs provided by each Source to fetch data. However, frequent requests to the APIs may exceed the Source API rate limits, leading to delays in data ingestion. To avoid this, follow these best practices to minimize API consumption and prevent disruption in data ingestion due to rate limits:

• Exclude any Source objects that are not required for data analysis from your Pipeline. You can exclude them during Pipeline creation or skip them after the Pipeline is created.

  Advantage: Fewer objects reduce the number of API calls made, resulting in faster ingestion.

• Reduce the sync frequency if near real-time updates are not critical.

  Advantage: Decreases the number of API requests made in a given time frame, minimizing the risk of exceeding rate limits.

• Increase the API quota in your Source if supported by the Source API provider. You can do this by upgrading to a higher subscription plan or contacting the provider.

  Advantage: A higher API quota allows your Pipelines to make more requests without triggering rate limit errors.

• Use separate API credentials for different Pipelines whenever possible.

  Advantage: Distributes the API load across multiple credentials, reducing the likelihood of exceeding rate limits on a single credential.