Data Pipelines

A data Pipeline, or Pipeline in Hevo, is a no-code data processing framework that loads data from any Source such as a database, SaaS application, or file into a Destination database or data warehouse, with built-in transformation and schema mapping capabilities.

For example, you can load data from your Facebook Ads account into a Google BigQuery data warehouse for analysis, without writing any code.

---

Pipelines in Hevo

Hevo Pipelines are designed for reliability and ease of use. Once configured, a Pipeline continuously ingests data from the Source, applies any configured Transformations, maps it to the Destination schema, and loads it — with no manual intervention required. You can view samples of incoming data in real time as it loads from your Source into your Destination.

Each Pipeline connects one Source to one Destination. However, the same Destination database or data warehouse can receive data from multiple Sources through separate Pipelines.

---

Pipeline Components

A Hevo Pipeline consists of the following components:

• Source: The database, API endpoint, or file storage system that holds the data you want to replicate. Hevo integrates with 150+ Sources including databases, SaaS applications, and cloud storage systems. Read Sources.

• Transformations: Python or drag-and-drop UI-based Transformations that let you clean, enrich, filter, or reshape data before it is loaded into your Destination. Transformations can be tested before being applied to live data. Read Transformations.

• Schema Mapper: Automatically detects and maps your Source schema to tables in your Destination. You can also configure mappings manually when needed. Read Schema Mapper.

• Destination: The data warehouse or database where the replicated data is stored and made available for analysis. Read Destinations.

---

Benefits of Creating Pipelines Using Hevo

Following are some of the benefits of creating Pipelines using Hevo:

• Near Real-Time Data Transfer: Hevo ingests and loads data continuously, making it available for analysis with minimal latency.

• 100% Complete and Accurate Data Transfer: Hevo tracks ingestion progress and handles failures automatically to ensure data is not lost or duplicated during replication.

• Scalable Infrastructure: Hevo supports integration with leading data warehouses and databases including Google BigQuery, Amazon Redshift, Snowflake, Amazon S3, MySQL, MongoDB, DynamoDB, and PostgreSQL, among others.

• Live Monitoring: You can monitor data movement across all stages of a Pipeline in real time. Hevo provides Data Ingestion Statuses and Pipeline Progress views, and supports Data Spike Alerts to notify you when Event consumption exceeds a defined threshold. Read Data Ingestion Statuses and Viewing Pipeline Progress.

• Transformations: Hevo provides built-in Transformation functions including Date and Control Functions, JSON manipulation, and Event Manipulation. Custom Python Transformations can also be configured and tested before being applied. Read Transformations.

• Schema Management: Hevo automatically detects the schema of incoming data and maps it to the Destination schema. When the Source schema changes, Hevo detects the change and updates the Destination schema accordingly.

---

Revision History