Pipelines

A Pipeline moves your data from a Source system to a Destination database or data warehouse. It has the following components:

• Source: A Source can be a database, a SaaS-based application (an API endpoint) or a file storage which has the data that you want to analyze. Hevo integrates with a variety of Sources. Read Sources.

• Transformations: Transformations are useful when you want to clean, enrich, or transform your data before loading it to your Destination. Read Transformations.

• Schema Mapper: Schema Mapper helps you map your Source schemas to tables in your Destination. Read Schema Mapper.

• Destination: Destination is a data warehouse or database where the data fetched from a Source is loaded. Read Destinations.

You can connect only one Source and Destination in a Pipeline. However, multiple Pipelines may load to the same Destination.

• Articles in this section
  • Data Flow in a Pipeline
  • Familiarizing with the Pipelines UI
  • Working with Pipelines
    • Best Practices for Creating Database Pipelines
    • Creating a Pipeline
      • Draft Pipelines
    • Scheduling a Pipeline
    • Modifying a Pipeline
    • Prioritizing a Pipeline
    • Viewing Pipeline Progress
    • Troubleshooting Data Replication Errors
    • Pausing and Deleting a Pipeline
    • Log-based Pipelines
      • Pausing a Log-based Pipeline
      • Handling Deletes in Log-based Pipelines
  • Pipeline Objects
    • Managing Objects in Pipelines
  • Transformations
    • Python Code-Based Transformations
      • Examples of Python Code-based Transformations
        • Adding Time to a Timestamp Value
        • Splitting an Event into Multiple Event Types
        • Splitting Multiple Values in a Key into Separate Events
        • Splitting Nested Events into Multiple Events
      • API References
        • Event Object
        • TimeUtils
        • Utils
    • Drag and Drop Transformations
      • Special Keywords
      • Transformation Blocks and Properties
        • Add a Field
        • Change Datetime Field Values
        • Change Field Values
        • Drop Events
        • Drop Fields
        • Find & Replace
        • Flatten JSON
        • Format Date to String
        • Format Number to String
        • Hash Fields
        • If-Else
        • Mask Fields
        • Modify Text Casing
        • Parse Date from String
        • Parse JSON from String
        • Parse Number from String
        • Rename Events
        • Rename Fields
        • Round-off Decimal Fields
        • Split Fields
      • Examples of Drag and Drop Transformations
    • Effect of Transformations on the Destination Table Structure
    • Transformation Reference
      • Datetime Field Values
      • Operators
      • Rounding Modes
      • Text Casing
      • Values and Formulas
    • Transformation FAQs
      • How does Transformation affect the structure of the Destination table?
      • Why do I not see the newly ingested Source data in the sample Event?
      • Can I import standard Python libraries in Transformation?
  • Schema Mapper
    • Using Schema Mapper
    • Mapping Statuses
    • Auto Mapping Event Types
      • Modifying Schema Mapping for Auto Mapped Event Types
    • Mapping a Source Event Type with a Destination Table
    • Mapping a Source Event Type Field with a Destination Table Column
    • Schema Mapper Actions
    • Fixing Unmapped Fields
    • Resolving Incompatible Schema Mappings
    • Resizing String Columns in the Destination
    • Schema Mapper Compatibility Table
  • Troubleshooting Failed Events in a Pipeline
    • Reasons for Event Failures
    • Resolving Event Failures
  • Mismatch in Events Count in Source and Destination
  • Pipeline FAQs
    • Does creation of Pipeline incur cost?
    • Can I have the same Source and Destination in the Pipeline?
    • Why are my new Pipelines in trial?
    • Can I connect tables from multiple Pipelines to a common Destination table?
    • Is my data stored after I delete a Pipeline?
    • What happens if I delete a Pipeline and re-create a similar one again?
    • When should I pause vs delete my Pipeline?
    • Why am I getting warnings while adding Pipelines?
    • Why is there a delay in my Pipeline?
    • Can I delete the skipped objects in a Pipeline so they don’t appear in the UI?
    • How do I disable Auto Mapping for my Pipeline?
    • How do I change the ingestion frequency for individual tables?
    • How do I change the query mode for Pipelines created with Table mode?
    • How does changing the query mode of a Pipeline in table mode affect data ingestion?
    • Why is my billable Events count so high even though I selected Delta Timestamp as the query mode?
    • How does the timing of Change Schedule work?
    • Can I drop multiple Destination tables in a Pipeline at once?
    • Does triggering ingestion using Run Now affect the scheduled ingestion frequency of the Pipeline?
    • Will pausing the ingestion of some objects increase the overall ingestion speed of the Pipeline?
    • Can I alphabetically sort Event Types listed in the Schema Mapper?
    • How do I include new tables in the Pipeline?
    • Can I see the historical load progress for my Pipeline?
    • My Historical Load Progress is still at 0%. What does it mean?
    • Why is the historical data not getting ingested?
    • How do I restart the historical load for all the objects at once?
    • How do I set a field as a primary key to avoid duplication?
    • How can I load only filtered Events from the Source to the Destination?
    • How can I make sure that each record is loaded only once?
    • Why is there a mismatch in the count of Events in the Source and the Destination even though I selected the Unique and Incrementing query mode?
    • How can I transfer Excel files using Hevo?
    • How can I load an XML file from an S3 folder using Hevo?