MongoDB Atlas

Configure MongoDB Atlas Connection Settings

Perform the following steps to configure MongoDB Atlas as a Source in Hevo:

1. Click PIPELINES in the Navigation Bar.

2. Click + CREATE PIPELINE in the Pipelines List View.

3. On the Select Source Type page, select MongoDB Atlas.

4. On the Configure your MongoDB Atlas Source page, specify the following:

   

   • Pipeline Name: A unique name for the Pipeline, not exceeding 255 characters.

   • General Connection Settings:

     • Paste Connection String:

       • Connection URI: The unique identifier for connecting to a MongoDB replica set or a sharded cluster.

         Note: If you are connecting to a sharded cluster, Hevo automatically extracts all the shards of the cluster.

     • Enter Connection Fields Manually:

       • Database Host: The MongoDB DNS name fetched from your MongoDB Atlas account.

         Note: To connect to a sharded cluster, specify only the parent cluster name. For example, if your host is the sharded cluster, cluster0-shard-00-00.t7l5k.mongodb.net, cluster0-shard-00-01.t7l5k.mongodb.net, cluster0-shard-00-02.t7l5k.mongodb.net, the parent hostname would be cluster0.t7l5k.mongodb.net. Hevo automatically extracts all the shards of the cluster.

       • Database User: The authenticated user who has permission to read collections in your database. Read Setting up Permissions to Read MongoDB Atlas Databases

         Note: It is recommended that only the read-only permission be provided to the user.

       • Database Password: The password for the database user.

   • Select an Ingestion Mode: The desired mode by which you want to ingest data from the Source. You can expand this section by clicking SEE MORE to view the list of ingestion modes to choose from. Default value: OpLog.

     

     The following ingestion modes are applicable to MongoDB Source:

     Note: If you are connecting to a sharded cluster, select Change Streams as the ingestion mode.

     • OpLog: Data is ingested using MongoDB’s OpLog. The OpLog contains individual, transaction-level details which helps replicas sync data from the primary instance.

       Note: OpLogs are present in data or standalone primary instances and replicas.

     • Change Streams: MongoDB’s Change Streams enables applications to stream real-time data changes without the complexity and risk of tailing the OpLog, for a single collection, a database, or an entire deployment. Change Streams are supported for all MongoDB configurations. However, for the clustered configuration, Change Streams works only if set up against a shard router (mongos).

       By default, in both Change Streams and OpLog modes, the data is read at the mongo instance level. To read data at the database level in Change Streams mode, disable the Load All Databases option while creating the Pipeline.

   • Connection Settings:

     • Connect through SSH: Enable this option to connect to Hevo using an SSH tunnel, instead of directly connecting your MongoDB Atlas database host to Hevo. This provides an additional level of security by not exposing your MongoDB setup to the public. Read Connecting Through SSH.

       If this option is disabled, you must whitelist Hevo’s IP addresses to allow Hevo to connect to your MongoDB host.

   • Advanced Settings:

     • Load All Databases: If enabled, Hevo fetches data from all the databases you have access to on the specified database host(s). Ensure that you have assigned the readAnyDatabase role or the listDatabases action on the admin database. Refer to Step 5 of section, Create a custom role above.

       If disabled, Hevo fetches the list of all the databases you have access to on the specified database host(s). From this list, you must select the databases you want to fetch data from.

       

       Warning! Once the Pipeline is created, if you drop any of these databases from your MongoDB Source, the Pipeline will fail.

     • Merge Collections: If enabled, collections with the same name across different databases are merged into a single Destination table. If disabled, separate tables are created and prefixed with the respective database name. Read Example - Merge Collections Feature.

     • Load Historical Data: If disabled, Hevo loads data written in your database after the time of creation of the Pipeline. If enabled, the entire table data is fetched during the first run of the Pipeline.

     • Include New Tables in the Pipeline:

       If enabled, Hevo automatically ingests data from tables created in the Source after the Pipeline has been built. These may include completely new tables or previously deleted tables that have been re-created in the Source.

       If disabled, new and re-created tables are not ingested automatically. They are added in SKIPPED state in the objects list, on the Pipeline Overview page. You can update their status to INCLUDED to ingest data.

       You can change this setting later.

5. Click TEST CONNECTION. This button is enabled once you specify all the mandatory fields. Hevo’s underlying connectivity checker validates the connection settings you provide.

6. Click TEST & CONTINUE to proceed for setting up the Destination. This button is enabled once you specify all the mandatory fields.

Data Replication

Note: The custom frequency must be set in hours as an integer value. For example, 1, 2, or 3 but not 1.5 or 1.75.

• Historical Data: In the first run of the Pipeline, Hevo ingests all available data for the selected objects from your Source database.

• Incremental Data: Once the historical load is complete, data is ingested as per the ingestion frequency.