MongoDB Atlas

On This Page

After you have selected MongoDB Atlas as the Source for creating the Pipeline, provide the connection settings and data replication details listed here in the Configure your MongoDB Atlas Source page. You can fetch the database settings from your MongoDB Atlas account.

You can connect to the MongoDB database in one of the following ways:

  • By specifying the individual connection fields such as database host, database port, username, and password.

  • By entering the connection URI to connect to your MongoDB replica set or sharded cluster.

    Connection URIs are of two types:

    • DNS Seedlist: This type of connection URI has the prefix mongodb+srv://.

      For example, mongodb+srv://Jerome:Hevo123@cluster0.t7l5k.mongodb.net/test.

      The +srv part indicates that the hostname corresponds to a DNS SRV record. The hostnames and port values for your MongoDB database are fetched from the DNS SRV record.

    • Standard Connection String: This type of connection URI has the prefix mongodb://. This contains a comma-separated list of host:port combinations.

      For example, mongodb://Jerome:Hevo123@cluster0-shard-00-00.t7l5k.mongodb.net:27017,cluster0-shard-00-01.t7l5k.mongodb.net:27017,cluster0-shard-00-02.t7l5k.mongodb.net:27017/test?replicaSet=atlas-uax6f3-shard-0&ssl=true&authSource=admin.

      In the above example, Jerome is the database user and Hevo123 is the password for the database user.

Prerequisites

Perform the following steps to configure your MongoDB Atlas Source:

Retrieve Database Settings

Perform the following steps to retrieve your MongoDB Atlas database settings:

  1. Log in to the MongoDB Atlas console.

  2. In the drop-down on the top left corner of the page, select the project whose data you want to replicate.

    Select project

  3. In the left navigation pane, under Data Storage, click Clusters.

    Click Clusters

    Check the MongoDB version of your cluster. Depending on the MongoDB version, your choices in the subsequent steps will vary.

  4. Click CONNECT.

  5. Click Connect using MongoDB Compass.

    Click Connect using MongoDB Compass

  6. Click I have MongoDB Compass.

    Note: Select this option even if you do not have MongoDB Compass. You do not need MongoDB Compass to configure MongoDB Atlas as a Source in Hevo.

    I have

  7. Choose your version of Compass:

    • If your MongoDB version is 3.6 or later, select any version of Compass.

      Note: For MongoDB v3.6 or later, the preferred option is 1.12 or later.

      1.12 or later

    • If your MongoDB version is earlier than 3.6, select 1.11 or earlier as the Compass version.

      1.11 or earlier

  8. Copy the connection string that appears in the code block and replace username with your database username and password with your database password. The connection string is determined by the Compass version you select, as displayed in the respective image.

    For example, in the following connection string for MongoDB Compass version as 1.11 or later, the username is replaced with jerome and the password with Hevo123:

    mongodb://jerome:Hevo123@cluster0-shard-00-00.t7l5k.mongodb.net:27017,cluster0-shard-00-01.t7l5k.mongodb.net:27017,cluster0-shard-00-02.t7l5k.mongodb.net:27017/test?replicaSet=atlas-uax6f3-shard-0&ssl=true&authSource=admin.

  9. In the Configure your MongoDB Atlas Source page in Hevo, specify the connection settings in one of the following ways:

    • Copy the modified connection string from Step 8 and paste it into the Connection URI field.

    • Specify the individual connection fields derived from the connection string.

      In the example above, the connection fields are:

      • Database Host: cluster0-shard-00-00.t7l5k.mongodb.net, cluster0-shard-00-01.t7l5k.mongodb.net, cluster0-shard-00-02.t7l5k.mongodb.net

        Note: Connection URIs of MongoDB Atlas contains mongodb.net at the end of the database host name.

      • Database User: jerome

      • Database Password: Hevo123

      • Database Port: 27017

Set up Permissions to Read MongoDB Atlas Databases

You need to assign the required permissions to the database user to access the different MongoDB Atlas databases.

To do this, you can create a custom role and assign it to the database user. Or, you can directly provide built-in MongoDB roles to the database user.

1. Create a custom role (optional)

To create a custom role:

  1. Log in to the MongoDB Atlas console.

  2. In the left navigation pane, under Security, click Database Access.

  3. In the Database Access page, click the Custom Roles tab.

    Add custom roles

  4. Click +ADD NEW CUSTOM ROLE.

  5. In the Edit Custom Role page, specify the following:

    • If the Pipeline mode is Change Streams:

      • Assign read role or the listCollections action on the local database.

      • Assign read role or the find and ChangeStream actions on the database(s) you want to ingest.

      • Assign readAnyDatabase role or the listDatabases action on the admin database. This is mandatory if you want to load data from all the databases; indicated by selecting the Load All Databases option during MongoDB Atlas Source configuration.

    • If the Pipeline mode is OpLog:

      • Assign read role or the find, listCollections, and listIndexes actions on the database(s) you want to ingest.

      • Assign read role on the local database.

  6. Click Add Custom Role.

2. Create a database user

To create a database user:

  1. Log in to the MongoDB Atlas console.

  2. In the left navigation pane, under Security, click Database Access.

  3. In the Database Access page, click the Database Users tab.

  4. Click + ADD NEW DATABASE USER.

    Set password

  5. Under Authentication Method, click Password.

  6. Under Password Authentication, enter the new username and password.

  7. Specify the following:

    • For custom roles created in the previous step:

      • In the Database User Privileges drop-down, select Select pre-defined custom role under Advanced Privileges.

    • For built-in MongoDB roles:

      1. In the Database User Privileges drop-down, select Grant specific privileges.

      2. Assign read role on the local database.

      3. Do one of the following:

        • Assign readAnyDatabase on the admin database. This is mandatory if you want to load data from all the databases; indicated by selecting the Load All Databases option during MongoDB Atlas Source configuration.

        • Assign individual read actions on the database(s) you want to ingest and listDatabases action on the admin database.

  8. Click Add User.

Note: After you create or modify a user in the MongoDB Atlas console, you must wait five minutes for the changes to take effect.

Whitelist Hevo’s IP Addresses

Access your MongoDB Atlas console and:

  1. Click Network Access.

  2. In the IP Whitelist tab, click Add IP Address.

    Whitelisting HEVO IP Addresses in MongoDB Atlas

  3. Provide the Hevo IP address you want to whitelist.

    Adding Hevo's IP addresses

    Note: To provide all IPs with access, enter 0.0.0.0.

  4. Click Confirm.

Configure MongoDB Atlas Connection Settings

In the Configure your MongoDB Atlas Source page, specify the following:

Configure Atlas as a Source

  • Pipeline Name: A unique name for the Pipeline, not exceeding 255 characters.

  • General Connection Settings:

    • Paste Connection String:

      • Connection URI: The unique identifier for connecting to a MongoDB replica set or a sharded cluster.

    • Enter Connection Fields Manually:

      • Database Host: The MongoDB DNS name fetched from your MongoDB Atlas account.

        Note: To connect to a sharded cluster, specify only the parent cluster name. For example, if your host is the sharded cluster, cluster0-shard-00-00.t7l5k.mongodb.net,cluster0-shard-00-01.t7l5k.mongodb.net,cluster0-shard-00-02.t7l5k.mongodb.net, the parent hostname would be cluster0.t7l5k.mongodb.net. Hevo automatically extracts all the shards of the cluster.

      • Database User: The authenticated user who has the permissions to read collections in your database. Read Setting up Permissions to Read MongoDB Atlas Databases

        Note: It is recommended that only read-only permissions be provided to the user.

      • Database Password: The password for the database user.

  • Select an Ingestion Mode: The desired mode by which you want to ingest data from the Source:

    • OpLog: Data is polled using MongoDB’s OpLog. The OpLog is a collection of individual, transaction-level details which help replicas sync data from the primary instance.

    Note: OpLogs are present in data/standalone primary instances and replicas.

    • Change Streams: MongoDB’s Change Streams enable applications to stream real-time data changes without the complexity and risk of tailing the OpLog, for a single collection, a database, or an entire deployment. Change Streams are supported for all MongoDB configurations. However, for the clustered configuration, Change Streams works only if set up against a shard router (mongos).

    By default, in both Change Streams and OpLog mode, the data is read at the mongo instance level. To read data at the database level in Change Streams mode, disable the Load All Databases option while creating the Pipeline.

  • Connection Settings:

    • Connect through SSH: Enable this toggle option to connect to Hevo using an SSH tunnel, instead of directly connecting your MongoDB Atlas database host to Hevo. This provides an additional level of security by not exposing your MongoDB setup to the public. Read Connecting Through SSH.

    If this option is disabled, you must whitelist Hevo’s IP addresses to allow Hevo to connect to your MongoDB host.

  • Advanced Settings:

    • Load All Databases: If enabled, Hevo fetches data from all the databases you have access to on the selected host. Ensure that you have assigned the readAnyDatabase role or the listDatabases action on the admin database. Refer to Step 5 of section, Create a custom role above.

      If disabled, provide the database name to fetch data from. In the case of OpLog Ingestion mode, you can specify a comma-separated list of database names.

    • Merge Collections: If enabled, collections with the same name across different databases are merged into a single Destination table. If disabled, separate tables are created and prefixed with the respective database name. Read Example - Merge Collections Feature.

    • Load Historical Data: If disabled, Hevo loads data written in your database after the time of creation of the Pipeline. If enabled, the entire table data is fetched during the first run of the Pipeline.

    • Include New Tables in the Pipeline:

      If enabled, Hevo automatically ingests data from tables created in the Source after the Pipeline has been built. These may include completely new tables or previously deleted tables that have been re-created in the Source.

      If disabled, new and re-created tables are not ingested automatically. They are added in SKIPPED state in the objects list, in the Pipeline Overview page. You can update their status to INCLUDED to ingest data.

      You can change this setting later.

Click TEST & CONTINUE to proceed for setting up the Destination.

Limitations

  • Hevo does not support configuring a standalone instance of MongoDB without a replica.

See Also

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Apr-21-2022 1.86 Updated section, Configure MongoDB Atlas Connection Settings.
Jan-03-2022 1.79 Updated the description of the Include New Tables in the Pipeline advance setting in the Configure MongoDB Atlas Connection Settings section.
Jul-12-2021 1.67 Added the field Include New Tables in the Pipeline under Source configuration settings.
Jun-28-2021 1.66 - Updated the page overview section.
- Updated the section Retrieve Database Settings to include latest steps.
- Updated the section Set up Permissions to Read MongoDB Atlas Databases to include latest steps.
- Updated the section Configure MongoDB Atlas Connection Settings to include the option to connect to the MongoDB Atlas database using connection string.
- Added section, Limitations.
Last updated on 21 Apr 2022
Was this page helpful?