Setting up MongoDB replication for Oplog and Change Streams

Hevo supports data ingestion for replication from MongoDB via OpLog and Change Streams. Change Streams internally uses OpLog for replication. To enable OpLog or Change Streams you will have to do the following changes.

  1. Changes to be made in the MongoDB server configuration: MongoDB configuration file i.e. mongod.conf is generally found in /etc/ directory in a Linux system. The options to configure are as follow.
    • replication.replSetName: The replica set this MongoDB is part of.
    • replication.oplogSizeMB: The max size of logs that mongo will persist keep this enough high we recommend 2048 MB (2GB).
    • net.bindIp: The IP this MongoDB server should listen to.
    • An example config will look like as follows.

      net:  
       bindIp: 0.0.0.0  
      replication:  
       replSetName: "repSet0"  
       oplogSizeMB: 2048
      
  2. Configuring replication through Mongo shell: Open your Mongo shell on the replication server and run the following commands
    • rs.initiate(): This command will initialize the replica set.
    • rs.conf(): This command will show you the replication configuration that has been set.
    • rs.status(): This command will show you the replication status.
  3. Allowing Hevo user to read mongo OpLog: In your Mongo shell run the following commands.
    • use <your admin db>
      
    • db.grantRolesToUser('<Hevo user>',[{ role: "read", db: "<db to replicate>" }])
      

      If you want to enable Historical Load, run the following for every db required.

    • db.grantRolesToUser('<Hevo user>',[{ role: "read", db: "<required db>" }])
      
  4. Allowing Hevo user to read Mongo Change Stream: In your Mongo shell run the following commands.
    •  use <your admin db>
      
    •  db.grantRolesToUser('<Hevo user>', [{role: "read", db: "local"}])
      

      If you want to load all databases, run the following command.

    •  db.grantRolesToUser('<Hevo user>', [{role: "readAnyDatabase", db: "admin"}])
      

      If you only want to use one database, run the following command for the required db.

    •  db.grantRolesToUser('<Hevo user>',[{ role: "read", db: "<required db>" }]
      
Last updated on 06 Aug 2020