Generic MySQL (Edge)

Prerequisites

• The MySQL server (not a local host) is running. To check, enter the following command in your MySQL instance:

  service mysql status
  

  The status must show as Active: active (running).

• MySQL version is 8.0 or higher. To check this, enter the following command in your MySQL instance:

  mysql -V
  

• Binary Log (BinLog) replication is enabled.

• Hevo’s IP address(es) for your region is added to your MySQL database IP Allowlist.

• The SELECT, RELOAD, SHOW DATABASES, and REPLICATION privileges are granted to the database user.

  Note: We recommend that you create a database user for configuring your MySQL Source in Hevo. However, if you already have one, refer to section Grant privileges to the user.

Perform the following steps to configure your Generic MySQL Source:

Set up MySQL Binary Logs for Replication

A binary log is a collection of log files that records information about data modifications and data object modifications made on a MySQL server instance. Typically binary logs are used for data replication and data recovery.

Hevo supports data ingestion for replication from MySQL servers via binary logs (BinLog). For this, binary logging must be enabled on your MySQL server. You can do this via the MySQL server configuration file or via server startup options to mysqld.

Follow these steps to set up BinLog replication:

1. Check if BinLog replication is already enabled

1. Access the MySQL database you want to check for BinLog activity:

    mysql -h hostname -u user -p database
   

2. Open a secure shell:

    ssh user@localhost
   

3. Enter the command:

   select @@log_bin;
   

   If this statement returns a value of 1, BinLog is active. If value returned is 0, this means that BinLog is disabled. To enable it, follow the steps below.

2. Enable BinLog replication

1. Log in to your MySQL server instance.

2. Check your MySQL Server configuration:

   sudo nano /etc/mysql/my.cnf
   

   This may be sudo nano /etc/my.cnf in some cases.

3. In the config file, ensure the following configurations are specified. If not specified, add them now.

    [mysqld]
   
    binlog_format=ROW
   
    binlog_row_image=FULL
   
    expire_logs_days=3  -- The retention period (`expire_log_days`) can also be set in seconds by using the command: `binlog_expire_logs_seconds=259200`
   
    log_bin=mysql-binlog  -- For ubuntu, use: `/var/log/mysql/mysql-bin.log`
       
    server-id=1 -- (only in the case of ubuntu) 
       
    log_slave_updates=1
   
    max_allowed_packet=16777216 -- The SQL statement that the server can accept from a client. It is specified in bytes.
   
    gtid_mode=ON
    
    enforce_gtid_consistency=ON
   

   Note:

   • The log_slave_updates setting is required only if you are connecting a read replica. When it is set to 1, the replica logs any updates received from the main database, maintaining a record of those changes in its log.

   • Enabling Global Transaction Identifiers (GTIDs) is recommended because it simplifies replication by uniquely identifying transactions. This makes it easier to track them and ensures that the primary and replica servers are in sync.

4. Restart the MySQL server using the following command:

    service mysql restart
   

5. After restart, log in to the MySQL server to check BinLog again:

    SELECT @@log_bin;
   

   The value returned is now 1, indicating that BinLog is active.

   Note: The retention period should ideally be at least 72 hours (3 days). This helps Hevo ensure that no log-file is missed from being read, specially when historical data loading is enabled.

6. Run the following command to check the current value of binlog_row_value_options variable:

      show global variables where variable_name = 'binlog_row_value_options';
   

7. If the value is set to PARTIAL_JSON, unset this variable by:

      set @@global.binlog_row_value_options="" ;
   

The replication reference guide on MySQL’s documentation portal provides a complete reference of the options available for replication and binary logging.

Allowlist Hevo IP addresses for your region

You need to allowlist the Hevo IP address for your region to enable Hevo to connect to your MySQL database. To do this:

1. Edit the MySQL server configuration:

    sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf
   

2. Scroll to [mysqld] and add:

    bind-address = 0.0.0.0
   

   Or

    bind-address = 10.2.7.152
   

   Check the Hevo IP address for your region.

3. Save the file.

Create a Database User and Grant Privileges

1. Create a database user (Optional)

2. Grant privileges to the user

The database user for Hevo requires the following privileges:

Connect to your MySQL database as an admin user with an SQL client tool, such as MySQL Workbench, and run the following script. These commands grant only the privileges required by Hevo to ingest data from your MySQL database.

# Grant Privileges to the Database User
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO <database_username>@'%';

# (Optional) View the grants for the user:
show grants for <database_username>@localhost;

# Finalize the User’s Permissions
FLUSH PRIVILEGES;

Note:

• Replace the placeholder values in the commands above with your own. For example, <database_username> with hevo.

• The SELECT, RELOAD, and SHOW DATABASES privileges are required only for the historical load.