Generic MySQL (Edge)

Prerequisites

• The MySQL server (not a local host) is running. To check, enter the following command in your MySQL instance:

  service mysql status
  

  The status must show as Active: active (running).

• The MySQL version is 5.7 or 8.0. To check this, enter the following command in your MySQL instance:

  mysql -V
  

• Binary Log (BinLog) replication is enabled.

  Note: Hevo currently does not support ingesting data from read replicas.

• Hevo’s IP address(es) for your region is added to your MySQL database IP Allowlist.

• The SELECT, RELOAD, SHOW DATABASES, and REPLICATION privileges are granted to the database user.

  Note: We recommend that you create a database user for configuring your MySQL Source in Hevo. However, if you already have one, refer to section Grant privileges to the user.

Perform the following steps to configure your Generic MySQL Source:

Set up MySQL Binary Logs for Replication

A binary log is a collection of log files that records information about data modifications and data object modifications made on a MySQL server instance. Typically binary logs are used for data replication and data recovery.

Hevo supports data ingestion for replication from MySQL servers via binary logs (BinLog). For this, binary logging must be enabled on your MySQL server. You can do this via the MySQL server configuration file or via server startup options to mysqld.

Follow these steps to set up BinLog replication:

1. Check if BinLog replication is already enabled

1. Access the MySQL database you want to check for BinLog activity:

    mysql -h hostname -u user -p database
   

2. Open a secure shell:

    ssh user@localhost
   

3. Enter the command:

   select @@log_bin;
   

   If this statement returns a value of 1, BinLog is active. If value returned is 0, this means that BinLog is disabled. To enable it, follow the steps below.

2. Enable BinLog replication

1. Log in to your MySQL server instance.

2. Check your MySQL Server configuration:

   sudo nano /etc/mysql/my.cnf
   

   This may be sudo nano /etc/my.cnf in some cases.

3. In the config file, ensure the following configurations are specified. If not specified, add them now.

    [mysqld]
   
    binlog_format=ROW
   
    binlog_row_image=FULL
   
    expire_logs_days=3  -- The retention period (`expire_log_days`) can also be set in seconds by using the command: `binlog_expire_logs_seconds=259200`
   
    log_bin=mysql-binlog  -- For Ubuntu, use: `/var/log/mysql/mysql-bin.log`
       
    server-id=1 -- (only in the case of Ubuntu) 
   
    max_allowed_packet=16777216 -- The SQL statement that the server can accept from a client. It is specified in bytes.
   
    gtid_mode=ON
    
    enforce_gtid_consistency=ON
   

   Note: Enabling Global Transaction Identifiers (GTIDs) is recommended because it simplifies replication by uniquely identifying transactions. This makes it easier to track them and ensures that the primary and replica servers are in sync.

4. Restart the MySQL server using the following command:

    service mysql restart
   

5. After restarting, log in to the MySQL server and do the following:

   1. Check the value of BinLog again by running the following command:

       SELECT @@log_bin;
      

      The value returned is now 1, indicating that BinLog is active.

      Note: The retention period should ideally be at least 72 hours (3 days). This helps Hevo ensure that no log-file is missed from being read, specially when historical data loading is enabled.

   2. Verify that the binlog_format setting is set to ROW by running the following command:

       SHOW VARIABLES LIKE '%binlog_format%';
      

      The expected output should be:

       +---------------+------+ 
       | Variable_name | Value| 
       +---------------+------+ 
       | binlog_format | ROW | 
       +---------------+------+
      

      Note: Hevo cannot replicate data using binary logs if the value is not set to ROW.

6. Run the following command to check the current value of binlog_row_value_options variable:

      show global variables where variable_name = 'binlog_row_value_options';
   

7. If the value is set to PARTIAL_JSON, unset this variable by:

      set @@global.binlog_row_value_options="" ;
   

The replication reference guide on MySQL’s documentation portal provides a complete reference of the options available for replication and binary logging.

Allowlist Hevo IP addresses for your region

You need to allowlist the Hevo IP address for your region to enable Hevo to connect to your MySQL database. To do this:

1. Edit the MySQL server configuration:

    sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf
   

2. Scroll to [mysqld] and add:

    bind-address = 0.0.0.0
   

   Or

    bind-address = 10.2.7.152
   

   Check the Hevo IP address for your region.

3. Save the file.

Create a Database User and Grant Privileges

1. Create a database user (Optional)

2. Grant privileges to the user

The database user for Hevo requires the following privileges:

Connect to your MySQL database as an admin user with an SQL client tool, such as MySQL Workbench, and run the following script. These commands grant only the privileges required by Hevo to ingest data from your MySQL database.

# Grant Privileges to the Database User
GRANT RELOAD, SHOW DATABASES, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO <database_username>@'%';

# (Optional) Grant Hevo access to all Tables in the Database
GRANT SELECT ON <database_name>.* TO <database_username>;

# Grant Hevo access to the Tables in the Database that you want to replicate
GRANT SELECT, SHOW VIEW ON <database_name>.<table_name> TO <database_username>;

# (Optional) View the grants for the user:
SHOW GRANTS FOR <database_username>@'%';

# Finalize the User’s Permissions
FLUSH PRIVILEGES;

Note:

• Replace the placeholder values in the commands above with your own. For example, <database_username> with hevo.

• The SELECT, RELOAD, and SHOW DATABASES privileges are required only for the historical load.