Google Cloud MySQL (Edge)

Prerequisites

• The Google Cloud MySQL instance (not localhost) is running. To check this, access your Google Cloud MySQL server and look for an Active indication next to the instance name.

  

• The MySQL version is 8.0 or higher. You can select the MySQL version while creating the instance.

• Binary Log (BinLog) replication is enabled.

• Hevo’s IP address(es) for your region is added to the Amazon Aurora MySQL database IP Allowlist.

• The SELECT, RELOAD, SHOW DATABASES, and REPLICATION privileges are granted to the database user.

• The username, password, database names, database hostname, and port number of the Source instance are available.

---

Perform the following steps to configure your Google Cloud MySQL Source:

Create a Read Replica (Optional)

---

Set up MySQL Binary Logs for Replication

A binary log is a collection of log files that records information about data modifications and data object modifications made on a MySQL server instance. Typically binary logs are used for data replication and data recovery.

Hevo supports data ingestion for replication from servers via binary logs (BinLog). For this, binary logging must be enabled on your Google Cloud MySQL server.

To do this:

1. Check if BinLog replication is already enabled

1. Access your Google Cloud MySQL instance.

2. In the left navigation pane, under Databases, select Backups.

3. Under Settings, check the status of Automated backups.

4. If the status is Disabled, provided below to enable BinLog replication. If the status is Enabled, ensure that the BinLog retention period is set to at least 3 days by following the steps provided below.

   

2. Enable BinLog replication and configure the Binlog retention period

1. In the left navigation pane, click Overview, and then click EDIT.

   

2. Scroll down to the Data Protection panel and select the Automated daily backups and Enable point-in-time recovery check boxes.

   

3. In the Flags panel, click ADD A DATABASE FLAG.

4. Click the Choose a flag field, and from the drop-down list, search and select the binlog_expire_logs_seconds flag.

5. Set the Value greater than or equal to 259200 (three days), and then click DONE.

   Note: Ensure that the Value of binlog_row_value_options flag is not set to PARTIAL_JSON.

   

6. Click SAVE.

This confirms that binary logging is now enabled for your Google Cloud MySQL instance.

The replication reference guide on MySQL’s documentation portal provides a complete reference of the options available for replication and binary logging.

3. Enabling GTIDs (Recommended)

Enabling GTID is recommended because it makes replication simpler and helps ensure that the primary and replica servers are in sync with each other.

1. Access your MySQL database server.

2. Run the following commands:

    SET GLOBAL enforce_gtid_consistency = ON;
    SET GLOBAL gtid_mode = OFF_PERMISSIVE;
    SET GLOBAL gtid_mode = ON_PERMISSIVE;
    SET GLOBAL gtid_mode = ON;
   

3. Confirm the changes:

    show global variables like '%GTID%';
   

---

Allowlist Hevo IP addresses for your region

You need to allowlist the Hevo IP address for your region to enable Hevo to connect to your Google Cloud MySQL database:

1. Access the Google Cloud MySQL Instances page and click the Instance ID that you want to use.

   

2. In the left navigation pane, click Connections.

   

3. On the Connections page, do the following:

   

   1. Go to the NETWORKING tab.

   2. Click the Public IP check box.

   3. Click the ADD A NETWORK and specify the following:

      • Name: A name for this connection. For example, all or Hevo IP address.

      • Network: The IP address of the site to grant access to. Specify 0.0.0.0/0 to authorize all sites or 10.2.7.152 to specifically whitelist Hevo’s IP address. Check the Hevo IP address for your region.

   4. Click DONE.

   This adds the IP address to the list of Authorized networks.

4. (Optional) Click Add A NETWORK to add another IP address.

5. Click SAVE.

---

Create a Database User and Grant Privileges

MySQL allows you to connect to Hevo after you configure your Google Cloud MySQL instance as an existing user or as a new user. To do this, select one of the following options:

Option 1. Configuring the user account using Google Cloud console

1. Click the primary Google Cloud MySQL instance.

   

2. In the left navigation pane, under PRIMARY INSTANCE, select Users, and then click + ADD USER ACCOUNT.

   

3. In the Add a user account to instance window, select one of the following methods to authenticate your user to access the instance:

   

   • Authentication method:

     • Built-in authentication: This uses a username and password to authenticate local database users

       • Host name: The host name determines which client hosts can connect to your instance with this user account. Default value: Allow any host (%).

     • Cloud IAM: This uses an IAM principal that provides access to the required resources in your Google Cloud MySQL instance.

4. Click ADD.

You will specify this username and password while creating your Hevo Pipeline.

Option 2. Configuring the user account using MySQL client

The database user specified in the Hevo Pipeline must have the following global privileges:

• SELECT - Select data from tables in your database which you want to replicate.

• RELOAD - Access to clear or reload internal caches, flush tables, or acquire locks.

• SHOW DATABASES - View list of database names in the server.

• REPLICATION CLIENT - Access to the MySQL server BinLog for replication.

• REPLICATION SLAVE - Access replication status and log details.

Note: The SELECT, RELOAD, and SHOW DATABASES privileges are only required during the historical load.

1. Connect to your Google Cloud MySQL database as a root user with an SQL client tool, such as MySQL workbench.

2. Create a database user:

   CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
   

   Note: We recommend creating a database user for connecting to your Google Cloud MySQL Source in Hevo. However, if you already have one, skip to step 3 below.

3. Grant the required privileges to the user:

   GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO '<username>'@'%';
   

   Note:

   • Replace the placeholder values in the commands above with your own. For example, <username> with hevo.

   • The REPLICATION SLAVE privilege is required only if you connect to a read replica. When it is granted to the authenticating user, the replica logs any updates received from the main database, maintaining a record of those changes in its log.

4. Finalize the user’s permissions:

   FLUSH PRIVILEGES;
   

---

Retrieve the Configuration Details (Optional)

---