Kafka Confluent Cloud

Prerequisites

• An active Confluent Cloud account from which data is to be ingested exists.

• One or more bootstrap servers are available in Kafka Confluent Cloud.

• The API key and secret are available in Kafka Confluent Cloud.

• You are assigned the Team Administrator, Team Collaborator, or Pipeline Administrator role in Hevo to create the Pipeline.

---

Obtain the Bootstrap Server Information

Perform the following steps to retrieve the Bootstrap server:

1. Log in to your Kafka Confluent Cloud account.

2. In the left navigation pane, click Environments and then click <Environment name>.

   

3. In the <Environment name> page, select the Clusters tab and click <Cluster name>.

   

4. In the left navigation pane, under Cluster Overview, click Cluster Settings.

   

5. In the Cluster settings page, copy the Bootstrap server endpoint and save it securely like any other password. Use this endpoint while configuring your Hevo Pipeline.

   

---

Create the API Key and Secret

Perform the following steps to create the API credentials (API key and secret):

1. Follow steps 1-3 from the above section.

2. In the left navigation pane, under Cluster Overview, click API Keys.

   

3. In the API keys page, click Create key.

   

4. In the Create key page, do the following:

   1. In the Access control tab, select Granular access scope and click Next.

      

   2. In the Service account tab, specify the following to create a service account and click Next.

      Note: ACLs are required to assign permissions for each category of the API key.

      

      • New service account name: A unique name for your service account.

      • Description: A brief description of your service account.

   3. In the Add ACLs to service account tab, specify the Cluster, Consumer group, and Topic ACLs from the table below the image :

      Note: You need to click + Add ACLs to define a new ACL.

      

      • Cluster API

        ACL Category	Operation	Permission
        Cluster	DESCRIBE	ALLOW

      • Consumer Group ACLs

        ACL Category	Consumer Group ID	Pattern Type	Operation	Permission
        Consumer Group	hevo-integration	PREFIXED	DESCRIBE	ALLOW
        Consumer Group	hevo-integration	PREFIXED	READ	ALLOW

      • Topic ACLs

        ACL Category	Topic Name	Pattern Type	Operation	Permission
        Topic	The topic for which you grant permission to Hevo. By default, all topics can be accessed.	LITERAL or PREFIXED. If the Topic Name specified is *, select LITERAL.	READ	ALLOW
        Topic	The topic for which you grant permission to Hevo. By default, all topics can be accessed.	LITERAL or PREFIXED. If the Topic Name specified is *, select LITERAL.	DESCRIBE_CONFIGS	ALLOW

   4. Click Next.

   5. In the Get your API key section, click the Copy icon corresponding to the Key and Secret to copy them, and save them securely like any other password.

      Note: Once you exit this screen, you cannot see the same API key and secret.

      

You can use these credentials while configuring your Hevo Pipeline.

---

Configure Kafka Confluent Cloud Connection Settings

Perform the following steps to configure Kafka Confluent Cloud as a Source in your Pipeline:

1. Click PIPELINES in the Navigation Bar.

2. Click + CREATE PIPELINE in the Pipelines List View.

3. In the Select Source Type page, select Kafka Confluent Cloud.

4. In the Configure your Kafka Confluent Cloud Source page, specify the following:

   

   • Pipeline Name - A unique name for your Pipeline, not exceeding 255 characters.

   • Bootstrap Server(s): The bootstrap server(s) extracted from Kafka Confluent Cloud.

   • API Key: The API key for your Kafka Confluent Cloud account.

   • API Secret: The API secret for your Kafka Confluent Cloud account.

   • Ingest Data From

     • All Topics: Select this option to ingest data from all topics. Any new topics that are created are automatically included.

     • Specific Topics: Select this option to manually specify a comma-separated list of topics. New topics are not automatically added in this option.

     • Topics Matching a Pattern (Regex): Select this option to specify a Regular Expression (regex) to match and select the topic names. This option also fetches data for new topics that match the pattern dynamically. You can test your regex patterns here.

   • Additional SSL Settings: (Optional) Enable this option if you are using a custom Certificate Authority (CA).

     • CA File: The file containing the CA of the SSL server.

5. Click TEST & CONTINUE.

6. Proceed to configuring the data ingestion and setting up the Destination.

---