Apache Kafka

Last updated on Sep 25, 2023

Prerequisites

The bootstrap server information is available in Apache Kafka.
The Certificate Authority (CA) file, the client certificate, and the client key are available, if Secure Sockets Layer (SSL) encryption is used.
Hevo’s IP addresses are whitelisted.

Note: Hevo recommends using Apache Kafka version 0.10.0.1 or higher.

You are assigned the Team Administrator, Team Collaborator, or Pipeline Administrator role in Hevo to create the Pipeline.

Perform the following steps to configure your Apache Kafka Source:

Locate the Bootstrap Server Information

To locate the bootstrap server information:

Open the server.properties file.
Note: The location of the server.properties file varies depending on the operating system being used. Contact your server administrator if you cannot locate this file.
Copy the entire line after bootstrap.servers: that contains the list of bootstrap servers and ports.

For example: Copy hostname1:9092, hostname2:9092 from the line below:
```
bootstrap.servers: hostname1:9092, hostname2:9092
```

Configure Secure Sockets Layer (SSL) (Optional)

In the context of SSL/TLS:

A KeyStore is a secure collection of identity certificates and private keys. By default, the KeyStore is in Java KeyStore (.jks) format.
CN represents the common name of the server protected by the SSL certificate.
FQDN represents the most complete domain name that identifies a host or server.

Hevo requires the CA file, the client certificate, and the client key to configure Apache Kafka as a Source if SSL encryption is enabled in Hevo. To generate these files, log in to the Kafka server folder where the Kafka TrustStore and KeyStore are located and perform the following steps:

1. Generate Client Certificate and Client Key

Enter the following command on the Kafka server command line:
```
keytool -keystore {key_store_name}.jks -alias {key_store_name_alias} -keyalg RSA -validity {validity} -genkey
```
Note: The alias is just a shorter name for the key store. The same alias needs to be reused throughout the steps.

You need to remember the passwords for each KeyStore or TrustStore to use later.
Provide the answers to questions that are displayed on the interactive prompt.

For the question What is your first and last name?, enter the CN for your certificate.

Note: The Common Name (CN) must match the fully qualified domain name (FQDN) of the server to ensure that Hevo connects to the correct server. Refer to this page to find the FQDN based on the type of the server.

2. Create your Certificate Authority

Note: This step is optional if you already have a CA to sign the certificates.

A Certificate Authority (CA) is responsible for signing certificates for each server in the cluster to prevent unauthorized access.

Run the following command on the Kafka server command line to create your own certificate authority:

openssl req -new -x509 -keyout ca-key -out ca-cert -days {validity}

3. Add the Certificate Authority to a TrustStore

A TrustStore is a secure collection of CA certificates that the broker can trust.

Run the following command on the Kafka server command line to add the CA to the broker’s truststore:

keytool -keystore {broker_trust_store}.jks -alias CARoot -importcert -file ca-cert

4. Sign the certificate with the CA file

Enter the following command on the Kafka server command line to sign the certificates with the CA file:

Export the certificate from the KeyStore:

keytool -keystore {key_store_name}.jks -alias {key_store_name_alias} -certreq -file cert-file

Sign the certificate with the CA:

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca-password}

Add the certificates back to the keystore:

keytool -keystore {key_store_name}.jks -alias CARoot -importcert -file ca-cert  
keytool -keystore {key_store_name}.jks -alias {key_store_name_alias} -importcert -file cert-signed

5. Extract the Client Certificate key from KeyStore

Before extracting the client certificate key from the KeyStore, you must convert the KeyStore file from its existing .jks format to the PKCS12 (.p12) format for interoperability.

Enter the following command in the Kafka server command line:

Convert the keystore from .jks format to .p12 format:

keytool -v -importkeystore -srckeystore {key_store_name}.jks -srcalias {key_store_name_alias} -destkeystore {key_store_name}.p12 -deststoretype PKCS12

Extract the client certificate key into a .pem file.
```
openssl pkcs12 -in {key_store_name}.p12 -nocerts -nodes > cert-key.pem  
```
This is the format in which Hevo understands the certificate keys.

You can now find the following files in the Kafka server folder where the Kafka TrustStore and KeyStore are located and upload them to Hevo to configure Apache Kafka as a Source.

Client Certificate: cert-signed
Client Key: cert-key.pem
CA File: ca-cert

Whitelist Hevo’s IP Addresses

You need to whitelist the Hevo IP address for your region to enable Hevo to connect to your Kafka server. To do this:

Open the Kafka server configuration file:
```
sudo nano /usr/local/etc/config/server.properties
```
Note: Depending on how Kafka was installed, this file may be in a different location.
In the file, scroll to listeners section. If it does not exist, add it on a new line.
Add the following under listeners:
```
 <protocol>://0.0.0.0:<port>
```
Or
```
 <protocol>://<hevo_ip>:<port>
```
Note: The <protocol> can be PLAINTEXT or SSL, and the port can be the same port used in your bootstrap server. The <hevo_ip> is the Hevo IP address for your region. If you are adding these to existing values, use comma to separate the entries.
Save the file.

Configure Apache Kafka Connection Settings

Perform the following steps to configure Apache Kafka as the Source in your Pipeline:

Click PIPELINES in the Navigation Bar.
Click + CREATE in the Pipelines List View.
In the Select Source Type page, select Kafka.
In the Select your Kafka Variant page, select Apache Kafka.
In the Configure your Kafka Source page, specify the following:
- Pipeline Name: A unique name for your Pipeline, not exceeding 255 characters.
- Bootstrap Server(s): The bootstrap server(s) extracted from Apache Kafka.
- Ingest Data From
  - All Topics: Select this option to ingest data from all topics. Any new topics that are created are automatically included.
  - Specific Topics: Select this option to manually specify a comma-separated list of topics. New topics are not automatically added in this option.
  - Topics Matching a Pattern (Regex): Select this option to specify a Regular Expression (regex) to match and select the topic names. This option also fetches data for new topics that match the pattern dynamically. You can test your regex patterns here.
- Use SSL: Enable this option to use an SSL-encrypted connection. Specify the following:
  - CA File: The file containing the SSL server certificate authority (CA).
  - Client Certificate: The client public key certificate file.
  - Client Key: The client private key file.
Click TEST & CONTINUE.
Proceed to configuring the data ingestion and setting up the Destination.

Data Replication

Default Pipeline Frequency	Minimum Pipeline Frequency	Maximum Pipeline Frequency
5 Mins	5 Mins	1 Hr

Note: The custom frequency must be set in hours, as an integer value. For example, 1, 2, 3 but not 1.5 or 1.75.

Note: Hevo only supports JSON Format for Kakfa.

Incremental Data: Once the Pipeline is created, all new and updated records are synchronized with your Destination as per the ingestion frequency.

If you restart an object via the Pipeline UI, Hevo ingests all the data available at that time in the Source.

For records that are structured as a list of records, Hevo ingests each record as an individual record. Each child record contains a common field called ref_id which is used to indicate a common parent record.

Additional Information

Read the detailed Hevo documentation for the following related topics:

Limitations

Limitations

Hevo only supports SSL/TLS-encrypted and plain text data in Apache Kafka.
Hevo only supports JSON data format in Apache Kafka.

Revision History

Refer to the following table for the list of key updates made to this page:

Date	Release	Description of Change
Dec-07-2022	NA	Updated section, Data Replication to reorganize the content for better understanding and coherence.
Sep-19-2022	NA	Added section, Data Replication.