Kafka

Kafka is open-source software that provides a framework for storing, reading, and analyzing streaming data.

In Kafka, a Topic is a category or a common name used to store and publish a particular stream of data. Topics in Apache Kafka are similar to tables in a database. Hevo reads your data from the topics created in your Apache Kafka instance.

A collection of topics is called a Kafka Cluster. The server that the topics are hosted on in Kafka is called a Broker. Therefore, a Kafka Cluster may consist of many Kafka Brokers on many servers.

Bootstrap servers are a comma-separated list of host and port pairs that represent the addresses of the brokers. In the context of your Hevo Pipeline, these are the initial servers in the cluster which Hevo can connect to establish the initial connection. If one fails, the others are used. The bootstrap server automatically redirects subsequent connections to appropriate servers in cluster.

Hevo supports two variations of Kafka as a Source. Both these variants offer the same functionality, with Confluent Cloud being the fully-managed version of Apache Kafka.

Click on each link below for steps to configure these as a Source in your Hevo Pipeline:

• Apache Kafka

• Kafka Confluent Cloud

Sample Source Event

[{ "name": "John", "age": 25 }, { "name": "Jack", "occupation": "chef" } ]

Sample Ingested Events

Event 1:

 { "__hevo_id": "abc1", "ref_id": "abcdef", "name": "John", "age": 25 }

Event 2:

 { "__hevo_id": "abc2", "ref_id": "abcdef", "name": "Jack", "occupation": "chef" }

Revision History