Service architecture^Beta

The Cloud Kafka cluster consists of several Zookeeper clusters and several brokers (Kafka Broker). The number of Zookeeper and Kafka Broker instances depends on the selected fault tolerance mode. To improve cluster performance, you can manually increase the number of brokers.

The service operates on the “publish/subscribe” principle: producers publish messages to topics, and consumers poll the service to receive new messages. To work with suppliers and consumers, a connection must be configured via load balancer bootstrap (Kafka bootstrap load balancer). The load balancer provides fault-tolerant connection between suppliers and consumers.

A Kafka Connect can be created for a Cloud Kafka cluster is a data streaming tool between Apache Kafka and other systems. The Cloud Kafka service supports connectors:

• JDBC,
• S3,
• Debezium,
• ClickHouse.

The Kafka UI tool is also deployed to work with the cluster. Its main functions are:

• view the list of Apache Kafka cluster topics and messages in them;
• launch connectors and monitor their operation in Kafka Connect;
• track the availability of brokers.