There is one Kubernetes property called replica, suppose we have 3 replicas(replica-0, replica-1, replica-2) and we are producing 100 messages in Kafka, Only one replica say (replica-0) is invoked at a time for consuming all the 100 messages, but the expected solution is all the 100 messages should be divided among 3 replicas(replica-0, replica-1, replica-2) instead keeping the load on one replica for consuming the messages

CodePudding user response：

You should clarify "invoked". Consumers poll; they are not externally "invoked"/started other than calling poll() method.

Consumers configured with the same group.id cannot subscribe to overlapping partitions. If you sent 100 records to one partition, then only one consumer will read that. If you want all applications to read a subset of the data, then you first will need a Kafka topic with at least 3 partitions (the default auto-created topics only have one), then send data amongst those.

Kubernetes has nothing to do with the Kafka consumer subscription.

CodePudding user response：

A Kubernetes Service provides an abstraction over a group of Pods. It creates a single endpoint that, by default, will balance requests across the backing pods selected by the Service.

Send the request to the Service endpoint instead of the individual pods.