I subscribe to

In short, here is what happens

1. There are 50 consumers subscribing to multiple topics by RegExp. All found topics has 52 partitions totally.
2. Each consumer tries to subscribe to 1 partition from each found topic. There are also two consumer subscribed to 1 partition from the single topic because the latter has 12 but not 10 partitions.
3. 12 consumers are working whilst 38 remains stale due to unavailable partitions to read.

Is there any way to force Kafka consumer to read 1 partition maximum with RegExp subscription? In this case, I can make all consumers start work. Or maybe there is a different approach that allows to read multiple topics respecting the number of partitions and running consumers?

CodePudding user response：

Well, it's about

Though you should be careful when you deploy new version with different partition.assignment.strategy. Suppose you have 5 running consumers with RangeAssignor strategy. Then you redeploy one with RoundRobinAssignor strategy. In this case, you get an exception on consumer replacement. Because all consumers in one consumer group should provide the same partition.assignment.strategy. The problem is described in this StackOverflow question

So, if you want to change the partition.assignment.strategy, you have several options:

1. Redeploy all consumers and then deploy new ones.
2. Specify new group.id for the new consumers.

Both of these ways have pros and cons.