We have installed Kafka on kubernetes , we want Kafka broker partitions needs to auto increase based on load from input source , so I want to know is there any possibile way to increase partitions based on load on kubernete
CodePudding user response：
Brokers don't have partitions that scale; topics do.
You'll have to build your own scripts for this because there are some caveats
Adding partitions to topics doesn't move data, and will not reduce producer load. It might increase broker cluster load because there would now be more data be able to get written/read. The only way to reduce broker load is to add more brokers, not just partitions
- If you add brokers, you'll need to have scripts that move partitions across brokers since that's not automatic either
It'll also remove any ordering guarantee for consumers, if they were relying on it (for example, compacted topics)