I have a spring boot application that sends messages into a Kafka topic, something like:
public class Message {
@JsonProperty("customer_id")
@SerializedName("customer_id")
private String customerId;
...
}
For the sake of some business logic, I do not want to process 2 messages of the same customer in parallel. I want to use the Kafka partition order as some kind of lock. As far as I could understand, I can use the customerId as a key to my message and use a custom partitioner (implements Kafka partitioner) to direct the message with the same key to the same partition. And though I assume I am not the first one in history to seek this kind of functionality I've found it hard to find some kind of library/documentation for implementing such a thing. Can anyone guide me to an existing solution to my needs?
CodePudding user response:
That is the behavior of the default partitioner; see its javadocs:
/**
* The default partitioning strategy:
* <ul>
* <li>If a partition is specified in the record, use it
* <li>If no partition is specified but a key is present choose a partition based on a hash of the key
* <li>If no partition or key is present choose the sticky partition that changes when the batch is full.
*
* See KIP-480 for details about sticky partitioning.
*/
public class DefaultPartitioner implements Partitioner {
Changing the number of partitions in the topic will change the destination partition so you might want to consider a custom partitioner.