In kafka can specify multiple consumer in a project-CodePudding

Or start the only set of processes for consumer, under the condition of the same consumer groups, multiple consumer can be in the process of a startup?

How to configure the need to develop the time?

CodePudding user response:

Certainly, just under the same consumer groups each consumer is a competitive relationship, can promote consumption performance,

CodePudding user response:

First of all, your topic must be multiple partitions, a partition, a consumer group ID (consumerGroupId) will only have a consumer instance to consumption data,
Topic1 has two partitions, for example, cgi1 have 3 consumers, only one of two consumers to buy their topic1 partitions, remaining 1 get less than consumer rights, there is no data to vomit to him,
Hang up when one of the consumers or processing delay more than configuration threshold value, the consumer will be the broker as a hang up, recover the consumer rights, free of the substituted for the hang up of consumers, from the last submitted offset began to hang up consumer spending again,
About the mechanism, to search [kafka isr] In Sync - up mechanism, copy, whether consumers or partitions are follow this mechanism, it is one of the most core knowledge kafka,

CodePudding user response:

So we have the most robust approach is n partition instance n + x consumers to spend, and then through the related business mechanism, control for hang up, substitute caused by repeated consumption, reduce the side effects,

CodePudding user response:

Quote: refer to the second floor LinkSe7en response:

Can explain, for example, how to avoid the substitute of the consumers, avoid duplication of consumption, consumption is said such as external database storage has been offset, or what can you do with kafka internal configuration?
CodePudding user response:
reference 4 floor doublepotato response:

Quote: refer to the second floor LinkSe7en response:

Can explain, for example, how to avoid the substitute of the consumers, avoid duplication of consumption, consumption is said such as external database storage has been offset, or what can you do with kafka internal configuration?

General practice is 1, the control of each batch message as little as possible (but not 1 1) of article 2, moderate both need to combine your overtime consumption logic tuning
As for external database check this to see how your business logic is binding, such as consumer 1 went to database records, to write 1 in messages need to be that the only field to distinguish, or change the database record, unable to identify, through field that is in need of some low cost but the only check method, rapid BloomFilter, for example, to record whether the offset was handled correctly,
Precisely a semantic in actual business to ensure that the cost is very high, we generally is based on the business logic of the consumer choice at (0-1) or at least once (1 - n) of the two weigh choose, you don't do any extra movement is at least once, BloomFilter check is the most time,
CodePudding user response:
reference 4 floor doublepotato response:

Quote: refer to the second floor LinkSe7en response:

Can explain, for example, how to avoid the substitute of the consumers, avoid duplication of consumption, consumption is said such as external database storage has been offset, or what can you do with kafka internal configuration?

The ultimate solution to avoid repeated consumption kafka is idempotent consumer realization was consumer spending, reduce the batch message consumption, improve throughput are tactical approach, considering the network is not reliable under the condition of this problem is no solution, just like the Byzantine generals problems