Springboot-Kafka: Bootstrap-servers with MultiNode Hosts-CodePudding

I am trying to connect to kafka consumers which uses multinode hosts. Following property works with one multinode host, but not able to read multiple multinode hosts.

Works fine:

spring.kafka.consumer.bootstrap-servers = example.dev.com:9092

Only reading from one host even-though mentioned two:

spring.kafka.consumer.bootstrap-servers = example.dev.com:9092,example2.dev.com:9092

Here:

example.dev.com:9092 = node:9092,node2:9092,node3:9092
example1.dev.com:9092 = node:9092,node2:9092,node3:9092

Is it possible to connect to multiple multinodes using one consumerFactory like below:

    @Bean
    public ConsumerFactory<String, String> consumerFactory()
    {
  
        // Creating a Map of string-object pairs
        Map<String, Object> config = new HashMap<>();
  
        // Adding the Configuration
        config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
                   "example.dev.com:9092,example1.dev.com:9092 ");
        config.put(ConsumerConfig.GROUP_ID_CONFIG,
                   "group_id");
        config.put(
            ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
            StringDeserializer.class);
        config.put(
            ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
            StringDeserializer.class);
  
        return new DefaultKafkaConsumerFactory<>(config);
    }

CodePudding user response：

The bootstrap server(s) are just used to, err, bootstrap the connections.

It only needs to connect to one of them to get information about the cluster.

That server then returns "advertised listeners" and the client will only actually connect to brokers that have partitions that the client wants to consume from (or send to).

So, if all the topics/partitions are located on just one server, that's the one you will connect to.

CodePudding user response：

The simplest hack would be check the servers and collect some info to make sure that topics/partition are created in multiple nodes. Once this is confirmed, you can just run your consumer and see if you are getting the topic/partition events.

List of bootstrap servers are mainly for resilience. It is not necessary to consume from each one of them every time.