I use spring-boot with transactional kafka. I have 4 topics where i need to send kafka messages. How could i check amount of kafka producers that creates spring for sending messages in different topics. I need to know : Is it single producer or multiple producers? Is it single producer for multiple topics and it is thread safe or maybe every thread create own transactional producer fow every topic?
I want to make some tests to know which is better way for performance : single tread safe producer for multiple topics or each producer has it's own topic.

CodePudding user response：

There are a few ways you can inspect the number of Kafka producers that Spring Boot creates when using transactional Kafka.

Use a Kafka monitoring tool: There are several open-source and commercial tools available that can monitor the number of producers and consumers, such as Kafka Manager, Prometheus and Grafana, and Confluent Control Center. These tools can provide detailed information about the number of producers, the number of messages produced, and the status of the producers.

Use the KafkaAdmin class: Spring Boot provides a KafkaAdmin class that allows you to programmatically inspect the state of your Kafka cluster. You can use this class to retrieve the number of producers and consumers, along with other information about the cluster.

Use log statements : You can also add log statements in your code to track the number of producers created. For example, you can add a log statement in the constructor of the KafkaProducer class, so that every time a new producer is created, a log statement is written to the log.

Regarding your question about performance, it depends on the scale of your application, the number of messages you need to produce, and the size of the messages. In general, having a single thread-safe producer for multiple topics can be more efficient than having multiple producers, as it reduces the overhead of creating and managing multiple connections. However, if your application requires a high throughput and low latency, it might be better to have multiple producers, each with its own topic.

It's recommended to test both options in your specific use case and measure the performance, to find which one is better for your needs.

Regarding your follow-up question:

How could i use log inside kafkaProducer? It is library class and i can't modify it. Thea are read only. Or how can i get information what i need from kafkaAdmin , have you any examples? Thx –

You're correct that the KafkaProducer class is a library class and cannot be modified. However, there are a few ways you can add logging to the KafkaProducer class without modifying it:

Use a Kafka interceptor: Kafka provides the ability to add interceptors to the producer and consumer classes. These interceptors can be used to add custom logic, such as logging, to the KafkaProducer class. You can create a custom interceptor that logs the number of producers created, and add it to the KafkaProducer class.

Use AOP (Aspect-Oriented Programming): You can use AOP to add logging to the KafkaProducer class. You can define a pointcut that matches the constructor of the KafkaProducer class and use it to add a logging statement. Spring Boot supports AOP out of the box, and you can use AspectJ to define pointcuts and advice.

Use a Decorator design pattern: You can create a decorator class that wraps the KafkaProducer class and adds the logging functionality. This way you can keep the original class as is, and use the decorator to add the logging.

Regarding KafkaAdmin, you can use the methods provided by this class to get information about the state of your Kafka cluster. Here are a few examples of how you can use the KafkaAdmin class:

To check the number of topics:

KafkaAdmin.listTopics()

To check the number of consumers:

kafkaAdmin.listConsumerGroups()

To check the number of producers:

kafkaAdmin.listProducerIdBlockList()

Keep in mind that these methods will give you the current state of the kafka cluster, and not the number of producers that spring created, but it can give you an idea about the current state of your kafka cluster. It's important to note that the KafkaAdmin class is not thread-safe, you should create a new instance for each thread.

It's also worth noting that the above code examples are not guaranteed to work as is, as the exact implementation of these methods depends on the version of spring-kafka and kafka that you are using. Please check the documentation and adapt the code to match the version you are using.