Hi I am new to Spring Boot @kafkaListener. Service A publishes message on kafka topic continuously. My service consume the message from that topic. Partitions of topic in both service (Service A and my service) is same, but rate of consuming the message is low as compare to publishing the message. I can see consumer lag in kafka.

How can I fill that lag? Or how can I increase the rate of consuming the message?

1. Can I have separate thread for processing message. I can consume a message in Queue (acknowledge after adding into queue) and another thread will read from that queue to process that message.

2. Is there any settings or property provides by Spring to increase the rate of consumption?

CodePudding user response：

Lag is something you want to reduce, not "fill".

Can you consume faster? Yes. For example, changing the consumer max.poll.records can be increased from the default of 500, per your I/O rates (do your own benchmarking) to fetch more data at once from Kafka. However, this will increase the surface area for consumer error handling.

You can also consume and immediately ack the offsets, then toss records into a queue for processing. There is possibility for skipping records in this case, though, as you move processing off the critical path for offset tracking.
Or you could only commit once per consumer poll loop, rather than ack every record, but this may result in duplicate record processing.

As mentioned before, adding partitions is the best way to scale consumption after distributing producer workload

CodePudding user response：

You generally will need to increase the number of partitions (and concurrency in the listener container) if a single consumer thread can't keep up with the production rate.

If that doesn't help, you will need to profile your consumer app to see where the bottleneck is.