I'm new to Kafka. currently, I have created a java consumer on a Kafka topic. What I want is to get only the new messages when I start my java application.

After checking on the internet I found that I can do that by setting the property: auto.offset.reset: earliest, but we don't want to use that. Also after doing some research I found that maybe we can do that by updating the consumer offset to the earliest one. Can anyone help to know how I can do that, please?

Note that I'm using KStream to subscribe to the topic. Kafka version 2.4.0 subscribe source code:

StreamsBuilder sb = new StreamsBuilder();
sb.stream("topic").foreach((t, u) -> {...})

CodePudding user response：

updating the consumer offset to the earliest one

That's the default behavior and configuration for auto.offset.reset when your application.id is brand new.

Note that i'm using KStream to subscribe the topic.

Then you can't seek at runtime (or at least, shouldn't, assuming you're using stateful processors)

Use the Application reset tool which helps manage all topics used within your topology.

CodePudding user response：

Can you specify why you don't want to use auto.offset.reset?

That would be the default approach to do what you want. Other than that, you'll need to go and get the latest offset for each partition of the topic and "point" your consumer to that offset on the partition you want to consume.