How can I query kafka and get the timestamp/offset/partition for the latest, most recent message given the topic and message key?

Our apps are in golang and java. However any solution is welcome. Thanks in advance.

CodePudding user response：

A Consumer cannot filter or only receive some messages from topics. The consumer will fetch messages in order.

At best we may be able to apply some filter to determine a specific partition. But within that partition you will have to dequeue to get the co-ordinates

If you are looking for a specific key , you will have to dequeue the message and then apply a filter on the customer side. For the ConsumerRecord you can then get the offset , partition etc.

A standard approach is to have the consumer write to the table for the key and the co-ordinates of the message , again something you may want to think about.

CodePudding user response：

You can't query Kafka by a key unless you build a KTable and use Interactive Queries in Kafka Streams / KsqlDB.

Otherwise, you can compute a partition for a key of a topic, and get the latest offset for that partition using GetOffsetShell java CLI tool included with Kafka binaries, or seek a consumer there via code after assigning the consumer to that partition.

If you need frequent key lookups from kafka, it would be best to consume the data into an actual key-value store or database