We are currently looking into how we can design our topics in Kafka, considering ordering and noise rate (percentage of messages to discard). We are using Spring Kafka and Avro as the serialisation mechanism for the record value.

One approach is, we put different event types in one topic. This would mean that we have some connected services to that topic that need to discard messages that they don't need. Our approach would be putting the event type in the record header and implement a RecordFilterStrategy that filters the events based on a white list.

I am wondering, if the deserialisation of the message takes place before we filter the message. Meaning do we have still the overhead of deserialisation of the message or not. I could not find any further information in the Spring Kafka documentation.

In case the deserialisation process is involved, is there a way in spring how we can intercept the message (based on headers) before the deserialisation starts.

Thanks for any help.

CodePudding user response：

Yes, deserialization has already happened before this RecordFilterStrategy is involved. This strategy is a part of MessageListener logic. The deserialization is done in the KafkaConsumer already. And that part is out of Spring control.

I see even ConsumerInterceptor is too late already since the record is deserialized before that in the Fetcher.parseRecord().

Only the way I see you can effect this is to implement a custom deserializer which could return null for non-desired value.