How to configure Kafka type mapping using spring boot, maven build success but contain errors?-CodePudding

I've looked at the documentation and found this:

spring.kafka.producer.properties.spring.json.type.mapping=cat:com.mycat.Cat,hat:com.myhat.Hat

My producer properties:

spring.kafka.producer.bootstrap-server=localhost:9092
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer 
spring.kafka.producer.properties.spring.json.type.mapping=com.producerservice.dto.UserDTO:UserDTO

The maven install run successfully.

The consumer properties:

spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer
spring.kafka.consumer.properties.spring.json.type.mapping=UserDTO:com.consumerservice.dto.UserDTO
spring.kafka.consumer.properties.spring.json.trusted.packages=*

maven is successful but there is an error, I don't undestand what it's passed when error exists:

java.lang.IllegalStateException: This error handler cannot process 'SerializationException's directly; please consider configuring an 'ErrorHandlingDeserializer' in the value and/or key deserializer

And the error continued, reporting about the producer class:

Caused by: org.springframework.messaging.converter.MessageConversionException: failed to resolve class name. Class not found [com.producerservice.dto.UserDTO]; nested exception is java.lang.ClassNotFoundException: com.producerservice.dto.UserDTO
    at org.springframework.kafka.support.mapping.DefaultJackson2JavaTypeMapper.getClassIdType(DefaultJackson2JavaTypeMapper.java:142) ~[spring-kafka-2.8.8.jar:2.8.8]
    at org.springframework.kafka.support.mapping.DefaultJackson2JavaTypeMapper.toJavaType(DefaultJackson2JavaTypeMapper.java:103) ~[spring-kafka-2.8.8.jar:2.8.8]
    at org.springframework.kafka.support.serializer.JsonDeserializer.deserialize(JsonDeserializer.java:572) ~[spring-kafka-2.8.8.jar:2.8.8]
    at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:1439) ~[kafka-clients-3.1.1.jar:na]
    ... 15 common frames omitted
Caused by: java.lang.ClassNotFoundException: com.producerservice.dto.UserDTO
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:636) ~[na:na]
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182) ~[na:na]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519) ~[na:na]
    at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
    at java.base/java.lang.Class.forName(Class.java:466) ~[na:na]
    at org.springframework.util.ClassUtils.forName(ClassUtils.java:284) ~[spring-core-5.3.22.jar:5.3.22]
    at org.springframework.kafka.support.mapping.DefaultJackson2JavaTypeMapper.getClassIdType(DefaultJackson2JavaTypeMapper.java:138) ~[spring-kafka-2.8.8.jar:2.8.8]
    ... 18 common frames omitted

class not found for the producer class? of course it's not found I'm in the consumer

UPDATE

I have changed the producer properties to the follow:

spring.kafka.producer.properties.spring.json.type.mapping=UserDTO:com.producerservice.dto.UserDTO

problem still persist with the same error.

CodePudding user response：

of course it's not found I'm in the consumer

Your producer and consumer need to share a dependency of your data classes.

You have a runtime error which Maven cannot detect since you have dynamic class-loading in your Kafka deserializer, but it could detect it, if you added UserDTO into your @KafkaListener method parameter.

CodePudding user response：

According to the doc a token should go before a classname for both consumer and producer:

senderProps.put(JsonSerializer.TYPE_MAPPINGS, "cat:com.mycat.Cat, hat:com.myhat.hat");
consumerProps.put(JsonDeSerializer.TYPE_MAPPINGS, "cat:com.yourcat.Cat, hat:com.yourhat.hat");

So pls update your producer from:

spring.kafka.producer.properties.spring.json.type.mapping=com.producerservice.dto.UserDTO:UserDTO

to:

spring.kafka.producer.properties.spring.json.type.mapping=UserDTO:com.producerservice.dto.UserDTO

Consumer looks correct now.

CodePudding user response：

So this solved at least the maven build:

userDto:com.my.example.UserDTO

I had

userDTO

So, the token was the problem, token is the variable name that is used in your listener consume method.

To understand that the main issue was with token this question helped: what is token in class type mapping

But that alone didn't fix, I had to stop zookeeper and broker delete all logs and guess what the build is successful on both side. Message will send tomorrow, too tired.