I'm trying to use Kafka ByteArrayDeserializer to read avro records from a Kafka topic. But getting below exception.

Caused by: java.lang.ClassCastException: [B cannot be cast to java.lang.String

My Code:

val ssc = new StreamingContext(spark.sparkContext, Seconds(1))

  val kafkaParams: Map[String, Object] = Map(
    "bootstrap.servers" -> "kafka-server:9092",
    "key.serializer" -> classOf[StringSerializer],
    "value.serializer" -> classOf[StringSerializer],
    "key.deserializer" -> classOf[StringDeserializer],
    "value.deserializer" -> classOf[ByteArrayDeserializer],
    "auto.offset.reset" -> "earliest",
    "enable.auto.commit" -> (false: java.lang.Boolean),
    "security.protocol" -> "SSL",
    "ssl.truststore.location" -> "truststore",
    "ssl.truststore.password" -> "pass",
    "ssl.keystore.location" -> "keystore.jks",
    "ssl.keystore.password" -> "pass",
    "group.id" -> "group1"
  )

val topics: Array[String] = Array("topics")

val kafkaDstream = KafkaUtils.createDirectStream(
    ssc,
    LocationStrategies.PreferConsistent,
    ConsumerStrategies.Subscribe[String, String](topics, kafkaParams)
  )

val schema = parser.parse(new String(Files.readAllBytes(Paths.get("avro2.avsc"))))
val datumReader = new SpecificDatumReader[GenericRecord](schema)

val processedStream = kafkaDstream.map(record => {
    val x = new ByteArrayInputStream(record.value().getBytes())  // throwing exception here
    val binaryDecoder = DecoderFactory.get.binaryDecoder(x, null)
    datumReader.read(null, binaryDecoder)
  })


processedStream.map(rec => rec.get("taskId")).print

Any help is appretiated.

Thank you.

CodePudding user response：

You're using Subscribe[String, String].

You want Subscribe[String, Array[Byte]]

Then record.value() is already a byte array, not having a getBytes method

CodePudding user response：

Caused by: java.lang.ClassCastException: [B cannot be cast to java.lang.String

This exception is to say that there's an object of type [B was cast to java.lang.String and failed.

[B is a string representation of Array[Byte]:

jshell> byte[] bytes = new byte[10]
bytes ==> byte[10] { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }

jshell> bytes.toString()
$2 ==> "[B@6e8cf4c6"

"Why does this even happen?" you ask. It's because the following line (silently) assumes that whatever you're doing can ever be a String.

val x = new ByteArrayInputStream(record.value().getBytes())  // throwing exception here

Final Note

Please use Spark Structured Streaming instead. You won't regret this decision.