Home > database >  How to map parquet record (json) to case class using alpakka
How to map parquet record (json) to case class using alpakka

Time:10-17

I have the data that saved in parquet file as json e.g {"name":"john", "age":23.5} I want to convert it to

case class Person(name: String, age: Double) so I can use pattern matching in my actor. this is what i got so far

val reader: ParquetReader[GenericRecord] =  AvroParquetReader.builder[GenericRecord](filePath).withConf(conf).build()

val source: Source[GenericRecord, NotUsed] = AvroParquetSource(reader)
  source.ask[WorkerAck](28)(workerActor)

I tried to replace the GenericRecord with Person but i got the following error :

inferred type arguments [com.common.Person] do not conform to method apply's type parameter bounds [T <: org.apache.avro.generic.GenericRecord]
  val source: Source[Person, NotUsed] = AvroParquetSource(reader)

CodePudding user response:

I think you have two options

  1. Use Avro code generator to generate the code for the Person DTO class. This will create a Person class that inherits from generic record. See this tutorial

  2. Add an actor that converts GenericRecord instances to Person:

Person((String)record.get("name"), (Double)record.get("age"))

  • Related