1

I have the data that saved in parquet file as json e.g {"name":"john", "age":23.5} I want to convert it to

case class Person(name: String, age: Double) so I can use pattern matching in my actor. This is what I got so far:

val reader: ParquetReader[GenericRecord] =  AvroParquetReader.builder[GenericRecord](filePath).withConf(conf).build()

val source: Source[GenericRecord, NotUsed] = AvroParquetSource(reader)
  source.ask[WorkerAck](28)(workerActor)

I tried to replace the GenericRecord with Person but I got the following error:

inferred type arguments [com.common.Person] do not conform to method apply's type parameter bounds [T <: org.apache.avro.generic.GenericRecord]
  val source: Source[Person, NotUsed] = AvroParquetSource(reader)
gianluca aguzzi
  • 1,734
  • 1
  • 10
  • 22
igx
  • 4,101
  • 11
  • 43
  • 88

1 Answers1

1

I think you have two options

  1. Use Avro code generator to generate the code for the Person DTO class. This will create a Person class that inherits from generic record. See this tutorial

  2. Add an actor that converts GenericRecord instances to Person:

Person((String)record.get("name"), (Double)record.get("age"))
0x26res
  • 11,925
  • 11
  • 54
  • 108
  • Thanks, I did manage to do it with mapping i.e `source.map{ record => Person(record.get("name"), record.get("age").toString.toDouble)` – igx Oct 16 '21 at 06:28
  • 1
    A little note... cast with (Type) is a Java thing. In Scala you have to use .asInstanceOf[Type] – gianluca aguzzi Oct 17 '21 at 09:34