How to to stream a directory of parquet files through akka/alpakka streams

Asked Oct 19 '21 at 07:33

Active Oct 19 '21 at 18:36

Viewed 107 times

I trying to stream a directory of parquet files through akka/alpakka streams, the problem is although I am setting the directory as the source, it streams only one file.

Any advice will be greatly appreciated This is my last experiment:

 val reader: ParquetReader[GenericRecord] =  AvroParquetReader.builder[GenericRecord](filePath).withConf(conf).build()

  val source: Source[GenericRecord, NotUsed] = AvroParquetSource(reader)

  source 
    .ask[WorkerAck](10)(workerActor)
    .runWith(Sink.ignore)

edited Oct 19 '21 at 18:36

Gaël J

11,274
4
17
32

asked Oct 19 '21 at 07:33

igx

4,101
11
43
88

Is `filePath` a `HadoopInputFile` / `InputFile` ? – ccheneson Oct 19 '21 at 14:17
@ccheneson it is a directory with parquet files – igx Oct 20 '21 at 07:31

How to to stream a directory of parquet files through akka/alpakka streams

0 Answers0