-2

I am trying to read multiple files in different paths at the same time. In sql server, the path "/mapr/ia1.comscore.com/output/cms/cmcm/227m/2018????/app/RunningApp/part-.txt" is telling the server to read all the files in running app folder for the month 227m( Nov 2018). The '2018????' means all the dates in Nov 2018. In this way, we don't have to tell sql server to read each date one by one. Here '2018????' means all the date from '20181101' to '20181130'. I want to do the same thing in spark.But it seems spark cannot recognize this format of path. I have also tried to change '2018????' too '2018', still it does not work.

Does any one know how to fix this?

Please note that "/mapr/ia1.comscore.com/output/cms/cmcm/227m/20181101/app/RunningApp/part-*.txt") works well in spark.

I am using zeppelin as the notebook.

Thanks a lot.

Vamsi Prabhala
  • 48,685
  • 4
  • 36
  • 58
Xixi
  • 117
  • 6

1 Answers1

0

The easiest way is to pass a list of paths, e.g. sparkContext.read.text(paths=[path1, path2, ...])

Mikhail Berlinkov
  • 1,624
  • 10
  • 15