I have a use case of reading sequence files as a Batch job in Flink Dataset. The files are stored in S3 bucket which I have to consume in a Flink Dataset. I am not able to read the files by providing comma(,) separated file paths to read in the Dataset. I cannot read the data in file using a loop as there are a lot of files in the bucket. Also the union function for Flink Datasets seems to fail after few iterations. Can someone help me out with creating a custom Sequence file reader which will work for this case as provided in Spark.
Asked
Active
Viewed 363 times
2
Abhinav Prakash
- 21
- 2