5

I writing an RDD to the local file system in text format by using the saveAsTextFile method. In the output directory, there is one .crc file for each part file even for the _SUCCESS file.

I am just looking for any inbuilt functionality or property of Hadoop/Spark that avoid generation of these metafiles(especially .crc)

I found following properties to avoid _SUCCESS file generation and .crc file generation for parquet file but looking for similar property for text file.

sc.hadoopConfiguration.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")

sc.hadoopConfiguration.set("parquet.enable.summary-metadata", "false")

Thanks in advance.

zero323
  • 322,348
  • 103
  • 959
  • 935
Vikash Pareek
  • 1,063
  • 14
  • 30
  • 1
    Did you try the parquet setting just to see if it would work? That is the only thing I've seen so far even related to these .crc files. – szeitlin Dec 03 '18 at 22:29
  • Have you tried `sc.hadoopConfigurtion.setBoolean("fs.file.impl.disable.cache", true)`? – Filip Feb 04 '20 at 19:20

0 Answers0