2

I am facing a weird issue and I've stuck with that for a while.

Basically in my spark application I have a method, where I create a file in MapR fs and save some content in this file.

The method is like below:

    def collector(value: String): Unit = {
        val conf = new Configuration()
        val fs= FileSystem.get(conf)
        val path = getConfig("path")
        Try {fs.create(new Path(path + s"${scala.util.Random.alphanumeric take 10 mkString}" +
          Calendar.getInstance.getTimeInMillis))
          .write(value.getBytes); 
          fs.close()}.getOrElse(logger.warn("File not saved"))
      }

This method is called from different object. When I run it with --master local[*], then the file is created in specific location in the FS with string that is passed in value. However, when I run it on cluster (--master yarn), then only empty file is saved. When I print value, its printing the string. But for some reason not saving it in the file.

I wonder if anybody has any idea why?

Thanks

Tomasz Krol
  • 596
  • 6
  • 23
  • 1
    This has most probably nothing to do with your problem, but using `System.currentTimeMillis` seems to return the same as `Calendar.getInstance.getTimeInMillis`, but without any mention of the `Calendar` legacy api. – Andrey Tyukin Jul 26 '18 at 22:39
  • I found something similar here: maybe it will help. – Pat S Jul 27 '18 at 08:12

0 Answers0