1

I am using Standalone clusters to run the ALS algorithm. The predictions are being stored to the textfile using:

saveAsTextFile(path)

But the text file is being stored on the clusters. I want to store the text file on the Master.

Shishir Anshuman
  • 1,115
  • 7
  • 23
  • Possible duplicate of [saveAsTextFile() to write the final RDD as single text file - Apache Spark](http://stackoverflow.com/questions/31145737/saveastextfile-to-write-the-final-rdd-as-single-text-file-apache-spark) – Alberto Bonsanto Mar 24 '16 at 12:06
  • @AlbertoBonsanto Writing to a single file is not the issue. I want to save the output file on the Master instead of the slaves. – Shishir Anshuman Mar 24 '16 at 21:09

1 Answers1

1

That is expected behavior. path is resolved on the machine it is executed, the slaves. I'd recommend to either use a cluster FS (e.g. HDFS) or .collect() your data so you can save them locally on the master. Beware of OOM if your data is large.

Reactormonk
  • 21,472
  • 14
  • 74
  • 123
  • I used it as `val data=predictions.collect()`, and now I cannot use `saveAsTextFile()` because I get this error: _saveAsTextFile is not a member of Array[String]_ . I need to convert `data` to **RDD**, so I tried `val rdd=sc.parallelize(data)` and `rdd.saveAsTextFile(path)`. But still the text file is being stored on the slaves machines. – Shishir Anshuman Mar 24 '16 at 19:02
  • Yeah, sure. `saveAsTextFile` only works on slaves. You'll have to do your own writing with java stuff, it's local now. – Reactormonk Mar 24 '16 at 20:35
  • I did not get it. I need more information on "own writing with java stuff". I am new to apache spark. – Shishir Anshuman Mar 25 '16 at 11:14
  • https://stackoverflow.com/questions/2885173/how-to-create-a-file-and-write-to-a-file-in-java – Reactormonk Mar 25 '16 at 11:22
  • Yes, thanks. To store RDD using printwriter I used `for (line <- rdd.toLocalIterator) {writer.println(line)}` from [here](http://www.russellspitzer.com/2015/12/15/Spark-Writing-Driver-FIleSystem/). Thanks a lot for the help. – Shishir Anshuman Mar 26 '16 at 07:31