-1

I have some calculated values and I want to save them in SparkR.

If I save it as a csv-file 

write.csv(data, file="/.../data.csv", row.names=FALSE)

it takes very long time for some reason. Is there a better way to do this ?

zero323
  • 322,348
  • 103
  • 959
  • 935
Ole Petersen
  • 670
  • 9
  • 21
  • 1
    How is that related to SparkR? `write.csv` doesn't work with Spark data frames. – zero323 Jul 29 '15 at 22:00
  • 1
    I am surprised it doesn't throw an error: did you already collect data to a data.frame? If it is a DataFrame, my guess is it only writes the pointer to a file? – Wannes Rosiers Jul 30 '15 at 06:21

1 Answers1

0

You can save the csv file in /tmp/ for temporary use. But when the cluster restarts, the file will be removed. Specify your the file name as file = "/tmp/filename.csv"

The other choice, you can register your table. see https://spark.apache.org/docs/latest/sparkr.html

Hsin
  • 1