0

I am training some models (random forest) using ml library in Spark, R, and sparklyr. Everything ok, but now I need to save those models, so they can be used to make predictions for new data. I call 

ml_save(rfW1,w$fileName)

where rfW1 is the model and w is a structure that contains some date, including the fileName where the model should be stored. I get a directory structure of several levels created, and the following error

Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 340.0 failed 4 times, most recent failure: Lost task 0.3 in stage 340.0 (TID 24502, 172.30.47.9, executor 4): java.io.IOException: Mkdirs failed to create file:/home/project1/churn/project120171226191000/model1/metadata/_temporary/0/_temporary/attempt_20171226202013_0340_m_000000_24502 (exists=false, cwd=file:/var/lib/spark-2.2.0-bin-hadoop2.7/work/app-20171226180043-0142/4)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
    at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
    at org.apache.spark.internal.io.SparkHadoopWriter.open(SparkHadoopWriter.scala:89)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133)
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:108)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Since the structure is created (up to /home/project1/churn/project120171226191000/model1/metadata/_temporary/0), it does not seems to be a permissions problem. Any hint?

molbdnilo
  • 64,751
  • 3
  • 43
  • 82
user2345448
  • 159
  • 2
  • 11
  • What version of spark are you using? My understanding is that you won't be able to save random forest models with Spark >= 1.6. See this post that I authored last year: https://github.com/rstudio/sparklyr/issues/449 – schristel Jan 11 '18 at 19:50
  • With sparklyr 0.7 you should be able to save a model with the syntax `ml_save(model, "path/to/model")` – kevinykuo Feb 08 '18 at 09:38

0 Answers0