Spark Invalid Checkpoint Directory

Question

I have a long run iteration in my program and I want to cache and checkpoint every few iterations (this technique is suggested to cut long lineage on the web) so I wont have StackOverflowError, by doing this

for (i <- 2 to 100) {
      //cache and checkpoint ever 30 iterations
      if (i % 30 == 0) {
        graph.cache
        graph.checkpoint
        //I use numEdges in order to start the transformation I need
        graph.numEdges
      }
      //graphs are stored to a list
      //here I use the graph of previous iteration to this iteration
      //and perform a transformation
}

and I have set the checkpoint directory like this

val sc = new SparkContext(conf)
sc.setCheckpointDir("checkpoints/")

However, when I finally run my program I get an Exception

Exception in thread "main" org.apache.spark.SparkException: Invalid checkpoint directory

I use 3 computers, each computer has Ubuntu 14.04, and I also use a pre-built version of spark 1.4.1 with hadoop 2.4 or later on each computer.

score 4 · Accepted Answer · answered Oct 24 '16 at 15:41

If you already set up HDFS on a cluster of nodes, you can find your hdfs address in "core-site.xml" located in the directory HADOOP_HOME/etc/hadoop. For me, the core-site.xml is set up as:

<configuration>
      <property>
           <name>fs.default.name</name>
           <value>hdfs://master:9000</value>
      </property>
</configuration>

Then you can create a directory on hdfs to save Rdd checkpoint files, let's name this directory RddChekPoint, by hadoop hdfs shell:

$ hadoop fs -mkdir /RddCheckPoint

If you use pyspark, after SparkContext is initialized by sc = SparkContext(conf), you can set checkpoint directory by

sc.setCheckpointDir("hdfs://master:9000/RddCheckPoint")

when an Rdd is checkpointed, in the hdfs directory RddCheckPoint, you can see the checkpoint files are saved there, to have a look:

$ hadoop fs -ls /RddCheckPoint

score 2 · Answer 2 · answered Sep 05 '15 at 09:07

2

The checkpoint directory needs to be an HDFS compatible directory (from the scala doc "HDFS-compatible directory where the checkpoint data will be reliably stored. Note that this must be a fault-tolerant file system like HDFS"). So if you have HDFS setup on those nodes point it to "hdfs://[yourcheckpointdirectory]".

answered Sep 05 '15 at 09:07

Holden

7,392
1
27
33

Is there any other way or do I need to have hdfs setup on my nodes? Also "hdfs://[whatDoIputHere]", if I have hdfs setup what directory do I use exactly? – Al Jenssen Sep 05 '15 at 10:48
You don't need HDFS (its needs to be HDFS compatable, but there are other possibilities too). If you have HDFS setup it should be a directory without anything important in it. – Holden Sep 05 '15 at 20:48

Spark Invalid Checkpoint Directory

2 Answers2