0

When I train a DNN network using CaffeOnSpark with 2 Spark executors with Ethernet connection, I'm getting an error. I run the job as the example on the https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_yarn

export SPARK_WORKER_INSTANCES=2 
export DEVICES=1
hadoop fs -rm -f hdfs:///mnist.model
hadoop fs -rm -r -f hdfs:///mnist_features_result
spark-submit --master yarn --deploy-mode cluster \
    --num-executors 2 \
    --files ${CAFFE_ON_SPARK}/data/lenet_memory_solver.prototxt,${CAFFE_ON_SPARK}/data/lenet_memory_train_test.prototxt \
    --conf spark.driver.extraLibraryPath="${LD_LIBRARY_PATH}" \
    --conf spark.executorEnv.LD_LIBRARY_PATH="${LD_LIBRARY_PATH}" \
    --class com.yahoo.ml.caffe.CaffeOnSpark  \
    ${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar \
        -train \
        -features accuracy,loss -label label \
        -conf lenet_memory_solver.prototxt \
        -devices 1 \
    -connection ethernet \
        -model hdfs:///mnist.model \
        -output hdfs:///mnist_features_result

Here is the error I'm getting.

When I see the log of the datanode, the error is as below. Log of datanode

Thank you very much for you answer.

YuJ.Xu
  • 1
  • 2

0 Answers0