When I train a DNN network using CaffeOnSpark with 2 Spark executors with Ethernet connection, I'm getting an error. I run the job as the example on the https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_yarn
export SPARK_WORKER_INSTANCES=2
export DEVICES=1
hadoop fs -rm -f hdfs:///mnist.model
hadoop fs -rm -r -f hdfs:///mnist_features_result
spark-submit --master yarn --deploy-mode cluster \
--num-executors 2 \
--files ${CAFFE_ON_SPARK}/data/lenet_memory_solver.prototxt,${CAFFE_ON_SPARK}/data/lenet_memory_train_test.prototxt \
--conf spark.driver.extraLibraryPath="${LD_LIBRARY_PATH}" \
--conf spark.executorEnv.LD_LIBRARY_PATH="${LD_LIBRARY_PATH}" \
--class com.yahoo.ml.caffe.CaffeOnSpark \
${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar \
-train \
-features accuracy,loss -label label \
-conf lenet_memory_solver.prototxt \
-devices 1 \
-connection ethernet \
-model hdfs:///mnist.model \
-output hdfs:///mnist_features_result
Here is the error I'm getting.
When I see the log of the datanode, the error is as below. Log of datanode
Thank you very much for you answer.