1

I am trying to get CaffeOnSpark running locally and I follow this procedure on CaffeOnSpark wiki:https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_local

The first 4 items are good to me.However, when i am doing the Make build in Step 5. I got this error on protoc:

The whole error stack:

jiaqi@jiaqi-ThinkPad-L450:~/github/CaffeOnSpark$ make build
cd caffe-public; make proto; make -j4 -e distribute; cd ..
make[1]: Entering directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
make[1]: Nothing to be done for 'proto'.
make[1]: Leaving directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
make[1]: Entering directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
CXX/LD -o .build_release/tools/convert_imageset.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_binary.bin
CXX/LD -o .build_release/tools/compute_image_mean.bin
CXX/LD -o .build_release/tools/caffe.bin
.build_release/tools/compute_image_mean.o: In function `std::string* google::MakeCheckOpString<int, int>(int const&, int const&, char const*)':
compute_image_mean.cpp:(.text._ZN6google17MakeCheckOpStringIiiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringIiiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/compute_image_mean.o: In function `std::string* google::MakeCheckOpString<unsigned long, int>(unsigned long const&, int const&, char const*)':
compute_image_mean.cpp:(.text._ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/compute_image_mean.o: In function `main':
compute_image_mean.cpp:(.text.startup+0x168): undefined reference to `google::SetUsageMessage(std::string const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/compute_image_mean.bin' failed
make[1]: *** [.build_release/tools/compute_image_mean.bin] Error 1
make[1]: *** Waiting for unfinished jobs....
.build_release/tools/convert_imageset.o: In function `std::string* google::MakeCheckOpString<unsigned long, int>(unsigned long const&, int const&, char const*)':
convert_imageset.cpp:(.text._ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/convert_imageset.o: In function `main':
convert_imageset.cpp:(.text.startup+0x347): undefined reference to `google::SetUsageMessage(std::string const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/convert_imageset.bin' failed
make[1]: *** [.build_release/tools/convert_imageset.bin] Error 1
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/upgrade_net_proto_binary.bin' failed
make[1]: *** [.build_release/tools/upgrade_net_proto_binary.bin] Error 1
.build_release/tools/caffe.o: In function `std::string* google::MakeCheckOpString<unsigned long, int>(unsigned long const&, int const&, char const*)':
caffe.cpp:(.text._ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc[_ZN6google17MakeCheckOpStringImiEEPSsRKT_RKT0_PKc]+0x50): undefined reference to `google::base::CheckOpMessageBuilder::NewString()'
.build_release/tools/caffe.o: In function `main':
caffe.cpp:(.text.startup+0x50): undefined reference to `google::SetVersionString(std::string const&)'
caffe.cpp:(.text.startup+0x83): undefined reference to `google::SetUsageMessage(std::string const&)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB**)'
.build_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
.build_release/lib/libcaffe.so: undefined reference to `leveldb::Status::ToString() const'
collect2: error: ld returned 1 exit status
Makefile:613: recipe for target '.build_release/tools/caffe.bin' failed
make[1]: *** [.build_release/tools/caffe.bin] Error 1
make[1]: Leaving directory '/home/jiaqi/github/CaffeOnSpark/caffe-public'
export LD_LIBRARY_PATH="/home/y/lib64:/home/y/lib64/mkl/intel64:/home/jiaqi/github/CaffeOnSpark/caffe-public/distribute/lib:/home/jiaqi/github/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64 "; mvn -B package
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.yahoo.ml:caffe-grid:jar:0.1-SNAPSHOT
[WARNING] The expression ${version} is deprecated. Please use ${project.version} instead.
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] caffe
[INFO] caffe-distri
[INFO] caffe-grid
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-distri 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (proto) @ caffe-distri ---
[INFO] Executing tasks

protoc:
     [exec] make[1]: Entering directory '/home/jiaqi/github/CaffeOnSpark/caffe-distri'
     [exec] make[1]: Leaving directory '/home/jiaqi/github/CaffeOmake[1]: *** No rule to make target '../caffe-public/distribute/proto/caffe.proto', needed by 'src/main/java/caffe/Caffe.java'. nSpark/caffe-distri'
     [exec]  Stop.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] caffe .............................................. SUCCESS [  0.019 s]
[INFO] caffe-distri ....................................... FAILURE [  2.425 s]
[INFO] caffe-grid ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.144 s
[INFO] Finished at: 2016-05-13T12:33:40+08:00
[INFO] Final Memory: 9M/238M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (proto) on project caffe-distri: An Ant BuildException has occured: exec returned: 2
[ERROR] around Ant part ...<exec failonerror="true" dir="/home/jiaqi/github/CaffeOnSpark/caffe-distri" executable="make">... @ 5:97 in /home/jiaqi/github/CaffeOnSpark/caffe-distri/target/antrun/build-protoc.xml
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :caffe-distri
Makefile:9: recipe for target 'build' failed
make: *** [build] Error 1

It seems protoc could not find its rule. Have you ever encoutered this issue? Please help me out!

My protoc's version is 2.5.0.

user3584021
  • 11
  • 5

3 Answers3

2

I encountered build errors with CaffeOnSpark before. I know this post is old, but anyways it might help someone.

From the logs I think that you have some dependency issues. CaffeOnSpark requires all dependencies of Caffe met. So the first question I want to ask is whether Caffe is working or not (if installed).

  1. If Caffe is not installed, you could try to install Caffe and verify its working by opening python and typing import caffe. If Caffe is not working, go to step 5.

  2. If caffe is working, verify that you executed the exact command to download the git:

    git clone https://github.com/yahoo/CaffeOnSpark.git --recursive

  3. Then please check your Makefile.config in CaffeOnSpark/caffe-public. Make sure that all the flags are properly given. After checking them, run these:

    cd CaffeOnSpark make clean make build

  4. If you encounter this error:

    make[1]: Leaving directory `/home/das/CaffeOnSpark/caffe-public'
export LD_LIBRARY_PATH="/usr/local/cuda-7.5/lib64:/home/das/CaffeOnSpark/caffe-public/distribute/lib:/home/das/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64 "; mvn -B package -DskipTests
/bin/sh: 1: mvn: not found
make: *** [build] Error 127

    run this command to install maven:

    sudo apt-get install maven

  5. If you are still getting errors, you might want to install reinstall/check the dependencies again. I will post the instructions just in case. Please note that many of these are not actually a requirement. I install these to set up my Deep Learning machine:

    sudo apt-get update
sudo apt-get install -y build-essential cmake g++ gfortran git pkg-config python-dev software-properties-common
sudo apt-get install -y python-pip python-dev python-numpy python-scipy python-nose python-h5py python-skimage python-matplotlib python-pandas python-sklearn python-sympy
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler libgflags-dev libgoogle-glog-dev liblmdb-dev

    and install the requirements from Caffe root folder:

    sudo pip install -r python/requirements.txt

  6. If there is a BLAS issue, install it properly:

    git clone https://github.com/xianyi/OpenBLAS.git
cd OpenBLAS
make FC=gfortran -j 4
sudo make PREFIX=/usr/local install

  7. Please make sure that the .bashrc file has all the paths correct.

    export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
export CAFFE_ROOT=/home/das/Caffe
export PYTHONPATH=$CAFFE_ROOT/python:$PYTHONPATH
export CAFFE_ON_SPARK=/home/das/CaffeOnSpark
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
export HADOOP_HOME=/usr/local/hadoop
export SPARK_HOME=/usr/local/spark
export LD_LIBRARY_PATH=${CAFFE_ON_SPARK}/caffe-public/distribute/lib:${CAFFE_ON_SPARK}/caffe-distri/distribute/lib

..and if there is still an error, please post it so that we can look into it.

Thanks.

Arun Das
  • 333
  • 3
  • 11
0

There should be an error message early on that says blas library cannot be found. By default Caffe uses atlas library which can be found in the Makefile.config: BLAS := atlas, if you have other blas library installed, say mkl, you should use BLAS := mkl

Xiangyu
  • 824
  • 9
  • 34
  • Hi, Xiangyu! I have edited my question with posting the whole error stack. However, I didn't find the BLAS library not found error. – user3584021 May 13 '16 at 04:43
  • It seems you are missing opencv as well as some other google libraries. Have you looked at the instructions at: http://caffe.berkeleyvision.org/install_apt.html – Xiangyu May 13 '16 at 13:52
-1

I try to compile CaffeOnSpark on Ubuntu 16.04, before the GetStarted_standalone reference I installed maven-3.0.5 and hadoop-2.7.1 and spark-2.0.0-bin-hadoop2.7 from their corresponding website. And I meet the similar problem, the error like:

[INFO] Compiling 10 source files to /home/bdg/usr/CaffeOnSpark/caffe-distri/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/bdg/usr/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[9919,31] cannot find symbol
  symbol:   class ProtocolStringList
  location: package com.google.protobuf
[ERROR] /home/bdg/usr/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[8280,24] cannot find symbol
  symbol:   class ProtocolStringList
  location: package com.google.protobuf
...
[ERROR] /home/bdg/usr/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[5655,26] cannot find symbol
  symbol:   method getUnmodifiableView()
  location: variable input_ of type com.google.protobuf.LazyStringList
[ERROR] /home/bdg/usr/CaffeOnSpark/caffe-distri/src/main/java/caffe/Caffe.java:[6395,26] cannot find symbol
  symbol:   method getUnmodifiableView()
  location: variable input_ of type com.google.protobuf.LazyStringList
...
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :caffe-distri
Makefile:19: recipe for target 'build' failed
make: *** [build] Error 1
BDGong
  • 1
  • 2