-1

I am trying to convert a dataset of images to LMDB format for use with Caffe, and I need to call the convert_imageset function for applying this conversion from inside Matlab.

I am using Linux, and I have created a shell (.sh) script with the needed parameters for running the conversion. Here is an example of how does my shell file look like:

GLOG_logtostderr=1 /usr/local/caffe-master2/build/tools/convert_imageset -resize_height=256 -resize_width=256 images_folder data_split/train.txt data_split/dataCNN_train_lmdb

When I simply run my script from the terminal like this:

./example_shell.sh

it works without any problem. But when I try to do it from Matlab using the system() function:

system('./example_shell.sh')

it seems it is not able to open/find my files, rising the following error for each image in train.txt:

I0917 18:15:13.637830 8605 convert_imageset.cpp:82] A total of 68175 images.
I0917 18:15:13.638947 8605 db.cpp:34] Opened lmdb data_split/dataCNN_train_lmdb
E0917 18:15:13.639143 8605 io.cpp:77] Could not open or find file ...
E0917 18:15:13.639143 8605 io.cpp:77] Could not open or find file ...
E0917 18:15:13.639143 8605 io.cpp:77] Could not open or find file ...

Here are some sample lines from train.txt file (do not mind the 0s, they are just dummy labels):

/media/user/HDD_2TB/Food_101_Dataset/images/beef_carpaccio/970563.jpg 0
/media/user/HDD_2TB/Food_101_Dataset/images/chocolate_mousse/1908117.jpg 0
/media/user/HDD_2TB/Food_101_Dataset/images/cup_cakes/632892.jpg 0
/media/user/HDD_2TB/Food_101_Dataset/images/garlic_bread/1498092.jpg 0
/media/user/HDD_2TB/Food_101_Dataset/images/ceviche/3115634.jpg 0

They are absolute paths, so there should be no problem.

Any idea you might have about what could be happening can be very helpful for me!

Thank you,

Marc

Marc Bolaños Solà
  • 11
  • 5
  • what is the working directory in matlab? what is the working directory when you run `./example_shell.sh`? it seems like relative paths are different and therefore you are unable to locate files. – Shai Sep 20 '15 at 12:29
  • I am testing it from the same working path, so that can't be the problem. Thank you for the suggestion. – Marc Bolaños Solà Sep 21 '15 at 07:10
  • I would try and add an explicit `cd` command at the first line of `example_shell.sh` that puts you explicitly in the directory you should be in. – Shai Sep 21 '15 at 07:18
  • Same result with the `cd` command. – Marc Bolaños Solà Sep 21 '15 at 07:39
  • Can you please post a few lines from `data_split/train.txt` file? are the paths there relative or absolute? – Shai Sep 21 '15 at 07:41
  • Ok course! :) I just updated the main post. – Marc Bolaños Solà Sep 21 '15 at 08:33
  • it seems like your images are located on a mounded device. is it possible it is not accessible by Matlab for some reason? can you open any of the images in matlab using a simple `imread`? can you change the `images_folder` in `example_shell.sh` to `/`? is it possible some of the paths contain symbolic links? – Shai Sep 21 '15 at 08:37
  • There are no symbolic links anywhere along the path, and Matlab has no problem reading any of the images in those folders. – Marc Bolaños Solà Sep 21 '15 at 08:43
  • My guess is that maybe it is some problem about file permissions instead of paths. The thing is that I have granted all permissions (rw and even x) to the images but there is still no change. – Marc Bolaños Solà Sep 21 '15 at 08:45
  • have you granted `rx` permissions to the folders as well? – Shai Sep 21 '15 at 08:46
  • another possibility is that for some reason caffe in matlab's shell is unable to run opencv routines. can you check `LD_LIBRARY_PATH` on your terminal and in matlab? The error is emitted from [this part of the code](https://github.com/BVLC/caffe/blob/master/src/caffe/util/io.cpp#L80). – Shai Sep 21 '15 at 08:50
  • Indeed, the folders have full permissions as well. Checking the LD_LIBRARY_PATH, I have this on my terminal: /usr/local/cuda/lib64 And a lot more paths in Matlab: /usr/local/MATLAB/R2014a/sys/os/glnxa64:/usr/local/MATLAB/R2014a/bin/glnxa64:/usr/local/MATLAB/R2014a/extern/lib/glnxa64:/usr/local/MATLAB/R2014a/runtime/glnxa64:/usr/local/MATLAB/R2014a/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/usr/local/MATLAB/R2014a/sys/java/jre/glnxa64/jre/lib/amd64/server:/usr/local/cuda/lib64 – Marc Bolaños Solà Sep 21 '15 at 09:00
  • I don't think it's a CUDA related issue, but **opencv** related. can you check for any opencv related folders in `LD_LIBRARY_PATH` in both terminal and matlab? – Shai Sep 21 '15 at 09:02
  • might be relevant: http://stackoverflow.com/questions/17647403/matlab-mex-opencv-links-and-compiles-correctly-but-cant-find-library-at-ru – Shai Sep 21 '15 at 09:05
  • 1
    Those are all the paths I have in `LD_LIBRARY_PATH`. What I suppose that means that Caffe probably is not using any of those for accessing OpenCV, it must have the link somewhere else. Although, by now I have managed to do a **weird** workaround for it to work. I am calling my Matlab program from Python and checking whenever it needs to call the `./example_shell.sh`. In that case Python is in charge of doing the conversion and then the Matlab execution continues... – Marc Bolaños Solà Sep 21 '15 at 10:06

1 Answers1

0

I have not been able to solve the specific problem with Matlab, but I have managed to do the following (weird) workaround by using .txt files for communication:

  1. Call the main Matlab's program from Python.
  2. Check whenever Matlab needs to call the ./example_shell.sh script.
  3. Python does the conversion calling ./example_shell.sh.
  4. Matlab execution continues.
Marc Bolaños Solà
  • 11
  • 5