0

I'm trying to run a Jupyter notebook that uses Caffe. Caffe is not included in datalab. I am trying to install that library from within the Jupyter notebook (as recommended in the datalab docs), but am running into problems.

I am new to datalab, and a novice with such things generally. Any advice would be very much appreciated.

The datalab documentation suggests 3 strategies for adding a python library that is not already included. I am concentrating on the first two of these strategies.

The platform for my datacloud instance is:

platform.platform() 'Linux-4.4.111+-x86_64-with-debian-stretch-sid'

Below I'll list various things I've tried and the error messages I got. For the first strategy, I tried these things in a cell of the same notebook.

(Attempt 1)

!pip install caffe
#results in the error:
#Collecting caffe
#  Could not find a version that satisfies the requirement caffe (from 
#versions: )
#No matching distribution found for caffe

!pip install caffe-cpu
#results in the same error as above

I realized from my research that caffe could not be installed with pip, so I tried:

(Attempt 2)

!apt-get install caffe
#results in the error:
#Reading package lists... Done
#Building dependency tree       
#Reading state information... Done
#E: Unable to locate package caffe

!apt-get install caffe-cpu
#results in the same error as above

Based on another stackoverflow question, I tried the following for both caffe and caffe-cpu:

(Attempt 3)

%bash
echo 'Y' | apt-get update
echo 'Y' | apt-get install caffe-cpu
#This results in output with a lot of warnings, but ends with the error:
#E: Unable to locate package caffe-cpu
#Stack Overflow prevented me from posting the entire thing, thinking it was spam

(Attempt 4)

Based on the second strategy recommended in the documentation, I tried running this code in a separate notebook:

%%bash
echo "pip install caffe" >> /content/datalab/.config/startup.sh
cat /content/datalab/.config/startup.sh
#This resulted in the error:
#bash: /usr/local/lib/libtinfo.so.5: no version information available (required by bash)

I got the same results when I ran:

%%bash
echo "apt-get install caffe" >> /content/datalab/.config/startup.sh
cat /content/datalab/.config/startup.sh
Joan Grau Noël
  • 3,084
  • 12
  • 21
Ben Bray
  • 3
  • 2

1 Answers1

0

I tried on my end to install caffe-cpu, and it seems that the file /etc/apt/sources.list doesn't have the needed repositories to install it, in the datalab instance. To workaround this issue, I used the following commands, in a created notebook:

!echo "deb http://deb.debian.org/debian stretch main\n\
deb-src http://deb.debian.org/debian stretch main\n\
deb http://deb.debian.org/debian-security/ stretch/updates main\n\
deb-src http://deb.debian.org/debian-security/ stretch/updates main\n\
deb http://deb.debian.org/debian stretch-updates main\n\
deb-src http://deb.debian.org/debian stretch-updates main" >> /etc/apt/sources.list

This will add the necessary debian/stretch repositories that include the caffe-cpu package.

Note: Strangely I didn't need to add the -e flag to the echo command in order to correctly read the newlines, you can check if the file has been correctly updated by doing !cat /etc/apt/sources.list.

Once that is done, run the following command:

!apt-get update && apt-get install caffe-cpu -y --allow-unauthenticated

Once the command finishes, the caffe-cpu package will be installed in your VM.

Joan Grau Noël
  • 3,084
  • 12
  • 21
  • Thank you so much, Joan. The debian stretch and security files do seem to be successfully appending to the sources.list file. However (and I tested this on a fresh instance, in both Python 2 and 3 notebooks), I seem to have the same problem. I try to import caffe using... caffe = __import__("caffe-cpu") ...in order to account for the hyphen, but still get the error "No module named 'caffe-cpu'". – Ben Bray Jan 10 '19 at 22:57
  • I think the issue is that the location of the `caffe` module is not in the python path. Try to add this line before importing it: `sys.path.insert(0,'/usr/lib/python3/dist-packages')`. In my end, running `!python3 -c "import sys;sys.path.insert(0,'/usr/lib/python3/dist-packages');import caffe"` did import the module. – Joan Grau Noël Jan 11 '19 at 10:45
  • That does get the import started, but I then get an error as it loads: – Ben Bray Jan 15 '19 at 03:45
  • ImportError: /usr/lib/python3/dist-packages/numpy/random/mtrand.cpython-35m-x86_64-linux-gnu.so: undefined symbol: PyFPE_jbuf In reading about this error, it seems like it would be a bit complicated to solve this issue. I hate to say it Joan, but I am starting to think of alternate approaches to this project that do not involve Caffe at all, so please don't spend too much energy :). However, if it's clear to you what's happening here, i would gladly try any suggestions you have. Thanks again for your help. – Ben Bray Jan 15 '19 at 03:54
  • This is the only thing I have found that could help, [this documentation](https://github.com/googledatalab/datalab/wiki/Getting-Started) uses a custom Docker image to build the datalab instance. I haven't been able to test it on my side, but it can work. The idea is to modify the `Dockerfile.in` file and add the lines `RUN apt update` and `RUN apt install caffe-cpu -y`, and then run the rest of the commands as stated in the docs. – Joan Grau Noël Jan 15 '19 at 14:09
  • Yeah, as I mentioned before, there were [three strategies](https://cloud.google.com/datalab/docs/how-to/adding-libraries) listed in the docs. A custom docker file was the third. I was avoiding it because I am having all kinds of problems getting docker to run on my machine (and I still am). The documentation you posted is clear and helpful, and I do agree that this is probably gonna be the way to go. If I manage to get it running on my machine, I will post the results. – Ben Bray Jan 17 '19 at 04:12