Caffe+GPU+Opencv3.1+Python3.5+Anaconda:fatal error: Python.h: No such file or directory

Question

Briefly I want to use Caffe these day for my project. My OS is Ubuntu 14.04, with Opencv3.1+Python3.5+Anaconda+GPU I have already passed all:

make all
make pycaffe
make test
make runtest

However when can try to make pycaffe, it cannot pass:

Python.h: No such file or directory

Here is my 'makefile.config', and I am sure the 'Python.h' has already in the path, which make me quite confused.

USE_CUDNN := 1
OPENCV_VERSION := 3
ANACONDA_HOME := $(HOME)/anaconda3
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
         $(ANACONDA_HOME)/include/python3.5m \
         $(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include \
PYTHON_LIB := $(ANACONDA_HOME)/lib
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib
USE_PKG_CONFIG := 1
PYTHON_LIBRARIES := boost_python3 python3.5m
PYTHON_INCLUDE := /usr/include/python3.5m \
                 /usr/lib/python3.5/dist-packages/numpy/core/include

Because I use Python3.5, so I uncomment the following:

PYTHON_INCLUDE := /usr/include/python2.7 \
        /usr/lib/python2.7/dist-packages/numpy/core/include
PYTHON_LIB := /usr/lib

I really appreciate someone could help,

score 4 · Answer 1 · answered Oct 05 '16 at 12:43

4

You have two definitions for PYTHON_INCLUDE: you need to decide if you go for the "python3" flavor, or the "anaconda" flavor...

Where is your python.h file anyway? try in shell

find / -name "Python.h" -type f

and see where it actually is. Then pick the correct settings for PYTHON_INCLUDE in your makefile.config

answered Oct 05 '16 at 12:43

Shai

111,146
38
238
371

1

Your help is really appreciate, I have solved the problem. – Kaku Oct 10 '16 at 13:03
@Kaku put yourself in the shoes of the next person that will have this problem, can s/he solve it with the information in this post? If not, you might want to consider posting your own answer describing how you solved this problem. – Shai Oct 10 '16 at 13:06

score 3 · Accepted Answer · edited Oct 10 '16 at 14:23

I almost spent(waisted) one week to configure caffe on Ubuntu 14.04, the reason why it too time consuming is that I am using the newest version of Opencv Python and anaconda. Here I want to share my experience.

Makefile.config

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1
# Uncomment if you’re using OpenCV 3
OPENCV_VERSION := 3
# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
-gencode arch=compute_20,code=sm_21 \
-gencode arch=compute_30,code=sm_30 \
-gencode arch=compute_35,code=sm_35 \
-gencode arch=compute_50,code=sm_50 \
-gencode arch=compute_50,code=compute_50
# BLAS choice: atlas for ATLAS (default)
BLAS := atlas
# We need to be able to find Python.h and numpy/arrayobject.h.
ANACONDA_HOME := $(HOME)/anaconda3
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
$(ANACONDA_HOME)/include/python3.5m \
$(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include \
# Uncomment to use Python 3 (default is Python 2)
PYTHON_LIBRARIES := boost_python3 python3.5m
# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := $(ANACONDA_HOME)/lib
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib
# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute
# The ID of the GPU that ‘make runtest’ will use to run unit tests.
TEST_GPUID := 0
# enable pretty build (comment to see full commands)
Q ?= @

/.bashrc

#Caffemake
export PYTHONPATH=~/caffe/python/:$PYTHONPATH
#Opencv
export LD_LIBRARY_PATH=/home/kaku/anaconda3/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=”/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH”

Notes:

1. library must be installed:
libboost-all-dev, although in some tutorial mentioned must install libboost1.55-all-dev.
protobuf-cpp-3.0.0-beta-2.zip or upper one
protobuf-python-3.0.0-beta-2.zip or upper one
http://blog.csdn.net/lien0906/article/details/51784191
https://github.com/google/protobuf/issues/1276

Other Debug: Details in my own blog.

Excellent! can you please highlight the **changes** you made from default Makefile? — Shai, Oct 10 '16 at 14:25
I still do not understand what is the root cause and why I cannot use the system default python2 or python3 include folder, what is really causing this issue, I am on Gentoo system and in same troubles. — kensai, May 23 '17 at 11:39

score 1 · Answer 3 · answered May 23 '17 at 13:00

After facing same issue and being on Gentoo system I tried something else. I have 2 python instances installed at same time via Gentoo slots:

ares ~ # eselect python list
Available Python interpreters, in order of preference:
  [1]   python3.4
  [2]   python2.7

My default was 2.7, so I tried to switch to 3.4. Well problem is that it required some changes in 2 files.

I note that similar changes with 2.7 simply didn't work, the path was correct, but something is broken underlying...

Makefile.config file I changed to work with Python 3 (3.4) :

PYTHON_LIBRARIES := boost_python3 python3.4m
PYTHON_INCLUDE := /usr/include/python3.4m \
                 /usr/lib64/python3.4/site-packages/numpy/core/include

Still, when you just change this, it won't work as soon as CMake still points to 2.7. I checked by doing:

mkdir build; cd build;cmake ..;

And the output was:

-- Python:
--   Interpreter       :   /usr/bin/python2.7 (ver. 2.7.12)
--   Libraries         :   /usr/lib64/libpython2.7.so (ver 2.7.12)
--   NumPy             :   /usr/lib64/python2.7/site-packages/numpy/core/include (ver 1.12.1)

So I changed this line in CMakeLists.txt file:

set(python_version "2" CACHE STRING "Specify which Python version to use")

To (change value 2 to 3):

set(python_version "3" CACHE STRING "Specify which Python version to use")

And did cmake again(after cleanup) and got finally:

-- Python:
--   Interpreter       :   /usr/bin/python3 (ver. 3.4.5)
--   Libraries         :   /usr/lib64/libpython3.4m.so (ver 3.4.5)
--   NumPy             :   /usr/lib64/python3.4/site-packages/numpy/core/include (ver 1.12.1)

Now the make -j8 command finish without issues. I note I used multithread option on compile (-j8) as I found on some forums suggesting to go only with -j1 (single thread), so this was not the case for me.

Thanks for this detailed answer. I was having another version of python and this solution worked for me. — mohamed_abdullah, Oct 10 '21 at 14:19

score 0 · Answer 4 · answered Sep 25 '17 at 03:30

I have the followings in my Make.config:

PYTHON_LIB := /usr/lib

PYTHON_INCLUDE :=  /usr/include/python2.7 \
                /usr/lib/python2.7/dist-packages/numpy/core/include

And the following in my ~/.bashrc:

export PYTHONPATH=$HOME/caffe/python
export CAFFE_ROOT=$HOME/caffe

You have to run the following in cd $CAFFE_ROOT: make all make pycaffe make test make runtest

My setup is in CentOS and for Python 2.7 but it should be a similar idea.

[jalal@ivcgpu1 caffe]$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.4.1708 (Core) 
Release:    7.4.1708
Codename:   Core
[jalal@ivcgpu1 caffe]$ uname -a
Linux ivcgpu1.bu.edu 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Caffe+GPU+Opencv3.1+Python3.5+Anaconda:fatal error: Python.h: No such file or directory

4 Answers4