Caffe/pyCaffe: set all GPUs

Question

is possible to set all GPUs for Caffe (especially pyCaffe)?

Something like:

caffe train -solver examples/mnist/lenet_solver.prototxt -gpu all

related: http://stackoverflow.com/q/33446612/1714410 – Shai Nov 16 '15 at 08:21 — Shai, Nov 16 '15 at 08:21

score 7 · Answer 1 · answered Jan 22 '16 at 18:25

Both forks have supported multi-GPU for a while now.

BVLC/caffe got support for multi-GPU on 08/13/2015 (see commit, issue).
NVIDIA/caffe got support for multi-GPU on 06/19/2015 (see release note).

You may be interested to know that there is a pretty serious outstanding issue with multi-GPU and PythonLayers (see issue, temporary fix).

score 6 · Accepted Answer · answered Nov 16 '15 at 07:03

6

AFAIK Caffe is not supporting multi gpu training at the moment. It is planned for future release. See a discussion here.
It seems like NVIDIA's branch of caffe has this functionality. See the issue here.

answered Nov 16 '15 at 07:03

Shai

111,146
38
238
371

Seems this is outdated info? – mrgloom Aug 23 '16 at 11:26
@Shai: Any update on this? is it still not possible to utilize multiple GPUs in Pycaffe? – Hossein Apr 26 '18 at 18:19
@Shai: Thanks alot :) – Hossein Apr 26 '18 at 18:27

score 4 · Answer 3 · answered Nov 18 '15 at 15:28

4

It seems that Caffe now starts supporting training on multi-GPUs for C++ interface. See the docs on CommandLine interface.

# train on GPUs 0 & 1 (doubling the batch size)
caffe train -solver examples/mnist/lenet_solver.prototxt -gpu 0,1
# train on all GPUs (multiplying batch size by number of devices)
caffe train -solver examples/mnist/lenet_solver.prototxt -gpu all

answered Nov 18 '15 at 15:28

beahacker

1,660
14
10

How much speedup did you gain? – nn0p Jul 17 '17 at 06:04
Hi @nn0p, the speedup is nearly linear. – beahacker Jul 19 '17 at 12:14
with or without NCCL? And I think the net's size also affect the speedup ratio. – nn0p Jul 19 '17 at 15:51
Hi @nn0p, NCCL is only a library to read data, not for computation. – beahacker Jul 21 '17 at 05:54

Caffe/pyCaffe: set all GPUs

3 Answers3

Linked