because my train model is on multiple GPU, during training , I use trainer.extend to validate my model every epoch, But chainer.training.extensions.Evaluator has only one device argument, but my model network parameter is on 2 GPUs. how to validate in this situation?
Asked
Active
Viewed 221 times
1 Answers
0
The device argument of Evaluator indicates to which device the input data should be put. It does nothing about model parameters.
If your model needs the input data loaded to multiple GPUs as well, Evaluator does not support such case directly so you have to customize it. The easiest way is setting a custom converter function (which is responsible on copying the data to appropriate devices). See the document of Evaluator for the details of converter. https://docs.chainer.org/en/stable/reference/generated/chainer.training.extensions.Evaluator.html
Seiya Tokui
- 341
- 2
- 3
-
Do you mean the model parameter and data can be on two different GPU? then how can they to calculate together in this situation?(for example Wx + b but W and x are on 2 different GPU)? – machen Sep 15 '17 at 02:36
-
I can't fully understand this – machen Sep 15 '17 at 02:37
-
In that case, you have to manually send x to the GPU on which W exist. – Seiya Tokui Oct 23 '17 at 01:15