While using tf.nn.softmax_cross_entropy_with_logits in training process, the result frequently gives nan or exceptionally big cross entropy values.
(Windows 7 64bit Python 3.6 (Anaconda 4.4.0) Tensorflow 1.4 NVIDIA Titan X Pascal CUDA 8.0 CUDNN 6.0.)
Epoch00 CrossEntropy: 1.288 L2_loss: 1247.340 Cost: 32.489
Epoch01 CrossEntropy: 0.936 L2_loss: 1019.474 Cost: 23.868
Epoch02 CrossEntropy: 0.550 L2_loss: 880.814 Cost: 14.669
Epoch03 CrossEntropy: 0.331 L2_loss: 796.639 Cost: 9.435
Epoch04 CrossEntropy: nan L2_loss: nan Cost: nan
Epoch05 CrossEntropy: nan L2_loss: nan Cost: nan
But this does not happen in another computer. Exactly the same code, the same data.
(Windows 7 64bit Python 3.6 (Anaconda 4.4.0) Tensorflow 1.3 NVIDIA GeForce GTX TITAN X CUDA 8.0 CUDNN 7.0)
Epoch00 CrossEntropy: 1.277 L2_loss: 1247.637 Cost: 32.244
Epoch01 CrossEntropy: 0.938 L2_loss: 1018.631 Cost: 23.917
Epoch02 CrossEntropy: 0.575 L2_loss: 878.269 Cost: 15.250
Epoch03 CrossEntropy: 0.345 L2_loss: 795.507 Cost: 9.766
Epoch04 CrossEntropy: 0.240 L2_loss: 741.619 Cost: 7.226
Epoch05 CrossEntropy: 0.193 L2_loss: 704.291 Cost: 6.051
I also tried in Linux Envinronment, but the results would give nan or big cross entropy values. Does anyone have any idea on this problem?
