0

Hi I am new to tensorflow. I want to debug Tensorflow (skflow) gmm_ops.py (Gaussian Mixture Model). I am getting ERROR:tensorflow:Model diverged with loss = NaN. How should I do it ? Is there any example?

    raise NanLossDuringTrainingError
tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training.
Rahul
  • 645
  • 1
  • 9
  • 21

1 Answers1

0

Usually a NanLoss means something overflowed or underflowed during training. Things such as normalizing the examples or processing a subset of the data tend to help debugging what could have caused this.

Alexandre Passos
  • 5,186
  • 1
  • 14
  • 19
  • I am already using a normalized and subset of my dataset My question is how to debug the skflow code line by line like a java code. I want to evaluate the values of tensors used in the code. Please find the dataset and sample code [here](http://stackoverflow.com/questions/42551421/tensorflow-gmm-errortensorflowmodel-diverged-with-loss-nan). – Rahul Mar 07 '17 at 02:09
  • You might find the [tensorflow debugger](https://www.tensorflow.org/programmers_guide/debugger) useful for that – Alexandre Passos Mar 07 '17 at 20:21