0

the same training code that works on one machine gets the following error on a different machine. The first machine is a SurfaceBook, the other is my desktop with a GTX 1080. Any idea what's going wrong here?

The error is:

Traceback (most recent call last): File "MultitaskAtis.py", line 162, in trainer_intent.train_minibatch(mb) File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py34\lib\site-packages\cntk\trainer.py", line 90, in train_minibatch updated = super(Trainer, self).train_minibatch(arguments, device) File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py34\lib\site-packages\cntk\cntk_py.py", line 1768, in train_minibatch return _cntk_py.Trainer_train_minibatch(self, *args) RuntimeError: Node '__v2libuid__Plus502__v2libname__Plus225' (Plus operation): DataFor: FrameRange's dynamic axis is inconsistent with matrix: {numTimeSteps:1, numParallelSequences:14, sequences:[{seqId:0, s:0, begin:0, end:1}, {seqId:1, s:1, begin:0, end:1}, {seqId:2, s:2, begin:0, end:1}, {seqId:3, s:3, begin:0, end:1}, {seqId:4, s:4, begin:0, end:1}, {seqId:5, s:5, begin:0, end:1}, {seqId:6, s:6, begin:0, end:1}, {seqId:7, s:7, begin:0, end:1}, {seqId:8, s:8, begin:0, end:1}, {seqId:9, s:9, begin:0, end:1}, {seqId:10, s:10, begin:0, end:1}, {seqId:11, s:11, begin:0, end:1}, {seqId:12, s:12, begin:0, end:1}, {seqId:13, s:13, begin:0, end:1}]} vs. {numTimeSteps:23, numParallelSequences:11, sequences:[{seqId:0, s:0, begin:0, end:16}, {seqId:1, s:1, begin:0, end:10}, {seqId:2, s:1, begin:10, end:21}, {seqId:3, s:2, begin:0, end:15}, {seqId:4, s:3, begin:0, end:23}, {seqId:5, s:4, begin:0, end:11}, {seqId:6, s:5, begin:0, end:14}, {seqId:7, s:6, begin:0, end:14}, {seqId:8, s:4, begin:11, end:22}, {seqId:9, s:7, begin:0, end:11}, {seqId:10, s:8, begin:0, end:14}, {seqId:11, s:9, begin:...

user1118332
  • 73
  • 6

1 Answers1

0

This is likely to be a bug with respect to GPU memory usage in the two device. Assuming you are able to run the code well on CPU device on both Surface book and Desktop.

Sayan Pathak
  • 870
  • 4
  • 7
  • Can you expand a bit more? how do you know it's memory usage? From the error message, I would have expected it to be to do with data sizes. – Anton Schwaighofer Jan 13 '17 at 10:21