root_dataset.cc:200] Optimization loop failed: CANCELLED: Operation was cancelled

Asked Dec 27 '22 at 14:24

Active Dec 27 '22 at 14:24

Viewed 210 times

I am getting this error when performing a custom training over a model, but the training seems to continue anyway... Is is something I should be worried about?

I tried with several batch size and learning rates, and more hyperparameter combination in general. The data is previously converted to tensorflow datasets before feeding them to the model

asked Dec 27 '22 at 14:24

SrTangente

These are just warning messages which will not affect code execution until there is any code error. To solve this issue, Please make sure the `batch_size` should not be larger and can also try running this code above all the code - gpus = `tf.config.experimental.list_physical_devices('GPU')`, `tf.config.experimental.set_memory_growth(gpu, True)`. Let us know if the issue still persists. Thank you. – Jan 18 '23 at 15:09

Tensorflow warning: W tensorflow/core/data/root_dataset.cc:200] Optimization loop failed: CANCELLED: Operation was cancelled

0 Answers0