Segmentation fault (core dumped) when training more than one Keras NN models

Question

I am optimizing the hyper-parameters of my neural-network, for which I am recursively training the network using different hyper-parameters. It works as expected until after some iterations, when creating a new network for training, it dies with the error "Segmentation fault (core dumped)".

Furthermore, I am using GPU for training and I am doing this on a Nvidia Jetson TX2 and Python3.5. Also, I am using Keras with TensorFlow backend.

If you can post the error stack, it would be easier for us to help you — pseudomonas, Jun 13 '18 at 17:19
The thing is it just dies, it does not print anything else besides that — StOchastiC_, Jun 13 '18 at 19:47
Actually I used gdb and obtained this: Thread 1 "python3.5" received signal SIGSEGV, Segmentation fault. 0x0000007f9ece1174 in tensorflow::TF_SessionReleaseCallable(TF_Session*, long, TF_Status*) () from /home/nvidia/.local/lib/python3.5/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so — StOchastiC_, Jul 13 '18 at 21:41

score 2 · Accepted Answer · answered Jul 11 '18 at 00:47

2

If you run K.clearsession() on a GPU with Keras 2, you may get a segmentation fault. If you have this in your code, try removing it!

answered Jul 11 '18 at 00:47

Megan Hardy

397
4
12

I did in fact have one on my code, but after removing it, I obtained the same result – StOchastiC_ Jul 13 '18 at 16:11
I found a second hidden K.clearsession(), after removing it, it works now! Thank you for your help – StOchastiC_ Jul 26 '18 at 14:54

Segmentation fault (core dumped) when training more than one Keras NN models

1 Answers1