9

I have created a wrapper class which initializes a keras.models.Sequential model and has a couple of methods for starting the training process and monitoring the progress. I instantiate this class in my main file and perform the training process. Fairly mundane stuff.

My question is:

How to free all the GPU memory allocated by tensorflow. I tried the following with no luck:

import keras.backend.tensorflow_backend as K
with K.get_session() as sess:
    K.set_session(sess)
    import tensorflow as tf
    from neural_net import NeuralNet
    with tf.device('/gpu:0'):
        nn = NeuralNet('config', train_db_path, test_db_path)
        nn.train(1000, 1)
        print 'Done'
    K._SESSION.close()
    K.set_session(None)

Even after the session has been closed and reset to None, nvidia-smi does not reflect any reduction in memory usage. Any ideas?

Idea

Would it be meaningful to add a __exit__ method to my NeuralNet class and instantiate it as:

with NeuralNet() as nn:
    nn.train(1000, 1)

How should I free up the resources of the keras model in this method?

Test environment

I'm using iPython Notebook on an Ubuntu 14.04 with 3 GTX 960 GPUs.

Reference:

  1. https://github.com/fchollet/keras/issues/2102
  2. https://groups.google.com/forum/#!topic/keras-users/MFUEY9P1sc8
Chintak
  • 365
  • 4
  • 14

1 Answers1

4

The following works for me to reinitialize the state of Keras layers in my Jupyter notebook for every run:

from keras import backend as K
K.clear_session()
sess = tf.Session()
K.set_session(sess)

Also, the graph is named and reset every time the notebook runs using:

graphr = K.get_session().graph
with graphr.as_default():
    #...graph building statements...

Note : I am still trying to wrap my head around the concepts of Keras and tensorflow ( I believe they are described poorly in documentation and examples ) but the above works.

xor007
  • 976
  • 2
  • 12
  • 21
  • Does clearing a session keep all existing models and their weights? – Daniel Möller May 30 '17 at 19:11
  • Weights are cleared, graphs are kept, sorry for late response. Somehow I did not receive notifications... – xor007 Jul 25 '17 at 12:31
  • @xor007: Correct me if I am wrong. If weights are cleared then what is the use of using `clear_session`? All the trained weights will be lost. – Srinivas Valekar Jun 17 '18 at 09:06
  • @SrinivasValekar, if you need to keep the weights, indeed you should not call `clear_session`. However, when I am experimenting with hyper parameters, I sometimes want to "forget" learned weights on a graph that is in a running Jupyter notebook. In this case, `clear_session` does it. – xor007 Jun 18 '18 at 14:48
  • @xor007: Thanks for the quick response, could you please tell me which will be the best way to tackle `MemoryError` for training a model? I am getting MemoryError after 1st epoch. Here is what I am doing - After the first epoch I am saving the model then using `clear_session` and loading the model and retraining it. Is this the correct way? – Srinivas Valekar Jun 19 '18 at 09:40