1

I'm using optuna package to optimize my model, and I'm using the option to run multiple job at a time. when I ran this I get:

Trial  failed because of the following error: ValueError('The name "dense" is used 3 times in the model. All layer names should be unique.'

But I didn't assign any names to the layers. I had the tensorflow.keras.backend.clear_session(), and when I remove it I don't get the error anymore. is this ok to remove it? what are the impacts? is there other solution to this?

Mr.O
  • 342
  • 2
  • 19
  • Layers name should be unique, for more clarification, you should share your model. – ashraful16 Sep 13 '20 at 02:01
  • @ashraful , As I said, I didn't give any names to the layers, keras is creating them automatically. when I using only job, I don't get the error. I can't provide the model as the code is complicated to understand and it contains a lot of modules. – Mr.O Sep 13 '20 at 02:07

2 Answers2

3
tf.keras.backend.reset_uids()

is all you need.

mowang niu
  • 46
  • 2
1

Based on similar experiences on a GPU-trained model in tensorflow, you are dealing with the fact that GPU memory is not automatically cleared out like CPU RAM. I would suggest the following solution (please let me know if it does not work):

from tensorflow.keras.backend import clear_session
import gc

and then call clear_session() before you define any algorithm_model that will be used for a cross-validation run. Also, at the end of that run/replicate, after you have returned the objective function score, call the following two lines to clear out your memory:

gc.collect()
del algorithm_model

For more details on a cross-validation that did not use Optuna, see this answer to a related question. And no, you cannot get rid of the call to clear_session() because that is what clears the model off your GPU memory!

brethvoice
  • 350
  • 1
  • 4
  • 14
  • 1
    Here is an example of using Optuna where they keep the clear_session() call: https://medium.com/@Minyus86/optkeras-112bcc34ec73 (note, they are using keras, not tf.keras, which appaently causes problems, so you might have to edit their code. – brethvoice Oct 06 '20 at 19:17