Loading multiple models sequentially leading to Python crash

Question

I am trying to load multiple models sequentially for some validation tests, but using K.clear_session() is causing Python to crash on Windows.

"Python is not working/resonding" Windows error.

If I don't use K.clear_session() I run into a memory error, where I assume the graphs continue to stack, and each model progressively takes longer to load.

I am attempting the suggestions here (below) on how to use K.clear_session(), but Python keeps crashing (I get no error, just a crash)

Extremely slow model load with keras

Here is my code:

>>> print(keras.__version__)
2.2.4
>>> import tensorflow as tf
>>> tf.__version__
'1.8.0'
>>> 

Windows 7



config = tf.ConfigProto()

# Don't pre-allocate memory; allocate as-needed
config.gpu_options.allow_growth = True

# Only allow a total of half the GPU memory to be allocated
config.gpu_options.per_process_gpu_memory_fraction = 0.8

# Create a session with the above options specified.
k.tensorflow_backend.set_session(tf.Session(config=config))



def evaluate_models_2(models_path_dir,):

    models_paths = [os.path.join(models_path_dir, model) for model in os.listdir(models_path_dir) if model.endswith(".hdf5")]
    models_pairs = get_model_key(models_paths, model_keys, MODEL_KEY)
    valid_gen = double_valid_generator()
    #global model_paths
    for num,model_pair in enumerate(models_pairs):
        model_path,model_key = model_pair
        print(num, ": ", model_path)
        evaluate_validation_data_2(model_path, model_key, generator = valid_gen)


def evaluate_validation_data_2(model_path,model_key, generator = None, show_mistakes = False):

    if generator == None:
        valid_gen = double_valid_generator()
    else:
        valid_gen = generator
    #load data generator
    print("about to load model")
    model = load_model_max_layer(model_path)
    #evaluate and then clear session
    print(model.evaluate_generator(valid_gen, steps =10))
    k.clear_session()

I don't seem to have another other sessions of Python running.

If I remove K.clear_session(), python doesn't crash However, as stated earlier the models take longer and longer to load and I have around 200 models to go through.