6

I can't save a keras model when using a lambda layer and shared variables. Here's a minimal code that gives this error :

# General imports.
import numpy as np

# Keras for deep learning.
from keras.layers.core import Dense,Lambda
from keras.layers import Input
from keras.models import Model
import keras.backend as K

n_inputs = 20
n_instances = 100

def preprocess(X,minimum,span):
    output = (X - minimum)/span
    return output

inputs = Input(shape=(n_inputs,),name='input_tensor')
maximum = K.max(inputs)
minimum = K.min(inputs)
span = maximum - minimum

x = Lambda(preprocess,arguments={'minimum':minimum,'span':span})(inputs)
x = Dense(units=100,activation='elu')(x)
outputs = Dense(units=n_inputs,activation='elu')(x)

model = Model(inputs=inputs,outputs=outputs)
model.compile(optimizer='adam', loss='mse')

x = np.array([np.random.randn(20) for i in range(n_instances)])
y = np.array([np.random.randn(20) for i in range(n_instances)])

model.fit(x,y,epochs=10)
model.save('test.h5')       # This line doesn't work.

And here's the full error I am getting :

Traceback (most recent call last):
  File "C:\Dropbox\HELMo_Gramme\M2\Stage\Programmation\Filter\sanstitre0.py", line 35, in <module>
    model.save('test.h5')       # This line doesn't work.
  File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py", line 2580, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "C:\ProgramData\Anaconda3\lib\site-packages\keras\models.py", line 111, in save_model
    'config': model.get_config()
  File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py", line 2421, in get_config
    return copy.deepcopy(config)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle _thread.lock objects

The model can be trained and can be used to predict, the issue only appears when saving. I have seen similar errors (e.g. this link) for people using lambda layers but I think my issue is slightly different (most of the times it was using seq2seq.py which I don't use). However it still seems to be linked with deep copy.

If I remove the lambda layer or the external variables it works. I am probably doing something I shouldn't be doing with the variables but I don't know how to do it properly. I need them to be outside of the scope of the preprocess function because I use those same variable in a postprocess function.

I understand that preprocessing inside the model is not the most efficient but I have my reasons to do it and performance is not an issue on this dataset.

Update

I forgot to clarify that I would like to be able to reuse maximum, minimum, and span in another Lambda layer, which is why they are defined outside the scope of preprocess.

Update 2

maxim's solution did help, but it still didn't work in my actual code. The difference is that I actually create my model inside a function and return it, and somehow this returns the same type of error.

Example code:

# General imports.
import numpy as np

# Keras for deep learning.
from keras.layers.core import Dense,Lambda
from keras.layers import Input
from keras.models import Model
import keras.backend as K

n_inputs = 101
n_instances = 100

def create_model(n_inputs):

    def preprocess(X):
        maximum = K.max(inputs)
        minimum = K.min(inputs)
        span = maximum - minimum
        output = (X - minimum)/span
        return output

    def postprocess(X):
        maximum = K.max(inputs)
        minimum = K.min(inputs)
        span = maximum - minimum
        output = X*span + minimum
        return output

    inputs = Input(shape=(n_inputs,),name='input_tensor')

    x = Lambda(preprocess)(inputs)
    x = Dense(units=100,activation='elu')(x)
    outputs = Dense(units=n_inputs,activation='elu')(x)
    outputs = Lambda(postprocess)(outputs)

    model = Model(inputs=inputs,outputs=outputs)
    model.compile(optimizer='adam', loss='mse')
    return model

x = np.array([np.random.randn(n_inputs) for i in range(n_instances)])
y = np.array([np.random.randn(n_inputs) for i in range(n_instances)])

model = create_model(n_inputs)
model.fit(x,y,epochs=10)
model.save('test.h5')       # This line doesn't work.

Error :

Traceback (most recent call last):
  File "C:\Dropbox\HELMo_Gramme\M2\Stage\Programmation\Filter\sanstitre0.py", line 46, in <module>
    model.save('test.h5')       # This line doesn't work.
  File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py", line 2580, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "C:\ProgramData\Anaconda3\lib\site-packages\keras\models.py", line 111, in save_model
    'config': model.get_config()
  File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py", line 2421, in get_config
    return copy.deepcopy(config)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "C:\ProgramData\Anaconda3\lib\copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle _thread.lock objects

I need to create my model in a function because I am optimizing hyperparameters and so I iterate over different models with different set of parameters (well I could probably do it in a loop but it's not as nice).

Maxim
  • 52,561
  • 27
  • 155
  • 209
Flabou
  • 115
  • 7

3 Answers3

2

The problem is with lambda arguments: minimum and span. They are deduced from the input, but when you define the lambda layer like this:

x = Lambda(preprocess,arguments={'minimum':minimum,'span':span})(inputs)

... they are considered independent arguments that need to be serialized (as a context for lambda). This results in error, because both of them are tensorflow tensors, not static values or a numpy arrays.

Change your code to this:

# `preprocess` encapsulates all intermediate values in itself.
def preprocess(X):
  maximum = K.max(X)
  minimum = K.min(X)
  span = maximum - minimum
  output = (X - minimum) / span
  return output

inputs = Input(shape=(n_inputs,), name='input_tensor')
x = Lambda(preprocess)(inputs)
Maxim
  • 52,561
  • 27
  • 155
  • 209
  • It does work but I should have clarified I would like to be able to reuse `maximum`, `minimum` and `span` in a second lambda function without having to recompute them (which is why I defined them outside the scope of preprocess). Is there a way to do this? – Flabou Mar 22 '18 at 10:25
  • @Flabou in this case they are fairly cheap to compute, so I wouldn't bother. – Maxim Mar 22 '18 at 10:28
  • agreed but it is still interesting to know what is the good practice to have. – Flabou Mar 22 '18 at 10:32
  • Lambdas are designed to be self-contained, so if I needed something more complicated, I'd use custom layers - https://keras.io/layers/writing-your-own-keras-layers/ – Maxim Mar 22 '18 at 10:39
  • I've encountered a second error in my actual code because the model is created inside a function (I've updated my question for this issue). If you have any idea I would be grateful! – Flabou Mar 22 '18 at 11:37
  • @Flabou OK. I've updated the answer. Now lambda is *really* self-contained. – Maxim Mar 22 '18 at 11:58
  • I'm sorry as I have not been very clear as to what my goal is, I've updated my code in the "Update 2" part. In `postprocess` I must use the max and min values of `input` so I can't use `X`. I don't understand why it works when not defined in a function and doesn't when defined in the function though. Is it not enough that `preprocess`, `postprocess` and `inputs` are in the same scope? – Flabou Mar 22 '18 at 12:26
0

I had the same problem (Lambda layer + multiple arguments on keras@2.1.5 with tensorflow@1.11.0 backend) and model.save_weights(...)instead of model.save(...) works - if you just want to load the trained weights later and you don't need to store the architecture.

Fabian
  • 3,139
  • 2
  • 23
  • 49
0

See my answer on a similar question here on Stackoverflow.

For your particular case, change these lines:

(...)
x = Lambda(preprocess)(inputs)
(...)
outputs = Lambda(postprocess)(outputs)
(...)

with these lines:

(...)
x = Lambda(lambda t: preprocess(t))(inputs)
(...)
outputs = Lambda(lambda t: postprocess(t))(outputs)
(...)
Alaroff
  • 2,178
  • 1
  • 15
  • 9