How to transfer weights from baseline model to federated model?

Question

def create_keras_model():
model = Sequential([
    Conv2D(16, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Conv2D(32, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Conv2D(64, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Flatten(),
    Dense(512, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.001)),
    Dropout(0.5),
    Dense(1, activation='sigmoid')
])

model.load_weights('/content/drive/My Drive/localmodel/weights')
return model

Tried something like this in Colab, but I get errno 21, is a directory.

Then I tried another method as shown below,

tff_model = create_keras_model() #now this function doesnt load weights, just returns a Sequential model   
tff.learning.assign_weights_to_keras_model(tff_model, model_with_weights)

Just like assign_weights_to_keras_model() transfers weights from tff_model to keras model, I want to transfer weights from keras model to tff_model. How can this be done?

*Re: errno 21*: `model.load_weights` is calling [`tf.keras.Model.load_weights()`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#load_weights) and it seems like it might be pointing at a path that was not generated by the paired [`tf.keras.Model.save_weights()`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#save_weights)? — Zachary Garrett, Jun 17 '20 at 02:36
This error is coming from the Keras library inside TF core, unrelated to TFF. It might be good to ask a question specifically about this error in the `tensorflow` tag. — Zachary Garrett, Jun 17 '20 at 15:33
Okay, I get it. And yes, I have already tagged tf. Lets hope! — gokulan vikash, Jun 17 '20 at 20:10

score 3 · Answer 1 · answered Jun 16 '20 at 12:12

3

here model_with_weights must be a TFF value representing the weights of a model for example:

def model_fn():

    keras_model = create_keras_model()

  return tff.learning.from_keras_model(keras_model)

fed_avg = tff.learning.build_federated_averaging_process(model_fn, ...)
state = fed_avg.initialize()
state = fed_avg.next(state, ...)
...
tff.learning.assign_weights_to_keras_model(keras_model, state.model)

answered Jun 16 '20 at 12:12

Ayness

127
1
1
8

Thanks @Ayness. But, I want to assign weights from a baseline model to a tff model, not the other way around. Lets say my baseline model gives an accuracy of 70%. I want my tff model to take it from there and start training. Any idea how? – gokulan vikash Jun 16 '20 at 22:45

score 1 · Answer 2 · answered Jun 17 '20 at 00:19

1

I just got to know how this can be done. The idea is to use:

tff.learning.state_with_new_model_weights(state, trainable_weights_numpy, non_trainable_weights_numpy)

Documentation here

where trainable weights are taken from baseline model and converted to numpy format.

trainable_weights = []

for weights in baseline_model.trainable_weights:
    trainable_weights.append(weights.numpy())

This is particularly useful when the server has part of the data and the client has similar data. May be this can be used for transfer learning.

answered Jun 17 '20 at 00:19

gokulan vikash

43
6

Should have caught this earlier--see also the [answer here](https://stackoverflow.com/questions/61786305/tff-loading-a-pre-trained-keras-model), pointing to the same solution. – Keith Rush Jun 17 '20 at 04:49
Thanks for the reference. Just had a look at it. Wish I had looked at it earlier. – gokulan vikash Jun 17 '20 at 10:10

How to transfer weights from baseline model to federated model?

2 Answers2