Random Seed for Tensorflow federated?

Question

I want to get reproducible results with my Tensorflow Federated code. For that I have implemented some seeds (random, numpy and tensorflow), but they aren't affecting Tensorflow Federated. The data processing steps are all reproducible, it has to be in the code snippet below.

I have read that Tensorflow Federated doesn't provide a global seed function and that my only possibility is to save the state. But I don't understand this argumentation. Is anyone aware of a method/function that can help me out or explain to me why I can't use seeds with Tensorflow Federated?

Appreciate every comment :) Thanks for your help.

nest_asyncio.apply()

seed_value = 0 
random.seed(seed_value)
np.random.seed(seed_value)
tf.random.set_seed(seed_value)

# designing the clients
client_train_data = collections.OrderedDict()

for i in range(1, num_clients+1): 
    client_name = "Client_{}".format(i)
    size = len(X_train)//num_clients
    start = size * (i-1)
    end = size * i 
    data = collections.OrderedDict((("label", y_train[start:end]),
                                    ("features", X_train[start:end])))
    client_train_data[client_name] = data

train_dataset = tff.simulation.FromTensorSlicesClientData(client_train_data)

def preprocess(dataset): 
    
    def batch_format(element): 
        return collections.OrderedDict(
            x = reshape(element["features"], [-1, 11]), 
            y = reshape(element["label"], [-1, 1]))

    return dataset.repeat(num_epochs).shuffle(shuffle_buffer).batch(
        batch_size).map(batch_format).prefetch(prefetch_buffer)

def make_federated_data(client_data, client_ids): 
    return [
        preprocess(client_data.create_tf_dataset_for_client(x))
        for x in client_ids
    ]

fl_train_data = make_federated_data(train_dataset, train_dataset.client_ids)

def create_keras_model(): 
    model = Sequential()
    model.add(Dense(15, input_dim=11, activation="relu"))
    model.add(Dense(15, activation="relu"))
    model.add(Dense(1, activation="sigmoid"))
    return model

def model_fl(): 
    keras_model = create_keras_model()
    return tff.learning.from_keras_model(
        keras_model,
        input_spec=fl_train_data[0].element_spec,
        loss=tf.keras.losses.BinaryCrossentropy(), 
        metrics=[tf.keras.metrics.BinaryAccuracy()])

fl_process = tff.learning.build_federated_averaging_process(
    model_fl,
    client_optimizer_fn=lambda: tf.keras.optimizers.SGD(learning_rate=0.01), 
    server_optimizer_fn=lambda: tf.keras.optimizers.SGD(learning_rate=1.00))
# initialize federated averaging
state = fl_process.initialize()
# federated rounds
for round in range(1, num_rounds+1):
    state, metrics = fl_process.next(state, fl_train_data)
    print("Runde {:2d}, metrics={}".format(round, metrics))

It would be helpful to copy/past the exact code, as where non-determinism is coming from could be a variety of places. Check that all layers of the model have deterministic initializers, any `tf.data.Dataset.shuffle()` calls set a seed, `ClientData.datasets()` calls set a seed, etc. — Zachary Garrett, Jun 15 '21 at 11:50
Thank you, I have posted the code. Aren't the Keras layers deterministic, because of the `tf.random.set_seed(seed_value)`? So, there is no global function for tff? — Stefan, Jun 15 '21 at 12:23
I'm not positive the Keras layers pickup a fixed seed from the graph. [`tf.keras.initializers.GlorotUniform`](https://www.tensorflow.org/api_docs/python/tf/keras/initializers/GlorotUniform) for example (the default for Dense layer kernels). It doesn't seem to use the graph global seed set by `tf.random.set_seed`. — Zachary Garrett, Jun 15 '21 at 18:07
Yes, you are right. If I set the initializer to `tf.keras.initializers.GlorotUniform(seed_value)` the whole FL approach is reproducible. Thanks a lot. But what confuses me is that the deep learning model was reproducible beforehand on central hosted data without setting the initializer and only using `tf.random.set_seed`. Do you know what's the difference? — Stefan, Jun 16 '21 at 07:32

score 0 · Answer 1 · answered Jun 23 '21 at 18:01

Just to second the comments above: Generally TFF best practices include using stateless random number generators in controlled ways in order to get reproducibility (eg. using tf.keras.initializers with random seeds to control model initialization).

Global random seeds set in eager mode will not guarantee exact reproducibility. This is because any time a tff.tf_computation is invoked, TFF effectively enters a unique tf.Session (see https://www.tensorflow.org/federated/faq#how_can_i_ensure_randomness_in_tff_matches_my_expectations). This is also why TFF and centralized TF simulations are different: In TFF, there is no expectation that we can control how TF sessions are entered.

Random Seed for Tensorflow federated?

1 Answers1