1

looking at tensorflow documentation (see, e.g., https://www.tensorflow.org/api_docs/python/tf/keras/initializers/GlorotNormal) a seed should guarantee that "multiple initializers will produce the same sequence when constructed with the same seed value"

The following easy experiment says otherwise

import tensorflow as tf
initializer = tf.keras.initializers.GlorotNormal(seed=123)
values = initializer(shape=(2, 2))
print(values)

initializer1 = tf.keras.initializers.GlorotNormal(seed=123)
values1 = initializer1(shape=(2, 2))
print(values1)

Giving the output

tf.Tensor(
[[-0.58071285 -0.06369764]
 [ 0.06184607 -1.2040431 ]], shape=(2, 2), dtype=float32)
tf.Tensor(
[[ 0.76186    -0.11021858]
 [-1.1184257  -1.430372  ]], shape=(2, 2), dtype=float32)

Interesting fact, if I run the python script multiple times I always get the same overall results. So the first seed somehow works, but when it is called a second time in the script it 'keeps advancing', although it should be fixed.

Any opinion about that? Do you think it is a bug? Do you think it is the intended behaviour (if yes could you explain me why)? It may be a problem of my TF installation? I have python 3.7.9 on Windows and Tensorflow version is 2.7.0

Of course, the same behaviour applies when inserting an initializer in a tf.keras.layer

x = tf.constant(6, shape=(2,3))
dense = tf.keras.layers.Dense(units=3, kernel_initializer=tf.keras.initializers.GlorotNormal(seed=123))
dense1 = tf.keras.layers.Dense(units=3, kernel_initializer=tf.keras.initializers.GlorotNormal(seed=123))
print(dense(x), '\n', dense1(x))

giving

tf.Tensor(
[[14.365635   3.3581433 -1.2498709]
 [14.365635   3.3581433 -1.2498709]], shape=(2, 3), dtype=float32)
 tf.Tensor(
[[10.644517  8.859441  5.136632]
 [10.644517  8.859441  5.136632]], shape=(2, 3), dtype=float32)

Thanks in advance for your time!

user17788510
  • 158
  • 1
  • 8

1 Answers1

1

If you go to the link you send you can read:

Note that a seeded initializer will not produce the same random values across multiple calls, but multiple initializers will produce the same sequence when constructed with the same seed value.

So yes is deterministic but not return the same value in a single build note that keras and tensorflow are keeping track of the calls you make if you want to do this in a single script you need to reset the backend for keras and is recommended use tf.keras.utils.set_random_seed to set the seed, here an example how to do this

import tensorflow as tf

seed = 123

tf.keras.utils.set_random_seed(
    seed
)
initializer = tf.keras.initializers.GlorotNormal()
values = initializer(shape=(2, 2))
print(values)
tf.keras.backend.clear_session()
tf.keras.utils.set_random_seed(
    seed
)

initializer1 = tf.keras.initializers.GlorotNormal()
values1 = initializer1(shape=(2, 2))
print(values1)

This will print :

tf.Tensor(
[[-0.7219447  -1.4678022 ]
 [-0.35725543 -1.1963991 ]], shape=(2, 2), dtype=float32)
tf.Tensor(
[[-0.7219447  -1.4678022 ]
 [-0.35725543 -1.1963991 ]], shape=(2, 2), dtype=float32)
Pedro Fillastre
  • 892
  • 6
  • 10
  • Thanks, your solution for sure solved my issue. I'm still convinced that the documentation is not completly clear (at least for me). I interpreted the part " a seeded initializer will not produce the same random values across multiple calls" in the following way: If I create one initializer and I call it multiple times then I will have different random values ` import tensorflow as tf initializer = tf.keras.initializers.GlorotNormal(seed=123) values = initializer(shape=(2, 2)) values1 = initializer(shape=(2, 2)) print(values) print(values1) ` – user17788510 Oct 17 '22 at 07:51
  • ... but if i make different initializers (see my original example) I expected to get the same sequences ("multiple initializers will produce the same sequence when constructed with the same seed value"). – user17788510 Oct 17 '22 at 07:58
  • Any opinions about that? – user17788510 Oct 17 '22 at 08:04
  • Well, the api have some strong opinions to keep the graph coherent, and you can also use constants as initializer if you want to have the same matrix in two layers at my side, I don't have any strong opinions ^^, not forget to set this as resolved if this is solved for you – Pedro Fillastre Oct 17 '22 at 14:28