Seed in tensorflow initializer (tf.keras.initializers) doesn't guarantees reproducible results-CodePudding

looking at tensorflow documentation (see, e.g., https://www.tensorflow.org/api_docs/python/tf/keras/initializers/GlorotNormal) a seed should guarantee that "multiple initializers will produce the same sequence when constructed with the same seed value"

The following easy experiment says otherwise

import tensorflow as tf
initializer = tf.keras.initializers.GlorotNormal(seed=123)
values = initializer(shape=(2, 2))
print(values)

initializer1 = tf.keras.initializers.GlorotNormal(seed=123)
values1 = initializer1(shape=(2, 2))
print(values1)

Giving the output

tf.Tensor(
[[-0.58071285 -0.06369764]
 [ 0.06184607 -1.2040431 ]], shape=(2, 2), dtype=float32)
tf.Tensor(
[[ 0.76186    -0.11021858]
 [-1.1184257  -1.430372  ]], shape=(2, 2), dtype=float32)

Interesting fact, if I run the python script multiple times I always get the same overall results. So the first seed somehow works, but when it is called a second time in the script it 'keeps advancing', although it should be fixed.

Any opinion about that? Do you think it is a bug? Do you think it is the intended behaviour (if yes could you explain me why)? It may be a problem of my TF installation? I have python 3.7.9 on Windows and Tensorflow version is 2.7.0

Of course, the same behaviour applies when inserting an initializer in a tf.keras.layer

x = tf.constant(6, shape=(2,3))
dense = tf.keras.layers.Dense(units=3, kernel_initializer=tf.keras.initializers.GlorotNormal(seed=123))
dense1 = tf.keras.layers.Dense(units=3, kernel_initializer=tf.keras.initializers.GlorotNormal(seed=123))
print(dense(x), '\n', dense1(x))

giving

tf.Tensor(
[[14.365635   3.3581433 -1.2498709]
 [14.365635   3.3581433 -1.2498709]], shape=(2, 3), dtype=float32)
 tf.Tensor(
[[10.644517  8.859441  5.136632]
 [10.644517  8.859441  5.136632]], shape=(2, 3), dtype=float32)

Thanks in advance for your time!

CodePudding user response：

If you go to the link you send you can read:

Note that a seeded initializer will not produce the same random values across multiple calls, but multiple initializers will produce the same sequence when constructed with the same seed value.

So yes is deterministic but not return the same value in a single build note that keras and tensorflow are keeping track of the calls you make if you want to do this in a single script you need to reset the backend for keras and is recommended use tf.keras.utils.set_random_seed to set the seed, here an example how to do this

import tensorflow as tf

seed = 123

tf.keras.utils.set_random_seed(
    seed
)
initializer = tf.keras.initializers.GlorotNormal()
values = initializer(shape=(2, 2))
print(values)
tf.keras.backend.clear_session()
tf.keras.utils.set_random_seed(
    seed
)

initializer1 = tf.keras.initializers.GlorotNormal()
values1 = initializer1(shape=(2, 2))
print(values1)

This will print :

tf.Tensor(
[[-0.7219447  -1.4678022 ]
 [-0.35725543 -1.1963991 ]], shape=(2, 2), dtype=float32)
tf.Tensor(
[[-0.7219447  -1.4678022 ]
 [-0.35725543 -1.1963991 ]], shape=(2, 2), dtype=float32)