I am training a neural network using Tensorflow with SimpleRNN layers. By default kernel_initializer='glorot_uniform'. Is there a difference between GlorotNormal and GlorotUniform? Which is best for RNN?

CodePudding user response：

As far as I understand the Normal and Uniform of Glorot are very similar. The main difference is in the random values that are pulled during the initialization. In the normal variation, the random values are pulled from a normal distribution centered around 0 (1) (which you also know as Gaussian) and in the uniform case from the uniform distribution with limit [-limit,limit], where limit = sqrt(6 / (fan_in fan_out))(2)

As for your second question which is better for RNN I'm not aware that there is a consensus which approach is better. There is still an ongoing discussion, but you can find a good insight in this answer in the datascience stackexchange.