I'm working on a project that aims to predict the next character. The predicted character Y is mapped to a One Hot encoding.

Here is my code:

HSIZE = 128
model = Sequential()
model.add(SimpleRNN(HSIZE, return_sequences=False, input_shape=(SEQLEN, nb_chars), unroll=True, use_bias=True)) 

And here is the number of parameters printed by model.summary():

I choose a hidden size of h = 128, and I'm wondering how to compute the number of parameters. I tried to do it by hand but I couldn't find the number printed by model.summary().

I know that there are three matrices U (for the input), W ( for the hidden state), and V (for the output), the biases, I ended up with:

Dim(U) Dim(W) Dim(V) = (128,60) (128,128) (60,128) (bias)60 128 128 = 32060.

Any ideas or did you spot any potential misunderstanding from my side?

CodePudding user response：

The SimpleRNN layer (same holds for the LSTM and GRU layers, and also using RNN along with the respective Cell classes) does not include an output transformation. You can actually kinda guess it by the fact that the summary lists an output shape of 128 units (the state size). It only computes the state sequence.

The number of parameters is thus simply 128*128 60*128 128 = 24192 (hidden-to-hidden matrix, input-to-hidden matrix, bias).