We can add layer normalization in Pytorch by doing: torch.nn.LayerNorm(shape). However, this is layer normalization with learnable parameters. I.e, it's the following equation:

Does Pytorch have builtin layer normalization without learnable parameters?

CodePudding user response：

You can use nn.LayerNorm, setting the elementwise flag to False. This way the layer won't have learnt parameters. See source code for additional reference.