Initialization¶
Embedding weight initialization routines.
- xavier_normal_(tensor, gain=1.0)[source]¶
Initialize weights of the tensor similarly to Glorot/Xavier initialization.
Proceed as if it was a linear layer with fan_in of zero and Xavier normal initialization is used. Fill the weight of input embedding with values values sampled from \(\mathcal{N}(0, a^2)\) where
\[a = \text{gain} \times \sqrt{\frac{2}{\text{embedding_dim}}}\]
- xavier_uniform_(tensor, gain=1.0)[source]¶
Initialize weights of the tensor similarly to Glorot/Xavier initialization.
Proceed as if it was a linear layer with fan_in of zero and Xavier uniform initialization is used, i.e. fill the weight of input embedding with values values sampled from \(\mathcal{U}(-a, a)\) where
\[a = \text{gain} \times \sqrt{\frac{6}{\text{embedding_dim}}}\]- Parameters
tensor – A tensor
gain (
float) – An optional scaling factor, defaults to 1.0.
- Returns
Embedding with weights by the Xavier uniform initializer.