Initialization
Embedding weight initialization routines.
- class LabelBasedInitializer(labels, pretrained_model_name_or_path='bert-base-cased', batch_size=None, max_length=None)[source]
An initializer using pretrained models from the transformers library to encode labels.
Example Usage:
Initialize entity representations as Transformer encodings of their labels. Afterwards, the parameters are detached from the labels, and trained on the KGE task without any further connection to the Transformer model.
from pykeen.datasets import get_dataset from pykeen.nn.init import LabelBasedInitializer from pykeen.models import ERMLPE dataset = get_dataset(dataset="nations") model = ERMLPE( embedding_dim=768, # for BERT base entity_initializer=LabelBasedInitializer.from_triples_factory( triples_factory=dataset.training, ), )
Initialize the initializer.
- Parameters
pretrained_model_name_or_path (
str) – the name of the pretrained model, or a path, cf.transformers.AutoModel.from_pretrained()batch_size (
Optional[int]) – >0 the (maximum) batch size to use while encoding. If None, use len(labels), i.e., only a single batch.max_length (
Optional[int]) – >0 the maximum number of tokens to pad/trim the labels to
- Raises
ImportError – if the transformers library could not be imported
- classmethod from_triples_factory(triples_factory, for_entities=True, **kwargs)[source]
Prepare a label-based initializer with labels from a triples factory.
- Parameters
triples_factory (
TriplesFactory) – the triples factoryfor_entities (
bool) – whether to create the initializer for entities (or relations)kwargs – additional keyword-based arguments passed to
LabelBasedInitializer.__init__()
- Return type
- Returns
A label-based initializer
- Raises
ImportError – if the transformers library could not be imported
- class PretrainedInitializer(tensor)[source]
Initialize tensor with pretrained weights.
Example usage:
import torch from pykeen.pipeline import pipeline from pykeen.nn.init import create_init_from_pretrained # this is usually loaded from somewhere else # the shape must match, as well as the entity-to-id mapping pretrained_embedding_tensor = torch.rand(14, 128) result = pipeline( dataset="nations", model="transe", model_kwargs=dict( embedding_dim=pretrained_embedding_tensor.shape[-1], entity_initializer=PretrainedInitializer(tensor=pretrained_embedding_tensor), ), )
Initialize the initializer.
- Parameters
tensor (
FloatTensor) – the tensor of pretrained embeddings.
- class RandomWalkPositionalEncodingInitializer(*, triples_factory=None, mapped_triples=None, edge_index=None, dim, num_entities=None, space_dim=0, skip_first_power=True)[source]
Initialize nodes via random-walk positional encoding.
The random walk positional encoding is given as
\[\mathbf{x}_i = [\mathbf{R}_{i, i}, \mathbf{R}^{2}_{i, i}, \ldots, \mathbf{R}^{d}_{i, i}] \in \mathbb{R}^{d}\]where \(\mathbf{R} := \mathbf{A}\mathbf{D}^{-1}\) is the random walk matrix, with \(\mathbf{D} := \sum_i \mathbf{A}_{i, i}\).
See also
Initialize the positional encoding.
One of triples_factory, mapped_triples or edge_index will be used. The preference order is:
triples_factory
mapped_triples
edge_index
- Parameters
triples_factory (
Optional[CoreTriplesFactory]) – the triples factorymapped_triples (
Optional[LongTensor]) – shape: (m, 3) the mapped triplesedge_index (
Optional[Tensor]) – shape: (2, m) the edge indexdim (
int) – the dimensionalitynum_entities (
Optional[int]) – the number of entities. If None, it will be inferred from edge_indexspace_dim (
int) – estimated dimensionality of the space. Used to correct the random-walk diagonal by a factor k^(space_dim/2). In euclidean space, this correction means that the height of the gaussian distribution stays almost constant across the number of steps, if space_dim is the dimension of the euclidean space.skip_first_power (
bool) – in most cases the adjacencies diagonal values will be zeros (since reflexive edges are not that common). This flag enables skipping the first matrix power.
- init_phases(x)[source]
Generate random phases between 0 and \(2\pi\).
Note
This method works on the canonical torch real representation of complex tensors, cf. https://pytorch.org/docs/stable/complex_numbers.html
- Parameters
x (
Tensor) – a tensor to initialize- Return type
- Returns
tensor with weights set by this initializer
See also
- xavier_normal_(tensor, gain=1.0)[source]
Initialize weights of the tensor similarly to Glorot/Xavier initialization.
Proceed as if it was a linear layer with fan_in of zero, fan_out of prod(tensor.shape[1:]) and Xavier Normal initialization is used, i.e. fill the weight of input tensor with values sampled from \(\mathcal{N}(0, \text{std}^2)\) where
\[\text{std} = \text{gain} \times \sqrt{\frac{2}{\text{fan\_out}}}\]Example: >>> w = torch.empty(3, 5) >>> pykeen.nn.init.xavier_normal_(w, gain=torch.nn.init.calculate_gain(“relu”))
See also
- xavier_uniform_(tensor, gain=1.0)[source]
Initialize weights of the tensor similarly to Glorot/Xavier initialization.
Proceed as if it was a linear layer with fan_in of zero, fan_out of prod(tensor.shape[1:]) and Xavier uniform initialization is used, i.e. fill the weight of input tensor with values sampled from \(\mathcal{U}(-a, a)\) where
\[a = \text{gain} \times \sqrt{\frac{6}{\text{fan\_out}}}\]Example: >>> w = torch.empty(3, 5) >>> pykeen.nn.init.xavier_uniform_(w, gain=torch.nn.init.calculate_gain(“relu”))
See also