TransformerInteraction

class TransformerInteraction(input_dim: int = 512, num_layers: int = 2, num_heads: int = 8, dropout: float = 0.1, dim_feedforward: int = 2048, position_initializer: str | ~typing.Callable[[~torch.Tensor], ~torch.Tensor] | type[~typing.Callable[[~torch.Tensor], ~torch.Tensor]] | None = <function xavier_normal_>)[source]

Bases: FunctionalInteraction[Tensor, Tensor, Tensor]

Transformer-based interaction, as described in [galkin2020].

Initialize the module.

Parameters:

  • input_dim (int) – >0 the input dimension

  • num_layers (int) – >0 the number of Transformer layers, cf. nn.TransformerEncoder.

  • num_heads (int) – >0 the number of self-attention heads inside each transformer encoder layer, cf. nn.TransformerEncoderLayer

  • dropout (float) – the dropout rate on each transformer encoder layer, cf. nn.TransformerEncoderLayer

  • dim_feedforward (int) – the hidden dimension of the feed-forward layers of the transformer encoder layer, cf. nn.TransformerEncoderLayer

  • position_initializer (HintOrType[Initializer]) – the initializer to use for positional embeddings

Methods Summary

func(r, t, transformer, position_embeddings, ...)

Evaluate the Transformer interaction function, as described in [galkin2020]..

Methods Documentation

func(r: Tensor, t: Tensor, transformer: TransformerEncoder, position_embeddings: Tensor, final: Module) → Tensor

Evaluate the Transformer interaction function, as described in [galkin2020]..

\[\textit{score}(h, r, t) = \textit{Linear}(\textit{SumPooling}(\textit{Transformer}([h + pe[0]; r + pe[1]])))^T t\]
Parameters:

  • h (Tensor) – shape: (*batch_dims, dim) The head representations.

  • r (Tensor) – shape: (*batch_dims, dim) The relation representations.

  • t (Tensor) – shape: (*batch_dims, dim) The tail representations.

  • transformer (TransformerEncoder) – the transformer encoder

  • position_embeddings (Tensor) – shape: (2, dim) the positional embeddings, one for head and one for relation

  • final (Module) – the final (linear) transformation

Returns:

The scores.

Return type:

Tensor