Perceptron-like modules.

class ConcatMLP(num_tokens, embedding_dim, dropout=0.1, ratio=2)[source]

A 2-layer MLP with ReLU activation and dropout applied to the concatenation of token representations.

This is for conveniently choosing a configuration similar to the paper. For more complex aggregation mechanisms, pass an arbitrary callable instead.

Initialize the module.

  • num_tokens (int) – the number of tokens

  • embedding_dim (int) – the embedding dimension for a single token

  • dropout (float) – the dropout value on the hidden layer

  • ratio (Union[int, float]) – the ratio of the embedding dimension to the hidden layer size.

forward(xs, dim)[source]

Forward the MLP on the given dimension.

  • xs (FloatTensor) – The tensor to forward

  • dim (int) – Only a parameter to match the signature of torch.mean / torch.sum this class is not thought to be usable from outside

Return type



The tensor after applying this MLP