- class ConcatMLP(num_tokens, embedding_dim, dropout=0.1, ratio=2)
A 2-layer MLP with ReLU activation and dropout applied to the concatenation of token representations.
This is for conveniently choosing a configuration similar to the paper. For more complex aggregation mechanisms, pass an arbitrary callable instead.
Initialize the module.
- forward(xs, dim)
Forward the MLP on the given dimension.
FloatTensor) – The tensor to forward
int) – Only a parameter to match the signature of torch.mean / torch.sum this class is not thought to be usable from outside
- Return type
The tensor after applying this MLP