Utilities
Utilities for neural network components.
- class TransformerEncoder(pretrained_model_name_or_path, max_length=None)[source]
A combination of a tokenizer and a model.
Initialize the encoder.
- Parameters
- Raises
ImportError – if the
transformers
library could not be imported
- encode_all(labels, batch_size=1)[source]
Encode all labels (inference mode & batched).
- Parameters
batch_size (
int
) – the batch size to use for encoding the labels.batch_size=1
means that the labels are encoded one-by-one, whilebatch_size=len(labels)
would correspond to encoding all at once. Larger batch sizes increase memory requirements, but may be computationally more efficient.
- Return type
FloatTensor
- Returns
shape: (len(labels), dim) a tensor representing the encodings for all labels