TransformerTextEncoder

class TransformerTextEncoder(pretrained_model_name_or_path: str = 'bert-base-cased', max_length: int = 512)[source]

A combination of a tokenizer and a model.

Initialize the encoder using transformers.AutoModel.

Parameters:

pretrained_model_name_or_path (str) – the name of the pretrained model, or a path, cf. transformers.AutoModel.from_pretrained()
max_length (int) – >0, default: 512 the maximum number of tokens to pad/trim the labels to

Raises:

ImportError – if the transformers library could not be imported

Methods Summary

Encode a batch of text.

Methods Documentation

forward_normalized(texts: Sequence[str]) → Tensor[source]

Encode a batch of text.