TokenizationRepresentation

class TokenizationRepresentation(assignment, token_representation=None, token_representation_kwargs=None, shape=None, **kwargs)

Bases: Representation

A module holding the result of tokenization.

Initialize the tokenization.

Parameters:

• assignment (LongTensor) – shape: (n, num_chosen_tokens) the token assignment.

• token_representation (Union[str, Representation, Type[Representation], None]) – shape: (num_total_tokens, *shape) the token representations

• token_representation_kwargs (Optional[Mapping[str, Any]]) – additional keyword-based parameters

• shape (Union[int, Sequence[int], None]) – The shape of an individual representation. If provided, has to match.

• kwargs – additional keyword-based parameters passed to Representation.__init__()

Raises:

ValueError – if there’s a mismatch between the representation size and the vocabulary size

Attributes Summary

num_tokens

Return the number of selected tokens for ID.

Methods Summary

from_tokenizer(tokenizer, num_tokens, ...[, ...])	Create a tokenization from applying a tokenizer.
iter_extra_repr()	Iterate over components for extra_repr().
save_assignment(output_path)	Save the assignment to a file.

Attributes Documentation

num_tokens

Return the number of selected tokens for ID.

Return type:

int

Methods Documentation

classmethod from_tokenizer(tokenizer, num_tokens, mapped_triples, num_entities, num_relations, token_representation=None, token_representation_kwargs=None, **kwargs)

Create a tokenization from applying a tokenizer.

Parameters:

• tokenizer (Tokenizer) – the tokenizer instance.

• num_tokens (int) – the number of tokens to select for each entity.

• token_representation (Union[str, Representation, Type[Representation], None]) – the pre-instantiated token representations, class, or name of a class

• token_representation_kwargs (Optional[Mapping[str, Any]]) – additional keyword-based parameters

• mapped_triples (LongTensor) – the ID-based triples

• num_entities (int) – the number of entities

• num_relations (int) – the number of relations

• kwargs – additional keyword-based parameters passed to TokenizationRepresentation.__init__

Return type:

TokenizationRepresentation

Returns:

A tokenization representation by applying the tokenizer

iter_extra_repr()

Iterate over components for extra_repr().

Return type:

Iterable[str]

save_assignment(output_path)

Save the assignment to a file.

Parameters:

output_path (Path) – the output file path. Its parent directories will be created if necessary.