TorchPrecomputedTokenizerLoader

class TorchPrecomputedTokenizerLoader

Bases: PrecomputedTokenizerLoader

A loader via torch.load.

Methods Summary

__call__(path)	Load tokenization from the given path.
save(path, order, anchor_ids)	Save tokenization to path.

Methods Documentation

__call__(path)

Load tokenization from the given path.

Return type:

Tuple[Mapping[int, Collection[int]], int]

Parameters:

path (Path) –

static save(path, order, anchor_ids)

Save tokenization to path.

Parameters:

• path (Path) – the output path

• order (ndarray) – shape: (num_entities, num_anchors) the sorted anchor_ids’ ids per entity

• anchor_ids (ndarray) – shape: (num_anchors,) the anchor entity IDs

Return type:

None