TorchPrecomputedTokenizerLoader

class TorchPrecomputedTokenizerLoader

Bases: PrecomputedTokenizerLoader

A loader via torch.load.

Methods Summary

__call__(path)	Load tokenization from the given path.
save(path, order, anchor_ids)	Save tokenization to path.

Methods Documentation

__call__(path: Path) → tuple[Mapping[int, Collection[int]], int]

Load tokenization from the given path.

Parameters:

path (Path)

Return type:

tuple[Mapping[int, Collection[int]], int]

static save(path: Path, order: ndarray, anchor_ids: ndarray) → None

Save tokenization to path.

Parameters:

• path (Path) – the output path

• order (ndarray) – shape: (num_entities, num_anchors) the sorted anchor_ids’ ids per entity

• anchor_ids (ndarray) – shape: (num_anchors,) the anchor entity IDs

Return type:

None