TorchPrecomputedTokenizerLoader

class TorchPrecomputedTokenizerLoader[source]

Bases: PrecomputedTokenizerLoader

A loader via torch.load.

Methods Summary

__call__(path)

Load tokenization from the given path.

save(path, order, anchor_ids)

Save tokenization to path.

Methods Documentation

__call__(path)[source]

Load tokenization from the given path.

Return type:

Tuple[Mapping[int, Collection[int]], int]

Parameters:

path (Path) –

static save(path, order, anchor_ids)[source]

Save tokenization to path.

Parameters:
  • path (Path) – the output path

  • order (ndarray) – shape: (num_entities, num_anchors) the sorted anchor_ids’ ids per entity

  • anchor_ids (ndarray) – shape: (num_anchors,) the anchor entity IDs

Return type:

None