TorchPrecomputedTokenizerLoader¶
- class TorchPrecomputedTokenizerLoader[source]¶
Bases:
PrecomputedTokenizerLoader
A loader via torch.load.
Methods Summary
__call__
(path)Load tokenization from the given path.
save
(path, order, anchor_ids)Save tokenization to path.
Methods Documentation