PrecomputedTokenizerLoader

class PrecomputedTokenizerLoader

Bases: ABC

A loader for precomputed tokenization.

Methods Summary

__call__(path)

Load tokenization from the given path.

Methods Documentation

abstract __call__(path: Path) → tuple[Mapping[int, Collection[int]], int]

Load tokenization from the given path.

Parameters:

path (Path)

Return type:

tuple[Mapping[int, Collection[int]], int]