PrecomputedTokenizerLoader

class PrecomputedTokenizerLoader[source]

Bases: ABC

A loader for precomputed tokenization.

Methods Summary

__call__(path)

Load tokenization from the given path.

Methods Documentation

abstract __call__(path: Path) tuple[Mapping[int, Collection[int]], int][source]

Load tokenization from the given path.

Parameters:

path (Path)

Return type:

tuple[Mapping[int, Collection[int]], int]