PrecomputedTokenizerLoader

class PrecomputedTokenizerLoader[source]

Bases: ABC

A loader for precomputed tokenization.

Methods Summary

__call__(path)

Load tokenization from the given path.

Methods Documentation

abstract __call__(path)[source]

Load tokenization from the given path.

Return type:

Tuple[Mapping[int, Collection[int]], int]

Parameters:

path (Path) –