GalkinPrecomputedTokenizerLoader

class GalkinPrecomputedTokenizerLoader[source]

Bases: PrecomputedTokenizerLoader

A loader for pickle files provided by Galkin et al.

Methods Summary

__call__(path)

Load tokenization from the given path.

Methods Documentation

__call__(path)[source]

Load tokenization from the given path.

Return type:

Tuple[Mapping[int, Collection[int]], int]

Parameters:

path (Path) –