PrecomputedPoolTokenizer
- class PrecomputedPoolTokenizer(*, path=None, url=None, download_kwargs=None, pool=None, randomize_selection=False, loader=None)[source]
Bases:
Tokenizer
A tokenizer using externally precomputed tokenization.
Initialize the tokenizer.
Note
the preference order for loading the precomputed pools is (1) from the given pool (2) from the given path, and (3) by downloading from the given url
- Parameters:
path (
Optional
[Path
]) – a path for a file containing the precomputed poolsurl (
Optional
[str
]) – an url to download the file with precomputed pools fromdownload_kwargs (
Optional
[Mapping
[str
,Any
]]) – additional download parameters, passed to pystow.Module.ensurepool (
Optional
[Mapping
[int
,Collection
[int
]]]) – the precomputed pools.randomize_selection (
bool
) – whether to randomly choose from tokens, or always take the first num_token precomputed tokens.loader (
Union
[str
,PrecomputedTokenizerLoader
,Type
[PrecomputedTokenizerLoader
],None
]) – the loader to use for loading the pool
- Raises:
ValueError – If the pool’s keys are not contiguous on \(0 \dots N-1\).
Methods Summary
__call__
(mapped_triples, num_tokens, ...)Tokenize the entities contained given the triples.
Methods Documentation