TokenizerConfig

Module: fast_llm.data.preparation.tokenizer

Fields

bos_token — core

Type: str or None    Default: None

BOS token to use if the tokenizer doesn't define one; must be an existing token.

path — core

Type: Path    Default: None

Path to the tokenizer file.

trust_remote_code — feature

Type: bool    Default: False

Allow this tokenizer to load custom Python code shipped with its repository. Has no effect unless --trust-remote-code is also passed on the command line; both are required so a config file alone cannot enable remote-code execution.

Used in

• tokenizer in FimConfig
• tokenizer in GPTFimSampledDatasetConfig
• tokenizer in GPTMemmapDatasetPreparatorConfig
• tokenizer in LmEvalEvaluatorConfig