TokenizerConfig

Module: fast_llm.data.preparation.tokenizer

Fields

bos_token — core

Type: str or None    Default: None

BOS token to use if the tokenizer doesn't define one; must be an existing token.

path — core

Type: Path    Default: None

Path to the tokenizer file.

Used in

• tokenizer in FimConfig
• tokenizer in GPTFimSampledDatasetConfig
• tokenizer in GPTMemmapDatasetPreparatorConfig
• tokenizer in LmEvalEvaluatorConfig