FimConfig

Module: fast_llm.data.dataset.gpt.config

Fields

rate — core

Type: float    Default: 0.0

FIM rate for each sample.

fragment_rate — feature

Type: float    Default: 0.0

FIM rate for each fragment when using fim_split_sample.

ignore_prefix — feature

Type: str or None    Default: None

Do not apply FIM to fragments that start with this prefix.

max_middle_len — feature

Type: int or None    Default: None

Maximum length of the middle segment in FIM.

middle_token — feature

Type: str    Default: "<fim_middle>"

TODO.

pad_token — feature

Type: str    Default: "<fim_pad>"

TODO.

prefix_token — feature

Type: str    Default: "<fim_prefix>"

TODO.

split_sample — feature

Type: str or None    Default: None

Split samples on this token and permute each fragment separately.

spm_rate — feature

Type: float    Default: 0.5

TODO.

suffix_token — feature

Type: str    Default: "<fim_suffix>"

TODO.

tokenizer — feature

Type: TokenizerConfig    Default: (sub-fields optional)

Configuration for the tokenizer.

truncate_or_pad — feature

Type: bool    Default: False

TODO.