TokenPreprocessingConfig

Module: fast_llm.data.document.config

Inherits from: LengthPreprocessingConfig, BatchPreprocessingConfig

Fields

causal

Type: bool    Default: True

distributed

Type: DistributedConfig    Default: (sub-fields optional)

return_cumulative_sequence_lengths

Type: bool    Default: False

return_document_count

Type: bool    Default: False

return_document_index

Type: bool    Default: False

return_max_sequence_lengths

Type: bool    Default: False

return_position_index

Type: bool    Default: False