TokenReaderConfig

Module: fast_llm.data.dataset.memmap.config

Variant of: MemmapReaderBaseConfig — select with type: token

Inherits from: MemmapReaderConfig, MemmapReaderBaseConfig

Fields

begin

Type: int    Default: (required)

data_type

Type: DataType    Default: (required)

end

Type: int    Default: (required)

num_documents

Type: int    Default: (required)

num_tokens

Type: int    Default: (required)

Used in

• tokens in LanguageModelReaderConfig