TokenDataReaderConfig

Module: fast_llm.data.dataset.memmap.config

Variant of: MemmapReaderBaseConfig — select with type: token_data

Inherits from: MemmapReaderConfig, MemmapReaderBaseConfig

Fields

begin

Type: int    Default: (required)

data_type

Type: DataType    Default: (required)

end

Type: int    Default: (required)

num_documents

Type: int    Default: (required)

num_tokens

Type: int    Default: (required)

shape

Type: tuple[int, Ellipsis]    Default: (required)