DatasetDiscoveryConfig

Module: fast_llm.data.preparation.dataset_discovery.config

Variant of: RunnableConfig — select with type: prepare_dataset_discovery

Variant of: DatasetPreparatorConfig — select with type: dataset_discovery

Inherits from: DatasetPreparatorConfig, RunnableConfig

Fields

directory — core

Type: Path    Default: (required)

Directory to search for datasets recursively

output — core

Type: Path    Default: (required)

Output path for the generated config YAML file

ignore_paths — optional

Type: list[Path]    Default: list()

List of paths to ignore during dataset discovery (can be absolute or relative to directory)