OptionalParameterConfig

Module: fast_llm.engine.config_utils.parameter

Inherits from: ParameterConfig, ModuleConfig

Fields

initialization — feature

Type: InitializationConfig    Default: (sub-fields optional)

If provided, override the default initialization method set by the parent layer.

lr_scale — feature

Type: float or None    Default: None

Scaling factor for the parameter learning rate. Combines multiplicatively with the scale set by the parent layer, if applicable.

weight_decay — feature

Type: float or bool or None    Default: None

Override the default weight decay for this parameter. Set to True or False to enable or disable the optimizer weight decay, or to a number to use a specific value.

enabled

Type: bool or None    Default: None

Used in

• bias in AffineLinearBaseConfig
• bias in AffineLinearConfig
• bias in CausalConv1dConfig
• output_scale in DecoderBlockConfig
• position_embeddings in LanguageModelEmbeddingsConfig
• router_per_expert_scale in MoEMLPConfig
• router_scale in MoEMLPConfig