ParameterConfig

Module: fast_llm.engine.config_utils.parameter

Inherits from: ModuleConfig

Fields

initialization — feature

Type: InitializationConfig    Default: (sub-fields optional)

If provided, override the default initialization method set by the parent layer.

lr_scale — feature

Type: float or None    Default: None

Scaling factor for the parameter learning rate. Combines multiplicatively with the scale set by the parent layer, if applicable.

Used in

• weight in AffineLinearBaseConfig
• weight in AffineLinearConfig
• weight in CausalConv1dConfig
• weight in GatedRMSNormalizationConfig
• word_embeddings in LanguageModelEmbeddingsConfig
• output_weight in LanguageModelHeadConfig
• weight in LayerNormalizationBaseConfig
• bias in LayerNormalizationConfig
• weight in LayerNormalizationConfig
• weight in LinearBaseConfig
• weight in LinearConfig
• weight in RMSNormalizationConfig