AffineLinearConfig

Module: fast_llm.layers.common.linear.config

Inherits from: AffineLinearBaseConfig, LinearConfig, LinearBaseConfig

Fields

bias — architecture

Type: OptionalParameterConfig    Default: (sub-fields optional)

Use bias.

weight — architecture

Type: ParameterConfig    Default: (sub-fields optional)

Configuration for the weight.

apply_peft — feature

Type: bool or None    Default: None

Wrap this layer . Otherwise, treat the layer as a non-peft layer (may be frozen). If not provided, the default set by the parent layer will be used.

lr_scale — feature

Type: float or None    Default: None

Scaling factor for the layer learning rate. Combines multiplicatively with the scale set by the parent layer and individual parameters, if applicable.

Used in

• dense_layer in AttentionConfig
• key_layer in AttentionConfig
• query_layer in AttentionConfig
• value_layer in AttentionConfig
• layer_1 in MLPConfig
• layer_2 in MLPConfig
• layer_1 in MoEMLPConfig
• layer_2 in MoEMLPConfig
• patch_embeddings in PatchEmbeddingsConfig