GatedRMSNormalizationConfig

Module: fast_llm.layers.common.normalization.config

Variant of: NormalizationConfig — select with type: gated_rms_norm

Inherits from: RMSNormalizationConfig, LayerNormalizationBaseConfig, NormalizationConfig

Fields

activation — core

Type: ActivationType    Default: "silu"

The MLP intermediate activation type. Default: SiLU for gated MLP, GeLU otherwise.

epsilon — architecture

Type: float    Default: 1e-05

Regularizer for the division.

weight — architecture

Type: ParameterConfig    Default: (sub-fields optional)

Configuration for the weight.

zero_centered — architecture

Type: bool    Default: False

Write the normalization weight as w = 1 + w', to improve numerical accuracy when close to one.

lr_scale — feature

Type: float or None    Default: None

Scaling factor for the layer learning rate. Combines multiplicatively with the scale set by the parent layer and individual parameters, if applicable.

implementation — performance

Type: NormalizationImplementation    Default: "auto"

The implementation to use for the normalization layer.