GradientScalerConfig¶

Module: fast_llm.engine.optimizer.config

Fields¶

constant — feature

Type: float or None Default: None

Constant multiplier applied to the loss. Setting this disables dynamic scaling.

hysteresis — feature

Type: int Default: 2

Number of failed updates to tolerate before lowering the learning rate in dynamic scaling (fp16).

initial — feature

Type: float Default: 65536

Initial loss scale for dynamic scaling (fp16).

minimum — feature

Type: float Default: 1.0

Minimum loss scale for dynamic scaling (fp16).

window — feature

Type: int Default: 1000

Interval between dynamic scaling growth (fp16).