ProportionalRotaryConfig

Module: fast_llm.layers.attention.rotary.config

Variant of: RotaryConfig — select with type: proportional

Inherits from: DefaultRotaryConfig, RotaryConfig, ModuleConfig

Fields

partial_rotary_factor — architecture

Type: float    Default: 1.0

Fraction of head dimensions to apply rotary embeddings to.

theta — architecture

Type: float    Default: 10000

Scale for the rotary positional embeddings