YarnRotaryConfig

Module: fast_llm.layers.attention.rotary.config

Variant of: RotaryConfig — select with type: yarn

Inherits from: DefaultRotaryConfig, RotaryConfig, ModuleConfig

Fields

attention_factor — architecture

Type: float or None    Default: None

beta_fast — architecture

Type: float    Default: 32.0

beta_slow — architecture

Type: float    Default: 1.0

original_context_length — architecture

Type: int    Default: 8192

scale_factor — architecture

Type: float    Default: 8.0

theta — architecture

Type: float    Default: 10000

Scale for the rotary positional embeddings