Skip to content

Llama3RotaryConfig

Module: fast_llm.layers.attention.rotary.config

Variant of: RotaryConfig — select with type: llama3

Inherits from: DefaultRotaryConfig, RotaryConfig, ModuleConfig

Fields

high_frequency_factor — architecture
Type: float    Default: 4.0
low_frequency_factor — architecture
Type: float    Default: 1.0
original_context_length — architecture
Type: int    Default: 8192
scale_factor — architecture
Type: float    Default: 8.0
theta — architecture

Type: float    Default: 10000

Scale for the rotary positional embeddings