LanguageModelConfig

Abstract

This class cannot be instantiated directly. Use one of the variants listed below.

Module: fast_llm.layers.language_model.config

Inherits from: BlockConfig, ModuleConfig

Fields

decoder — architecture

Type: BlockSequenceConfig    Default: (sub-fields optional)

Configuration for the language model decoder.

embeddings — architecture

Type: LanguageModelEmbeddingsConfig    Default: (sub-fields optional)

Configuration for the language model embeddings.

head — architecture

Type: LanguageModelHeadConfig    Default: (sub-fields optional)

Configuration for the language model head(s).

hidden_size — architecture

Type: int    Default: 1024

Size of the model's main hidden dimension, e.g., for its input and output layers.

tied_embedding_weight — architecture

Type: bool    Default: False

Tie the output weights (logits) with the vocabulary embedding.

lr_scale — feature

Type: float or None    Default: None

Scaling factor for the layer learning rate. Combines multiplicatively with the scale set by the parent and child layers, if applicable.