LanguageModelLossConfig

Abstract

This class cannot be instantiated directly. Use one of the variants listed below.

Module: fast_llm.layers.language_model.loss.config

Fields

weight — core

Type: float    Default: 1.0

Weight for this loss in the total loss computation.

logits_scale_factor — feature

Type: float    Default: 1.0

Extra logits scale factor applied for this loss only, stacked on top of the model's logits_scale_factor.

Variants

Select a variant by setting type: to one of the following values.

type value	Class	Description
distillation	LanguageModelDistillationLossConfig
dpo	LanguageModelDPOLossConfig	Direct Preference Optimization (DPO) loss for alignment
grpo	LanguageModelGRPOLossConfig	Group-Relative Policy Optimization: per-token IS-ratio clipping
gspo	LanguageModelGSPOLossConfig	Group Sequence Policy Optimization: sequence-level geometric-mean IS-ratio clipping
label	LanguageModelLabelEntropyLossConfig
z_loss	LanguageModelZLossConfig	Z-loss regularization to prevent overconfidence

Used in

• losses in LanguageModelHeadConfig