LanguageModelDistillationLossConfig

Module: fast_llm.layers.language_model.loss.config

Variant of: LanguageModelLossConfig — select with type: distillation

Inherits from: LanguageModelLossConfig

Fields

loss_type — core

Type: EntropyLossType    Default: "cross_entropy"

Type of loss to use.

weight — core

Type: float    Default: 1.0

Weight for this loss in the total loss computation.

reference_model — feature

Type: str    Default: "teacher"

Name of the reference model for knowledge distillation.

temperature — optional

Type: float    Default: 1.0

Temperature for teacher softmax.

use_triton — expert

Type: bool or None    Default: None

Enable triton implementation. Default: use if available.