LLM Distillation

LLM Distillation is a process used to transfer reasoning capabilities from a large, powerful model to a smaller, more efficient model.

Process

1. Teacher Model: A large reasoning model generates a dataset of “Input-Output Pairs”. The output includes the full reasoning chain/thought process.
2. Fine-Tuning Student Model: The small model undergoes Supervised Fine-Tuning (SFT) on this generated dataset.