Quantization (DeepSeek)

Quantization in the context of DeepSeek-V3 involves converting model weights and activations to lower precision (e.g., FP8) to accelerate training and inference while managing memory.

DeepSeek Innovations

DeepSeek implemented several innovations in quantization, specifically for FP8 training:

1. Mixed Precision Framework: Combining different precisions strategically.
2. Fine-Grained Quantization: improving the granularity of quantization scale factors.
3. Increasing Accumulation Precision: ensuring high-precision accumulation to avoid overflow/underflow.
4. Mantissa or Exponents: innovative handling of floating point components.
5. Online Quantization: quantizing data on the fly.

These techniques allow for massive scale training on limited hardware resources.