NVIDIA has showcased the enhancement of the Mistral NeMo 12B language model, introduced in July, which has reduced its parameter size to 8B without significant performance loss. The result is the Mistral-NeMo-Minitron 8B model, which has successfully outperformed similar-level competitors like Llama 3.1 8B and Gemma 7B in the benchmark AI tests.
The techniques used by NVIDIA to reduce model size involve Model Pruning, where the number of layers or neurons is reduced, requiring retraining of the model to maintain accuracy. Additionally, Model Distillation transfers knowledge from a larger “teacher” model to a smaller “student” model, creating a compact model while preserving the original model’s properties, resembling a light retraining process post-pruning.
By employing depth pruning followed by distillation, NVIDIA has achieved the Mistral-NeMo-Minitron 8B model, detailed in the paper “Compact Language Models via Pruning and Knowledge Distillation” from NVIDIA.
TLDR: NVIDIA has successfully reduced the Mistral NeMo 12B model to 8B without compromising performance by utilizing techniques like model pruning and distillation, resulting in the Mistral-NeMo-Minitron 8B model outperforming competitors.
Leave a Comment