ESFT: Expert-Specialized Fine-Tuning for MoE
Fine-tune only the experts a task routes to and freeze the rest; matches or beats full fine-tuning of a DeepSeek MoE at lower cost.
Finds that routing for a given task concentrates on few experts, then tunes just those. Works best with fine-grained experts like DeepSeekMoE's. Adapter checkpoints for DeepSeek-V2-Lite were posted on Hugging Face.
- Date
- Tuesday, 2 July 2024
- Lab
- DeepSeek
- Kind
- paper
- Access
- open weights
Figures
| Measure | Value | Measured by |
|---|---|---|
| Paper | arXiv 2407.01906 Submitted 2024-07-02; claims tuning efficiency up and performance on par with or above full-parameter fine-tuning | company |
Hugging Face ESFT adapter repos were created 2024-07-04. Results are the authors' own, on DeepSeek-V2-Lite.
Sources
- arxiv.org/abs/2407.01906
- huggingface.co/api/models?author=deepseek-ai&sort=createdAt&direction=-1&limit=1000
This record was checked against its sources on 6 October 2026. How we check