Researchers have open-sourced the code and weights for two knowledge distillation models, SD-Small and SD-Tiny. These models are designed to mimic the performance of larger language models while using significantly fewer parameters. The code is available on the Hugging Face website. The researchers claim that these models can be used as a starting point for building smaller, more efficient language models. --- Why it matters: Engineers working with limited computational resources or memory constraints will find this open-source release useful, as it provides a smaller and more efficient alternative to larger language models without sacrificing too much performance. Source: https://huggingface.co/blog/sd_distillation