Hugging Face has released an update to improve the deduplication of Parquet files on their model hub. The update addresses a common issue where duplicate models are uploaded, wasting storage space and causing confusion for users. According to Hugging Face, this improvement will reduce the number of duplicates by up to 90%, making it easier for developers to find and use existing models. --- Why it matters: This matters because duplicate models can lead to wasted resources and make it harder for researchers to track and build upon each other's work. Source: https://huggingface.co/blog/improve_parquet_dedupe