Rearchitecting Hugging Face Uploads and Downloads

As part of Hugging Face's Xet team’s work to improve Hugging Face Hub’s storage backend, we analyzed a 24 hour window of Hugging Face upload requests to better understand access patterns. On October 11th, 2024, we saw:

• Uploads from 88 countries
• 8.2 million upload requests
• 130.8 TB of data transferred

The map below visualizes this activity, with countries colored by bytes uploaded per hour.

Currently, uploads are stored in an S3 bucket in us-east-1 and optimized using S3 Transfer Acceleration. Downloads are cached and served using AWS Cloudfront as a CDN. Cloudfront’s 400+ convenient edge locations provide global coverage and low-latency data transfers. However, like most CDNs, it is optimized for web content and has a file size limit of 50GB.

While this size restriction is reasonable for typical internet file transfers, the ever-growing size of files in model and dataset repositories presents a challenge. For instance, the weights of meta-llama/Meta-Llama-3-70B total 131GB and are split across 30 files to meet the Hub’s recommendation of chunking weights into 20 GB segments. Additionally, to enable advanced deduplication or compression techniques for both uploads and downloads requires a reimagining of how we handle file transfers.

A Custom Protocol for Uploads and Downloads

To push Hugging Face infrastructure beyond its current limits, we are redesigning the Hub’s upload and download architecture. We plan to insert a content-addressed store (CAS) as the first stop for content distribution. This enables us to implement a custom protocol built on a guiding philosophy of dumb reads and smart writes. Unlike Git LFS, which treats files as opaque blobs, our approach analyzes files at the byte level, uncovering opportunities to improve transfer speeds for the massive files found in model and dataset repositories.

Meanwhile, in the new design, reads will take the following path:

and finally here is the updated write path: