Migrating the Hub from Git LFS to Xet

In January of this year, Hugging Face's Xet Team deployed a new storage backend, and shortly after shifted ~6% of Hub downloads through the infrastructure. This represented a significant milestone, but it was just the beginning. In 6 months, 500,000 repositories holding 20 PB joined the move to Xet as the Hub outgrows Git LFS and transitions to a storage system that scales with the workloads of AI builders.

Today, more than 1 million people on the Hub are using Xet. In May, it became the default on the Hub for new users and organizations. With only a few dozen GitHub issues, forum threads, and Discord messages, this is perhaps the quietest migration of this magnitude.

How? For one, the team came prepared with years of experience building and supporting the content addressed store (CAS) and Rust client that provide the system's foundation. Without these pieces, Git LFS may still be the future on the Hub. However, the unsung heroes of this migration are:

1. An integral piece of infrastructure known internally as the Git LFS Bridge
2. Background content migrations that run around the clock

Together, these components have allowed us to aggressively migrate PBs in the span of days without worrying about the impact to the Hub or the community. They're giving us the peace of mind to move even faster in the coming weeks and months (skip to the end 👇 to see what's coming).

Bridges and backward compatibility

In the early days of planning the migration to Xet, we made a few key design decisions:

• There would be no "hard cut-over" from Git LFS to Xet
• A Xet-enabled repository should be able to contain both Xet and LFS files
• Repository migrations from LFS to Xet don't require "locks"; that is, they can run in the background without disrupting downloads or uploads

Every time a file needs to be migrated from LFS to Xet, a webhook is triggered, pushing the event to a distributed queue where it is processed by an orchestrator. The orchestrator: