Improving Parquet Dedupe on Hugging Face Hub

The Xet team at Hugging Face is working on improving the efficiency of the Hub's storage architecture to make it easier and quicker for users to store and update data and models. As Hugging Face hosts nearly 11PB of datasets with Parquet files alone accounting for over 2.2PB of that storage, optimizing Parquet storage is of pretty high priority.

Most Parquet files are bulk exports from various data analysis pipelines or databases, often appearing as full snapshots rather than incremental updates. Data deduplication becomes critical for efficiency when users want to update their datasets on a regular basis. Only by deduplicating can we store all versions as compactly as possible, without requiring everything to be uploaded again on every update. In an ideal case, we should be able to store every version of a growing dataset with only a little more space than the size of its largest version.

Our default storage algorithm uses byte-level Content-Defined Chunking (CDC), which generally dedupes well over insertions and deletions, but the Parquet layout brings some challenges. Here we run some experiments to see how some simple modifications behave on Parquet files, using a 2GB Parquet file with 1,092,000 rows from the FineWeb dataset and generating visualizations using our dedupe estimator.

Background

Parquet tables work by splitting the table into row groups, each with a fixed number of rows (for instance 1000 rows). Each column within the row group is then compressed and stored:

Intuitively, this means that operations which do not mess with the row grouping, like modifications or appends, should dedupe pretty well. So let's test this out!

Append

Here we append 10,000 new rows to the file and compare the results with the original version. Green represents all deduped blocks, red represents all new blocks, and shades in between show different levels of deduplication.

Modification

In this case, the new file is only 89% deduped, requiring 230MB of additional storage.