Parquet Content-Defined Chunking

Reduce Parquet file upload and download times on Hugging Face Hub by leveraging the new Xet storage layer and Apache Arrow’s Parquet Content-Defined Chunking (CDC) feature enabling more efficient and scalable data workflows.

TL;DR: Parquet Content-Defined Chunking (CDC) is now available in PyArrow and Pandas, enabling efficient deduplication of Parquet files on content-addressable storage systems like Hugging Face's Xet storage layer. CDC dramatically reduces data transfer and storage costs by uploading or downloading only the changed data chunks. Enable CDC by passing the use_content_defined_chunking argument:

import pandas as pd
import pyarrow.parquet as pq

df.to_parquet("hf://datasets/{user}/{repo}/path.parquet", use_content_defined_chunking=True)
pq.write_table(table, "hf://datasets/{user}/{repo}/path.parquet", use_content_defined_chunking=True)

Table of Contents

• Introduction
• Data Preparation
• Different Use Cases for Parquet Deduplication
  • 1. Re-uploading an Exact Copies of the Table
  • 2. Adding and Removing Columns from the Table
  • 3. Changing Column Types in the Table
  • 4. Appending New Rows and Concatenating Tables
  • 5. Inserting / Deleting Rows in the Table
  • 6. Using Different Row-group Sizes
  • 7. Using Varying File-Level Splits
• Using Parquet CDC feature with Pandas
• References
• Conclusion

Introduction

Apache Parquet is a columnar storage format that is widely used in the data engineering community.

As of today, Hugging Face hosts nearly 21 PB of datasets, with Parquet files alone accounting for over 4 PB of that storage. Optimizing Parquet storage is therefore a high priority. Hugging Face has introduced a new storage layer called Xet that leverages content-defined chunking to efficiently deduplicate chunks of data reducing storage costs and improving download/upload speeds.

Let's explore the performance benefits of the new Parquet CDC feature used alongside Hugging Face's Xet storage layer.