Introducing the SQL Console on Datasets

Datasets use has been exploding and Hugging Face has become the default home for many datasets. Each month, as the amount of datasets uploaded to the Hub increases, so does the need to query, filter and discover them.

Datasets created on Hugging Face Hub each month

We are very excited to announce that you can now run SQL queries on your datasets directly in the Hugging Face Hub!

Introducing the SQL Console for Datasets

On every dataset you should see a new SQL Console badge. With just one click you can open a SQL Console to query that dataset.

All the work is done in the browser and the console comes with a few neat features:

• 100% Local: The SQL Console is powered by DuckDB WASM, so you can query your dataset without any dependencies.
• Full DuckDB Syntax: DuckDB has full SQL syntax support, along with many built in functions for regex, lists, JSON, embeddings and more. You'll find DuckDB syntax to be very similar to PostgreSQL.
• Export Results: You can export the results of your query to parquet.
• Shareable: You can share your query results of public datasets with a link.

How it works

Parquet Conversion

Most datasets on Hugging Face are stored in Parquet, a columnar data format that is optimized for performance and storage efficiency. The Dataset Viewer on Hugging Face and the SQL Console load the data directly from the datasets Parquet files. And if the dataset is in another format, the first 5GB is auto-converted to Parquet. You can find more information about the Parquet conversion process in the Dataset Viewer Parquet API documentation.

Using the Parquet files, the SQL Console creates views for you to query based on your dataset splits and configs.

DuckDB WASM 🦆

DuckDB WASM is the engine that powers the SQL Console. It is an in-process database engine that runs on Web Assembly in the browser. No server or backend needed.

You may be wondering, "Will it work for big datasets?" and the answer is, "Yes!".

You can see we received results for a simple filter query in under 3 seconds.