Huggy Lingo: Using Machine Learning to Improve Language Metadata on the Hugging Face Hub

Huggy Lingo: Using Machine Learning to Improve Language Metadata on the Hugging Face Hub

tl;dr: We're using machine learning to detect the language of Hub datasets with no language metadata, and librarian-bots to make pull requests to add this metadata.

The Hugging Face Hub has become the repository where the community shares machine learning models, datasets, and applications. As the number of datasets grows, metadata becomes increasingly important as a tool for finding the right resource for your use case.

In this blog post, I'm excited to share some early experiments which seek to use machine learning to improve the metadata for datasets hosted on the Hugging Face Hub.

Language Metadata for Datasets on the Hub

There are currently ~50K public datasets on the Hugging Face Hub. Metadata about the language used in a dataset can be specified using a YAML field at the top of the dataset card.

All public datasets specify 1,716 unique languages via a language tag in their metadata. Note that some of them will be the result of languages being specified in different ways i.e. en vs eng vs english vs English.

For example, the IMDB dataset specifies en in the YAML metadata (indicating English):

Section of the YAML metadata for the IMDB dataset

It is perhaps unsurprising that English is by far the most common language for datasets on the Hub, with around 19% of datasets on the Hub listing their language as en (not including any variations of en, so the actual percentage is likely much higher).

The frequency and percentage frequency for datasets on the Hugging Face Hub

What does the distribution of languages look like if we exclude English? We can see that there is a grouping of a few dominant languages and after that there is a pretty smooth fall in the frequencies at which languages appear.

Distribution of language tags for datasets on the hub excluding English.