Evaluating Language Model Bias with 🤗 Evaluate

While the size and capabilities of large language models have drastically increased over the past couple of years, so too has the concern around biases imprinted into these models and their training data. In fact, many popular language models have been found to be biased against specific religions and genders, which can result in the promotion of discriminatory ideas and the perpetuation of harms against marginalized groups.

To help the community explore these kinds of biases and strengthen our understanding of the social issues that language models encode, we have been working on adding bias metrics and measurements to the 🤗 Evaluate library. In this blog post, we will present a few examples of the new additions and how to use them. We will focus on the evaluation of causal language models (CLMs) like GPT-2 and BLOOM, leveraging their ability to generate free text based on prompts.

If you want to see the work in action, check out the Jupyter notebook we created!

The workflow has two main steps:

• Prompting the language model with a predefined set of prompts (hosted on 🤗 Datasets)
• Evaluating the generations using a metric or measurement (using 🤗 Evaluate)

Let's work through bias evaluation in 3 prompt-based tasks focused on harmful language: Toxicity, Polarity, and Hurtfulness. The work we introduce here serves to demonstrate how to utilize Hugging Face libraries for bias analyses, and does not depend on the specific prompt-based dataset used. Critically, remember that recently introduced datasets for evaluating biases are initial steps that do not capture the vast range of biases that models may produce (see the Discussion section below for more details).

Toxicity

Although we define these prompts directly for the sake of example here, more can be extracted directly from the WinoBias dataset using the Hugging Face dataset library's load_dataset function; see the provided code in the Jupyter notebook for more details.