SyncAI.news, a Varaisys broadcasting
A Scalable Entity-Based Framework for Auditing Bias in Large Language Models
AE

Akram Elbouanani, Aboubacar Tuo, Adrian Popescu

· 1 min read

ResearcharXiv cs.CL

A Scalable Entity-Based Framework for Auditing Bias in Large Language Models

arXiv:2601.12374v3 Announce Type: replace Abstract: Existing approaches to bias evaluation in large language models (LLMs) trade ecological validity for statistical control, relying either on artificial prompts that poorly reflect real-world use or on naturalistic tasks that lack scale and rigor. We introduce a scalable bias-auditing framework that uses named entities as controlled probes to measure systematic disparities in model behavior. Synthetic data enables us to construct diverse, controlled inputs, and we show that it reliably reproduces bias patterns observed in natural text, supporting its use for large-scale analysis. Using this framework, we conduct the largest bias audit to date, comprising 1.9 billion data points across multiple entity types, tasks, languages, models, and prompting strategies. We find consistent patterns: models penalize right-wing politicians and favor left-wing politicians, prefer Western and wealthier countries over the Global South, favor Western companies, and penalize firms in the defense and pharmaceutical sectors. While instruction tuning reduces bias, increasing model scale amplifies it, and prompting in Chinese or Russian does not mitigate Western-aligned preferences. These findings highlight the need for systematic bias auditing before deploying LLMs in high-stakes applications. Our framework is extensible to other domains and tasks, and we make it publicly available to support future work.

Original source

This story was published by arXiv cs.CL and written by Akram Elbouanani, Aboubacar Tuo, Adrian Popescu. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News