Ethics and Society Newsletter #6: Building Better AI: The Importance of Data Quality

In February, Reddit announced a new content partnership with Google where they would provide data that would power the new Generative AI based search engine using Retrieval Augmented Generation (RAG). That attempt did not go as planned, and soon, people were seeing recommendations like adding glue to pizza:

In the age of artificial intelligence, massive amounts of data fuel the growth and sophistication of machine learning models. But not all data is created equal; AI systems require high-quality data to produce high-quality outputs.

So, what makes data "high-quality," and why is it crucial to prioritize data quality from the outset? Achieving data quality is not just a matter of accuracy or quantity; it requires a holistic, responsible approach woven throughout the entire AI development lifecycle. As data quality has garnered renewed attention, we explore what constitutes "high quality" data, why prioritizing data quality from the outset is crucial, and how organizations can utilize AI for beneficial initiatives while mitigating risks to privacy, fairness, safety, and sustainability.

In this article, we first provide a high-level overview of the relevant concepts, followed by a more detailed discussion.

What is Good, High-Quality Data?

Good data isn't just accurate or plentiful; it's data fit for its intended purpose. Data quality must be evaluated based on the specific use cases it supports. For instance, the pretraining data for a heart disease prediction model must include detailed patient histories, current health status, and precise medication dosages, but in most cases, should not require patients' phone numbers or addresses for privacy. The key is to match the data to the needs of the task at hand. From a policy standpoint, consistently advocating for a safety-by-design approach towards responsible machine learning is crucial. This includes taking thoughtful steps at the data stage itself. Desirable aspects of data quality include (but are not limited to!):