Exploring Representational Harms of Generative AI Systems upon Historically Marginalized Populations often Overlooked in Fairness Research

Abstract

As I write this, the year is 2026, and the world finds itself firmly in the midst of a Generative AI (GAI) summer -- a period of abundant investment, research, and interest around the development of Artificial Intelligence tools. Since the public release of OpenAI's ChatGPT, GAI tools have seen widespread adoption across the public and private sectors. Capable of generating artificial text or images based on user input, such tools have also been known to replicate societal power dynamics and hierarchies. This has prompted a surge in research aimed at understanding the depth of this problem, with a particular focus on historically marginalized populations. However, such research has taken a fairly narrow view of “historically marginalized populations" in this context, overlooking vast sections of marginalized populations who continue to suffer GAI-mediated harm. In this Dissertation, I shed light on the plights of some such populations, in a larger case for more expansive coverage of AI fairness research. I first begin by demonstrating, through a comprehensive literature review of the field, how AI fairness research, intended to show how such technologies cause harm to historically marginalized populations, effectively limits itself to studying only a few ways in which populations face marginalization. I then provide three case studies of historically marginalized populations overlooked in fairness research, being stereotyped and facing harm through the outputs of GAI tools. First, I study six low-resource languages -- Bengali, Farsi, Malay, Tagalog, Thai, and Turkish -- connected by the fact that all these languages do not have gendered pronouns. Examining translations to and from English by the GAI tool ChatGPT, I show how translations insert assumed binary gender pronouns reflecting stereotypical associations of gender with actions and occupations. Then, I examine how the text-to-image GAI tool Stable Diffusion portrays aspects of Indian culture and its various subcultures, seen through the eyes of 5 focus groups of Indian AI users, accompanied by further qualitative examination of images. This reveals the presence of several problematic stereotypes of the Western gaze within Stable Diffusion outputs, patterns for which existing taxonomies of AI harm were inadequate and necessitated the addition of two novel terms not previously named in existing AI harm taxonomies: exoticism and cultural misappropriation. Finally, I study how Stable Diffusion represents caste-oppressed communities in India, documenting how the same patterns of `castelessness' prevalent in Indian society/diasporas also emerge within its outputs. I finally outline approaches towards a research agenda that better focuses on the harmful outcomes borne by historically marginalized populations currently overlooked in fairness research as a result of GAI tools being popularized, and conclude with a larger vision of AI for social good that will improve and address the marginalization of several communities of people across the world through AI-based practices.