When two fictional names were fed into anartificial intelligence (AI)system, the results were disturbingly real. In a controlled research prompt,GPT-4was given nothing more than a list of professions and two Indian surnames, Usha Bansal and Pinky Ahirwar. The system swiftly assigned Bansal roles such as scientist, dentist and financial analyst. Ahirwar, by contrast, was linked to manual scavenger, plumber and construction worker.

There was no biographical data, no education history, no geographic context. Only names.

In India, surnames often function as social signifiers, quiet markers of caste, community and perceived status. “Bansal" is commonly associated with upper-caste trading or Brahmin communities, while “Ahirwar" is linked to Dalit identity. GPT-4 appeared to draw on those embedded associations. Researchers say the model absorbed social hierarchies from the vast corpus of data it was trained on.

The findings were not an isolated case. Across thousands of prompts, multiple large language models and independent academic studies, a similar pattern emerged, thatAI systemswere internalising caste hierarchies embedded in society.

According to a TOI report, sociologists Anoop Lal, Associate Professor at St Joseph University in Bengaluru, said, “Caste in India has a way of sticking on. Even when Indians convert to religions with no caste in their foundation, the caste identities continue. I am not surprised that AI models are biased." Another sociologist remarked, “Is AI really wrong? After all, it’s learning from us."

The implications extend far beyond text generation. As AI systems are increasingly deployed in hiring, credit scoring, education, governance and healthcare, concerns are mounting that embedded bias could influence decision-making in subtle but consequential ways. Researchers warn that discrimination need not be explicit. Even if a system does not directly reject applicants from marginalised backgrounds, its internal mathematical associations, linking certain surnames with lower ability or status, could influence rankings, recommendations or risk assessments.

In a paper titled “DECASTE", researchers from IBM, Dartmouth College and other institutions argued that while discussions on algorithmic fairness have grown globally, caste-based bias in large language models (LLMs) remains under-explored. “If left unchecked, caste-related biases could perpetuate or escalate discrimination in subtle and overt forms," the authors wrote.

LLMs convert words into high-dimensional numerical vectors known as “embeddings". The proximity between these vectors determines how closely concepts are associated. If certain caste identities consistently appear closer to negative traits or lower-status professions in this embedding space, structural bias exists, even if overtly discriminatory outputs are filtered.

In the DECASTE study, models including GPT-4 were asked to assign professions based solely on Indian surnames. Positive descriptors such as “fair", “refined" and “fashionable" were more frequently associated with upper-caste names. Words like “dark", “messy" and “sweaty" clustered around marginalised caste identities. Prestigious institutions like “IIT", “IIM", “medical college" were linked to Brahmin names, while “government school", “anganwadi" and “remedial class" were associated with Dalit surnames.

In another experiment, two fictional architects, identical in qualifications and experience but differing in caste identity, were described to GPT-4. The Brahmin character was assigned “innovative, eco-friendly building design" work. The Dalit character was tasked with “cleaning and organising design blueprints".

Source: Tech News in news18.com, Tech Latest News, Tech News