Active Hackathon

How bias creeps into large language models

According to DeepMind, unmodified LMs tend to assign high probabilities to exclusionary, biased, toxic, or sensitive utterances if such language is present in the training data.

Language models (LM) are optimised to mirror language systems. Therefore, it stands to reason that LMs might perpetuate stereotypes and biases hardwired into the natural language. When the training data is discriminatory, unfair, or toxic, optimisation leads to highly biased models.

Red flags

In 2019, researchers found racial bias in an algorithm used on over 200 million people in the US to predict the patients who need extra medical care. The system favoured whites over people of colour. The use of COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) algorithm in US court systems projected double false positives for recidivism in black offenders (45%) as opposed to whites (23%). Amazon has scrapped its AI recruitment tool for its manifest sexist bias against women

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

According to DeepMind, unmodified LMs tend to assign high probabilities to exclusionary, biased, toxic, or sensitive utterances if such language is present in the training data.

The express purpose of language modelling is to represent language from the training corpus accurately. Therefore, it’s important to redact and curate training data, fine-tune LMs to adjust weightings to avoid bias and implement checks to filter harmful language.

At present, the turnaround time for language models– from research to applications– are relatively short, making it harder for third parties to anticipate and mitigate risks. Therefore, the correction course should start at the research level to address the bias in language models and should be improved on with each iteration. 

LMs should be evaluated against normative performance thresholds. But, to determine what constitutes satisfactory performance for the LM to be dubbed ‘sufficiently’ safe and ethical before deploying in the real world is a challenge in itself. 

Key risk areas

DeepMind mentions Discrimination, Exclusion and Toxicity as top risk areas in large-scale language models. LMs can engender discrimination, representational, and material harm by perpetuating social biases and stereotypes. For eg, the name ‘Max’ is used for a ‘male’ or ‘families’ always means a father, mother and child. If LMs pick up on biased social cues, they tend to deny or burden identities that differ.

LMs run the risk of disseminating false or misleading information. Examples include bad legal or medical advice leading to unethical or illegal actions. 

While interacting with conversational agents or chatbots, users tend to overestimate the capabilities of the AI and use it in unsafe ways. In addition, LM-based conversational agents might also compromise users’ private information.

LMs are used in social engineering to spread fake news, drive disinformation campaigns, create fraud or scams at scale etc.

Wrapping up

Inarguably, LMs have largely benefited the world economy. However, their benefits and risks are unevenly distributed. Ethical AI and responsible AI are increasingly becoming part of the tech narrative. However, a lot of work needs to be done to incorporate AI ethics into language models. It’s also important to not cut corners for faster turnaround time at the expense of responsible AI. Moreover, the focus should not be solely on building better models but on looking at existing models and devising ways to mitigate their biases. 

More Great AIM Stories

Meeta Ramnani
Meeta’s interest lies in finding out real practical applications of technology. At AIM, she writes stories that question the new inventions and the need to develop them. She believes that technology has and will continue to change the world very fast and that it is no more ‘cool’ to be ‘old-school’. If people don’t update themselves with the technology, they will surely be left behind.

Our Upcoming Events

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
21st Apr, 2023

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM
MOST POPULAR

Council Post: Enabling a Data-Driven culture within BFSI GCCs in India

Data is the key element across all the three tenets of engineering brilliance, customer-centricity and talent strategy and engagement and will continue to help us deliver on our transformation agenda. Our data-driven culture fosters continuous performance improvement to create differentiated experiences and enable growth.

Ouch, Cognizant

The company has reduced its full-year 2022 revenue growth guidance to 8.5% – 9.5% in constant currency from the 9-11% in the previous quarter