A pre-trained BERT model fine-tuned on a dataset consisting of news articles would inherit the political bias existing in the articles.

Description

Transformers can learn universal language representations. They learn useful patterns and information from the dataset. Pre-trained models such as BERT and GPT-2 are trained on large quantities of unsupervised data. However, they can sometimes pick up undesirable and nuanced knowledge from the dataset they are trained on. For example, if the dataset that the model is trained on has a negative sentiment towards a particular entity, the model can learn that and inherit this negative sentiment. This gives rise to bias in these pre-trained models which can be in the form of gender or toward particular political entities. If not tackled early on, this bias could cause serious problems when these models are deployed in the real world. As Natural Language Processing (NLP) techniques become more and more popular in the world, it is more important to address this kind of social & political bias.

In this project, I studied distilBERT, a pre-trained language model, and examined if there exists any inherent political bias. The main aim of this study is to check if BERT is biased toward a particular political entity such as Democrats or Republicans. After fine-tuning BERT on a set of political news articles, I tested the predictions of that model on a set of validation sentences that contain groups of covid-related topics. Moreover, I compute the sentiment of the MASK word being predicted by the model, to check if the model is more positively or negatively inclined towards different political entities - Democrats or Republicans.