Handling imbalanced data with class weights in logistic regression

Logistic Regression is one of the supervised machine learning techniques that are used for classification tasks. Classification datasets most of the time will have a class imbalance with a certain class with more samples and certain classes with a very less number of samples. Using an imbalanced dataset for the model building would account for the wrong prediction and would be more favorable to classes with more samples. So in this article let us try to understand the importance of class weights in logistic regression and why a balance of class weights is necessary to yield a reliable model.

Table of Contents

1. What are class weights?
2. Problems associated with imbalanced class weights
3. Understanding the importance of class weights
4. Steps to computing class weight 
5. Summary

What are class weights?

Class weights are terminology used for classification tasks where each category of the dataset will be provided with certain weights according to the frequency of occurrence of each category. So class weights will be responsible for giving equal weights for all categories on gradient updates. The usage of unbalanced class weights will be responsible for bias towards the most occurring categories in the data. To obtain a more reliable and unbiased classification model it is important to have a uniform distribution of class weights. Uniform distribution of class weights will also yield various parameters like precision, recall, and F1 score as the class weights would be balanced.

Now let us try to understand the problems associated with an imbalance in class weights.

Are you looking for a complete repository of Python libraries used in data science, check out here.

Problems associated with imbalanced class weights

The main problem associated with imbalanced class weights is accuracy. The accuracy produced in the world is high but what matters is the rightness of accuracy. For imbalance class weights the accuracy yielded would be high generally as it would be biased towards the most occurring class as it would account for higher class weights.

Say suppose healthcare data or business-driven data is in use and there is an imbalance of class weights. So if the class imbalance in the data is not addressed it would account for misinterpretations from the model. Moreover, certain parameters like False-positive and False negatives would turn out to be 0 as the model will be inclined more toward the frequently occurring category.

Understanding the importance of class weights 

To understand the importance of class weights let us consider a classification dataset which is having an imbalance in the target variable distribution.

Here a healthcare dataset is being used and let's analyze the uneven distribution of the target variable through a count plot.

sns.countplot(y)

Here we can see that the target variable is hugely imbalanced where class 0 is having higher class weights when compared to class 1. So let us build a logistic regression with the imbalance target variable and try to evaluate certain parameters from the model.

X=df.drop('stroke',axis=1)
y=df['stroke']
from sklearn.model_selection import train_test_split
X_train,X_test,Y_train,Y_test=train_test_split(X,y,test_size=0.2,random_state=42)
from sklearn.linear_model import LogisticRegression
lr_imb=LogisticRegression(random_state=42)
lr_imb_model=lr_imb.fit(X_train,Y_train)
y_pred=lr_imb_model.predict(X_test)
print('Classification report of imbalanced logistic regression \n',classification_report(Y_test,y_pred))

plot_confusion_matrix(lr_imb_model,X_test,Y_test)

From the classification report, we can observe that the harmonic mean (F1-score) is more for the class with more weightage, and for the class with lesser weightage the harmonic mean and other parameters are 0. Moreover, the model has yielded an accuracy score of 94% which means that the model would make wrong predictions and would be more inclined to the frequently occurring classes. So this is the major problem associated with imbalanced class weights and it turns out to balance the class weights to yield an unbiased and more reliable model.

Balancing class weights

The class weights can be balanced using the logistic regression model by just declaring the class_weight parameter as balanced in the logistic regression model. The class weights can be balanced automatically bypassing the standard parameter as balanced in class weights or random weights to each of the classes can be provided to each of the categories in the data.

Now let us look into how to balance the weights using the predefined “balanced parameter” of the scikit learn library.

Using the “balanced” parameter for class weights

Now the logistic regression model is being fitted with class weights as a standard parameter as “balanced”. The parameter is readily made available in scikit-learn models. Let us see how to use this parameter and obtain a logistic regression model and evaluate certain parameters.

lr_bal=LogisticRegression(random_state=42,class_weight='balanced')
lr_bal_model=lr_bal.fit(X_train,Y_train)
y_pred_bal=lr_bal_model.predict(X_test)
print('Classification report for balanced classes \n',classification_report(Y_test,y_pred_bal))

plot_confusion_matrix(lr_bal_model,X_test,Y_test)

Here we can see that after using the class weights as balanced while fitting the logistic regression model we can see that the accuracy has reduced when compared to the imbalance model but the harmonic mean for the lower occurring class has seen an increase. Other parameters like precision, recall, and F1-score have also seen an increase which helps machine learning engineers to suitably make interpretations from the model. 

This model can be further hyper tuned to yield even better parameters and performance. Moreover, if the confusion matrix is also evaluated we can see that we are able to yield the right parameters like False positive and false negative which shows that the model is not inclined or biased towards the frequently occurring class.

Balancing class weights using the dictionary as a parameter

Class weights will be balanced using a dictionary where the dictionary keys are the classes of the dataset and the keys of the dictionary would be the percentage of weights that would be assigned to each of the classes of the data. So let us look into how to use a dictionary as a parameter for class weights and evaluate certain parameters of the model.

lr_bal2=LogisticRegression(random_state=42,class_weight={0: 0.2,1: 0.8}).fit(X_train,Y_train)
y_pred_bal2=lr_bal2.predict(X_test)
print(classification_report(Y_test,y_pred_bal2))

plot_confusion_matrix(lr_bal2,X_test,Y_test)

So after using a random distribution of weights among the classes in the data we have seen an increase in the model accuracy when compared to using the “balanced” parameter but here a balance of class weights with a percentage of 80 and 20 is shown. So if the classes are balanced equally or near to equal the model will yield performance similar to the model obtained by using the “balanced” parameter.

So this is how imbalance and balance of class weights account for the model performance. In general, it is a good practice to use balanced data to yield a reliable model and obtain the right predictions from the model. Now let us try to understand that class weights are being calculated for different class weight parameters.

Steps to compute the class weights 

The class weights for any classification problems can be obtained using standard libraries of scikit-learn. But it is important to understand how scikit-learn internally computes the class weights. The class weights are generally calculated using the formula shown below.

w(j)=n/Kn(j)

w(j) = weights of the classes

n = number of observations

K = Total number of classes

n(j) = Number of observations in each class

So the scikit-learn utils library internally uses this formula to compute the class weights with different sets of parameters being used for class weights. Now let us look into how to use the scikit- learn utils library for calculating the class weights at different instances.

Calculating imbalance class weights

For calculating the class weights the “compute_class_weight” inbuilt function is being used as shown below and correspondingly the class weights for minority and majority classes can be computed as shown below.

# Calculate weights using sklearn

sklearn_weights1 = class_weight.compute_class_weight(class_weight=None,y=df['stroke'],classes=np.unique(y))
sklearn_weights1

Here the weights assigned to both the classes are equal. Let us compute the weights for each of the classes.

# Compare the values
print(f'The weights for the majority class is {sklearn_weights1[0]*2:.3f}')
print(f'The weights for the minority class is {sklearn_weights1[1]*2:.3f}')

Here we can see that for imbalanced classes the weights assigned to both minority and majority classes are the same which accounts for the bias of the model toward the majority classes.

Calculating class weights after using the “balanced” parameter

The class weights can be calculated after using the “balanced” parameter as shown below.

sklearn_weights2 = class_weight.compute_class_weight(class_weight='balanced',y=df['stroke'],classes=np.unique(y))
Sklearn_weights2

Here we can see that more weightage is given to class 1 as it has a lesser number of samples when compared to class 0. So let us try to obtain the weights which can be easily interpreted as shown below.

# Compare the values
print(f'The weights for the majority class is {sklearn_weights2[0]*2:.3f}')
print(f'The weights for the minority class is {sklearn_weights2[1]*2:.3f}')

So here we can see that the “balanced” parameter has provided more weightage to the minority class when compared to the majority class which helps us to yield a better reliable model.

Calculating class weights by using the dictionary as a parameter

Here the class weights will be calculated by providing random percentages of distribution of weights to each class of the data as shown below.

# Calculate weights using sklearn
dict1={0: 0.2,1: 0.8}
sklearn_weights3 = class_weight.compute_class_weight(class_weight=dict1,y=df['stroke'],classes=np.unique(y))
sklearn_weights3

Here we can see that as specified 20% of the weights is applied to class 0 and 80% of weightage is applied to class 1 of the dataset. Now let us try to interpret the class weights as shown below for easy understanding.

print(f'The weights for the majority class is {sklearn_weights3[0]*2:.3f}')
print(f'The weights for the minority class is {sklearn_weights3[1]*2:.3f}')

So this is how class weights for different classes in the dataset are enforced by using the weightage that is provided in the dictionary for each of the classes.

Summary

Class weights play a very important role in any of the classification machine learning models. So in this article, we have seen how class weights and the balance of class weights are important to obtain a reliable model. Class weight balance is very essential to obtain a bias-free model that can be taken up for the right predictions. The imbalance of class weights accounts for faulty predictions and false interpretations from the model. So it is very important to balance the class weights to obtain a reliable model that can be used for predictions in real-time.

Reference

• Scikit Learn Class Weight Official Documentation
• Colab Notebook