Evaluation of a machine learning model is crucial to measure its performance. Numerous metrics are used in the evaluation of a machine learning model. Selection of the most suitable metrics is important to fine-tune a model based on its performance. In this article, we discuss the mathematical background and application of evaluation metrics in classification problems.
We can start discussing evaluation metrics by building a machine learning classification model. Here breast cancer data from sklearn’s in-built datasets is used to build a random forest binary classification model.
Import necessary libraries and packages to prepare the required environment.
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn import metrics
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import seaborn as sns
sns.set_style('darkgrid')
Load data, split it into train-test set, build and train the model, and make predictions on test data.
# choose a binary classification problem data = load_breast_cancer() # develop predictors X and target y dataframes X = pd.DataFrame(data['data'], columns=data['feature_names']) y = abs(pd.Series(data['target'])-1) # split data into train and test set in 80:20 ratio X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2, random_state=1) # build a RF model with default parameters model = RandomForestClassifier(random_state=1) model.fit(X_train, y_train) preds = model.predict(X_test)
Confusion Matrix
Without a clear understanding of the confusion matrix, it is hard to proceed with any of classification evaluation metrics. The confusion matrix provides a base to define and develop any of the evaluation metrics. Before discussing the confusion matrix, it is important to know the classes in the dataset and their distribution.
plt.figure(figsize=(7,7)) y.value_counts().plot.pie(ylabel=' ', autopct = '%0.1f%%') plt.title(f'0 - Not cancerous (negative)\n 1 - Cancerous (positive) ', size=14, c='green') plt.tight_layout(); plt.show()
There are two classes in the dataset. 0 refers to ‘Benign’: a non-cancerous state, we simply denote it as ‘negative’. 1 refers to ‘Malignant’: a cancerous state, we simply denote it as ‘positive’. In the dataset, there are 357 negative cases and 212 positive cases. It is clear that class distribution is highly imbalanced.
Knowledge of the following terms will be of more use to proceed further with metrics.
True Positive: Actually positive (ground truth), predicted as positive (correctly classified)
True Negative: Actually negative (ground truth), predicted as negative (correctly classified)
False Positive: Actually negative (ground truth), predicted as positive (misclassified)
False Negative: Actually positive (ground truth), predicted as negative (misclassified)
To plot a confusion matrix,
metrics.plot_confusion_matrix(model, X_test, y_test, display_labels=['Negative', 'Positive'])
Here out of 114 total test samples, 72 are True Negatives (TN), 37 are True Positives (TP), 5 are False Negatives (FN), and there are no False Positives (FP).
confusion = metrics.confusion_matrix(y_test, preds) confusion.ravel()
yields the output array([72, 0, 5, 37])
Most of the evaluation metrics are defined with the terms found in the confusion matrix.
Accuracy
Accuracy can also be defined as the ratio of the number of correctly classified cases to the total of cases under evaluation. The best value of accuracy is 1 and the worst value is 0.
In python, the following code calculates the accuracy of the machine learning model.
accuracy = metrics.accuracy_score(y_test, preds) accuracy
It gives 0.956 as output. However, care should be taken while using accuracy as a metric because it gives biased results for data with unbalanced classes. We discussed that our data is highly unbalanced, hence the accuracy score may be a biased one!
Precision
Precision can be defined with respect to either of the classes. The precision of negative class is intuitively the ability of the classifier not to label as positive a sample that is negative. The precision of positive class is intuitively the ability of the classifier not to label as negative a sample that is positive. The best value of precision is 1 and the worst value is 0.
In Python, precision can be calculated using the code,
precision_positive = metrics.precision_score(y_test, preds, pos_label=1) precision_negative = metrics.precision_score(y_test, preds, pos_label=0) precision_positive, precision_negative
which gives (1.000, 0.935) as output.
Recall
Recall can also be defined with respect to either of the classes. Recall of positive class is also termed sensitivity and is defined as the ratio of the True Positive to the number of actual positive cases. It can intuitively be expressed as the ability of the classifier to capture all the positive cases. It is also called the True Positive Rate (TPR).
Recall of negative class is also termed specificity and is defined as the ratio of the True Negative to the number of actual negative cases. It can intuitively be expressed as the ability of the classifier to capture all the negative cases. It is also called True Negative Rate (TNR).
In python, sensitivity and specificity can be calculated as
recall_sensitivity = metrics.recall_score(y_test, preds, pos_label=1) recall_specificity = metrics.recall_score(y_test, preds, pos_label=0) recall_sensitivity, recall_specificity
which gives (0.881, 1.000) as output. The best value of recall is 1 and the worst value is 0.
F1-score
F1-score is considered one of the best metrics for classification models regardless of class imbalance. F1-score is the weighted average of recall and precision of the respective class. Its best value is 1 and the worst value is 0.
In python, F1-score can be determined for a classification model using
f1_positive = metrics.f1_score(y_test, preds, pos_label=1) f1_negative = metrics.f1_score(y_test, preds, pos_label=0) f1_positive, f1_negative
It gives an output of (0.937, 0.966)
Accuracy, Precision, Recall, and F1-score can altogether be calculated using the method classification_report in python
print(metrics.classification_report(y_test, preds))
outputs
Here, the macro average of any metric is calculated as the mean of respective values of all classes by giving equal weightage to all classes. On the other hand, the weighted average of any metric is calculated by giving weightage based on the number of data points in respective classes. In the above output, numbers 0 and 1 denote negative and positive classes respectively and column support refers to the number of data points in those classes.
ROC and AUC score
ROC is the short form of Receiver Operating Curve, which helps determine the optimum threshold value for classification. The threshold value is the floating-point value between two classes forming a boundary between those two classes. Here in our model, any predicted output above the threshold is classified as class 1 and below it is classified as class 0.
ROC is realized by visualizing it in a plot. The area under ROC, famously known as AUC is used as a metric to evaluate the classification model. ROC is drawn by taking false positive rate in the x-axis and true positive rate in the y-axis. The best value of AUC is 1 and the worst value is 0. However, AUC of 0.5 is generally considered the bottom reference of a classification model.
In python, ROC can be plotted by calculating the true positive rate and false-positive rate. The values are calculated in steps by changing the threshold value from 0 to 1 gradually.
sns.set_style('darkgrid')
preds_train = model.predict(X_train)
# calculate prediction probability
prob_train = np.squeeze(model.predict_proba(X_train)[:,1].reshape(1,-1))
prob_test = np.squeeze(model.predict_proba(X_test)[:,1].reshape(1,-1))
# false positive rate, true positive rate, thresholds
fpr1, tpr1, thresholds1 = metrics.roc_curve(y_test, prob_test)
fpr2, tpr2, thresholds2 = metrics.roc_curve(y_train, prob_train)
# auc score
auc1 = metrics.auc(fpr1, tpr1)
auc2 = metrics.auc(fpr2, tpr2)
plt.figure(figsize=(8,8))
# plot auc
plt.plot(fpr1, tpr1, color='blue', label='Test ROC curve area = %0.2f'%auc1)
plt.plot(fpr2, tpr2, color='green', label='Train ROC curve area = %0.2f'%auc2)
plt.plot([0,1],[0,1], 'r--')
plt.xlim([-0.1, 1.1])
plt.ylim([-0.1, 1.1])
plt.xlabel('False Positive Rate', size=14)
plt.ylabel('True Positive Rate', size=14)
plt.legend(loc='lower right')
plt.show()
Tuning ROC to find the optimum threshold value: Python guides find the right value of threshold (cut-off) with the following codes.
# creating index
i = np.arange(len(tpr1))
# extracting roc values against different thresholds
roc = pd.DataFrame({'fpr':fpr1, 'tpr':tpr1, 'tf':(tpr1-1+fpr1), 'thresholds':thresholds1}, index=i)
# top 5 best roc occurrences
roc.iloc[(roc.tf-0).abs().argsort()[:5]]
Precision-Recall Curve
To find the best threshold value based on the trade-off between precision and recall, precision_recall_curve is drawn.
pre, rec, thr = metrics.precision_recall_curve(y_test, prob_test)
plt.figure(figsize=(8,4))
plt.plot(thr, pre[:-1], label='precision')
plt.plot(thr, rec[1:], label='recall')
plt.xlabel('Threshold')
plt.title('Precision & Recall vs Threshold', c='r', size=16)
plt.legend()
plt.show()
Trade-off performed by our random forest model between Precision and Recall can be visualized using the following codes:
fig, ax = plt.subplots(1,1, figsize=(8,8)) metrics.plot_precision_recall_curve(model, X_test, y_test, ax=ax)
Hamming Loss
Hamming loss is the fraction of targets that are misclassified. The best value of the hamming loss is 0 and the worst value is 1. It can be calculated as
hamming_loss = metrics.hamming_loss(y_test, preds) hamming_loss
to give an output of 0.044.
Jaccard Score
Jaccard score is defined as the ratio of the size of the intersection to the size of the union of label classes between predicted labels and ground truth labels. It is considered a similarity coefficient to compare the predicted classes and true classes. The value of 1 denotes the best classification and 0 denotes the worst. Jaccard loss is considered a poor choice if the class distribution is imbalanced.
jaccard = metrics.jaccard_score(y_test, preds) jaccard
gives an output of 0.881
Cross-entropy loss
Cross-entropy loss, also known as log loss, becomes famous in deep neural networks because of its ability to overcome vanishing gradient problems. It measures the impurity caused by misclassification. The cross-entropy loss is calculated as the summation of the logarithmic value of prediction probability distribution for misclassified data points.
In python, cross-entropy loss can be calculated using the code,
# Entropy loss cross_entropy_loss = metrics.log_loss(y_test, prob_test) cross_entropy_loss
which gives an output of 0.463, where 0 denotes perfect classification or zero impurity.
Other metrics
Though we have covered most of the evaluation metrics for classification in this article, few metrics meant only for multi-class classification are left untouched. Interested readers can refer to the official documentation of metrics used by Scikit-Learn, TensorFlow, and PyTorch.
