One vs One, One vs Rest with SVM for multi-class classification

In machine learning, binary classification algorithms become one of the most important and used algorithms when things come into the accuracy part of modelling. In most cases, we can see a support vector machine is a preferable option for data scientists in their projects. One thing which lags here is that these binary classifiers are not capable of performing multiclass classification. One-vs-rest and One-vs-one are two methods that make binary classifiers work as multiclass classifiers. In this article, we are going to discuss the one-vs-one and one-vs-rest or one-vs-all method with a support vector machine for multi-class classification. The major points to be discussed in the article are listed below.

Table of contents 

1. Binary classification to multiclass classification
2. What is One-vs-Rest (OvR) or One-vs-All(OvA)?
   1. Implementation of  One-vs-Rest (OvR)
3. What is One-vs-One (OvO)?
   1. Implementation of  One-vs-One (OvO)

Let’s first discuss how Binary classifiers can be used for multi-class classification. 

Binary classification to multiclass classification

Generally, we see the usage of algorithms like SVM and logistic regression in binary classification problems in which using these algorithms we are required to predict one class out of two for any sample data. Predicting one class out of two classes is the reason to consider classification as binary classification. 

In one of our articles, we have discussed that a support vector machine is the most used method for classification because it can create non-linear boundaries by projecting the data with higher dimensions in the space using its nonlinear function. But these models are only capable of dealing with binary classification problems.  

In multiclass classification problems, we need to deal with more than two classes which means the algorithm which we are using should be capable of working with multiple classes. There are various models available for this and some methods are also available that can make support vector machines capable of dealing with more than two classes.  We call these methods heuristic methods. There are two types of heuristic methods:

1. One-vs-Rest (OvR) or One-vs-All(OvA)
2. One-vs-One (OvO) 

Let's take a closer look at One-vs-Rest (OvR) or One-vs-All(OvA).

Are you looking for for a complete repository of Python libraries used in data science, check out here.

What is One-vs-Rest (OvR) or One-vs-All(OvA)?

We can think of One-vs-Rest (OvR) or One-vs-All(OvA) as an approach to making binary classification algorithms capable of working as multiclass classification algorithms. This approach mainly splits the multiclass data as binary classification data so that the binary classification algorithms can be applied to convert binary classification data. 

The procedure of conversion of the data can be understood using an example of iris data where we have three classes as follows:

• Setosa
• Versicolor 
• Virginica

The converted data as binary classification data will look like the following:

• Setosa vs [Versicolor, Virginica]
• Versicolor vs [Setosa, Virginica]
• Virginica vs [Setosa, Versicolor]

By looking at the conversion we can think that there is a requirement of three models but with the large datasets creating three models can be a tough and non-accurate approach to modeling. Here One-vs-Rest (OvR) or One-vs-All(OvA) comes to save us where binary classifiers can be trained to predict any class as positive and other classes as negative. 

We mainly use binary classifiers that can give membership probability or probability-like scores because argmax of these scores can be utilized to predict a class out of multiple classes. Let's see how we can implement binary classifiers using the One-vs-Rest (OvR) or One-vs-All(OvA) method for a multiclass classification problem.

Implementation of One-vs-Rest (OvR)

For implementing the One-vs-Rest (OvR) method the scikit-learn libraries provide a method called OneVsRestClassifier under the multiclass package. In this article, we are going to see how we can implement a support vector classifier for the multiclass classification of IRIS data using the OneVsRestClassifier. Let’s import libraries first.

from sklearn import datasets
from sklearn.svm import SVC
from sklearn.multiclass import OneVsRestClassifier

Here in the above, we can see that we have just called SVC models, the OneVsRestClassifier model, and iris data. Let's make independent and target variables using the iris data.

iris = datasets.load_iris()
X = iris.data
y = iris.target

Let's instantiate the SVC model.

svc = SVC()

Instantiating the One-Vs-Rest Classifier for the SVC model.

o_vs_r = OneVsRestClassifier(svc)

Let's train the model 

o_vs_r.fit(X, y)

Predicting values from the model. 

yhat = o_vs_r.predict(X)
yhat

Output:

Here in the above, we can see that the support vector classifier is predicting 3 classes and we did not require to use three binary classifiers.  Let's take a closer look at One-vs-One (OvO).

What is One-vs-One (OvO)?

This method can also be considered as an approach to making binary classification algorithms capable of working as multiclass classification algorithms.it is similar to the One-Vs-Rest method because it also works based on splitting the data but the splitting behavior of this method is different from the One-Vs-Rest method. This method includes the split of data for each class where each class has every other class as its opponent. We can again use the iris data set to understand the data split behavior of this method. 

In the iris, we have the following classes:

• Setosa
• Versicolor 
• Virginica

Split iris data using the One-vs-One (OvO) methods will look like the following:

• Setosa vs Versicolor
• Setosa vs Virginica
• Versicolor vs Virginica
• Virginica vs Versicolor

Here we can see using this data we have binary classification data that includes one class with every other class. We can find out the number of data split using the following formula 

Split of data = (number of classes X (number of classes - 1))/2

Other functions of this method are similar to the One-vs-Rest method. Let’s see how we can implement a support vector classifier for multiclass classification using the One-vs-One method. 

Implementation of One-vs-One (OvO)

To implement this method we can use the scikit-learn library where the OneVsOneClassifier method is provided under the multiclass package. In this section, we will look at how we can apply this method to the support vector classifier to classify the IRIS data. Let’s import the library 

from sklearn import datasets
from sklearn.svm import SVC
from sklearn.multiclass import OneVsOneClassifier

Let's call the data.

iris = datasets.load_iris()
X = iris.data
y = iris.target

Instantiating support vector classifier 

svc = SVC()

Instantiating One-Vs-One method

o_vs_o = OneVsOneClassifier(svc)

Training the model

o_vs_o.fit(X, y)

Predicting from model

yhat = o_vs_o.predict(X)
yhat

Output:

Here we can see our support vector classifier can predict more than two classes using the One-vs-One method. 

Final words

In this article, we have discussed the two heuristic methods that make a binary classifier capable of working on multi-class data. These methods can help us in applying binary classification methods to a multiclass classification using the split of the data. 

References 

• Link for the codes