In machine learning, binary classification algorithms become one of the most important and used algorithms when things come into the accuracy part of modelling. In most cases, we can see a support vector machine is a preferable option for data scientists in their projects. One thing which lags here is that these binary classifiers are not capable of performing multiclass classification. One-vs-rest and One-vs-one are two methods that make binary classifiers work as multiclass classifiers. In this article, we are going to discuss the one-vs-one and one-vs-rest or one-vs-all method with a support vector machine for multi-class classification. The major points to be discussed in the article are listed below.
Table of contents
- Binary classification to multiclass classification
- What is One-vs-Rest (OvR) or One-vs-All(OvA)?
- Implementation of One-vs-Rest (OvR)
- What is One-vs-One (OvO)?
- Implementation of One-vs-One (OvO)
Let’s first discuss how Binary classifiers can be used for multi-class classification.
Sign up for your weekly dose of what's up in emerging technology.
Binary classification to multiclass classification
Generally, we see the usage of algorithms like SVM and logistic regression in binary classification problems in which using these algorithms we are required to predict one class out of two for any sample data. Predicting one class out of two classes is the reason to consider classification as binary classification.
In one of our articles, we have discussed that a support vector machine is the most used method for classification because it can create non-linear boundaries by projecting the data with higher dimensions in the space using its nonlinear function. But these models are only capable of dealing with binary classification problems.
In multiclass classification problems, we need to deal with more than two classes which means the algorithm which we are using should be capable of working with multiple classes. There are various models available for this and some methods are also available that can make support vector machines capable of dealing with more than two classes. We call these methods heuristic methods. There are two types of heuristic methods:
- One-vs-Rest (OvR) or One-vs-All(OvA)
- One-vs-One (OvO)
Let’s take a closer look at One-vs-Rest (OvR) or One-vs-All(OvA).
Are you looking for for a complete repository of Python libraries used in data science, check out here.
What is One-vs-Rest (OvR) or One-vs-All(OvA)?
We can think of One-vs-Rest (OvR) or One-vs-All(OvA) as an approach to making binary classification algorithms capable of working as multiclass classification algorithms. This approach mainly splits the multiclass data as binary classification data so that the binary classification algorithms can be applied to convert binary classification data.
The procedure of conversion of the data can be understood using an example of iris data where we have three classes as follows:
The converted data as binary classification data will look like the following:
- Setosa vs [Versicolor, Virginica]
- Versicolor vs [Setosa, Virginica]
- Virginica vs [Setosa, Versicolor]
By looking at the conversion we can think that there is a requirement of three models but with the large datasets creating three models can be a tough and non-accurate approach to modeling. Here One-vs-Rest (OvR) or One-vs-All(OvA) comes to save us where binary classifiers can be trained to predict any class as positive and other classes as negative.
We mainly use binary classifiers that can give membership probability or probability-like scores because argmax of these scores can be utilized to predict a class out of multiple classes. Let’s see how we can implement binary classifiers using the One-vs-Rest (OvR) or One-vs-All(OvA) method for a multiclass classification problem.
Implementation of One-vs-Rest (OvR)
For implementing the One-vs-Rest (OvR) method the scikit-learn libraries provide a method called OneVsRestClassifier under the multiclass package. In this article, we are going to see how we can implement a support vector classifier for the multiclass classification of IRIS data using the OneVsRestClassifier. Let’s import libraries first.
from sklearn import datasets from sklearn.svm import SVC from sklearn.multiclass import OneVsRestClassifier
Here in the above, we can see that we have just called SVC models, the OneVsRestClassifier model, and iris data. Let’s make independent and target variables using the iris data.
iris = datasets.load_iris() X = iris.data y = iris.target
Let’s instantiate the SVC model.
svc = SVC()
Instantiating the One-Vs-Rest Classifier for the SVC model.
o_vs_r = OneVsRestClassifier(svc)
Let’s train the model
Predicting values from the model.
yhat = o_vs_r.predict(X) yhat
Here in the above, we can see that the support vector classifier is predicting 3 classes and we did not require to use three binary classifiers. Let’s take a closer look at One-vs-One (OvO).
What is One-vs-One (OvO)?
This method can also be considered as an approach to making binary classification algorithms capable of working as multiclass classification algorithms.it is similar to the One-Vs-Rest method because it also works based on splitting the data but the splitting behavior of this method is different from the One-Vs-Rest method. This method includes the split of data for each class where each class has every other class as its opponent. We can again use the iris data set to understand the data split behavior of this method.
In the iris, we have the following classes:
Split iris data using the One-vs-One (OvO) methods will look like the following:
- Setosa vs Versicolor
- Setosa vs Virginica
- Versicolor vs Virginica
- Virginica vs Versicolor
Here we can see using this data we have binary classification data that includes one class with every other class. We can find out the number of data split using the following formula
Split of data = (number of classes X (number of classes – 1))/2
Other functions of this method are similar to the One-vs-Rest method. Let’s see how we can implement a support vector classifier for multiclass classification using the One-vs-One method.
Implementation of One-vs-One (OvO)
To implement this method we can use the scikit-learn library where the OneVsOneClassifier method is provided under the multiclass package. In this section, we will look at how we can apply this method to the support vector classifier to classify the IRIS data. Let’s import the library
from sklearn import datasets from sklearn.svm import SVC from sklearn.multiclass import OneVsOneClassifier
Let’s call the data.
iris = datasets.load_iris() X = iris.data y = iris.target
Instantiating support vector classifier
svc = SVC()
Instantiating One-Vs-One method
o_vs_o = OneVsOneClassifier(svc)
Training the model
Predicting from model
yhat = o_vs_o.predict(X) yhat
Here we can see our support vector classifier can predict more than two classes using the One-vs-One method.
In this article, we have discussed the two heuristic methods that make a binary classifier capable of working on multi-class data. These methods can help us in applying binary classification methods to a multiclass classification using the split of the data.