How to use genetic algorithm for hyperparameter tuning of ML models?

During the development of ML models, properly exploring the hyperparameter space with optimisation algorithms can find the ideal hyperparameters for the models. A hyperparameter is a parameter in machine learning specified before the learning process begins. Fine-tuning the model hyperparameters improves the model's performance on a validation set. This article will be focused on fine-tuning hyperparameters for a classifier using a Genetic algorithm. Following are the topics to be covered.

About hyperparameter tunning
The working of hyperparameter tunning
How Genetic algorithm is used in hyperparameter optimization?
Implementing GA for tunning model

The hyperparameter could be classified as continuous, discrete or categorical depending upon the learning rate, number of clusters, etc. Let’s have a look at hyperparameter tunning.

About hyperparameter tunning

In basic circumstances, all hyperparameters can have unconstrained real values, and the viable set of hyperparameters can be an n-dimensional vector space with real values. However, because an ML model's hyper-parameters can take on values from multiple domains and have distinct constraints, their optimization issues are frequently complicated constrained optimization problems.

For example, in a decision tree, the number of examined features should be in the range of 0 to the number of features, and the number of clusters in k-means should not be greater than the number of data points. Furthermore, categorical characteristics, like the activation function and optimizer of a neural network, can frequently only take a few specific values. As a result, the feasible domain of a collection of hyperparameters frequently has a complicated structure, increasing the problem's complexity.

There are four primary components to the hyper-parameter tuning method.

An estimator with a goal function.
A search area.
A method of searching or optimising for hyper-parameter combinations.
An evaluation function for comparing the performance of various hyper-parameter combinations.

Are you looking for a complete repository of Python libraries used in data science, check out here.

The working of hyperparameter tunning

Hyperparameter optimization aims to obtain optimal or near-optimal model performance by modifying hyper-parameters within the constraints of the budget. The function's mathematical formulation varies based on the goal function of the chosen ML algorithm and the performance metric function. Model performance may be measured using a variety of measures, including accuracy, RMSE, F1-score, and false alarm rate. In reality, however, time budgets are an important restriction for improving hyperparameter optimization models and must be considered. Maximizing the objective function of an ML model with a decent number of hyper-parameter configurations frequently takes a long time.

The primary process of Hyperparameter optimization is as follows:

Choose an objective function and performance metrics;
Identify the hyper-parameters that need to be tuned, describe their kinds, and identify the best optimization approach.
As the baseline model, train the ML model using the default hyper-parameter setup or common values.
Begin the optimization process by selecting a broad search space as the hyperparameter viable domain based on manual testing and/or domain expertise.
Narrow the search space based on the areas of currently-tested well-performing hyper-parameter values, or, if required, explore additional search spaces.
As the final answer, return the best-performing hyper-parameter configuration.

How Genetic algorithm is used in hyperparameter optimization?

One of the most prevalent metaheuristic algorithms is the genetic algorithm (GA), which is based on the evolutionary idea that people with the highest survival potential and adaptation to the environment are more likely to survive and pass on their qualities to future generations. The qualities of their parents will be passed on to the following generation, which may include both good and bad people. Better people will be more likely to live and create more capable children, whereas the worst people will progressively fade away. The individual with the most adaptability will be chosen as the global optimum after multiple generations.

To use GA to hyperparameter optimization issues, each chromosome or person represents a hyper-parameter, and its decimal value reflects the hyper-real parameter's input value in each evaluation. Every chromosome has multiple genes, which are binary digits, and these genes are subsequently subjected to crossover and mutation activities. The population contains all potential values within the initialised chromosome/parameter ranges, whereas the fitness function characterises the parameter assessment metrics.

Since the spontaneous parameter values frequently do not include the optimal parameter values, additional operations, like selection, crossover, and mutation, must be conducted on the well-performing chromosomes to discover the optimums. Chromosome selection is carried out by choosing chromosomes with high fitness function values. To keep the population size constant, chromosomes with high fitness function values are more likely to be passed on to the next generation, where they develop new chromosomes with the best traits of their parents.

Chromosome selection ensures that beneficial traits from one generation are transmitted down to subsequent generations. Crossover is a method of creating new chromosomes by swapping a proportion of genes between chromosomes. Mutation procedures are also employed to create new chromosomes by randomly modifying one or more chromosomal genes. Crossover and mutation processes allow for alternative features in following generations and lessen the likelihood of missing beneficial qualities.

Implementing GA for tunning model

This article builds a classifier which can categorise water for drinking based on different features of water. The hyperparameter will be optimized with the help TPOT classifier which is a machine learning model based on a Genetic Algorithm (GA).

Let’s start with installing TPOT.

!pip install tpot

Importing necessary libraries

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split,RandomizedSearchCV
from tpot import TPOTClassifier
 
import warnings
warnings.filterwarnings('ignore')

Reading the dataset

data=pd.read_csv('water_potability.csv')
data_utils=data.dropna(axis=0)
data_utils[:5]

X=data_utils.drop('Potability',axis=1)
y=data_utils['Potability']

Splitting the data for train and test in standard ratio 70:30 respectively.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42,shuffle=True)

Building the optimizer

For the article use a Random Forest classifier for the labelling of different water samples. So defining the parameters for the optimization of the hyperparameter.

n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
max_features = ['auto', 'sqrt','log2']
max_depth = [int(x) for x in np.linspace(10, 1000,10)]
min_samples_split = [2, 5, 10,14]
min_samples_leaf = [1, 2, 4,6,8]
param = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
              'criterion':['entropy','gini']}

After the parameters are defined then the tpot classifier would be built by defining the number of generations required, population size, offspring size and the model with hyperparameters which is needed to be optimized. This article uses the scoring technique as an accuracy.

tpot_classifier = TPOTClassifier(generations= 5, population_size= 24, offspring_size= 12,
                                 verbosity= 2, early_stop= 12,
                                 config_dict={'sklearn.ensemble.RandomForestClassifier': param}, 
                                 cv = 4, scoring = 'accuracy')
tpot_classifier.fit(X_train,y_train)

In this article, the generation is set at 5, but as observed in the above result a generation of 3 was enough for the optimization because the cv score is increasing after the 3rd generation. Since GA is a sequential algorithm so parallel processing is a difficult task due to this the computational time is high. The number of generations is directly proportional to the computational time.

The best hyperparameter for the classifier would be returned in this format from the argument passed as a parameter for the classifier while defining the GA.

accuracy = tpot_classifier.score(X_test, y_test)
print(accuracy)

Conclusion

Since its selection, crossover, and mutation procedures reduce the danger of missing the global optimum, the Genetic Algorithm is simple to implement and does not need adequate initializations. The fundamental constraint of the Genetic Algorithm is that it provides new hyper-parameters to set, such as the fitness function type, population size, crossover rate, and mutation rate. With this article, we have understood the implementation of genetic algorithms for hyperparameter optimization.