How To Use Tensorboard HPrams In Hyperparameter Tuning And Visualization

Hyperparameters are the parameter that controls the performance of a predictive model. These are often tuned using different methods of tuning to achieve higher performance and accuracy. Tensorboard is similar to a dashboard that gives us different visualizations related to the model like model training and model loss. Tensorboard Hprams can be used to check the performance of a model by tweaking parameters like no of neurons in layers, using different optimization techniques, or by adding regularization techniques such as batch normalization and dropouts. 

Through this article, we will explore more about Tensorboard HPrams. We will build a model using a different number of neurons for each layer and different no dropouts and compute the performance of the model. After the training is done we will check different visualizations in the TensorBoard. We will make use of MNIST data for the experiment whereas we can perform this tuning on any dataset. The dataset has 60,000 images in the training sample and 10,000 in the testing sample. Each image is of 28X28 pixels. 

What we will learn from this article?

1. Hyperparameter Tuning  
2. Building a classification model by initializing multiple values for hyperparameters
3. Different Tensorboard Hprams Visualization 

1. Hyperparameter Tuning

When we build different predictive models either in machine learning or in deep learning we often define different sets of hyperparameters for the learning of the algorithms. These are used to compute the real ability of the model to generalize. We tune them with different methods to get to the highest accuracy or to improve the model performance. It is very important to define a good set of hyperparameters for achieving good results. We have different methods for tuning these hyperparameters like Keras Tuner, etc. Hprams is also a way in which we can compute the best parameter for our model. Hprams visualization is shown in Tensorboard.

2. Building a classification model with multiple values for hyperparameters

We will now build a classification model over MNIST data to classify handwritten digits. The dataset can be either downloaded directly from Kaggle where it is publicly available or can be directly imported from Keras. We will be directly importing it, use the below code to import the libraries required and the dataset. 

import tensorflow as tf

from tensorboard.plugins.hparams import api as HP

(X_train, y_train),(X_test, y_test) = tf.keras.datasets.mnist.load_data()

We have now imported the data and store training and testing images with labels. Now we will load the tensorboard notebook. Use the below code for the same.

%load_ext tensorboard

Since we have loaded the tensorboard now we will define the different combinations on which we want to train the model. We will now tweak no of neurons in the layer, dropouts, and different optimizers whereas you can try for other hyperparameters like learning rate, etc. Use the below code for the same. 

neurons_hp = HP.HParam('num_units', HP.Discrete([64,128]))

dropout_hp = HP.HParam('dropout', HP.RealInterval(0.20, 0.25))

optimizer_hp = HP.HParam('optimizer', HP.Discrete(['adam', 'sgd']))

 metrics = 'accuracy'

We have defined the value of neurons to be 64 and 128, dropouts to be 20% and 25%, and also two different optimizers. The model will pick these combinations and will get trained on. Now we will create the model networks using the below code to define the same. 

def model_t(hparams):

    model = tf.keras.models.Sequential([

                                        tf.keras.layers.Flatten(),

                                        tf.keras.layers.Dense(hparams[neurons_hp], activation='relu'),

                                        tf.keras.layers.Dense(hparams[neurons_hp], activation='relu'),

                                        tf.keras.layers.Dropout(hparams[dropout_hp]),

                                        tf.keras.layers.Dense(10, activation='softmax'),

                                        ])

    model.compile(optimizer=hparams[optimizer_hp],loss='categorical_crossentropy',metrics=['accuracy'],)

    model.fit(X_train, y_train, epochs=5)

    _, accuracy = model.evaluate(X_test, y_test)

    return accuracy

We have defined dense with activation function as relu and output layers having 10 classes with a softmax activation function. Also, we have defined the no of epochs to be 5 whereas you can train it for anyone of epochs. Now we will define the summary for the model. Use the below code for the same. 

def run(dir, hparams):

  with tf.summary.create_file_writer(dir).as_default():

    HP.hparams(hparams)

    accuracy = model_t(hparams)

    tf.summary.scalar(metrics, accuracy, step=1)

Now we will train this model for different sets of hyperparameter combinations that we defined in the starting. Use the below code for doing the same.

session_num = 0

for num_units in neurons_hp.domain.values:

  for dropout_rate in (dropout_hp.domain.min_value, dropout_hp.domain.max_value):

    for optimizer in optimizer_hp.domain.values:

      hparams = {

          neurons_hp: num_units,

          dropout_hp: dropout_rate,

          optimizer_hp: optimizer,

      }

      run_name = "run-%d" % session_num

      print({h.name: hparams[h] for h in hparams})

      run('logs/hparam_tuning/' + run_name, hparams)

      session_num += 1

Output:

The above training snapshot is just for 2 combinations whereas this process would be repeated for several other combinations.

3. Different Tensorboard Hprams Visualization 

Now we will visualize the log dir of the hyperparameters using a tensorboard. Use the below code to do so. Once we run the below code tensorboard dashboard will automatically come up. 

%tensorboard --logdir logs/hparam_tuning

This is the first view of the tensorboard dashboard that we will get. We need to click on Hparams to check different visualizations. There are mainly three different views in Hparams that are Table view, Parallel Coordinates view, and Scatter plot matrix view.  Check the below images for these views. 

Table View:

Parallel Coordinates view: 

Scatter plot matrix view:

Conclusion

Through this article, we explored how effective Hprams can be for the performance of the model. How Hprams computes training for different numbers of hyperparameter combinations that are defined by the programmer. We explored different Hparms tensorboard visualization for computing what hyperparameters result in higher accuracy. It also gives the table view for all the sets of combinations. This type of hyperparameter technique can be used to check what would be the best hyperparameter for the respective model to achieve the highest accuracy. 

Also, check similar articles on Hyperparameter tuning that is titled as “Keras Tuner for Hyperparameter tuning”.