Active Hackathon

How To Use Tensorboard HPrams In Hyperparameter Tuning And Visualization

Through this article, we will explore more about Tensorboard HPrams. We will build a model using a different number of neurons for each layer and different no dropouts and compute the performance of the model.

Hyperparameters are the parameter that controls the performance of a predictive model. These are often tuned using different methods of tuning to achieve higher performance and accuracy. Tensorboard is similar to a dashboard that gives us different visualizations related to the model like model training and model loss. Tensorboard Hprams can be used to check the performance of a model by tweaking parameters like no of neurons in layers, using different optimization techniques, or by adding regularization techniques such as batch normalization and dropouts. 

Through this article, we will explore more about Tensorboard HPrams. We will build a model using a different number of neurons for each layer and different no dropouts and compute the performance of the model. After the training is done we will check different visualizations in the TensorBoard. We will make use of MNIST data for the experiment whereas we can perform this tuning on any dataset. The dataset has 60,000 images in the training sample and 10,000 in the testing sample. Each image is of 28X28 pixels. 

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

What we will learn from this article?

  1. Hyperparameter Tuning  
  2. Building a classification model by initializing multiple values for hyperparameters
  3. Different Tensorboard Hprams Visualization 
  1. Hyperparameter Tuning

When we build different predictive models either in machine learning or in deep learning we often define different sets of hyperparameters for the learning of the algorithms. These are used to compute the real ability of the model to generalize. We tune them with different methods to get to the highest accuracy or to improve the model performance. It is very important to define a good set of hyperparameters for achieving good results. We have different methods for tuning these hyperparameters like Keras Tuner, etc. Hprams is also a way in which we can compute the best parameter for our model. Hprams visualization is shown in Tensorboard.

  1. Building a classification model with multiple values for hyperparameters

We will now build a classification model over MNIST data to classify handwritten digits. The dataset can be either downloaded directly from Kaggle where it is publicly available or can be directly imported from Keras. We will be directly importing it, use the below code to import the libraries required and the dataset. 

import tensorflow as tf

from tensorboard.plugins.hparams import api as HP

(X_train, y_train),(X_test, y_test) = tf.keras.datasets.mnist.load_data()

We have now imported the data and store training and testing images with labels. Now we will load the tensorboard notebook. Use the below code for the same.

%load_ext tensorboard

Since we have loaded the tensorboard now we will define the different combinations on which we want to train the model. We will now tweak no of neurons in the layer, dropouts, and different optimizers whereas you can try for other hyperparameters like learning rate, etc. Use the below code for the same. 

neurons_hp = HP.HParam('num_units', HP.Discrete([64,128]))

dropout_hp = HP.HParam('dropout', HP.RealInterval(0.20, 0.25))

optimizer_hp = HP.HParam('optimizer', HP.Discrete(['adam', 'sgd']))

 metrics = 'accuracy'

We have defined the value of neurons to be 64 and 128, dropouts to be 20% and 25%, and also two different optimizers. The model will pick these combinations and will get trained on. Now we will create the model networks using the below code to define the same. 

def model_t(hparams):

    model = tf.keras.models.Sequential([

                                        tf.keras.layers.Flatten(),

                                        tf.keras.layers.Dense(hparams[neurons_hp], activation='relu'),

                                        tf.keras.layers.Dense(hparams[neurons_hp], activation='relu'),

                                        tf.keras.layers.Dropout(hparams[dropout_hp]),

                                        tf.keras.layers.Dense(10, activation='softmax'),

                                        ])

    model.compile(optimizer=hparams[optimizer_hp],loss='categorical_crossentropy',metrics=['accuracy'],)

    model.fit(X_train, y_train, epochs=5)

    _, accuracy = model.evaluate(X_test, y_test)

    return accuracy

We have defined dense with activation function as relu and output layers having 10 classes with a softmax activation function. Also, we have defined the no of epochs to be 5 whereas you can train it for anyone of epochs. Now we will define the summary for the model. Use the below code for the same. 

def run(dir, hparams):

  with tf.summary.create_file_writer(dir).as_default():

    HP.hparams(hparams)

    accuracy = model_t(hparams)

    tf.summary.scalar(metrics, accuracy, step=1)

Now we will train this model for different sets of hyperparameter combinations that we defined in the starting. Use the below code for doing the same.

session_num = 0

for num_units in neurons_hp.domain.values:

  for dropout_rate in (dropout_hp.domain.min_value, dropout_hp.domain.max_value):

    for optimizer in optimizer_hp.domain.values:

      hparams = {

          neurons_hp: num_units,

          dropout_hp: dropout_rate,

          optimizer_hp: optimizer,

      }

      run_name = "run-%d" % session_num

      print({h.name: hparams[h] for h in hparams})

      run('logs/hparam_tuning/' + run_name, hparams)

      session_num += 1

Output:

The above training snapshot is just for 2 combinations whereas this process would be repeated for several other combinations.

  1. Different Tensorboard Hprams Visualization 

Now we will visualize the log dir of the hyperparameters using a tensorboard. Use the below code to do so. Once we run the below code tensorboard dashboard will automatically come up. 

%tensorboard --logdir logs/hparam_tuning

This is the first view of the tensorboard dashboard that we will get. We need to click on Hparams to check different visualizations. There are mainly three different views in Hparams that are Table view, Parallel Coordinates view, and Scatter plot matrix view.  Check the below images for these views. 

Table View:

Parallel Coordinates view: 

Scatter plot matrix view:

Conclusion

Through this article, we explored how effective Hprams can be for the performance of the model. How Hprams computes training for different numbers of hyperparameter combinations that are defined by the programmer. We explored different Hparms tensorboard visualization for computing what hyperparameters result in higher accuracy. It also gives the table view for all the sets of combinations. This type of hyperparameter technique can be used to check what would be the best hyperparameter for the respective model to achieve the highest accuracy. 

Also, check similar articles on Hyperparameter tuning that is titled as “Keras Tuner for Hyperparameter tuning”.

More Great AIM Stories

Rohit Dwivedi
I am currently enrolled in a Post Graduate Program In Artificial Intelligence and Machine learning. Data Science Enthusiast who likes to draw insights from the data. Always amazed with the intelligence of AI. It's really fascinating teaching a machine to see and understand images. Also, the interest gets doubled when the machine can tell you what it just saw. This is where I say I am highly interested in Computer Vision and Natural Language Processing. I love exploring different use cases that can be build with the power of AI. I am the person who first develops something and then explains it to the whole community with my writings.

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
21st Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM