How to deploy and monitor your Keras model with Comet?

Comet is an online experimentation platform for artificial intelligence and machine learning used by Researchers and Data Scientists which provides an easy platform for tracking the ML models and comparing the results of the model on the server. Comet provides an excellent dashboard to visualize the various parameters of the model deployed in the interface and keeps track of the model internally. So this article provides a brief overview of how to deploy a Keras model on the online platform of Comet and make suitable interpretations from the model.

Table of Contents

1. Introduction to Comet
2. Implementing a Keras model from scratch
3. Logging in to the Comet Interface
4. Integrating the Keras model into the comet interface
5. Obtaining predictions from comet interface
6. Mounting the model in the comet server
7. Visualizing the model deployed on the server
8. Summary

Let’s first briefly discuss about the Comet. 

Introduction to Comet

Comet is an online API that serves as an online platform to deploy machine learning or deep learning models and continuously validate all the stages of a model deployment lifecycle until production. The model on the platform can be deployed as public or can be limited for certain access for analyzing the parameters of the model deployed on their platform. With the concerned access, comet provides the flexibility to access the model at any time on the server and monitor the models deployed on their server.

Are you looking for a complete repository of Python libraries used in data science, check out here.

Implementing a Keras model from scratch

Keras module provides useful datasets for deep learning model development and here we have used the MNIST Digits dataset for model development and deployment of the model developed on the Comet server.

So initially the dataset was acquired and split into train and test and the split data was visualized using subplots. Later a simple Sequential model was built with appropriate input dimensions and activation functions. The dependent features of the dataset were suitably reshaped and the target variables were encoded according to the number of digits present in the MNIST dataset. After suitable preprocessing the model was compiled with categorical_crossentropy as the loss function and accuracy as the metrics to evaluate from the model developed. Also, the model was validated for its training and testing accuracy and loss. The complete steps to follow to implement a basic digit classification model for the MNIST dataset are shown below.

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import random
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.preprocessing import image
from tensorflow.keras.optimizers import RMSprop
from tensorflow.keras.utils import to_categorical
 
(X_train,y_train),(X_test,y_test)=mnist.load_data()
print('Number of training records',X_train.shape)
print('Number of testing records',y_test.shape)

plt.figure(figsize=(5,10))
 
for i in range(9):
   plt.subplot(3,3,i+1)
   num = random.randint(0, len(X_train))
   plt.imshow(X_train[num], cmap='gray', interpolation='none')
   plt.title("Class {}".format(y_train[num]))
   #plt.tight_layout()
   plt.axis('off')
 plt.show()

X_test_original=X_test.copy()
# preprocess and normalize the data
X_train = X_train.reshape(60000, 28*28)
X_test = X_test.reshape(10000, 28*28)
X_train1=X_train/255.
X_test1=X_test/255.
## Encoding the target variable
y_train=to_categorical(y_train,10)
y_test=to_categorical(y_test,10)

With all these suitable preprocessing model building was taken up as shown below.

Model Building

model=Sequential()
 
model.add(Dense(units=512,activation='relu',input_dim=784))
model.add(Dense(units=256,activation='relu'))
model.add(Dense(units=125,activation='relu'))
model.add(Dense(units=10,activation='softmax'))
 
model.summary()

model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])
model_res=model.fit(X_train1,y_train,epochs=5,validation_data=(X_test,y_test))

Evaluating the model performance

plt.figure(0)
plt.plot(model_res.history['accuracy'],label='training accuracy')
plt.plot(model_res.history['val_accuracy'],label='testing accuracy')
plt.title('Accuracy')
plt.xlabel('epochs')
plt.ylabel('accuracy')
plt.legend()
plt.show()
 
plt.figure(1)
plt.plot(model_res.history['loss'],label='training loss')
plt.plot(model_res.history['val_loss'],label='testing loss')
plt.title('Loss')
plt.xlabel('epochs')
plt.ylabel('loss')
plt.legend()
plt.show()

model.evaluate(X_train,y_train)
model.evaluate(X_test,y_test)

Logging in to the Comet Interface

Firstly the comet module can be easily installed in the working environment using the pip commands and the module can be loaded using the import statements. So once when the module is successfully installed in the working environment a random name has to be given to the model that would be deployed onto the comet server using the init() built-in function. 

So a link will be generated in that particular run instance and a unique API Key will be generated which can be stored in the form of a string character in the working environment and utilized if required. The steps involved are shown below.

import comet_ml
from comet_ml import Experiment
comet_ml.init('comet-keras')

Initiating the comet interface

keras_exp=Experiment(project_name='comet-keras',api_key=API_key)

So now this comet interface can be used to train, test, and evaluate the parameters of the model on the comet server. Now the interface and the login with the appropriate API key can be validated using the display() function as shown below.

keras_exp.display()

Integrating the Keras model in the comet interface

The integration of the model into the server starts with primarily hashing the training dataset using the log_dataset_hash() function and the experiment can be trained on the comet interface using the train() inbuilt function of the comet. The steps involved in hashing the dataset and loading the model are shown below.

keras_exp.log_dataset_hash(X_train)
with keras_exp.train():
model_res=model.fit(X_train1,y_train,batch_size=32,epochs=10,validation_data=(X_test1,y_test))

Evaluating model parameters on the Comet interface

Now using the test() inbuilt function of the comet the model deployed on the server can be evaluated on various parameters. Here the steps involved to evaluate the training and testing loss and accuracy are shown below.

with keras_exp.test():
   test_loss, test_accuracy = model.evaluate(X_test1, y_test)
   train_loss, train_accuracy = model.evaluate(X_train1, y_train)
   metrics = {'test_loss':test_loss,'test_accuracy':test_accuracy,'train_loss':train_loss,'train_accuracy':train_accuracy}
}

Logging various metrics in the Comet Interface

The log_metrics() inbuilt function is utilized to deploy or log in the metrics used for evaluating the model on the comet server.

keras_exp.log_metrics(metrics)

Now let us see how to obtain predictions from the model deployed onto the comet interface.

Obtaining predictions from comet interface

For obtaining the predictions from the comet interface the log_figure() function is used which provides outputs for the model’s prediction for each digit being classified as shown below. But for obtaining predictions the original TensorFlow model has to be used and to visualize the model performance the inbuilt function of Comet can be used.

model_pred = model.predict(X_test1[:10]).argmax(-1)
 
# remember our copy of x_test? This is where it comes into play again!
# Since we flatted x_test during pre-processing, we need to preserve an unflattened version
plt.figure(figsize=(16,8))
for i in range(10):
   plt.subplot(1, 10, i+1)
   plt.imshow(X_test_original[i], interpolation='nearest')
   plt.text(0, 0, model_pred[i], color='black',
            bbox=dict(facecolor='white', alpha=1))
   plt.axis('off')
 
 
keras_exp.log_figure('Predicted MNIST Digits from Comet Interface',plt)

By clicking on the web link from the plots obtained the output of the model deployed on the server can be downloaded if required for documentation along with the REST api information. Now the experiment on the comet server has to be ended to visualize how the comet interface has monitored the model deployed on its server

Logging out the model deployed on the server

Now the experiment carried out on the server has to be ended using the end() inbuilt function. While ending the experiment on the server, several information pertaining to the Data, Metrics used to evaluate the model, and various other information will be provided as shown below.

keras_exp.end()

Mounting the model in the comet server

The model can be saved in an HDF5 (h5) format and the model can be mounted onto the comet server using log_model() function which may facilitate using the model weights for any parameter evaluation or using the model weights for any other tasks. The steps involved to mount the model on the comet server are shown below.

model.save('keras-comet-new.h5')
keras_exp.log_model('Saved Keras model',"/content/keras-comet-new.h5")

Visualizing the model deployed on the server

The display() function of the comet can be used to visualize the parameters and to monitor the model functionality on the server.

keras_exp.display()

The entire code that is being used to deploy on the server can be seen in the code section of the server where the entire code is made visible to the person with access.

All the experiments carried out on the server with its logging time and the parameters evaluated can be visualized in the metrics pane as shown below.

Along with this, the hardware resource usage can also be visualized in the server as shown below.

Summary

So this article provides a brief overview of how to develop a Keras model and deploy it on the Comet server to continuously monitor the model. Easily interpretable reports and graphs are produced with respect to the model performance on the server and the model’s memory consumption for each of the hardware resources used in the premise can also be visualized using the system metrics. So by using the Comet server the model can be made available on any platform and utilized accordingly to monitor the model on the servers.