Guide To Catalyst - A PyTorch Framework For Accelerated Deep Learning

Catalyst is a PyTorch framework developed with the intent of advancing research and development in the domain of deep learning. It enables code reusability, reproducibility and rapid experimentation so that users can conveniently create deep learning models and pipelines without writing another training loop.

Catalyst framework is part of the PyTorch ecosystem - a collection of numerous tools and libraries for AI development. It is also a part of the Catalyst Ecosystem - an MLOps ecosystem that expedites training, analysis and deployment of deep learning experiments through Catalyst, Alchemy and Reaction frameworks respectively.

Highlighting features of Catalyst

It enables creating deep learning pipelines with just a few lines of code.
It is compatible with Python 3.6+ and PyTorch 1.3+ versions.
It enables the creation of a universal train/inference loop with features such as evaluation metrics, model checkpointing and early-stopping included.
All the source code, as well as environment variables, remain saved thereby enabling code reproducibility.
It enables creating configuration files for storing the model’s hyperparameters.
It also has a provision for ‘callbacks’ - ways to reuse parts of a train/inference pipeline with required customizations.
It supports some of the best deep learning R&D practices such as Stochastic Weight Averaging (SWA), Ranger optimizer, one-cycle training, fp16 precision, distributed training and so on.

Practical implementation

Here’s a demonstration of handling image classification task using Catalyst. We have used the well-known MNIST dataset having 10 output classes (for classifying images of handwritten digits from 0 to 9). The code has been implemented using Google colab with Python 3.7.10 and Catalyst 21.3 versions. Step-wise explanation of the code is as follows:

Install Catalyst library using pip command

!pip install -U catalyst

(-U option installs the updated version of library)

Import required libraries and modules

 import os
 from torch import nn, optim
 from torch.utils.data import DataLoader
 from catalyst import dl, utils
 from catalyst.data.transforms import ToTensor
 from catalyst.contrib.datasets import MNIST

Define the neural network model, loss function and optimizer to be used

Here, we use a simple architecture comprising an input layer and a hidden layer each having 28 neurons while the output layer has 10 neurons since we have 10 possible output classes.

net = nn.Sequential(nn.Flatten(), nn.Linear(28 * 28, 10))

We have used the cross entropy loss function of PyTorch which combines LogSoftmax and NLLLoss in one class.

loss = nn.CrossEntropyLoss()

We are using Adam optimization algorithm

opt = optim.Adam(net.parameters(), lr=0.02) #’lr’ denotes learning rate

Load data for training and validation

Using DataLoader() method of PyTorch, we load the MNIST dataset from contrib API of Catalyst. We create a dictionary to load both training and validation sets at once by specifying the ‘train’ parameter of MNIST() as True or False respectively.

 data_loader = {
     "training": DataLoader(MNIST(os.getcwd(), train=True, download=True, 
     transform=ToTensor()), batch_size=32),
     "validation": DataLoader(MNIST(os.getcwd(), train=False,     
      download=True, 
      transform=ToTensor()), batch_size=32),
 }

ToTensor() method of ‘Data’ API converts a NumPy array to tensor representation.

Define an experiment runner for handling experiments with supervised model using SupervisedRunner() method of Runners API.

exp_runner = dl.SupervisedRunner(input_key="features", output_key="logits", target_key="targets", loss_key="loss")

Model training using Runners.train() method

 exp_runner.train(
     model=net,  #model to be trained
     criterion=loss, #loss function for training
     optimizer=opt, #optimizer for training

#dictionary with data loaders for training and validation
     loaders=data_loader, 

     num_epochs=1,
   #list with Catalyst callbacks
     callbacks=[  
        #callback for accuracy computation
         dl.AccuracyCallback(input_key="logits", target_key="targets",  
         topk_args=(1, 3, 5)),
        #callback for plotting confusion matrix to the loggers
         dl.ConfusionMatrixCallback(input_key="logits", 
         target_key="targets", 
         num_classes=10),
     ],
     logdir="./logs",  #path to output directory
  #name of loader to compute metrics and save the checkpoints
     valid_loader="valid",  
 #key to the metric name by which to select the checkpoints
     valid_metric="loss",
 #flag indicating whether to minimize the valid metric specified above
     minimize_valid_metric=True,
     verbose=True,  #display status of model training to console
 #load the best checkpoint state as per the validation metrics
     load_best_on_end=True,
 )

Sample output:

 Hparams (experiment): {}

 1/1 * Epoch (train): 100% 1875/1875 [00:20<00:00, 92.78it/s, 

accuracy=0.938, accuracy01=0.938, accuracy03=1.000, accuracy05=1.000, loss=0.098, lr=0.020, momentum=0.900]

 train (1/1) accuracy: 0.8802833557128906 | accuracy/std: 0.07081212792674185 | accuracy01: 0.8802833557128906 | accuracy01/std: 0.07081212792674185 | accuracy03: 0.9749666452407837 | accuracy03/std: 0.03580065525942475 | accuracy05: 0.9921166896820068 | accuracy05/std: 0.02176691186632642 | loss: 0.5139051675796509 | loss/std: 0.3664878010749817 | lr: 0.02 | momentum: 0.9
 
1/1 * Epoch (valid): 100%
313/313 [00:03<00:00, 83.65it/s, accuracy=0.875, accuracy01=0.875, accuracy03=1.000, accuracy05=1.000, loss=0.608, lr=0.020, momentum=0.900]
 
valid (1/1) accuracy: 0.8496999740600586 | accuracy/std: 0.08438007466043124 | accuracy01: 0.8496999740600586 | accuracy01/std: 0.08438007466043124 | accuracy03: 0.9573000073432922 | accuracy03/std: 0.04357493405029723 | accuracy05: 0.9886000156402588 | accuracy05/std: 0.019559607813700718 | loss: 0.8412032127380371 | loss/std: 0.5866137742996216 | lr: 0.02 | momentum: 0.9

Note: By default, top-1, top-3 and top-5 relevant sets of samples are considered for metrics computation. So the output displays average accuracy as well as accuracy01, accuracy03 and accuracy05 accordingly.

These accuracy results for training and validation sets get stored in ‘training.csv’ and ‘validation.csv’ files which appear as follows:

Sample ‘training.csv’

Sample ‘validation.csv’

Code source
Google colab notebook of the above implementation is available here.

References

Refer to the following sources for understanding the Catalyst framework in detail: