Guide To TensorLy: A Python Library For Tensor Learning

TensorLy is an open-source Python library that eases the task of performing tensor operations. It provides a high-level API for dealing with deep tensorized neural networks and tensor methods. It was created in 2015 by a senior research scientist at NVIDIA Research Group named Jean Kossaifi. It was presented at the NeurIPS workshop and was later published byJean Kossaifi, Yannis Panagakis, Anima Anandkumar and Maja Pantic in a JMLR paper in February 2019.

The project is supported by the following universities and organizations:

• NVIDIA
• INRIA
• Imperial College, London
• Caltech
• National and Kapodistrian University of Athens

TensorLy makes it easy to handle tensor decomposition, tensor learning and tensor algebra. It’s robust backend system enables users to perform computations with NumPy, TensorFlow, PyTorch, MXNet, CuPy or JAX. It can run methods that can scale on multiple CPUs or GPUs.

TensorLy was initially built on the top of NumPy and SciPy with a soft dependency on plotting library Matplotlib. A backend system was added to it later on for combining tensor methods with deep learning and for enabling transparent switching between various libraries and platforms.. It provides a simple and efficient API such as that of scikit-learn. However, scikit-learn deals with observations in terms of vectors whereas TensorLy represent them as higher-order arrays called tensors. 

The methods, functionalities and libraries supported be summarized in the following diagrams:

Installation of TensorLy

NOTE: TensorLy has been developed and tested only on Python 3 version.

1. Install the library using pip command:

pip install -U tensorly 

(option -U above is optional; use it for packages updates)

2. Install it using conda from tensorly channel:

conda install -c tensorly tensorly

Practical implementation

Here, we demonstrate how to use this library for basic tensor operations, tensor decomposition and tensor regression. Step-wise explanation for code of each of the use cases is as follows: 

Some basic tensor operations

1. Import required libraries

 import numpy as np
 import tensorly as tl
 from tensorly.testing import assert_array_equal
 #assert_array_equal() raises an error if two array-like objects are unequal 

2. Create a tensor (it is nothing but a numpy array)

orig_tensor = tl.tensor(np.arange(40).reshape((5, 2, 4)))

A 3D array containing elements from 0 to 39 will be created’

Print the created tensor

 orig_tensor = tl.tensor(np.arange(40).reshape((5, 2, 4)))
 print('Original tensor:\n{}'.format(orig_tensor)) 
Output:

3. Unfolding the tensor

 for m in range(tensor.ndim):  #for each dimension of the tensor
      print('Mode-{} unfolding:\n{}'.format(m, tl.unfold(tensor, m))) 

unfold() method performs unfolding starting from mode 0, and unfolds till (n-1)th dimension (where, ‘n’ is the dimension of the tensor).

4. Refolding the tensor

 for mode in range(tensor.ndim):  #for each dimension of the tensor
     unf = tl.unfold(tensor, mode) #unfold the tensor first
     f = tl.fold(unf, mode, tensor.shape) #re-fold the tensor
 #give an error if the refolded tensor and original tensor are unequal
     assert_array_equal(f, tensor)  

The unfolded tensor after executing above lines of code:

The re-folded tensor will be as follows:

Tensor decomposition example

Here, we demonstrate compression of colored image of a raccoon using parafac decomposition and Tucker decomposition techniques.

1. Install required libraries

 import matplotlib.pyplot as plt
 import tensorly as tl
 import numpy as np
 from scipy.misc import face #for the raccoon’s face image
 #to zoom the image represented in the form of an array
 from scipy.ndimage import zoom 
 from tensorly.decomposition import parafac
 from tensorly.decomposition import tucker
 from math import ceil 

2. Load the raccoon face image

 img = face()
 #Represent the image as a tensor
 img = tl.tensor(zoom(face(), (0.3, 0.3, 1)), dtype='float64') 

3. Define a function to convert image tensor’s datatype from float to unsigned integer

 def to_img(tensor):
     image = tl.to_numpy(tensor) #Convert the tensor to a numpy array
  #Subtract minimum element from the array from each of the elements
     image -= image.min() 
 #Divide the modified elements by the maximum array element
     image /= image.max()
 #Multiply the elements by 255 (0-255 is pixel range for colored images)
     image *= 255
 #Change the datatype to unit8 (unsigned integer)
     return image.astype(np.uint8) 

4. Define ranks of decomposition

 # Rank of CP decomposition
 cp_rank = 25
 # Rank of Tucker decomposition
 tucker_rank = [100, 100, 2] 

‘Rank’ here refers to the number of directions required to describe it.

5. Perform CP decomposition

 weights, factors = parafac(img, rank=cp_rank, init='random', tol=10e-6)
 # Reconstruct the image from the factors
 cp_rec = tl.cp_to_tensor((weights, factors)) 

6. Perform Tucker decomposition

 core, factors = tucker(img, rank=tucker_rank, init='random', tol=10e-5, 
 random_state=12345)
 #Convert the tucker tensor into a full tensor
 tucker_rec = tl.tucker_to_tensor((core, factors)) 

7. Plot the original image

 fig = plt.figure()
 ax = fig.add_subplot(1, 3, 1) 
 #arguments represent (number of rows, number of columns, index)
 ax.set_axis_off() #Turn the X and Y axes off
 ax.imshow(to_img(img))  #Display the plot
 ax.set_title('Original Image') #Title of the plot 

8. Similarly, plot the image compressed using CP decomposition and Tucker decomposition.

 ax = fig.add_subplot(1, 3, 2)
 ax.set_axis_off()
 ax.imshow(to_img(cp_rec))
 ax.set_title('CP decomposition')
 ax = fig.add_subplot(1, 3, 3)
 ax.set_axis_off()
 ax.imshow(to_img(tucker_rec))
 ax.set_title('Tucker Decomposition')
 #Adjust the padding between and around the subplots
 plt.tight_layout()
 #Display the plots
 plt.show() 

   Output:

Tensor regression example

1. Import required libraries

 import matplotlib.pyplot as plt
 from tensorly.base import tensor_to_vec, partial_tensor_to_vec
 from tensorly.datasets.synthetic import gen_image
 from tensorly.random import check_random_state
 from tensorly.regression.cp_regression import CPRegressor
 import tensorly as tl 

2. Set image parameters

 img_ht = 25 #height of the image
 img_width = 25 #width of the image
 # shape of the images
 patterns = ['swiss', 'circle']
 # ranks for testing
 ranks = [1, 2, 3, 4, 5] 

3. Generate random samples

 r = check_random_state(1)
 #Tensor representation of the samples
 X = tl.tensor(r.normal(size=(1000, img_ht, img_width), loc=0,  
 scale=1))
 #numpy.random.RandomState.normal() draws random samples from a  
 normal distribution 

4. Parameters of the plot

 n_rows = len(patterns)
 n_columns = len(ranks) + 1 

5. Plot the images before and after applying regression

 fig = plt.figure()
 for i, pattern in enumerate(patterns):
      # Generate the original image for regression testing
        wt_img = gen_image(region=pattern, image_height=img_ht,    
               image_width=img_width)
               wt_img = tl.tensor(wt_img) #Convert the image into tensor
        # Generate the labels
        y = tl.dot(partial_tensor_to_vec(X, skip_begin=1),    
        tensor_to_vec(wt_img))
        # Plot the original weights
        ax = fig.add_subplot(n_rows, n_columns, i*n_columns + 1)
        ax.imshow(tl.to_numpy(wt_img), cmap=plt.cm.OrRd,    
        interpolation='nearest')
        ax.set_axis_off()
        if i == 0:
            ax.set_title('Original\nweights')
       for j, rank in enumerate(ranks):
    # Create a tensor Regressor estimator
         est = CPRegressor(weight_rank=rank, tol=10e-7, n_iter_max=100,   
          reg_W=1, verbose=0)
          # Fit the estimator to the data
          est.fit(X, y)
          #Plot the images after regression
 ax = fig.add_subplot(n_rows, n_columns, i*n_columns + j + 2)
          ax.imshow(tl.to_numpy(est.weight_tensor_), cmap=plt.cm.OrRd,   
          interpolation='nearest')
          ax.set_axis_off()
          if i == 0:
              ax.set_title('Learned\nrank = {}'.format(rank))
 plt.suptitle("CP tensor regression") #Add a centered title to the figure
 plt.show() #Display the plot 

Output:

• Google colab notebooks of the above implementations:

• Basic operations
• Decomposition
• Regression

References

Refer to the following sources to get an in-depth understanding of TensorLy Python library:

• Research paper
• Official website
• GitHub repository