Open-source machine learning framework PyTorch has released its version 1.9 just two months after the release of version 1.8. The new release consists of 3,400 new commits made by 398 contributors.
Highlights:
- Improvements to functions such as torch.linalg, torch.special, and Complex Autograd to support scientific computing
- Beta release of Mobile Interpreter to execute PyTorch programs in edge devices
- Upstreaming of TorchElastic into Pytorch Core
- Major updates to the PyTorch RPC framework
- New APIs for performance optimization
- Support for distributed training, GPU utilization, and SM efficiency in the PyTorch Profiler.
Increased support for scientific computing
In the new version, torch.linalg has been moved to a stable release. This module extends PyTorch support for linear algebra by implementing several functions such as NumPy’s linear algebra module, torch.linalg.matrix_norm and torch.linalg.householder_product.
The Complex Autograd feature has also been promoted to a stable state after being released as a beta in v1.8. In the latest release, the team has extended support for Complex Autograd for 98 percent operators, added more OpInfos, and TorchAudio has been migrated to native complex tensors.
Complex Autograd provides users the ability to calculate complex gradients and work with complex variables. It is an important feature for users who need to work with complex numbers in PyTorch toolkits such as TorchAudio, ESPNet, FastMRI, and Asteroid.
PyTorch 1.9 also includes torch.use_determinstic_algorithms option for debugging and writing reproducible programs.
The torch.special module, which contains functions for scientific computing and helps in working with distributions like iv, ive, erfcx, logerfc, and logerfcx, has been made available in beta. Along with torch.special module, nn.Model parameterization is also now available in beta. It helps to reduce the space required for parameters without using any special optimization methods.
PyTorch Mobile
Mobile Interpreter in beta to execute PyTorch programs in edge devices is one of the major announcements. Mobile Interpreter will significantly reduce the binary size as compared to the current on-device runtime. “The current pt size with MobileNetV2 in arm64-v8a Android is 8.6 MB compressed and 17.8 MB uncompressed. Using Mobile Interpreter, we are targeting at the compressed size below 4 MB and uncompressed size below 8MB,” the team said in the blog.
PyTorch is releasing a new PyTorch Video library-based video app and an update on the speech recognition app based on torchaudio and wave2vec model. With the addition of these two apps, PyTorch now offers a full demo apps suite that cover image, text, audio, and video.
Distributed training
TorchElastic is a coordinator for PyTorch worker processes that handles scaling events. It was open-sourced more than a year ago and has been used in various distributed torch use-cases such as deepspeech.pytorch, pytorch-lighting, and Kubernetes CRD. In v1.9, TorchElastic has been made part of PyTorch core.
Other beta features made available in the distributed package include–CUDA support in RPC, ZeroRedundancyOptimizer, and support for profiling distributed collectives.
Performance optimization
Module Freezing is a process for optimising and specialising programs; freezing is a recommended practice for model deployment. It inlines module parameters and attribute values as constants into the TorchScript internal representation.
PyTorch Profiler, Inference mode API, and torch.package have been graduated to beta.
Other updates
Along with the release of PyTorch v1.9, the team also announced updates to a couple of PyTorch libraries which include new releases for domain libraries such as TorchVision, TorchText, and TorchAudio.
Highlights:
- TorchVision: Addition of new SSD and SSDLite models, quantized kernels for object detection, and iOS support
- TorchAudio: Several new additions including wav2vec 2.0 model, options for quality control in sampling, and performance improvements in spectral operations and resampling.
- TorchText – Introduction of a new high-performance Vocab module for common functional APIs for NLP workflows.
