This Latest Model Serving Library Helps Deploy PyTorch Models At Scale

Model Serving Library

PyTorch has become popular within organisations to develop superior deep learning products. But building, scaling, securing, and managing models in production due to lack of PyTorch’s model server was keeping companies from going all in. The robust model server allows loading one or more models and automatically generating prediction API, backed by a scalable web server. Besides, it also offers production-critical features like logging, monitoring, and security.

Until now, TensorFlow Serving and Multi-Model Server catered to the needs of developers in production, but the lack of a model server that could effectively manage the workflows with PyTorch was causing hindrance among users. Consequently, to simplify the model development process, Facebook and Amazon collaborated to bring TorchServe, a PyTorch model serving library, that assists in deploying trained PyTorch models at scale without having to write custom code.

TorchServe & TorchElastic

Motivated by the request from Alex Wong on GitHub, Facebook and AWS released the much-needed service for PyTorch enthusiasts. TorchServe will be available as part of the PyTorch open-source project. Users can not only bring their models to production quicker for low latency prediction API, but also embed default handlers for the most common applications, such as object detection and text classification.


Sign up for your weekly dose of what's up in emerging technology.

TorchServe also includes multi-model serving, model versioning for A/B testing, monitoring metrics, and RESTful endpoints for application integration. Developers can leverage the model server on various machine learning environments, including Amazon SageMaker, container services, and EC2 (Amazon Elastic Computer Cloud).

You can install TorchServe using the pip package manager:

Download our Mobile App

pip install torch torchtext torchvision sentencepiece

pip install torchserve torch-model-archiver

Follow the link to get started with TorchServe.

Increasing Adoption Of PyTorch Within Companies

Many blue-chip companies are using PyTorch for their machine learning products, and critical use of this deep learning framework is being employed by the Toyota Research Institute Advanced Development (TRI-AD). The company is now able to simplify the model deployment and management process in production using TrochServe.

“Due to the absence of PyTorch’s model server, we were spending significant engineering effort in creating and maintaining software for deploying PyTorch models to our fleet of vehicles and cloud servers. However, with TrochServe, now we have a lightweight model server that is helping us continuously optimize our computer vision model effortlessly,” said Yusuke Yachide, Lead of ML Tools at TRI-AD.

The framework by Facebook witnessed a widespread adoption even without the model server due to several advantages, such as simple usability, readability, and distributed training. With the new PyTorch 1.5 that further focused on enhancing the speed in gaming and 3D graphics designing, it is poised to gain momentum among developers.

“Our team and the community has pushed PyTorch forward in recent years. A large number of researchers and enterprise users are leveraging AWS. Therefore, we collaborated with AWS and pushed PyTorch together,” said Bill Jia to TechCrunch.

Facebook has been enhancing PyTorch and making it more reliable within enterprises. More recently, it also announced TorchElastic’s integration with Kubernetes. This will enable developers to train models on a cluster of compute nodes that can dynamically change without disrupting the training workflow. Besides, the capability of built-in-fault-tolerant TorchElastic maintains the continuity even if nodes go down during the training.


Although the service is under experimental release, TorchServe has opened up a clear path to deploying PyTorch models for scalable production. However, it will have to enhance the service further to match up with TensorFlow Serving, which is a flexible and high-performance end-to-end serving system that can extend to a wide range of frameworks. 

TensorFlow has a considerable user base and exceptional community with more than 80.9k fork and 2,474 contributors on GitHub, whereas PyTorch has only 9.7k fork and 1,374 contributors on the platform. However, continuous advancement in the framework is allowing PyTorch to cover new grounds and become the go-to framework for all machine learning initiatives.

More Great AIM Stories

Rohit Yadav
Rohit is a technology journalist and technophile who likes to communicate the latest trends around cutting-edge technologies in a way that is straightforward to assimilate. In a nutshell, he is deciphering technology. Email:

AIM Upcoming Events

Early Bird Passes expire on 3rd Feb

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox