- Published on May 15, 2022
- In AI Features
Automating model parallelism with just one line of code
The difference between these two approaches maps naturally to the heterogeneity of a typical compute cluster.
- By Avi Gopani
Researchers from Google, Amazon Web Services, UC Berkeley, Shanghai Jiao Tong University, Duke University and Carnegie Mellon University have published a paper titled “Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning” at OSDI 2022. The paper introduces a new method for automating the complex process of parallelising a model with only one line of code. So how does Alpa work?
Model parallelism
Data parallelism is a technique where model weights are duplicated across accelerators while only partitioning and distributing the training data. The dataset is split into ‘N’ parts in data parallelism with ‘N’ being the quantity of GPUs. The parts are assigned to parallel computational machines after which gradients are calculated for each copy of
📣 Want to advertise in AIM? Book here
Avi Gopani
Avi Gopani is a technology journalist that seeks to analyse industry trends and developments from an interdisciplinary perspective at Analytics India Magazine. Her articles chronicle cultural, political and social stories that are curated with a focus on the evolving technologies of artificial intelligence and data analytics.
Related Posts
Top 10 Companies That Crowned Hyderabad as India’s Greenfield GCC Leader in 2025
Telangana has attracted over 75 greenfield GCCs in 2025, compared with 40-plus in Karnataka.
The AI Coding Gold Rush Ends Where Harness Begins
“Only 30% of software engineering happens on the laptop. The real 70% starts after you commit the code,” says Jyoti
How Gradient-Boosting is Quietly Powering India’s Research Push
From groundwater and slopes to carbon sinks, tools like CatBoost are enabling Indian scientists to extract insights and drive sustainability.
India’s Data Centre Boom Is Running Into a Talent Wall
With capacity expected to more than double this decade, the industry is investing in training as graduates struggle to meet
This Firm Wants to be the ‘Next Big Disruptor’ in Networking
Arrcus positions itself as a horizontal software layer that can run across different types of networking hardware.
Will 2026 be the year of AI IPOs?
With CoreWeave’s listing and Fractal Analytics going for an IPO, an array of AI companies are now looking to raise
Fighting Deepfakes May Not Be a Technology Problem
Defenders must be active at all times, while attackers need only one opportunity.
India’s Data Centre Expansion Is Decentralising
Without compute buildup beyond metros, the next wave of digital adoption will be constrained
Download the easiest way to
stay informed
stay informed
