Active Hackathon

All The ML Papers Submitted By Apple At ICML 2021

We have rounded up all the machine learning research papers submitted by Apple at ICML 2021.

International Conference On Machine Learning (ICML) 2021 is back again with its 38th edition and is held virtually from July 18 – 24, 2021. The conference sees presentations of papers on all topics related to machine learning.

The 38th ICML Conference is sponsored by Apple. Here, we have rounded up all the machine learning research papers submitted by Apple at ICML 2021.


Sign up for your weekly dose of what's up in emerging technology.

Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling

The recent work from Vitaly Feldman, Audra McMillan and Kunal Talwar show how random shuffling strengthens differential privacy guarantees of locally randomised data. Such amplification implies significantly higher privacy guarantees for systems where data is contributed anonymously, which has sparked interest in the shuffle model of privacy. Researchers have provided a new analysis of privacy amplification by shuffling.

When is Memorisation of Irrelevant Training Data Necessary for High-Accuracy Learning?

Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith and Kunal Talwar worked on the research. Training algorithms operate on a huge range of prediction tasks, from image classification to language translation, often involving highly sensitive data. To succeed, models must contain information about the data they were trained on. These models can clearly be said to memorise at least part of their training data. Commonly, however, memorisation is an implicit, unintended side effect. In this paper, researchers aim to understand when this sort of memorisation is unavoidable. They give natural prediction problems in which every reasonably accurate training algorithm must encode, in the prediction model, information about a large subset of its training examples.

Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

Etai Littwin, Omid Saremi, Shuangfei Zhai, Vimal Thilak, Hanlin Goh, Joshua M. Susskind and Greg Yang investigated the effect of applying a bottleneck in an otherwise infinite width network. “The theoretical analysis reveals novel insights regarding the behaviour of input-output Jacobians, both at initialisation and training. Though stated for shallow, single hidden layer networks post bottleneck, we expect our results to hold in more general cases. Empirically, we observe that infinite width networks with bottlenecks train much faster than their fully infinite counterparts, while typically achieving better overall performance.”

Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks

Researchers including Shih-Yu Sun, Vimal Thilak, Etai Littwin, Omid Saremi and Joshua M. Susskind studied implicit regularisation induced by deep linear networks at autoencoder bottlenecks, revealing that latent codes are biased towards low-rank structures through greedy learning. Further, they showed that orthogonal initialisation removes prior spectral bias and significantly improves training stability across linear network depths when combined with principled learning speed adjustment.

Learning Neural Network Subspaces

Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi and Mohammad Rastegari have worked on the neural network optimisation landscape. Recent observations have surfaced the existence of paths of high accuracy containing diverse solutions and wider minima offering improved performance. Researchers aim to leverage both these properties with a single method and in a single training run as opposed to previous methods that require multiple training runs.

Lossless Compression of Efficient Private Local Randomisers

Locally Differentially Private (LDP) Reports are often used for the collection of machine learning and statistics. The most well-known LDP algorithms frequently necessitate transmitting unreasonably large messages from the client to the server (such as when constructing histograms over large domains or learning a high-dimensional model). This communication costs of LDP algorithms can be minimised, but it results in utility loss. Vitaly Feldman and Kunal Talwar have come up with a general approach that compresses every effective LDP algorithm with low loss in privacy and utility guarantees under standard cryptographic assumptions.

Private Adaptive Gradient Methods for Convex Optimisation

Researchers including Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht and Kunal Talwar introduce Pagan (Private AdaGrad with Adaptive Noise), a new differentially private variant of stochastic gradient descent and AdaGrad. They proposed a new private adaptive optimisation algorithm that analogises AdaGrad, showing that under certain natural distributional assumptions for the problems—similar to those that separate AdaGrad from non-adaptive methods [LD19] — the private versions of adaptive methods significantly outperform the standard non-adaptive private algorithms.

Private Stochastic Convex Optimisation: Optimal Rates in ℓ1 Geometry

Convex optimisation is one of the most well-studied problems in private data analysis. Existing works have largely studied optimisation problems over ℓ2-bounded domains. However, several machine learning applications, such as LASSO and minimisation over the probability simplex, involve optimisation over ℓ1-bounded domains. In this work, researchers including Hilal Asi, Vitaly Feldman, Tomer Koren and Kunal Talwar study the problem of differentially private stochastic convex optimisation (DP-SCO) over ℓ1-bounded domains.

Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning

In this work, researchers leveraged uncertainty estimation to detect, and down-weight OOD backups in the Bellman squared loss for offline RL. The proposed technique – UWAC, achieves superior performance and improved training stability without introducing any additional model or losses. Furthermore, they experimentally demonstrate the effectiveness of dropout uncertainty estimation at detecting OOD samples in offline RL. UWAC also can be applied to stabilise other actor-critic methods.

More Great AIM Stories

kumar Gandharv
Kumar Gandharv, PGD in English Journalism (IIMC, Delhi), is setting out on a journey as a tech Journalist at AIM. A keen observer of National and IR-related news.

Our Upcoming Events

Conference, Virtual
Genpact Analytics Career Day
3rd Sep

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
21st Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM

Council Post: How to Evolve with Changing Workforce

The demand for digital roles is growing rapidly, and scouting for talent is becoming more and more difficult. If organisations do not change their ways to adapt and alter their strategy, it could have a significant business impact.

All Tech Giants: On your Mark, Get Set – Slow!

In September 2021, the FTC published a report on M&As of five top companies in the US that have escaped the antitrust laws. These were Alphabet/Google, Amazon, Apple, Facebook, and Microsoft.

The Digital Transformation Journey of Vedanta

In the current digital ecosystem, the evolving technologies can be seen both as an opportunity to gain new insights as well as a disruption by others, says Vineet Jaiswal, chief digital and technology officer at Vedanta Resources Limited

BlenderBot — Public, Yet Not Too Public

As a footnote, Meta cites access will be granted to academic researchers and people affiliated to government organisations, civil society groups, academia and global industry research labs.