Top Machine Learning Research Papers

Advances in machine learning and deep learning research are reshaping our technology. Machine learning and deep learning have accomplished various astounding feats, and key research articles have resulted in technical advances used by billions of people. The research in this sector is advancing at a breakneck pace and assisting you to keep up. Here is a collection of the most important scientific study papers in machine learning.

Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training

The authors of this work examined why ACGAN training becomes unstable as the number of classes in the dataset grows. The researchers revealed that the unstable training occurs due to a gradient explosion problem caused by the unboundedness of the input feature vectors and the classifier's poor classification capabilities during the early training stage. The researchers presented the Data-to-Data Cross-Entropy loss (D2D-CE) and the Rebooted Auxiliary Classifier Generative Adversarial Network to alleviate the instability and reinforce ACGAN (ReACGAN). Additionally, extensive tests of ReACGAN demonstrate that it is resistant to hyperparameter selection and is compatible with a variety of architectures and differentiable augmentations.

This article is ranked #1 on CIFAR-10 for Conditional Image Generation.

For the research paper, read here.

For code, see here.

Dense Unsupervised Learning for Video Segmentation

The authors presented a straightforward and computationally fast unsupervised strategy for learning dense spacetime representations from unlabeled films in this study. The approach demonstrates rapid convergence of training and a high degree of data efficiency. Furthermore, the researchers obtain VOS accuracy superior to previous results despite employing a fraction of the previously necessary training data. The researchers acknowledge that the research findings may be utilised maliciously, such as for unlawful surveillance, and that they are excited to investigate how this skill might be used to better learn a broader spectrum of invariances by exploiting larger temporal windows in movies with complex (ego-)motion, which is more prone to disocclusions.

This study is ranked #1 on DAVIS 2017 for Unsupervised Video Object Segmentation (val).

For the research paper, read here.

For code, see here.

Temporally-Consistent Surface Reconstruction using Metrically-Consistent Atlases

The authors offer an atlas-based technique for producing unsupervised temporally consistent surface reconstructions by requiring a point on the canonical shape representation to translate to metrically consistent 3D locations on the reconstructed surfaces. Finally, the researchers envisage a plethora of potential applications for the method. For example, by substituting an image-based loss for the Chamfer distance, one may apply the method to RGB video sequences, which the researchers feel will spur development in video-based 3D reconstruction.

This article is ranked #1 on ANIM in the category of Surface Reconstruction. 

For the research paper, read here.

For code, see here.

EdgeFlow: Achieving Practical Interactive Segmentation with Edge-Guided Flow

The researchers propose a revolutionary interactive architecture called EdgeFlow that uses user interaction data without resorting to post-processing or iterative optimisation. The suggested technique achieves state-of-the-art performance on common benchmarks due to its coarse-to-fine network design. Additionally, the researchers create an effective interactive segmentation tool that enables the user to improve the segmentation result through flexible options incrementally.

This paper is ranked #1 on Interactive Segmentation on PASCAL VOC

For the research paper, read here.

For code, see here.

Learning Transferable Visual Models From Natural Language Supervision

The authors of this work examined whether it is possible to transfer the success of task-agnostic web-scale pre-training in natural language processing to another domain. The findings indicate that adopting this formula resulted in the emergence of similar behaviours in the field of computer vision, and the authors examine the social ramifications of this line of research. CLIP models learn to accomplish a range of tasks during pre-training to optimise their training objective. Using natural language prompting, CLIP can then use this task learning to enable zero-shot transfer to many existing datasets. When applied at a large scale, this technique can compete with task-specific supervised models, while there is still much space for improvement.

This research is ranked #1 on Zero-Shot Transfer Image Classification on SUN

For the research paper, read here.

For code, see here.

CoAtNet: Marrying Convolution and Attention for All Data Sizes

The researchers in this article conduct a thorough examination of the features of convolutions and transformers, resulting in a principled approach for combining them into a new family of models dubbed CoAtNet. Extensive experiments demonstrate that CoAtNet combines the advantages of ConvNets and Transformers, achieving state-of-the-art performance across a range of data sizes and compute budgets. Take note that this article is currently concentrating on ImageNet classification for model construction. However, the researchers believe their approach is relevant to a broader range of applications, such as object detection and semantic segmentation.

This paper is ranked #1 on Image Classification on ImageNet (using extra training data).

For the research paper, read here.

For code, see here.

SwinIR: Image Restoration Using Swin Transformer

The authors of this article suggest the SwinIR image restoration model, which is based on the Swin Transformer. The model comprises three modules: shallow feature extraction, deep feature extraction, and human-recognition reconstruction. For deep feature extraction, the researchers employ a stack of residual Swin Transformer blocks (RSTB), each formed of Swin Transformer layers, a convolution layer, and a residual connection.

This research article is ranked #1 on Image Super-Resolution on Manga109 - 4x upscaling.

For the research paper, read here.

For code, see here.

Artificial Replay: A Meta-Algorithm for Harnessing Historical Data in Bandits

Ways to incorporate historical data are still unclear: initialising reward estimates with historical samples can suffer from bogus and imbalanced data coverage, leading to computational and storage issues—particularly in continuous action spaces. The paper addresses the obstacles by proposing ‘Artificial Replay’, an algorithm to incorporate historical data into any arbitrary base bandit algorithm. 

Read the full paper here. 

Bootstrapped Meta-Learning

Author(s) – Sean R. Sinclair et al.

The paper proposes an algorithm in which the meta-learner teaches itself to overcome the meta-optimisation challenge. The algorithm focuses on meta-learning with gradients, which guarantees performance improvements. Furthermore, the paper also looks at how bootstrapping opens up possibilities. 

Read the full paper here.

LaMDA: Language Models for Dialog Applications

Author(s) – Sebastian Flennerhag et al.

The research describes the LaMDA system which caused chaos in AI this summer when a former Google engineer claimed that it had shown signs of sentience. LaMDA is a family of large language models for dialogue applications based on Transformer architecture. The interesting feature of the model is its fine-tuning with human-annotated data and the possibility of consulting external sources. This is a very interesting model family, which we might encounter in many applications we use daily. 

Read the full paper here.

Competition-Level Code Generation with AlphaCode

Author(s) – Yujia Li et al.

Systems can help programmers become more productive. The following research addresses the problems with incorporating innovations in AI into these systems. AlphaCode is a system that creates solutions for problems that require deeper reasoning. 

Read the full paper here.

Privacy for Free: How does Dataset Condensation Help Privacy?

Author(s) – Tian Dong et al.

The paper focuses on Privacy Preserving Machine Learning, specifically deducting the leakage of sensitive data in machine learning. It puts forth one of the first propositions of using dataset condensation techniques to preserve the data efficiency during model training and furnish membership privacy.

Read the full paper here.

Why do tree-based models still outperform deep learning on tabular data?

Author(s) – Léo Grinsztajn, Edouard Oyallon and Gaël Varoquaux

The research answers why deep learning models still find it hard to compete on tabular data compared to tree-based models. It is shown that MLP-like architectures are more sensitive to uninformative features in data compared to their tree-based counterparts. 

Read the full paper here.

Multi-Objective Bayesian Optimisation over High-Dimensional Search Spaces 

Author(s) – Samuel Daulton et al.

The paper proposes ‘MORBO’, a scalable method for multiple-objective BO as it performs better than that of high-dimensional search spaces. MORBO significantly improves the sample efficiency and, where existing BO algorithms fail, MORBO provides improved sample efficiencies over the current approach. 

Read the full paper here.

A Path Towards Autonomous Machine Intelligence Version 0.9.2

Author(s) – Yann LeCun

The research offers a vision about how to progress towards general AI. The study combines several concepts: a configurable predictive world model, behaviour driven through intrinsic motivation, and hierarchical joint embedding architectures trained with self-supervised

learning. 

Read the full paper here. 

TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data

Author(s) –  Shreshth Tuli, Giuliano Casale and Nicholas R. Jennings

This is a specialised paper applying transformer architecture to the problem of unsupervised anomaly detection in multivariate time series. Many architectures which were successful in other fields are, at some point, also being applied to time series. The research shows improved performance on some known data sets. 

Read the full paper here.

Differentially Private Bias-Term only Fine-tuning of Foundation Models

Author(s) – Zhiqi Bu et al. 

In the paper, researchers study the problem of differentially private (DP) fine-tuning of large pre-trained models—a recent privacy-preserving approach suitable for solving downstream tasks with sensitive data. Existing work has demonstrated that high accuracy is possible under strong privacy constraints yet requires significant computational overhead or modifications to the network architecture.

Read the full paper here. 

ALBERT: A Lite BERT

Usually, increasing model size when pretraining natural language representations often result in improved performance on downstream tasks, but the training times become longer. To address these problems, the authors in their work presented two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. The authors also used a self-supervised loss that focuses on modelling inter-sentence coherence and consistently helped downstream tasks with multi-sentence inputs. According to results, this model established new state-of-the-art results on the GLUE, RACE, and squad benchmarks while having fewer parameters compared to BERT-large. 

Check the paper here.

Beyond Accuracy: Behavioral Testing of NLP Models with CheckList

Microsoft Research, along with the University of Washington and the University of California, in this paper, introduced a model-agnostic and task agnostic methodology for testing NLP models known as CheckList. This is also the winner of the best paper award at the ACL conference this year. It included a matrix of general linguistic capabilities and test types that facilitate comprehensive test ideation, as well as a software tool to generate a large and diverse number of test cases quickly. 

Check the paper here.

Linformer

Linformer is a Transformer architecture for tackling the self-attention bottleneck in Transformers. It reduces self-attention to an O(n) operation in both space- and time complexity. It is a new self-attention mechanism which allows the researchers to compute the contextual mapping in linear time and memory complexity with respect to the sequence length. 

Read more about the paper here.

Plug and Play Language Models

Plug and Play Language Models (PPLM) are a combination of pre-trained language models with one or more simple attribute classifiers. This, in turn, assists in text generation without any further training. According to the authors, model samples demonstrated control over sentiment styles, and extensive automated and human-annotated evaluations showed attribute alignment and fluency. 

Check the paper here.

Reformer 

The researchers at Google, in this paper, introduced Reformer. This work showcased that the architecture of a Transformer can be executed efficiently on long sequences and with small memory. The authors believe that the ability to handle long sequences opens the way for the use of the Reformer on many generative tasks. In addition to generating very long coherent text, the Reformer can bring the power of Transformer models to other domains like time-series forecasting, music, image and video generation. 

Check the paper here.

An Image is Worth 16X16 Words

The irony here is that one of the popular language models, Transformers have been made to do computer vision tasks. In this paper, the authors claimed that the vision transformer could go toe-to-toe with the state-of-the-art models on image recognition benchmarks, reaching accuracies as high as 88.36% on ImageNet and 94.55% on CIFAR-100. For this, the vision transformer receives input as a one-dimensional sequence of token embeddings. The image is then reshaped into a sequence of flattened 2D patches. The transformers in this work use constant widths through all of its layers.

Check the paper here.

Unsupervised Learning of Probably Symmetric Deformable 3D Objects

Winner of the CVPR best paper award, in this work, the authors proposed a method to learn 3D deformable object categories from raw single-view images, without external supervision. This method uses an autoencoder that factored each input image into depth, albedo, viewpoint and illumination. The authors showcased that reasoning about illumination can be used to exploit the underlying object symmetry even if the appearance is not symmetric due to shading.

Check the paper here.

Generative Pretraining from Pixels

In this paper, OpenAI researchers examined whether similar models can learn useful representations for images. For this, the researchers trained a sequence Transformer to auto-regressively predict pixels, without incorporating knowledge of the 2D input structure. Despite training on low-resolution ImageNet without labels, the researchers found that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and low-data classification. On CIFAR-10, it achieved 96.3% accuracy with a linear probe, outperforming a supervised Wide ResNet, and 99.0% accuracy with full fine-tuning and matching the top supervised pre-trained models. An even larger model, trained on a mixture of ImageNet and web images, is competitive with self-supervised benchmarks on ImageNet, achieving 72.0% top-1 accuracy on a linear probe of their features.

Check the paper here.

Deep Reinforcement Learning and its Neuroscientific Implications

In this paper, the authors provided a high-level introduction to deep RL, discussed some of its initial applications to neuroscience, and surveyed its wider implications for research on brain and behaviour and concluded with a list of opportunities for next-stage research. Although DeepRL seems to be promising, the authors wrote that it is still a work in progress and its implications in neuroscience should be looked at as a great opportunity. For instance, deep RL provides an agent-based framework for studying the way that reward shapes representation, and how representation, in turn, shapes learning and decision making -- two issues which together span a large swath of what is most central to neuroscience. 

Check the paper here.

Dopamine-based Reinforcement Learning

Why humans doing certain things are often linked to dopamine, a hormone that acts as the reward system (think: the likes on your Instagram page). So, keeping this fact in hindsight, DeepMind with the help of Harvard labs, analysed dopamine cells in mice and recorded how the mice received rewards while they learned a task. They then checked these recordings for consistency in the activity of the dopamine neurons with standard temporal difference algorithms. This paper proposed an account of dopamine-based reinforcement learning inspired by recent artificial intelligence research on distributional reinforcement learning. The authors hypothesised that the brain represents possible future rewards not as a single mean but as a probability distribution, effectively representing multiple future outcomes simultaneously and in parallel. 

Check the paper here.

Lottery Tickets In Reinforcement Learning & NLP

In this paper, the authors bridged natural language processing (NLP) and reinforcement learning (RL). They examined both recurrent LSTM models and large-scale Transformer models for NLP and discrete-action space tasks for RL. The results suggested that the lottery ticket hypothesis is not restricted to supervised learning of natural images, but rather represents a broader phenomenon in deep neural networks.

Check the paper here.

What Can Learned Intrinsic Rewards Capture

In this paper, the authors explored if the reward function itself can be a good locus of learned knowledge. They proposed a scalable framework for learning useful intrinsic reward functions across multiple lifetimes of experience and showed that it is feasible to learn and capture knowledge about long-term exploration and exploitation into a reward function. 

Check the paper here.

AutoML- Zero

The progress of AutoML has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks, or similarly restrictive search spaces. In this paper, the authors showed that AutoML could go further with AutoML Zero, that automatically discovers complete machine learning algorithms just using basic mathematical operations as building blocks. The researchers demonstrated this by introducing a novel framework that significantly reduced human bias through a generic search space.

Check the paper here.

Rethinking Batch Normalization for Meta-Learning

Batch normalization is an essential component of meta-learning pipelines. However, there are several challenges. So, in this paper, the authors evaluated a range of approaches to batch normalization for meta-learning scenarios and developed a novel approach — TaskNorm. Experiments demonstrated that the choice of batch normalization has a dramatic effect on both classification accuracy and training time for both gradient-based and gradient-free meta-learning approaches. The TaskNorm has been found to be consistently improving the performance.

Check the paper here.

Meta-Learning without Memorisation

Meta-learning algorithms need meta-training tasks to be mutually exclusive, such that no single model can solve all of the tasks at once. In this paper, the authors designed a meta-regularisation objective using information theory that successfully uses data from non-mutually-exclusive tasks to efficiently adapt to novel tasks.

Check the paper here.

Understanding the Effectiveness of MAML

Model Agnostic Meta-Learning (MAML) consists of optimisation loops, from which the inner loop can efficiently learn new tasks. In this paper, the authors demonstrated that feature reuse is the dominant factor and led to ANIL (Almost No Inner Loop) algorithm -- a simplification of MAML where the inner loop is removed for all but the (task-specific) head of the underlying neural network. 

Check the paper here.

Your Classifier is Secretly an Energy-Based Model

This paper proposed attempts to reinterpret a standard discriminative classifier as an energy-based model. In this setting, wrote the authors, the standard class probabilities can be easily computed. They demonstrated that energy-based training of the joint distribution improves calibration, robustness, handout-of-distribution detection while also enabling the proposed model to generate samples rivalling the quality of recent GAN approaches. This work improves upon the recently proposed techniques for scaling up the training of energy-based models. It has also been the first to achieve performance rivalling the state-of-the-art in both generative and discriminative learning within one hybrid model.

Check the paper here.

Reverse-Engineering Deep ReLU Networks

This paper investigated the commonly assumed notion that neural networks cannot be recovered from its outputs, as they depend on its parameters in a highly nonlinear way. The authors claimed that by observing only its output, one could identify the architecture, weights, and biases of an unknown deep ReLU network. By dissecting the set of region boundaries into components associated with particular neurons, the researchers showed that it is possible to recover the weights of neurons and their arrangement within the network.

Check the paper here.

Cricket Analytics and Predictor

Authors: Suyash Mahajan,  Salma Shaikh, Jash Vora, Gunjan Kandhari,  Rutuja Pawar,

Abstract:  The paper embark on predicting the outcomes of Indian Premier League (IPL) cricket match using a supervised learning approach from a team composition perspective. The study suggests that the relative team strength between the competing teams forms a distinctive feature for predicting the winner. Modeling the team strength boils down to modeling individual player‘s batting and bowling performances, forming the basis of our approach.

Research Methodology: In this paper, two methodologies have been used. MySQL database is used for storing data whereas Java for the GUI. The algorithm used is Clustering Algorithm for prediction. The steps followed are as

• Begin with a decision on the value of k being the number of clusters.
• Put any initial partition that classifies the data into k clusters.
• Take every sample in the sequence; compute its distance from centroid of each of the clusters. If sample is not in the cluster with the closest centroid currently, switch this sample to that cluster and update the centroid of the cluster accepting the new sample and the cluster losing the sample.

For the research paper, read here

2.Real Time Sleep / Drowsiness Detection - Project Report

Author: Roshan Tavhare

Institute: University of Mumbai

Abstract: The main idea behind this project is to develop a nonintrusive system which can detect fatigue of any human and can issue a timely warning. Drivers who do not take regular breaks when driving long distances run a high risk of becoming drowsy a state which they often fail to recognize early enough.

Research Methodology: A training set of labeled facial landmarks on an image. These images are manually labeled, specifying specific (x, y) -coordinates of regions surrounding each facial structure.

2. Priors, more specifically, the probability on distance between pairs of input pixels. The pre-trained facial landmark detector inside the dlib library is used to estimate the location of 68 (x, y)-coordinates that map to facial structures on the face.

For the research paper, read here.

A Study of Various Text Augmentation Techniques for Relation Classification in Free Text

Authors: Chinmaya Mishra Praveen Kumar and Reddy Kumar Moda,  Syed Saqib Bukhari and Andreas Dengel

Institute: German Research Center for Artificial Intelligence (DFKI), Kaiserslautern, Germany

Abstract: In this paper, the researchers explore various text data augmentation techniques in text space and word embedding space. They studied the effect of various augmented datasets on the efficiency of different deep learning models for relation classification in text.

Research Methodology: The researchers implemented five text data augmentation techniques (Similar word, synonyms, interpolation, extrapolation and random noise method)  and explored the ways in which we could preserve the grammatical and the contextual structures of the sentences while generating new sentences automatically using data augmentation techniques.

For the research paper, read here.

Smart Health Monitoring and Management Using Internet of Things, Artificial Intelligence with Cloud Based Processing

Author: Prateek Kaushik

Institute: G D Goenka University, Gurugram

Abstract: This research paper described a personalised smart health monitoring device using wireless sensors and the latest technology.

Research Methodology: Machine learning and Deep Learning techniques are discussed which works as a catalyst to improve the  performance of any health monitor system such supervised machine learning algorithms, unsupervised machine learning algorithms, auto-encoder, convolutional neural network and restricted boltzmann machine.

For the research paper, read here.

Internet of Things with BIG DATA Analytics -A Survey

Author: A.Pavithra,  C.Anandhakumar and V.Nithin Meenashisundharam

Institute: Sree Saraswathi Thyagaraja College,

Abstract: This article we discuss about Big data on IoT and how it is interrelated to each other along with the necessity of implementing Big data with IoT and its benefits, job market

Research Methodology: Machine learning, Deep Learning, and Artificial Intelligence are key technologies that are used to provide value-added applications along with IoT and big data in addition to being used in a stand-alone mod.

For the research paper, read here.

Single Headed Attention RNN: Stop Thinking With Your Head 

Author: Stephen Merity

In this work of art, the Harvard grad author, Stephen “Smerity” Merity, investigated the current state of NLP, the models being used and other alternate approaches. In this process, he tears down the conventional methods from top to bottom, including etymology.

The author also voices the need for a Moore’s Law for machine learning that encourages a minicomputer future while also announcing his plans on rebuilding the codebase from the ground up both as an educational tool for others and as a strong platform for future work in academia and industry.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Authors: Mingxing Tan and Quoc V. Le 

In this work, the authors propose a compound scaling method that tells when to increase or decrease depth, height and resolution of a certain network.

Convolutional Neural Networks(CNNs) are at the heart of many machine vision applications. 

EfficientNets are believed to superpass state-of-the-art accuracy with up to 10x better efficiency (smaller and faster).

Deep Double Descent By OpenAI

Authors: Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal

In this paper, an attempt has been made to reconcile classical understanding and modern practice within a unified performance curve. 

The “double descent” curve overtakes the classic U-shaped bias-variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. 

The Lottery Ticket Hypothesis

Authors: Jonathan Frankle, Michael Carbin

Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. 

The authors find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, they introduce the "lottery ticket hypothesis:"

On The Measure Of Intelligence 

Authors: Francois Chollet

This work summarizes and critically assesses the definitions of intelligence and evaluation approaches, while making apparent the historical conceptions of intelligence that have implicitly guided them.

The author, also the creator of keras, introduces a formal definition of intelligence based on Algorithmic Information Theory and using this definition, he also proposes a set of guidelines for what a general AI benchmark should look like. 

Zero-Shot Word Sense Disambiguation Using Sense Definition Embeddings via IISc Bangalore & CMU

Authors: Sawan Kumar, Sharmistha Jat, Karan Saxena and Partha Talukdar

Word Sense Disambiguation (WSD) is a longstanding  but open problem in Natural Language Processing (NLP).  Current supervised WSD methods treat senses as discrete labels  and also resort to predicting the Most-Frequent-Sense (MFS) for words unseen  during training.

The researchers from IISc Bangalore in collaboration with Carnegie Mellon University propose  Extended WSD Incorporating Sense Embeddings (EWISE), a supervised model to perform WSD  by predicting over a continuous sense embedding space as opposed to a discrete label space.

Deep Equilibrium Models 

Authors: Shaojie Bai, J. Zico Kolter and Vladlen Koltun 

Motivated by the observation that the hidden layers of many existing deep sequence models converge towards some fixed point, the researchers at Carnegie Mellon University present a new approach to modeling sequential data through deep equilibrium model (DEQ) models. 

Using this approach, training and prediction in these networks require only constant memory, regardless of the effective “depth” of the network.

IMAGENET-Trained CNNs are Biased Towards Texture

Authors: Robert G, Patricia R, Claudio M, Matthias Bethge, Felix A. W and Wieland B

Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. The authors in this paper, evaluate CNNs and human observers on images with a texture-shape cue conflict. They show that ImageNet-trained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence.

A Geometric Perspective on Optimal Representations for Reinforcement Learning 

Authors: Marc G. B , Will D , Robert D , Adrien A T , Pablo S C , Nicolas Le R , Dale S, Tor L, Clare L

The authors propose a new perspective on representation learning in reinforcement learning

based on geometric properties of the space of value functions. This work shows that adversarial value functions exhibit interesting structure, and are good auxiliary tasks when learning a representation of an environment. The authors believe this work to open up the possibility of automatically generating auxiliary tasks in deep reinforcement learning.

Weight Agnostic Neural Networks 

Authors: Adam Gaier & David Ha

In this work, the authors explore whether neural network architectures alone, without learning any weight parameters, can encode solutions for a given task. In this paper, they propose a search method for neural network architectures that can already perform a task without any explicit weight training. 

Stand-Alone Self-Attention in Vision Models 

Authors: Prajit Ramachandran, Niki P, Ashish Vaswani,Irwan Bello Anselm Levskaya, Jonathon S

In this work, the Google researchers verified that content-based interactions can serve the vision models. The proposed stand-alone local self-attention layer achieves competitive predictive performance on ImageNet classification and COCO object detection tasks while requiring fewer parameters and floating-point operations than the corresponding convolution baselines. Results show that attention is especially effective in the later parts of the network. 

High-Fidelity Image Generation With Fewer Labels 

Authors: Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Z, Olivier B and Sylvain Gelly 

Modern-day models can produce high quality, close to reality when fed with a vast quantity of labelled data. To solve this large data dependency, researchers from Google released this work, to demonstrate how one can benefit from recent work on self- and semi-supervised learning to outperform the state of the art on both unsupervised ImageNet synthesis, as well as in the conditional setting.

The proposed approach is able to match the sample quality of the current state-of-the-art conditional model BigGAN on ImageNet using only 10% of the labels and outperform it using 20% of the labels.

ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations

Authors: Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin G, Piyush Sharma and Radu S

The authors present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT and to address the challenges posed by increasing model size and GPU/TPU memory limitations, longer training times, and unexpected model degradation

As a result, this proposed model establishes new state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks while having fewer parameters compared to BERT-large.

GauGANs-Semantic Image Synthesis with Spatially-Adaptive Normalization 

Author: Taesung Park, Ming-Yu Liu, Ting-Chun Wang and Jun-Yan Zhu

Nvidia in collaboration with UC Berkeley and MIT proposed a model which has a spatially-adaptive normalization layer for synthesizing photorealistic images given an input semantic layout.

This model retained visual fidelity and alignment with challenging input layouts while allowing the user to control both semantic and style.