Top Works In Neural Architecture Search

Currently employed neural network architectures have mostly been developed manually by human experts, which is a time-consuming and error-prone process. This is when Neural architecture search, a subset of AutoML, came to the rescue. Neural Architecture Search (NAS) is the process of automating architecture engineering.


Here we list top research works in Neural Architecture Search based on their popularity on Github. These works have set new baselines, resulted in new networks and more.

This work proposes Efficient Neural Architecture Search (ENAS), a fast and inexpensive approach for automatic model design. In this approach, a controller (RNN), trained with policy gradient, learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. The authors state that ENAS is fast, delivers strong empirical performances using much fewer GPU-hours than all existing automatic model design approaches, and notably, 1000x less expensive than standard Neural Architecture Search. 

AIM Daily XO

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Link to the paper.

The Evolved Transformer

In this work, the researchers from Google Brain attempt to apply NAS to search for a better alternative to the Transformer. They first constructed a large search space and then ran an evolutionary architecture search by seeding the initial population with the Transformer. The results showed that the architecture — the Evolved Transformer — demonstrated consistent improvement over the Transformer on four well-established language tasks.

Download our Mobile App

Link to the paper.


In this work, the authors state to have achieved substantial improvements in the latency-accuracy trade-off by incorporating regular convolutions in the search space and effectively placing them in the network via neural architecture search. This work resulted in a family of object detection models, MobileDets, that achieve state-of-the-art results across mobile accelerators.

Link to the paper

In this work, the authors propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. This approach uses a sequential model-based optimisation (SMBO) strategy. Results show that this method is up to 5 times more efficient than the popular RL methods in terms of the number of models evaluated, and 8 times faster in terms of total compute.

Link to the paper.


This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Instead of applying evolution or reinforcement learning over a discrete and non-differentiable search space, this method is based on the continuous relaxation of the architecture representation, allowing an efficient search of the architecture using gradient descent. Experimental results show that this algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modelling.

Link to the paper.


With MorphNet, the researchers aim to automate the design of neural network structures. MorphNet iteratively shrinks and expands a network. In contrast to previous approaches, MorphNet is scalable to large networks, adaptable to specific resource constraints and capable of increasing the network’s performance. When applied to standard network architectures on a wide variety of datasets, this approach discovers novel structures in each domain, obtaining higher performance.

Link to the paper.

Neural Architecture Search Without Labels

This paper tries to explore the most pressing question: can we do away with human annotations of images? Can we find high-quality neural architectures using only images? To answer this, the researchers first defined a new setup called Unsupervised Neural Architecture Search (UnNAS). They then trained a large number of diverse architectures with either supervised or unsupervised objectives and found that the architecture rankings produced with and without labels are highly correlated. The results reveal that labels are not necessary, and the image statistics alone may be sufficient to identify good neural architectures.

Link to the paper.


This paper states that convolutional neural networks do not perform well for tasks requiring simultaneous recognition and localisation. The encoder-decoder architectures are proposed to resolve this by applying a decoder network onto a backbone model designed for classification tasks. In this work, the authors argue encoder-decoder architecture is ineffective. So, they propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search. Using similar building blocks, SpineNet models outperform ResNet-FPN models.

Link to the paper.

Randomly Wired Neural Networks

The researchers at Facebook explored a more diverse set of connectivity patterns with Randomly Wired Neural Networks for Image Recognition. Randomly wired networks are founded on random graph models in graph theory.

According to the authors, randomly wired neural networks managed to outperform human-designed networks such as ResNet and ShuffleNet with a typical computation budget and can go toe to toe with the increased computational regime. 

Read more about this here.

To keep yourself up to date with the latest developments in NAS, check this.

Sign up for The Deep Learning Podcast

by Vijayalakshmi Anandan

The Deep Learning Curve is a technology-based podcast hosted by Vijayalakshmi Anandan - Video Presenter and Podcaster at Analytics India Magazine. This podcast is the narrator's journey of curiosity and discovery in the world of technology.

Ram Sagar
I have a master's degree in Robotics and I write about machine learning advancements.

Our Upcoming Events

24th Mar, 2023 | Webinar
Women-in-Tech: Are you ready for the Techade

27-28th Apr, 2023 I Bangalore
Data Engineering Summit (DES) 2023

23 Jun, 2023 | Bangalore
MachineCon India 2023 [AI100 Awards]

21 Jul, 2023 | New York
MachineCon USA 2023 [AI100 Awards]

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

Is Foxconn Conning India?

Most recently, Foxconn found itself embroiled in controversy when both Telangana and Karnataka governments simultaneously claimed Foxconn to have signed up for big investments in their respective states