Article Complete

Alibaba open-sources KNAS, a low consumption AutoML algorithm

KNAS uses a gradient kernel as a proxy for model quality and consumes less computing resources compared to standard techniques.

sharathkumar.nair

Published 2022-02-14T05:30:00.000+05:30Updated 2022-02-14T05:30:00.000+05:30

Key takeaways

Alibaba and Peking University have open-sourced the Kernel Neural Architecture Search algorithm, aiming for a more sustainable AutoML solution.
KNAS uses a gradient kernel to evaluate model quality, significantly reducing computational resource consumption compared to traditional methods.
The algorithm selects the most accurate model from a few top candidates without the need for extensive training on each architecture.
Traditional neural architecture search methods require complete training of each model, leading to longer overall training times.

5 min

FollowPreferred Source Google News WhatsApp Telegram

Researchers at Alibaba Group and Peking University have open-sourced an efficient AutoML algorithm called Kernel Neural Architecture Search. The study looked for a green NAS (Neural Architecture Search) solution that evaluates architectures without training.

KNAS uses a gradient kernel as a proxy for model quality and consumes less computing resources compared to standard techniques. The team proposed the hypothesis: "Gradients can be used as a coarse-grained proxy of downstream training to evaluate randomly-initialized architectures." The researchers found a gradient kernel (the mean of the Gram matrix (MGM) of gradients) has a strong correlation with a model's accuracy. The KNAS algorithm computes the MGM for each proposed model architecture, keeping only the best few, calculating the model accuracy for those candidates, and selecting the model with the highest accuracy as the final result.

Usually, neural architecture search systems are used to find the best deep-learning model architecture for a task. The system does this by finding an architecture that is well-suited to deliver the best performance metric on the given task dataset and search space of possible architectures. But, this method demands training each proposed model completely on the dataset, resulting in longer training times.