Recently, Google Brain researchers have introduced a new way of programming automated machine learning (AutoML) based on symbolic programming. The researchers also proposed PyGlove, a general symbolic programming library for Python, to implement the symbolic formulation of AutoML.
AutoML, designed to fill the machine learning industry’s talent gap, is gaining traction among various organisations. Here, we discuss the challenges AutoML faces and how to resolve them using symbolic programming.
Why This Research
According to the researchers, existing machine learning software libraries are quite limited in handling the dynamic interactions among automated machine learning components. For instance, efficient neural architecture search (NAS) algorithms, such as DARTs, ENAS, demand an implementation coupling between the search algorithm and search space.
It is also challenging to implement a complex search flow, such as searching architectures within a loop of searching hardware configurations. The researchers stated, “We believe the main challenge lies in the programming paradigm mismatch between existing
software libraries and AutoML.
“Most existing libraries are built on the premise of immutable programs, where a fixed program is used to process different data. On the contrary, AutoML requires programs to be mutable, as they must be dynamically modified by another program whose job is exploring the search space,” they added.
Due to such inconsistency, predefined interfaces for search algorithms and search spaces struggle to accommodate the unanticipated interactions, making it challenging for AI researchers to try new AutoML approaches. In light of these limitations, the researchers have developed a new method to program automated ML.
Use of Symbolic Programming
Symbolic programming, originated from LISP, allows a program to manipulate its components as if they were plain data. The researchers reformulated AutoML as an automated process to manipulate ML programs symbolically.
Under the new formulation, programs act as mutable objects that can be cloned as well as modified. These mutable objects can be expressed as standard machine learning concepts where all parts of the machine learning program are mutable. Through symbolic programming, programs can be modified, resulting in dynamic interactions between the child program, search space, and search algorithm in an AutoML program.
How It Works
AutoML is an automated process of searching for a child program from a search space to maximise a reward. The researchers broke down the process into a sequence of symbolic operations. Meaning, a child program is turned into a symbolic child program. The symbolic program is further hyperified into a search space by replacing some of the fixed parts with to-be-determined specifications.
During the search, the search space materialises into different child programs based on search algorithm decisions. It can also be rewritten into a super-program to apply complex search algorithms such as efficient NAS (ENAS).
PyGlove is a general symbolic programming library on Python. Using this library, Python classes, as well as functions, can be made mutable through brief Python annotations, making it easier to write AutoML programs.
The library also allows AutoML techniques to be quickly dropped into preexisting machine learning pipelines while benefiting open-ended research which requires extreme flexibility. PyGlove implements various popular search algorithms, such as PPO, Regularised Evolution and Random Search.
- Reformulated AutoML under the symbolic programming paradigm. The researchers simplified the programming interface for automated machine learning by accommodating unanticipated interactions among the child programs, search spaces and search algorithms through a mutable object model.
- Introduced PyGlove, a general symbolic programming library for Python that implements the symbolic formulation of AutoML. With this library, automated ML can be quickly dropped into preexisting ML programs, with all program parts searchable, allowing rapid exploration on different dimensions of AutoML.
- Demonstrated the expressiveness of PyGlove in real-world search spaces. The research showed how PyGlove allows AutoML practitioners to change search spaces, search algorithms and search flows with minimum code.
Researchers said PyGlove library could reduce the engineering cost when exploring complex search flows.
They concluded, ‘We see a big potential in symbolic programming/PyGlove in making machine learning researchers more productive. Symbolic programming and PyGlove makes it easier to develop search-based programs which could be used in a broad spectrum of research and product areas.’
Read the paper here.