Now Reading
Guide to Scalable and Robust Bayesian Optimization with Dragonfly

Guide to Scalable and Robust Bayesian Optimization with Dragonfly

Aishwarya Verma
Dragonfly Bayesian Optimization

Dragonfly, an open-source python framework for scalable and robust Bayesian optimization, is developed by researchers from Carnegie Mellon University, Pittsburgh : Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabas Poczos, Eric P. Xing. The paper was submitted to Journal of Machine Learning Research in April 2020 titled “ Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly.

Machine Learning tasks are cast as black-box problems, where we have given some input and black-box does some processing and outputs the value for the given input. To enhance the performance of such a black-box system, we tend to optimize it to its optimum. An example of it is Hyperparameter Tuning where we search for the best parameters to find the best model for a given task. According to domain knowledge, this searching is often done by grid search or random search or manual settings(according to the domain knowledge). Formally, we define a black box as a function f : X → R over some domain X and try to find the optimum value of f via evaluating it repeatedly. Each evaluation, in this case, is expensive and requires huge computation. That’s when Bayesian Optimization(BO) comes into the picture. BO maximises the function f with a few evaluations by calculating the next point’s evaluation via previous observation. In probability terms, in order to determine the next evaluation, given the current evaluations as a point to start for the next evaluation, BO uses posterior. BO also showed some promising results in some hyperparameters tasks. Optimization tasks in today’s applications are posed to new threats that are difficult to tackle by traditional approaches. Dragonfly is primarily focused on scalable Bayesian Optimization and its robustness.

 • Scalability: This includes handling higher dimensional domains, multi-fidelity evaluations, optimising over neural network architectures, handling parallel evaluations. Apart from this, Dragonfly also contains some evolutionary algorithms that are capable of optimizing complex domains.

 • Robustness: Includes a set of values for a set of acquisition choices and model parameters instead of one for the entire optimization process. Hence, lowering the computational cost and adding robustness to the optimization system.

Requirements & Installation

Dragonfly is supported by Python 2(>=2.7) and Python 3(>=3.6) as well. It is fully compatible with Mac, Linux & Windows: You can install dragonfly via PyPI.

pip install dragonfly-opt -v

To check the installation of Dragonfly by following code :

 #installation testing
 from dragonfly import minimise_function
 # The first argument below is the function, the second is the domain, and the third is the budget.
 min_val, min_pt, history = minimise_function(lambda x: x ** 4 - x**2 + 0.1 * x, [[-10, 10]], 10);  
 print(min_val, min_pt) 

Quick Start with Dragonfly

Dragonfly is supported by command line, python and ask-tell mode. This guide will contain all of the three approaches to dragonfly.

1. Dragonfly with Command line

Since, we can run command line on Colab Notebook, this demo contains the code for colab command line. Dragonfly can be used explicitly by calling dragonfly-script.py or by importing python code through maximize_function in the main library. Examples are given here in this directory. If you have installed dragonfly via pip, you have to clone the git repository to access the examples.

To use dragonfly through the command line, specify the optimization problem and the optimization parameters(can be specified in JSON (recommended) or protocol buffer format). You can check  examples/synthetic/branin/config.json and examples/synthetic/branin/config.pb for examples. Then, specify the optimisation parameters in an options file, in the format shown in examples/options_example.txt

Basic Usage : The example shown below is the Branin benchmark for global optimization.

 %cd examples
 !python /content/dragonfly/bin/dragonfly-script.py --config synthetic/branin/config.json --options options_files/options_example.txt
 !python /content/dragonfly/bin/dragonfly-script.py --config synthetic/branin/config.pb --options options_files/options_example.txt 

Minimization : By default setting, dragonfly is set to maximize the function. But we can change that setting by usingthe  max_or_min flag. You can check the example here.

!python dragonfly-script.py --config synthetic/branin/config.json --optionsoptions_files/options_example_for_minimisation.txt

Multi-fidelity Optimisation : Multi fidelity version of Branin benchmark is shown below

 #multi-fidelity optmization
 !python dragonfly-script.py --config synthetic/branin/config_mf.json --options options_files/options_example.txt 

You can find all types of optimization that can be run through cli here.

2. Dragonfly with Python Code

API’s for dragonfly are initialized in dragonfly/__init__.py and are defined in the dragonfly/apis folder. For example of using dragonfly API in python:

 from dragonfly import minimise_function, maximise_function
 func = lambda x: x ** 4 - x**2 + 0.1 * x
 domain = [[-10, 10]]
 max_capital = 100
 min_val, min_pt, history = minimise_function(func, domain, max_capital)
 print(min_val, min_pt)
 max_val, max_pt, history = maximise_function(lambda x: -func(x), domain, max_capital)
 print(max_val, max_pt) 

Here,

func : It is the optimization function which is to be maximized.

domain : Over which the above func needs to be maximized.

max_capital : capital available for optimization(max. Number of available functions depends on cost or other constraints).

maximise_function : Returns the history with optimum value and the optimum point.

history.query_points : Represents the points evaluated by algorithm

history.query_vals : It contains the function values.

You can check other examples as well :

Similarly, you call use multiple objective optimization API. The basic structure of it is shown below : 

 from dragonfly import multiobjective_maximise_functions
 pareto_max_values, pareto_points, history = multiobjective_maximise_functions(funcs, domain, max_capital) 

where,

funcs :  list of functions to be maximised, 

pareto_values :  Pareto optimal function values 

See Also

pareto_points :  The corresponding points in domain. 

You can check multi-objective branin example and multi-objective hartmann example.

You can check all other optimization available in dragonfly for python here.

3. Dragonfly with Ask-tell

It  enables step-by-step optimization by calculating the next point to be evaluated in a loop of BO.

Two main components of this interface are: 

  1. A function caller 
    1.  EuclideanFunctionCaller : Used when domain is limited to euclidean spaces
    2. CPFunctionCaller : else use this caller
  2. An optimizer.  Here, <domain> is replaced by Euclidean or CP(depending on the domain).
    1. <domain>GPBandit
    2. <domain>GAOptimiser
    3. <domain>RandomOptimiser

An example of it is shown below:

 from dragonfly.exd import domains
 from dragonfly.exd.experiment_caller import CPFunctionCaller, EuclideanFunctionCaller
 from dragonfly.opt import random_optimiser, cp_ga_optimiser, gp_bandit
 max_capital = 100
 objective = lambda x: x[0] ** 4 - x[0]**2 + 0.1 * x[0]
 domain = domains.EuclideanDomain([[-10, 10]])
 func_caller = EuclideanFunctionCaller(None, domain)
 opt = gp_bandit.EuclideanGPBandit(func_caller, ask_tell_mode=True)
 #call initialise() on the created optimizer to begin using the interface
 opt.initialise()
 for i in range(max_capital):
     #ask returns the next point to be evaluated in a numpy array.   
     x = opt.ask()
     y = objective(x)
     print("x:", x, ", y:", y)
     #Once the point is evaluated by calling the objective function,
     #tell the point back to the optimiser
     opt.tell([(x, y)]) 

The above example represents the maximization problem. For minimization, take the negative of the objective function and rest of the procedure is same.

Check all the details here.

Conclusion

In this post, we have given an overview of Dragonfly, an optimized framework for  Bayesian Optimization that is scalable and robust as well. This framework is built keeping in mind the optimization of expensive optimization functions. It outperformed many existing systems available for BO.

Colab Notebook Dragonfly Demo

Official codes, docs & Tutorial are available at : 

What Do You Think?

Join Our Telegram Group. Be part of an engaging online community. Join Here.

Subscribe to our Newsletter

Get the latest updates and relevant offers by sharing your email.

Copyright Analytics India Magazine Pvt Ltd

Scroll To Top