A Beginners Guide To Dopamine Reinforcement Learning Framework

Reinforcement learning algorithm, soon becoming the workhorse of machine learning is known for its act of rewarding and punishing an agent. This acts as a bridge between human behaviour and artificial intelligence, enabling leading researchers to work on artistic discoveries in this domain. The recent success of Deepmind’s AlphaGo in defeating the world champion at Go and OpenAI’s Dota 2 bots thrashing the game’s veteran players with just six months of training is a notable achievement in the area of Reinforcement Learning. This versatile research platform required an environment to test the new ideas and to play with the models in the mind of researchers. This is the reason why Dopamine was built and to enhance the work of individuals and teams passionate about reinforcement learning. Dopamine, the newest research framework released by Google, is geared at fast prototyping development of reinforcement learning algorithms. It provides that key missing piece for researchers, that is, benchmarking abilities with 60 different atari arcade games. Agents such as DQN, C51, Rainbow Agent and Implicit Quantile Network are the four-values based agents currently available. Some of the key features Google is focusing on are

• Easy experimentation: Making the environment more clarity and simplicity for better understanding.
• Flexible development: Experimenting new ideas easy implementation.
• Compact and reliable: Provide implementations for a few, battle-tested algorithms.
• Reproducible: Facilitate reproducibility in results.

Getting Started

We outline the steps which one can follow to install the dopamine framework on the virtual environment if required, else users can ignore it. With the available dependencies in the environment one can control the framework. Below are the steps to install the Dopamine framework on Python 2.7.

Setting Up the Virtual Environment

sudo apt-get update && sudo apt-get install virtualenv virtualenv --python=python2.7 dopamine-env source dopamine-env/bin/activate This will enable the dopamine-env virtual environment, where one can use the activate command to activate the environment. Later, install the required dependencies. sudo apt-get update && sudo apt-get install cmake zlib1g-dev pip install absl-py atari-py gin-config gym opencv-python tensorflow Also ignore the error message tensorflow 1.10.1 has requirement numpy<=1.14.5,>=1.13.3, but you'll have numpy 1.15.1 which is incompatible. Finally clone the Dopamine repository from GitHub, git clone https://github.com/google/dopamine.git

Test Run

One can test if the framework is working fine with this code, after cloning the repo cd dopamine export PYTHONPATH=${PYTHONPATH}:. python tests/atari_init_test.py To test the basic DQN agent, the python file train.py python -um dopamine.atari.train \ --agent_name=dqn \ --base_dir=/tmp/dopamine \ --gin_files='dopamine/agents/dqn/configs/dqn.gin' This will start an experiment with millions of frames. The output statistics can be visualised in the command line interface. Alternative Install - with Library sudo apt-get update && sudo apt-get install cmake pip install dopamine-rl pip install atari-py

Test Run

Once inside the root directory, the library can be tested with, python -um tests.agents.rainbow.rainbow_agent_test With this framework researchers can implement their models and test on various Atari games. This also provides the full training data in the format of Python pickles and JSON data files. Below is an example of the graph on seaquest with 4 agest as mentioned above, tested on one of the Atari 2600 game. The github repo of the Dopamine framework can be found here.

Conclusion

The goal of this framework is to provide easy-to-use models for researchers to play with. Also allowing the user community to work and understand the aspects of reinforcement learning. With this being provided as open source, we believe the research in reinforcement learning will accelerate in the near future.