Reinforcement learning is responsible for most of the breakthroughs in the field of emerging technology. Beating famous Go players, mastering chess and even poker sounded like conceptual ideas only a few years ago but with the advent of RN, they have been converted into reality. RL is now almost single-handedly mastering the gaming space.
Recently, researchers from Alphabet’s DeepMind introduced OpenSpiel, a reinforcement learning framework for games. It is basically a collection of environments and algorithms for research in general RL and search and planning in games.
OpenSpiel is a framework for writing games and algorithms and evaluating them on a variety of benchmark games. It contains an implementation of more than 20 different sorts of games like matrix and gridworld, among others. The implementation of games, core APIs, as well as the algorithms, are written in C++ and are wrapped in Python language.
Since the games have been written in C++, this makes the framework fast and memory-efficient for implementing basic algorithms. Currently, the framework is only tested on Linux and a subset of the library has been ported to Swift. In this framework, most of the learning algorithms written in Python use TensorFlow.
OpenSpiel supports n-player (single and multi-agent) zero-sum, cooperative and general-sum, one-shot and sequential, strictly turn-taking and simultaneous-move, perfect and imperfect information games, as well as traditional multiagent environments such as (partially and fully-observable) grid worlds and social dilemmas. It also includes tools to analyse learning dynamics and other common evaluation metrics.
There are several algorithms which are implemented within the OpenSpiel framework such as classical search algorithms which are minimax (and alpha-beta) search, and Monte Carlo tree search (MCTS). OpenSpiel includes some basic optimisation algorithms which are applied to games.
There are also three traditional single-agent reinforcement learning algorithms and they are Deep Q-Networks (DQN), Advantage Actor-Critic (A2C) and Ephemeral Value Adjustments (EVA). OpenSpiel utilises α-Rank algorithm as a tool for visualisation and evaluation. α-Rank is an algorithm which leverages evolutionary game theory to rank AI agents interacting in multiplayer games.
This framework includes two major design criteria, they are mentioned below:
- Simplicity: This framework provides codes which are readable, usable, extendable by non-experts in the programming languages and especially to researchers from potentially different fields. It provides reference implementations which are used to learn from and prototype with, rather than fully-optimised or high-performance code that would require additional assumptions or advanced language features
- Avoiding Dependencies: The OpenSpiel framework does not introduce dependencies as according to the researchers, dependencies can be problematic for long-term compatibility, maintenance, and easy to use.
Purpose of This Framework
Researchers have stated that the purpose of OpenSpiel is to promote general multiagent RL across different game types. This framework stresses heavy emphasis on learning. The researchers said, “We hope that OpenSpiel could have a similar effect on general RL in games as the Atari Learning Environment has had on single-agent RL.”
Below are the commands following which one can clone the repository and build OpenSpiel on Debian or Ubuntu Linux.
sudo apt-get install git
git clone https://github.com/deepmind/open_spiel.git
./install.sh # Install various dependencies
pip3 install –upgrade -r requirements.txt # Install Python dependencies
With the increasing demand for RL in the field of emerging technologies, DeepMind is not the only player for open-sourcing RL algorithms. Earlier, chip manufacturer Intel had also released RL Coach, an open-source framework for training and evaluating reinforcement learning agents.