Hands-on Guide To Creating RL Agents Using OpenAI Gym Retro

The goal of any Reinforcement learning agent is to maximize the cumulative rewards based on the goals for the provided environment. The learner is not told which actions to take but must discover which actions yield the most rewards by trying them. To develop such an agent, it is obligatory to allow the agents to experience a diverse set of environments so they can develop problem-solving skills that can be leveraged in different tasks.

The RL research community is keenly focused on using Games as an environment to train agents for a diverse set of tasks. Last year, DeepMind released OpenSpiel, with a purpose to promote the multi-agent RL systems across different games with an emphasis to learn and not to compete.

As any game is learned with experience, it provides the perfect training ground (environment) for agents whereas complex games also provide the learning ground for agents to solve hard tasks in real-life situations.

Subscribe to our Newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

In this article, we will familiarise ourselves with the two most popular Reinforcement learning software platforms for developing and comparing RL agents for games.

Overview Of OpenAI – Universe

The universe was released in 2016 and allows the agent to use the computer as a human does use a virtual keyboard and a mouse. The universe supports more than 1000 different tasks including games, browser tasks, and flash games.

Every environment in the Universe is packed as a docker container that hosts two different servers. A VNC(Virtual Network Computing) server to send the mouse and keyboard activities and a Websocket server to send the reward signals.

For the complete infrastructure understanding of the universe please refer OpenAI-Universe

Hands-on With OpenAI – Gym Retro

Gym Retro allows turning classic video games into Reinforcement learning environments. It comes with 1000+ games and supports adding various emulators which make adding new games as an environment fairly easy.

As of now, Gym Retro supports the following emulators.

  • Sega
  • Nintendo
  • NEC
  • Atari

However, it doesn’t come with preloaded ROM(game files) and we need to download the ROMs.

For more details on Gym Retro, kindly refer OpenAI – Gym Retro

The universe and Gym Retro are easy to program and need less than 10 lines of code to create an agent and test it our choice of games(environments). The predefined environments can simply be plugged in before creating an agent.

The below code snippet creates an agent which can play the Airstriker-Genesis game.

import retro

env = retro.make(game='Airstriker-Genesis', record='.') env.reset()

done = False

while not done:


obs, rew, done, info = env.step(env.action_space.sample())

if done:

obs = env.reset() env.close()

Here is one of the agent which just samples random moves from the state space with no prior training. The gameplay of the agent is displayed in the video below.

Better than the random agent is a rational agent that presses the buttons that do well in the game.

The rational agent can be said as a Brute force approach to the respective game and does well at lots of retro games. We have used a Brute force agent, the gameplay is displayed in the video below.

Both the platforms are based on OpenAI Gym, which is a toolkit for developing and comparing RL algorithms and was released in April 2016.

As OpenAI has deprecated the Universe, let’s focus on Retro Gym and understand some of the core features it has to offer.

Retro Gym provides python API, which makes it easy to interact and create an environment of choice.

Install Gym Retro

pip3 install gym-retro

Create Gym Environment

import retro

env = retro.make(game='Airstriker-Genesis', record='.')

That’s it you have created the learning ground for the agent.

Let’s use a random agent to play the Airstriker.

done = False

while not done:


obs, rew, done, info = env.step(env.action_space.sample())

if done:

obs = env.reset()


Congratulations, you have created an agent using OpenAI Gym Retro which can now play the game.

Importing ROMs

Game ROMs can be imported and added as an environment using the following command 

python3 -m retro.import /path/to/your/ROMs/directory/

Multiplayer Support

env = retro.make(game='Pong-Atari2600', players=2)

Both libraries support 1000+ game environments to train and compare RL algorithms. As most of the RL practitioners have to design the environments themselves to test the agents, Gym Retro ceases them from reinventing the wheel.

Even though the Real-life situations can go very complex, games can represent it very closely to practice and learn the Reinforcement Learning algorithms. Until now there are no such platforms for aspiring RL communities to quickly prototype and test the algorithms as we have for Machine Learning and Deep learning community.

OpenAI initiatives in this space will surely help the RL community to evolve and benchmark state of the art algorithms.

Anurag Upadhyaya
Experienced Data Scientist with a demonstrated history of working in Industrial IOT (IIOT), Industry 4.0, Power Systems and Manufacturing domain. I have experience in designing robust solutions for various clients using Machine Learning, Artificial Intelligence, and Deep Learning. I have been instrumental in developing end to end solutions from scratch and deploying them independently at scale.

Download our Mobile App

MachineHack | AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIMResearch Pioneering advanced AI market research

With a decade of experience under our belt, we are transforming how businesses use AI & data-driven insights to succeed.

The Gold Standard for Recognizing Excellence in Data Science and Tech Workplaces

With Best Firm Certification, you can effortlessly delve into the minds of your employees, unveil invaluable perspectives, and gain distinguished acclaim for fostering an exceptional company culture.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox