What’s Neural Population Learning?

NeuPL is an efficient general framework that learns and represents policies in symmetric zero-sum games within a single conditional network.

The need for diverse policies for strategy games like StarCraft and poker are addressed by growing a robust policy population by iteratively training new policies against the existing ones. However, the approach has two challenges: Firstly, under a limited budget, the best response operators need truncating, resulting in under-trained good responses. Secondly, repeated learning of basic skills is wasteful and becomes intractable against stronger opponents.

Now, DeepMind and University College London have developed Neural Population Learning (NeuPL) to solve both issues. The researchers discovered that NeuPL guarantees the best responses under mild assumptions by showcasing a single conditional model policy. Moreover, NeuPL helps in transfer learning across policies. The research showed NeuPL can improve performance across various test domains. Additionally, it helps us understand how novel strategies are more accessible when the neural population increases. 

THE BELAMY

Sign up for your weekly dose of what's up in emerging technology.

RTS and NeuPL

Classical game theory is crucial to learning population strategies. The study uses “rock-paper-scissors” as an example where two strategies (rock, paper) is obtainable. Meanwhile, a distinct population (scissors) can be defeated when both are set in opposition or revealed. It is also shown in Policy Space Response Oracle (PSRO). New policies are trained to respond to an amalgamation of policies with the help of a meta-strategy solver. Finally, a PSRO variation was used to overpower StarCraft in 2019.

In turn-based games, improving strategies means winning or losing. However, performance can sometimes be consequential when a population of pure strategies works against a single population. For example, picking strategies first can always beat a player going second in the meta-game.

There are two implications when the opponent cannot differentiate between good responses and globally optimal best responses. First, as a result, approximate best-response operators are often truncated prematurely as per hand-crafted schedules. Second, real-world games require strategy-agnostic transitive skills, prerequisites to strategic reasoning. However, studying these skills from zero against skilful opponents is difficult.

Using the computational infrastructure of simple self-play makes NeuPL an efficient and general framework that learns and represents policies in symmetric zero-sum games within a single conditional network. Most importantly, NeuPL allows transfer learning across policies that discover ways to overcome strong opponents that were previously inaccessible to comparable baselines.

When a game is completely transitive, every policy shares a similar best response policy. Self-play offers a natural curriculum that converges to this best response. Nevertheless, it is infeasible in real-world games since strategic cycles cannot be ruled out without an exhaustive policy search. In these games, NeuPL retains the ability to capture strategic cycles and falls back to self-play if the game appears transitive.

Previously, similar attempts were made to make population learning scalable. This study proposed pipeline PSRO (P2SRO), which learns iterative best responses simultaneously in a staggered, hierarchical way. It offered a principled way to use additional computation resources while retaining the convergence guarantee of PSRO.

Although it does not induce efficient learning computation per unit, it focuses on the lack of transfer learning across different domains with basic skills re-learned together. Instead, it proposed “Mixed-Oracles”, where knowledge acquired over previous iterations is accumulated via an ensemble of policies. Under this, each policy is trained to respond to a pure meta-game strategy rather than a mixture strategy suggested by the meta-strategy solver.

In comparison, NeuPL allows the transfer and optimises Bayes-optimal objectives head-on. The team also suggested an efficient, general and moral framework to learn and represent various policies in real-world games using a single conditional model. DeepMind and UCL claim the study is a step toward achieving scalable policy space. Furthermore, it seeks to go beyond the symmetric zero-sum setting as a possible direction for future research in this area.

More Great AIM Stories

Akashdeep Arul
Akashdeep Arul is a technology journalist who seeks to analyze the advancements and developments in technology that affect our everyday lives. His articles primarily focus upon the business, cultural, social and entertainment side of the technology sector.

Our Upcoming Events

Masterclass, Virtual
How to achieve real-time AI inference on your CPU
7th Jul

Masterclass, Virtual
How to power applications for the data-driven economy
20th Jul

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

Conference, Virtual
Deep Learning DevCon 2022
29th Oct

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM
MOST POPULAR

What can SEBI learn from casinos?

It is said that casino AI technology comes with superior risk management systems compared to traditional data analytics that regulators are currently using.

Will Tesla Make (it) in India?

Tesla has struggled with optimising their production because Musk has been intent on manufacturing all the car’s parts independent of other suppliers since 2017.