How Chess Has Become The Test Bed For AI Researchers 

Efforts have been made over the past decade to teach artificial intelligence to beat humans at almost every game that involves strategy. All AI breakthroughs in previous benchmark games, be it checkers, chess, Go, 2 player poker, StarCraft 2 or Dota 2, AI was successful because it attempted to estimate widely popular strategy: Nash equilibrium.

Chess has long been a benchmark of progress in machine intelligence, from Alan Turing’s 1951 program for playing the game (written on paper) through Garry Kasparov’s defeat at the hands of IBM’s Deep Blue.


Sign up for your weekly dose of what's up in emerging technology.

Deep Blue is the chess machine that defeated then-reigning World Chess Champion Garry Kasparov in a six-game match in 1997. There were a number of factors that contributed to this success including:

  • a single-chip chess search engine
  • a massively parallel system with multiple levels of parallelism
  • a strong emphasis on search extensions
  • a complex evaluation function and
  • effective use of a Grandmaster game database

One less talked about aspect of this entire ‘machine beats man’ grand tale is that Deep Blue-I was defeated by Kasparov in 1996. The lessons learnt here helped Deep Blue- II to triumph in 1997. This achievement was a culmination of efforts spanning over decades of making machines to learn logic step by step. 

More recently, the Alphabet subsidiary DeepMind demonstrated a chess variant of AlphaGo, the program capable of teaching itself to play the ancient Chinese board game Go. 

Now, a team of researchers at University College London went a step ahead and developed a chess algorithm, called SentiMATE, which can evaluate the quality of chess moves by analysing the reaction of expert commentators. Yes. 

All one has to do is to make the algorithm to read the commentaries of top games of grandmasters. Winning moves and blunders obviously get a stark contrast reaction from the commentators and these highs and lows can be identified by the machine and use it to make a decision on whether to make a move or not.

Designing SentiMATE

For this experiment, the team analysed the text of 2,700 chess game commentaries available online. They pruned out commentary that didn’t relate to high-quality moves and examples that were too ambiguous. Then they used a special type of recurrent neural network and word embeddings, trained on another state-of-the-art model for analysing language.

Data from different chess websites was scraped, which included information regarding the moves being made, and a qualitative assessment of the moves themselves in the form of commentary, written by a wide range of chess players; resulting in a large database of moves with annotated commentary.

The data was cleaned so that only commentaries which were deemed to describe the ‘quality’ of the moves were employed, through the use of a preliminary ‘Quality vs Non- Quality’ classifier.

A novel Sentiment Analysis model was built, given that the nature of chess commentary is different to that of, for example, movie reviews, thus making sentiment classifiers such as that trained on the IMDb’s database unsuitable for this problem. 2,090 moves were annotated for their ‘sentiment’, and a bi-directional LSTM using stacked BERT and GloVe embeddings was trained on this commentary. The resulting dataset was then fed into novel evaluation function for chess moves trained on sentiment, which is tested against other chess engines.

The results of this research demonstrate that annotated moves and their respective sentiment is sufficient for the model to outperform a random player and a (restrained) depth-one DeepChess engine. Furthermore, the provided classification models exhibit high accuracy, making the expansion of the dataset both feasible and timely.

Future Direction

Strategic games like chess or GO are known for having a peek into the human intellect. So it is only obvious that if AI is modelled around these games, they only get better eventually being able to perform not so sophisticated jobs. For instance, NLP used for chess as discussed above can find itself get smarter at applications like conversational agents, chatbots, home pods, speech assistants, translation and many more.

More Great AIM Stories

Ram Sagar
I have a master's degree in Robotics and I write about machine learning advancements.

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

Conference, in-person (Bangalore)
Cypher 2023
20-22nd Sep, 2023

3 Ways to Join our Community

Whatsapp group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our newsletter

Get the latest updates from AIM