How Chess Has Become The Test Bed For AI Researchers 

Efforts have been made over the past decade to teach artificial intelligence to beat humans at almost every game that involves strategy. All AI breakthroughs in previous benchmark games, be it checkers, chess, Go, 2 player poker, StarCraft 2 or Dota 2, AI was successful because it attempted to estimate widely popular strategy: Nash equilibrium.

Chess has long been a benchmark of progress in machine intelligence, from Alan Turing’s 1951 program for playing the game (written on paper) through Garry Kasparov’s defeat at the hands of IBM’s Deep Blue.

Subscribe to our Newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Deep Blue is the chess machine that defeated then-reigning World Chess Champion Garry Kasparov in a six-game match in 1997. There were a number of factors that contributed to this success including:

  • a single-chip chess search engine
  • a massively parallel system with multiple levels of parallelism
  • a strong emphasis on search extensions
  • a complex evaluation function and
  • effective use of a Grandmaster game database

One less talked about aspect of this entire ‘machine beats man’ grand tale is that Deep Blue-I was defeated by Kasparov in 1996. The lessons learnt here helped Deep Blue- II to triumph in 1997. This achievement was a culmination of efforts spanning over decades of making machines to learn logic step by step. 

More recently, the Alphabet subsidiary DeepMind demonstrated a chess variant of AlphaGo, the program capable of teaching itself to play the ancient Chinese board game Go. 

Now, a team of researchers at University College London went a step ahead and developed a chess algorithm, called SentiMATE, which can evaluate the quality of chess moves by analysing the reaction of expert commentators. Yes. 

All one has to do is to make the algorithm to read the commentaries of top games of grandmasters. Winning moves and blunders obviously get a stark contrast reaction from the commentators and these highs and lows can be identified by the machine and use it to make a decision on whether to make a move or not.

Designing SentiMATE

For this experiment, the team analysed the text of 2,700 chess game commentaries available online. They pruned out commentary that didn’t relate to high-quality moves and examples that were too ambiguous. Then they used a special type of recurrent neural network and word embeddings, trained on another state-of-the-art model for analysing language.

Data from different chess websites was scraped, which included information regarding the moves being made, and a qualitative assessment of the moves themselves in the form of commentary, written by a wide range of chess players; resulting in a large database of moves with annotated commentary.

The data was cleaned so that only commentaries which were deemed to describe the ‘quality’ of the moves were employed, through the use of a preliminary ‘Quality vs Non- Quality’ classifier.

A novel Sentiment Analysis model was built, given that the nature of chess commentary is different to that of, for example, movie reviews, thus making sentiment classifiers such as that trained on the IMDb’s database unsuitable for this problem. 2,090 moves were annotated for their ‘sentiment’, and a bi-directional LSTM using stacked BERT and GloVe embeddings was trained on this commentary. The resulting dataset was then fed into novel evaluation function for chess moves trained on sentiment, which is tested against other chess engines.

The results of this research demonstrate that annotated moves and their respective sentiment is sufficient for the model to outperform a random player and a (restrained) depth-one DeepChess engine. Furthermore, the provided classification models exhibit high accuracy, making the expansion of the dataset both feasible and timely.

Future Direction

Strategic games like chess or GO are known for having a peek into the human intellect. So it is only obvious that if AI is modelled around these games, they only get better eventually being able to perform not so sophisticated jobs. For instance, NLP used for chess as discussed above can find itself get smarter at applications like conversational agents, chatbots, home pods, speech assistants, translation and many more.

Ram Sagar
I have a master's degree in Robotics and I write about machine learning advancements.

Download our Mobile App

MachineHack | AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIMResearch Pioneering advanced AI market research

With a decade of experience under our belt, we are transforming how businesses use AI & data-driven insights to succeed.

The Gold Standard for Recognizing Excellence in Data Science and Tech Workplaces

With Best Firm Certification, you can effortlessly delve into the minds of your employees, unveil invaluable perspectives, and gain distinguished acclaim for fostering an exceptional company culture.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox