Yamaha develops an AI system that converts dance movements into piano music during the concert

Artificial intelligence (AI) can detect a dancer's movements in real-time by evaluating information from four different types of sensors attached to the dancer's body.

When seeing a skilled dancer perform, the music and movement may appear so in sync that the song appears to be magically flowing from the performer’s body. That was the case at a concert in Tokyo in November 2017. Titled “Mai Hi Ten Yu,” the concert was sponsored by the Tokyo University of the Arts and the Tokyo University of the Arts COI. It wasn’t, however, magic that made it happen. 

Kaiji Moriyama, a world-renowned dancer, showcased an artificial intelligence system that can convert dance movements into piano music during the concert. The “Dance to Piano Performance System,” developed by Yamaha, can detect movement in real-time utilising four types of sensors affixed to a dancer’s body. The AI then converts those movements into musical data and delivers it to a Yamaha Disklavier player piano in real-time.

MIDI (Musical Instrument Digital Interface) serves as a framework for converting movement into music. 

While the idea of utilising motion as an input to trigger MIDI notes and play music on virtual instruments isn’t new, Yamaha has managed to refine their approach to a point where it’s far superior to anything heard or seen before. MIDI is a digital standard for transmitting musical notes and other instrument setting information. MIDI is an acronym for Musical Instrument Digital Interface, which was created in the early 1980s to set a standard across firms for music production devices. A musical keyboard is the most basic and traditional method of MIDI input, but since the 1980s, creative people have been working on new ways to translate anything into new ways of playing music.


Sign up for your weekly dose of what's up in emerging technology.

How does the technology work?

The artificial intelligence (AI) used in the system, which is still in development, can detect a dancer’s movement in real-time by evaluating information from four different types of sensors attached to the dancer’s body. In addition, this system contains an original database that connects melody and movements. The AI on the system uses this database to generate appropriate melody data from the dancer’s movements in real-time. The MIDI data is then sent to a Yamaha DisklavierTM player piano and then converted into music. 

The Yamaha DisklavierTM is essential for converting dance motions into a musical expression. It can replicate a wide range of sounds with high precision with only minor variations in keyboard touch. Furthermore, at the concert, Yamaha employed a customised Disklavier that was configured based on Yamaha’s flagship model CFX concert grand piano to fully and thoroughly convey the talented dancer Moriyama’s performance.

Source: Yamaha

What’s next for Yamaha? 

Yamaha recently unveiled its first comprehensive AI Piano. The AI System from Yamaha is made up of a Disklavier player piano controlled by unique software that allows it to think for itself. Yamaha’s engineers first set the goal of having the system play any piece of music in the style of Glenn Gould to demonstrate the technology. The project was dubbed “Dear Glenn” with the help of the Glenn Gould Foundation. 

Studying the pianist’s performances was the first step. Over 100 hours of Gould’s recordings were analysed using deep learning technology (a mathematical modelling tool that employs neural networks for data processing). In addition, Yamaha studied the performances of numerous other pianists acknowledged to be exponents of Gould’s style. With all of the data gathered, the AI System could play practically any piece of piano music as if Gould himself performed it. 
Yamaha, on the other hand, did not stop there. They also used their proprietary AI Music Ensemble Technology (first seen back in 2016). As a result, the system was able to communicate with other musicians. This way, the AI pianist may be a “genuine” band member.

More Great AIM Stories

Abhishree Choudhary
Abhishree is a budding tech journalist with a UGD in Political Science. In her free time, Abhishree can be found watching French new wave classic films and playing with dogs.

Our Upcoming Events

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan, 2023

Conference, in-person (Bangalore)
Rising 2023 | Women in Tech Conference
16-17th Mar, 2023

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
27-28th Apr, 2023

Conference, in-person (Bangalore)
MachineCon 2023
23rd Jun, 2023

Conference, in-person (Bangalore)
Cypher 2023
20-22nd Sep, 2023

3 Ways to Join our Community

Whatsapp group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our newsletter

Get the latest updates from AIM