Yamaha develops an AI system that converts dance movements into piano music during the concert

When seeing a skilled dancer perform, the music and movement may appear so in sync that the song appears to be magically flowing from the performer's body. That was the case at a concert in Tokyo in November 2017. Titled "Mai Hi Ten Yu," the concert was sponsored by the Tokyo University of the Arts and the Tokyo University of the Arts COI. It wasn't, however, magic that made it happen. 

Kaiji Moriyama, a world-renowned dancer, showcased an artificial intelligence system that can convert dance movements into piano music during the concert. The "Dance to Piano Performance System," developed by Yamaha, can detect movement in real-time utilising four types of sensors affixed to a dancer's body. The AI then converts those movements into musical data and delivers it to a Yamaha Disklavier player piano in real-time.

MIDI (Musical Instrument Digital Interface) serves as a framework for converting movement into music. 

While the idea of utilising motion as an input to trigger MIDI notes and play music on virtual instruments isn't new, Yamaha has managed to refine their approach to a point where it's far superior to anything heard or seen before. MIDI is a digital standard for transmitting musical notes and other instrument setting information. MIDI is an acronym for Musical Instrument Digital Interface, which was created in the early 1980s to set a standard across firms for music production devices. A musical keyboard is the most basic and traditional method of MIDI input, but since the 1980s, creative people have been working on new ways to translate anything into new ways of playing music.

How does the technology work?

The artificial intelligence (AI) used in the system, which is still in development, can detect a dancer's movement in real-time by evaluating information from four different types of sensors attached to the dancer's body. In addition, this system contains an original database that connects melody and movements. The AI on the system uses this database to generate appropriate melody data from the dancer's movements in real-time. The MIDI data is then sent to a Yamaha Disklavier^TM player piano and then converted into music. 

The Yamaha Disklavier^TM is essential for converting dance motions into a musical expression. It can replicate a wide range of sounds with high precision with only minor variations in keyboard touch. Furthermore, at the concert, Yamaha employed a customised Disklavier that was configured based on Yamaha's flagship model CFX concert grand piano to fully and thoroughly convey the talented dancer Moriyama's performance.

Source: Yamaha

What's next for Yamaha? 

Yamaha recently unveiled its first comprehensive AI Piano. The AI System from Yamaha is made up of a Disklavier player piano controlled by unique software that allows it to think for itself. Yamaha's engineers first set the goal of having the system play any piece of music in the style of Glenn Gould to demonstrate the technology. The project was dubbed "Dear Glenn" with the help of the Glenn Gould Foundation. 

Studying the pianist's performances was the first step. Over 100 hours of Gould's recordings were analysed using deep learning technology (a mathematical modelling tool that employs neural networks for data processing). In addition, Yamaha studied the performances of numerous other pianists acknowledged to be exponents of Gould's style. With all of the data gathered, the AI System could play practically any piece of piano music as if Gould himself performed it. 
Yamaha, on the other hand, did not stop there. They also used their proprietary AI Music Ensemble Technology (first seen back in 2016). As a result, the system was able to communicate with other musicians. This way, the AI pianist may be a "genuine" band member.