While we all hum tunes randomly when we are nostalgic but have we ever thought that, what if the hummed tunes automatically gets converted to a beautiful musical piece. Google‘s new machine learning algorithm experiment has created a new tool — Tone Transfer, that makes this possible.
The workflow is quite simple — one has to go to the Tone Transfer site from any Android phone or desktop (Windows or Mac), select “Add your own,” to record your 15-second hum or tune. One can either use their voice, make a sound by tapping surfaces or actually play an instrument. Google’s machine learning algorithm will convert that captured 15 seconds tune into a digital signal, which will then convert it into a tune with Flute, Saxophone, Violin, or Trumpet.
While this sounds all dreamy, the quality of output is actually dependent on the user’s mic and background noise while recording. So, one might have to do a few tries for them to get a good recording. Nevertheless, it is indeed a fun tool and will get users hooked on to trying out new recordings.
According to the news, Google’s Magenta AI team, which builds open source technologies to explore the use of machine learning in art, built a ‘Differentiable Digital Signal Processing (DDSP)’ library to make this tool possible. As explained in this blog post, the key idea is to leverage simple interpretable DSP elements to create complex, realistic signals by accurately controlling their parameters. However, it is difficult to control all of these parameters dynamically, and that’s why synthesisers with simple controls often sound unnatural and “synthetic”.
Further, with DDSP, the company uses a neural network to convert a user’s input into complicated DSP controls that can produce more realistic signals. This input could be any form of the control signal, including features extracted from the audio itself. Since the DDSP units are differentiable, the team could then train the neural network to adapt to a dataset through standard backpropagation.
According to the company, with this sophisticated DDSP library, Google’s team can train audio synthesis models with fewer parameters and less data. These models can help tools such as Tone Transfer create high-quality audio from user input.
Check out the tool here.