Whistle a jaunty tune, and artificial intelligence can convert it into the style of a Mozart symphony played by an entire orchestra. Facebook has built an AI that can translate any style of music played on any instrument into a Mozart symphony, a Haydn string quartet, or even a Bach cantata with a full orchestra and chorus.
The system consists of two main parts: an encoder, which extracts the features of the input music, and a decoder, which makes those features match whatever style of music it鈥檚 being translated to.
Advertisement
The encoder doesn鈥檛 have a perfect representation of the input. Instead, it randomly changes some of the tones so that the decoder cannot simply regurgitate the exact notes back to you, only with different instruments.
鈥淚t makes it speak a more global, universal language of music instead of just recreating the notes,鈥� says says at Facebook.
[video_player id=鈥滾qsUwTiB鈥� access_level=鈥漵ubscriber鈥漖
The decoder was trained on six different styles of classical music: Mozart鈥檚 symphonies, Haydn鈥檚 string quartets, Beethoven鈥檚 piano sonatas, Bach鈥檚 string cantatas for orchestra, chorus, and soloists, Bach鈥檚 organ works, and Bach鈥檚 keyboard works as played on a harpsichord.
The team tested their network by having it convert segments of music 鈥� some Bach, some Mozart, and some swing jazz, metal guitar riffs, and instrumental Chinese music 鈥� into piano sonatas. They then employed three human musicians to do the same task. The two conversions were then scored out of five by 40 people recruited online.
For now, humans are still better. The human conversions got average ratings above a 4, while the network鈥檚 scores were closer to 3. However, in tests where the researchers presented six clips of music 鈥� one original and five translations to other styles 鈥� to freelancers and musicians, almost everyone had a hard time identifying the original.
The network isn鈥檛 like other music synthesisers, says Google software engineer Adam Roberts. 聽鈥淲hat鈥檚 new here is being able to do the modification end-to-end directly from the audio鈥� without converting the input into individual musical notes first, he says. You could sing directly into a microphone and the system will spit classical music right back out.
Journal Reference:
