UAE: Chinese scientist bands with robots to create music

A Chinese computer scientist and professional musician is redefining the level of human-computer interaction in the creation of music.

Prof. Gus G. Xia, a visiting professor at Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), is using machine learning to make robots improvise and contribute ideas to deliver new musical work. If a human drummer or a pianist is unwell to take part in a performance, Prof. Xia will save the day with his AI robot.

“I use interactive intelligent systems and technologies to extend human musical expression and capabilities,” said Prof. Xia, who is currently an assistant professor in computer science at NYU Shanghai. He received his PhD in the machine learning department at Carnegie Mellon University, US, where he studied machine learning and computer music.

Prof. Xia’s research work equips the automatic accompaniment system with machine learning, robotic embodiment and improvisation on folk, pop or popular songs or melodies. The existing system of automatic accompaniment is where the computer is fed with a predefined score and once a person plays a piece, the system delivers a performance complementing the dynamics and tempo.

Robot AI musicians to improvise

Prof. Xia has been doing research on human-computer interaction for the past 10 years or so.

“There are several levels of application. In the first level, the computer knows what I am going to play. In real time, I may speed up or slow down, the algorithm tracks my tempo, and delivers the complement. In the second level, when I am doing my expressive performance, the machine can learn from the rehearsal to better anticipate my next movement. In the third level, there is robot embodiment. The robot can have its own body gestures and facial expressions according to my musical expression and timings,” said Prof. Xia, director, the Music X Lab.

Currently, Prof. Xia is taking it to the next level by enabling the system to improvise, merge its ideas with the musical played by a human to produce a new composition.

“In the fourth level, that I am doing now, is improvisation. In this case, the robot doesn’t fully know what I am going to play. It’s not a predefined score. It’s a predefined lead sheet (a form of musical notation) only with a raw sketch of what I am going to do. I may improvise and play an existing piece or something new, and based on that material, the robot will improvise. It will merge some of its own ideas to my music and deliver a new thing. There are different layers, and I am building the fifth layer.”

System to assist, not replace, humans

Prof. Xia’s research work involves trying out the system on piano, saxophone and flute. He uses algorithms, which are rooted in the field of computer vision and computer graphics.

“We can use algorithms to analyse music, to say what genre it is, where the piece, the chords, from very low level to high level.”

By creating such a robotic system, Prof. Xia aims to assist humans and to perform with them.

“I don’t intend to substitute humans with my system. But the system could be part of a band.”

He is optimistic that the AI robot system can take a request from the audience and play a tune.

Couldn’t afford a pianist, created a robot

Prof. Xia started learning music when he was 3 years old, started playing the saxophone at 10 years and the Chinese flute when he was 12. He was the prime soloist of the Chinese Music Institute of Peking University, China, and played a solo concert in 2010. His inventions were part of the exploits and experiments done during his PhD days.

“One of the reasons why I created this robot to do a complement for me is because during my PhD studies I couldn’t afford a professional for my performance and decided to invent my own. I am planning to hold a theme concert in Dubai next month. The system hopefully will be part of it.”

In the coming months, Prof. Xia will be interacting with students and doing research work at MBZUAI in Abu Dhabi.

ALSO READ:

• UAE: MBZUAI ranked 30th globally in universities specialising in Artificial Intelligence

• UAE's MBZUAI, IBM to advance AI research with new centre of excellence