Creative Computing

Article Content

This story was published in the Spring 2023 issue of UC San Diego Magazine.

Walk into any jazz club and you’ll be blown away by the musicians’ skills and ability to riff off one another. The performances are dynamic, powerful and improvisational. The experience itself is multidimensional and uniquely human, a captivating conversation in music. But could computers, with their somewhat rigid programming, be capable of playing alongside humans in a creative exchange?

“Today, there are various machine learning applications to create art,” says Shlomo Dubnov, a UC San Diego professor with appointments in the departments of music and computer science. “But the question is, can we go beyond these tools and generate a sort of joint man-and-machine creativity?”

To answer this, Dubnov is working with researchers in Europe, including Gérard Assayag and Marc Chemillier, on Project REACH: Raising Co-creativity in Cyber-Human Musicianship and has already seen results. A human-computer team was able to recreate the improvisation style of the legendary Belgian jazz musician Toots Thielemans. Sifting through archived recordings, the team members used a new method of separating the musician’s playing of the chromatic harmonica from the rest of the band and then used that to train the computer in Toots’ unique style to generate a new improvisation.

The next step will be to get the computer to devise its own countermelodies from multiple options and then decide when and how often to interject into the larger group. To do this, Dubnov is developing a method to evaluate how much causal relationship exists between two improvising musicians. This would allow the machine to better tune in, listen and respond to its human musician counterpart, guiding the musical choices the computer makes.

“Not only should the human musician be interested in what the machine is doing, but the machine should also be interested in what the person is doing,” says Dubnov. “It needs to be able to analyze what’s happening and decide when it’s going to improvise with its human partners and when it’s going to improvise on its own. It needs agency and empowerment to make its own artistic choices in synergy with the human.”

UC San Diego Computer Science Professor Taylor Berg-Kirkpatrick and graduate students Ke Chen and Vignesh Gokul are also collaborators on this project. “The research goal for human-machine improvisation is interactive AI: we give feedback to the machine, the machine can receive it and change, and vice versa,” says Chen. “This enables the machine to function like an assistant rather than an independent artist.”

And while Dubnov and others continue to make advancements in artificial intelligence, these blurred lines between human and computer may expand our own creativity in areas we cannot yet imagine.