Capturing Creativity with Computation for Music AI

Article Content

We know it when we see it, but what is creativity and can it be quantified? In a paper that could help guide future artificial intelligence (AI) development, a team from UC San Diego’s Jacob School of Engineering, Department of Music and Qualcomm Institute (QI) has discovered answers in the context of musical collaboration.

“It’s a hard problem, because not a lot of people agree on what is creative,” said study co-author Vignesh Gokul, a computer science and engineering Ph.D. candidate at UC San Diego. “Our main hypothesis was that the musical score or output that is most creative is the one that conveys the most information. The contribution of this paper is a method to calculate this total information flow between a human and an agent (or another human) playing music.”

The paper’s senior author Shlomo Dubnov, who is professor in both UC San Diego’s Music Department and Computer Science and Engineering Department as well as a QI affiliate, added, “This is a new concept that highlights the importance of communication and collaboration that occurs between musicians or between musicians and musical artificial intelligence agents as a fundamental factor in achieving music creativity.”

The paper, “Evaluating Co-Creativity Using Total Information Flow,” is authored by Gokul, Dubnov and computer science and engineering master’s student Chris Francis, who spearheaded experimental infrastructure and operation. The paper is being presented at this week’s Mathemusical Encounters in Singapore : a Diderot Legacy conference.

The project was funded by Project REACH : Raising Co-creativity in Cyber-Human Musicianship, a European Research Council Advanced grant that promotes the study of "shared musicality" at the intersection of the physical, human and digital spheres; it is designed to produce models and tools to better understand and encourage human creativity in a context where it is increasingly intertwined with computation.

Generating a Creativity Score

In this research, the UC San Diego team decided to evaluate musical co-creativity by using an pre-trained large language neural network model called Multitrack Music Transformer to estimate the amount of interaction between different musical voices in tracks containing a piano melody and its piano accompaniment.

The team derived an equation to effectively compute the information flow between the two signals in this musical interaction system, as a quantitative score. According to the hypothesis, the higher the score, the greater the creativity. If one of the musical voices in a pair ignored or repeated the other, the score would go down because little information would be exchanged. If, however, the voices went back and forth integrating each other’s musical information with their own, the score would rise.

Next, the team looked for a way to check that this score was a meaningful measure of creativity.

“The hardest part was definitely validating this framework,” Gokul said. “We had come up with the hypothesis; we had come up with a method to compute the score based on the hypothesis. But validating is hard because humans have different subjective preferences.”

The team decided to lean into human subjectivity and compare the computed score with five expert musicians’ evaluations of 84 musical duets, including some that were randomly generated. The results demonstrated that the computational score matched human perception.

Gokul cautioned there is still work to be done. He notes there’s a tendency for pre-trained models to prefer their own generations, and the team found that to be the case in this work with the Multitrack Music Transformer model. And, Gokul notes, this is only one step toward a musical system in which humans and AI can interact creatively.

However, the researchers are already planning future applications of the current work, including a collaboration with neuroscience researchers who investigate the cognitive load of listening to music or performing musical tasks. 

As Gokul transitions from his Ph.D. studies to a job in the tech sector, he is grateful for all the human creative collaboration he experienced at UC San Diego, including with Dubnov. “He has been a fantastic advisor,” Gokul said. “He’s very knowledgeable in information theory, music, generative AI, and we have published about seven works together. It's been great to receive his guidance on these projects.”

Read more about UC San Diego’s Qualcomm Institute, Department of Music or Department of Computer Science and Engineering.

Learn more about research and education at UC San Diego in: Artificial Intelligence