Mentorship in the Sonic Arts Strikes a Chord at the Qualcomm Institute

Meanwhile, Burnett had also been drawn in by one of Yadegari’s classes, which had “the craziest title [they’d] ever seen.” This and Yadegari’s interdisciplinary “Crossing Boundaries” course represented a unique blend of technology, art and philosophy that Burnett had never seen before. With Burnett showing interest in pursuing and experimenting with the electronics behind music, Yadegari extended an offer to join his growing team.

As new members of the Sonic Arts group, Kestler, Burnett and Pisha worked on a software program designed to create realistic acoustics in virtual worlds. Called “Space3D,” the program manipulates frequency, amplitude and sound delay through headphones and speakers to trick our brain into believing that sound is traveling from far across a room, through a closed door or from a moving object.

“There’s been a huge amount of effort put into developing virtual graphics, but audio is a lot harder,” said Pisha. “And we get a lot of information about our surroundings from sound.” 

As Pisha learned graphics processing unit (GPU) programming to help design a new, dynamic algorithm for Space3D, Kestler focused on mapping a person’s head movement to object movement in VR. In October 2020, Yadegari, Burnett and Pisha, along with fellow Sonic Arts student Siddarth Atre, published the Space3D’s diffraction algorithm in the Journal of the Acoustical Society of America (JASA). Their paper made the front cover. 

A Journey in VR

That same fall, the Sonic Arts group gained a new undergraduate member with interests in music, technology and game design. Like Burnett, Pisha and Kestler, Eito Murakami found a unique niche for his passions in Sonic Arts. 

Murakami grew up training in piano. He moved to the San Francisco Bay Area from Japan when he was 14. In school, he befriended people who produced computer music and subsequently built his own DIY speakers for a school project. From then until his enrollment at UC San Diego as a political science and international relations student, he thought of music as a hobby. 

It wasn’t until he began taking music as an elective and enrolled in a virtual class with Yadegari that Murakami decided to pursue his first interest as a major. Once he joined Sonic Arts, Murakami learned Unreal Engine, which creates photorealistic landscapes for gaming, VR and research, under Burnett. 

The effort was building up to “Becoming,” a musical experience in VR directed by Yadegari and based on the work of the 13^th-century Persian poet Mowlana Rumi. Arising from a partnership at the 2019 Opera Hack, a 48-hour music theater hackathon in San Diego, “Becoming” was meant to adapt Rumi’s work on the evolution of the human spirit into an all-encompassing experience engaging the senses of sight, hearing and touch with Space3D and haptic (touch-based) technology. 

The Opera Hack team, consisting of Yadegari, Burnett and Pisha, with University of North Carolina at Chapel Hill Professor Emerita Francesca Talenti and McGill University Associate Professor of Electrical and Computer Engineering Jeremy Cooperstock, won $10,000 from the hackathon. As a new part of the “Becoming” team, Murakami worked on sound design while Burnett focused on the development of the play’s environment and fluid transitions between the shifts in the viewer’s surroundings.

The team members traveled to the SIGGRAPH 2022 Conference in Vancouver, Canada, in August 2022 to formally debut the complete piece. Features like haptic technology allowed visitors to interact with the piece, and have it respond in turn, making the human sense of touch seem cutting-edge. Burnett remarked that the team’s booth had attendees lining up to experience their work.

Taking it to the Next Step

Since graduating from Yadegari’s lab, Kestler, Pisha, Murakami, Burnett and other Sonic Arts alumni have leveraged the experience in their career paths.

Burnett spent the latter part of their time with Sonic Arts working on 3D videos and immersive music for performances. Simultaneously, they led the technical coordination of performances with QI’s Initiative for Digital Exploration of Arts and Sciences (IDEAS) series. Taken together, these experiences and the exposure to new and emerging technologies in music prepared them for a two-year contract as a software engineer with Meta Reality Labs Audio Research. In the future, they would like to continue creating immersive spaces at large scales.

“I’ve always been very inspired by the path [Yadegari] took, because I’ve always felt like a bit of an outlier in whatever program I’ve been in,” Burnett said. “[Yadegari] has shown me a way to [balance art and science] while being very encouraging of my work.” 

Kestler, inspired by building the base technology and algorithms for Sonic Arts initiatives, is now pursuing a Ph.D. in physics with a speciality in quantum computing at UC San Diego. To him, the work is a welcome continuation of the tool-building he did for Sonic Arts, but for quantum simulators and technologies. 

“[Yadegari] had the biggest effect on what I want to do,” said Kestler. “I can’t emphasize how crucial he has been in my career.” 

Murakami, likewise, enrolled in a master’s program at the Center for Computer Research in Music and Acoustics at Stanford University. In the years since, he has kept his ties with QI, most recently through a collaboration on the IDEAS VR performance “Pigments of Imagination.” 

Pisha, building on his experience learning and applying GPU programming to Space3D’s algorithm, soon secured two internships at artificial intelligence hardware and software provider NVIDIA. He now works as a senior GPU software engineer at the company Cruise Self-Driving Cars. 

For Yadegari, seeing the lab’s alumni grow has been “absolutely fantastic and inspiring.” 

“[Sonic Arts] creates space to innovate and solve problems with a more open mind,” he said. “We need innovative thinking, and the arts provide the context to practice and flex that muscle.  QI offers the space for students to learn in that world while thinking and practicing outside of the box.”

The success of the Sonic Arts lab bodes well for a new master’s program Yadegari is developing at QI. If the program receives the green light, prospective students can expect to apply for a formal degree in Master of Science in Audio Technologies in fall 2025.