Making Art with AI

Students learn to use computer algorithms to create artworks

Could a computer pick up where Mozart left off? Or could a machine make videos that look like pieces by Picasso? Students enrolled in UC San Diego’s first-ever Machine Learning for the Arts course used methods rooted in engineering to try to answer these types of questions and make art in the process.

Robert Twomey, a postdoctoral researcher in the Arthur C. Clarke Center for Human Imagination, taught the class, which is offered by the Department of Electrical and Computer Engineering. Twomey teaches students how to use tools commonly associated with artificial intelligence applications in computer science and engineering—to create and modify text, images, drawings, videos and more.

“I am an artist and an engineer exploring the intersection of human and machine perception,” Twomey said. He earned an MFA in Visual Arts at the University of California San Diego and a Ph.D. from the University of Washington in digital arts and experimental media. And he double-majored in art and biomedical engineering as an undergraduate at Yale. He brings this multi-disciplinary approach to the class.

Twomey uses technology both to explore new possibilities and to reveal or comment on its limits in his artwork. For example, he created and chatted with a computer simulation that mimicked his grandmother, who suffered from Alzheimer’s disease.

“This is a hot emerging field,” said Twomey. “But it’s pretty unique to be teaching this class in a STEM context.”

A handful of other institutions including Carnegie Mellon University and New York University offer similar classes, but these usually are geared toward art students.

Twomey’s goal in the class is to give students hands-on experience with contemporary machine learning techniques, and use them to create digital art. He also hopes the students will think critically about these tools and what they mean for our culture and our society.

On a recent Friday afternoon, Twomey asked students to gather in small groups and discuss the projects they had prepared for that week, which focused on using artificial intelligence tools to create visual pieces. The tools centered on neural networks, a type of computing system inspired by the way neurons are organized in the brain. These neural networks come in different flavors. For example, in generative adversarial networks, or GANs, two neural networks are trained on a data set and then compete to generate new data. Students also used Google’s Deep Dream, a visualization technique where neural networks create dream-like, surreal patterns within existing images and videos.

Megan Visaya, a senior majoring in electrical engineering, trained a GAN with images of players from the last 20 seasons of the National Hockey League and then got the neural network to generate new images of imaginary players. The project was a commentary on how people conceptualize hockey players and what they look like, Visaya said.

She decided to take the class after interning at Lab 41, a research lab run by the nonprofit venture capital firm In-Q-Tel, in Washington, D.C. The experience was thrilling and Visaya decided to focus on machine learning during her last year at UC San Diego. “It’s really different, to use engineering tools for art,” she said. She would like to work in data science after she graduates.

Ryan Bez, a senior computer engineering major, said he signed up for the class as an elective because it looked interesting. He struggled to put his visual art project together, as it required handling massive datasets that the tools he used could not always process effectively. “It’s nice to be able to produce something for a class that isn’t just an engineering project,” he said—something his friends who aren’t engineers can understand.

Watching engineering students stretch their abilities and create art projects is wonderful, Twomey said. He will be teaching the class again this fall. He hopes that it will become part of the regular curriculum.