Research & Innovation

This Software Will Play Live Jazz — and Help Siri Understand You Better

Stevens researcher Kelland Thomas uses music as a window into artificial intelligence

Dean Kelland Thomas featured with a saxophone.
Kelland Thomas, dean of the College of Arts & Letters at Stevens Institute of Technology

Stevens Institute of Technology researcher Kelland Thomas is helping design the artificial intelligence of the future.

With a little help from Charlie Parker, Miles Davis and John Coltrane along the way.

Thomas, dean of Stevens' College of Arts & Letters — as well as a computer scientist and longtime professional musician — is heading a five-year project with collaborators from the University of Illinois, the University of Arizona and Southern Methodist University to teach computers to study tendencies in human creation, communicate with us, then anticipate and create something original in response.

"We're building information systems that can model human-computer communication and can improvise," explains Thomas. "The system has to be able to do inference based on what it's hearing, then update its 'mental model' and make a statement back, a decision, based on what it thinks is being said."

And the Department of Defense is listening.

Wanted: More creative, communicative computers

The project is known as the Musical Improvising Collaborative Agent, or MUSICA. Supported by a $2 million award from the U.S. Department of Defense's DARPA (Defense Advanced Research Projects Agency), Thomas' research team will analyze musical data at scale, feeding thousands of jazz 'phrases' into a transcription engine as well as hand-curating famed solos by some of history's most celebrated jazz players.

MUSICA's software will then mine patterns and themes from that database, create a database, make probabilistic models, then play new music in real time by predicting and reacting to live jazz musicians who are, themselves, improvising new music.

"This is classic AI, classic knowledge engineering," notes Thomas. "We're trying to achieve a recipe of statistical techniques to extract relevant patterns that inform the model."

While musically interesting, the research is much more than an artistic venture: if successful, it also has wide implications for defense, industry and an increasingly digital society, says Thomas.

"The system's ability to anticipate and create chord changes and new, never-tried melodic riffs will go far beyond current computers' ability," he explains.

That's important, because computing researchers worldwide are working to design and test new AI systems that not only accept human instructions and data sets but also proactively anticipate instructions — or even reach out with questions, concerns and suggestions. This concept, known as computational creativity, requires AI systems to build knowledge, learn on the fly and interact with people in ways that are cognitively similar to the ways we interact with them.

Creative computing would enable much more useful human interactions with devices, automobiles, homes and intelligent services such as Siri, Google Now, Amazon Echo and Cortana.

And the building of MUSICA, Thomas says, could help create a step in that direction.

The question of what constitutes creation is an interesting one," notes Thomas, "as we normally think of creation as a wholly human enterprise. However, if we're talking about something that appears to the best experts to be original and appropriate within the given context, it may be fair to say that a system has 'created' something novel."

Defense, medical and digital applications

The Department of Defense agrees, seeing wide applications for any successful technology that proves it can bridge that gap between human and computer.

Potential applications could include healthcare and medicine — intelligent software detecting patterns in medical data and communicating those in new ways to physicians, researchers and the public, for instance — as well as much more useful interactions with smartphones and intelligent services.

"It happens that we chose jazz as the language and the data for this investigation, but we are really addressing basic questions about building the next level of computing and artificial intelligence," he concludes. "In this age, that touches nearly every one of us in one way or another."

Thomas also previously collaborated with University of Arizona professor Bryan Carter on the Virtual Harlem Project, a 3D virtual environment replicating the golden era of jazz in a small section of that New York City neighborhood.