Throughout its storied history as the world’s most famous research organization, Bell Labs has transformed businesses and enriched the lives of people around the globe through the scientific inventions and ingenuity of its researchers. Less known perhaps is the company’s pioneering contributions in the arts.
In the 1960s, Bell Labs collaborated with New York City artists and composers, such as dancer Merce Cunningham and composer John Cage, to create new works in music, theater and the media arts as part of Bell Labs researchers’ pioneering exploration of the intersection of technology and art at the dawn of the new digital era, which became known as “E.A.T.” (Experiments in Arts and Technology).
Now, as networks increasingly form the digital fabric that underpins everything we do, Bell Labs reached out to Stevens professor Rob Harari to collaborate in Bell Labs’ revitalized E.A.T. program. Through its collaborative research effort with Harari and with Stevens, Nokia Bell Labs is following in the company’s long and distinguished tradition in the creation and production of the arts with a new area of research that fuses human movement with media and digital art.
Harari, industry associate professor in the Music and Technology program at Stevens, was tapped by Paul Albin Wilford, senior director, Audiovisual Technologies Research, Nokia Bell Labs, to collaborate on the new project in November 2015, when Wilford showed him a slide of the conceptual representation for the Human Digital Orchestra™* and asked, “Can you do anything with this?”
The product of that collaboration was presented to an audience of roughly 500 invited guests at the sprawling industrial research campus of Nokia Bell Labs in Murray Hill, New Jersey, on April 28 with the world premiere of the Human Digital Orchestra as part of a two-day celebration — the Bell Labs Shannon Conference on the Future of the Information Age — to mark the 100th birthday of scientist Claude Shannon, widely regarded as the father of information theory and digital computing.
“As researchers, we are always trying to understand the human side of a problem,” Wilford said. “Claude Shannon understood this very well. With the collaboration between Bell Labs and Stevens Institute of Technology, we embraced this sentiment from Shannon and sought to express his work in ways that a wide audience could appreciate. By interweaving arts and science, we get a deeper and richer insight into the problems we should be working on that can have a wide impact on the way we live.”
Interactive technology driven by the audience
Stevens’ involvement with the Human Digital Orchestra follows a 2014 collaboration between Nokia Bell Labs and Stevens that culminated in a research display “Butterflies Alight!” at the W Hotel in Hoboken.
Harari would go on to serve as artistic director, producer and master of ceremonies for the performance, titled “The Shannon Effect,” which tells the story of Shannon, a larger-than-life figure whose influence on the modern world is felt on everything from cell phones to high-definition television.
“Every element of the performance was designed with a Shannon metaphor in mind — Shannon the conductor of new ways to communicate; Shannon the builder, inquisitive of structural design through a whimsical lens; Shannon the problem-solver through invention,” he said.
In the five months following the meeting with Wilford, Harari worked with Bell Labs leaders to meld the artistic vision with a compelling script, and to bring together performers, technologists and the involvement of WorldStage, an audio, video, lighting and event staging company that also produces Broadway and international shows.
Together with Wilford’s team, Harari designed a performance orchestrated by the movements of both the performers and the audience and inspired by the immersive Bell Labs facility referred to as “The Anomaly,” which resides on the Nokia Bell Labs Murray Hill campus.
“By designing the venue using a white tent, we could project everywhere to visually show what the Human Digital Orchestra does by having the performers wear bracelet accelerometers to measure and transmit motion data from the tap dancer and conductor. In conjunction with video analytics used to trace-map the pianist’s hands, the data streams were processed and then generated into command codes to control lighting, audio spatialization and computer graphics,” Harari said.
A multi-faceted collaboration
Performers included world-renowned artists such as tap dancer Andrew J. Nemr and jazz musician Dan Levinson. Two of the performers were Harari’s Stevens colleagues at the College of Arts and Letters (CAL): conductor and composer Andy Brick, director of the music and technology program, and world-renowned concert pianist Aysegul Durakoglu, an associate professor at CAL.
Students from CAL were also involved in the production of “The Shannon Effect.” Music and technology majors and current Stevens graduate students Brian Voyer ’16 and David Estes-Smargiassi ’16, both interns at Nokia Bell Labs, co-composed. Harry Patterson, Class of 2018, also a music and technology major, created the conceptual graphics for the staging. In addition, Julian Chaves ’15, now a member of the WorldStage crew, contributed as a projection designer and systems engineer.
The Bell Labs research team led by Wilford and Susanne Arney worked non-stop for five months creating the algorithms that provide the functionality of the Human Digital Orchestra. The team grew to double digits as the exploration of Shannon’s persona kept providing new story lines to display technologically.
Part of the story being told, noted Harari, was the complexity of staging a live show.
“The audience is exposed to all that is usually hidden in live entertainment. We created and lit a satellite stage for all the technologists that were running the sound, lighting, projection, computers and so forth. There were spotlights on those guys at different points in the performance, because what they were doing was just as intricate and intense and impactful to the audience as what we were doing on the stage.”
Audience members were not only spectators, but also participants in the orchestra. Through an app downloaded onto their cell phones, their collective behavior had a direct impact on the music being heard as data from their movements was transmitted into a wireless stream, then blended algorithmically into the music as it played.
The successful staging of “The Shannon Effect,” according to Harari, served as a proving ground for a new disruptive technology in live production.
“In the entertainment world, everything is sort of pre-programmed,” he noted. “What has been lacking is real-time automated control. The research stemming from this collaboration has resulted in the creation of a dynamic control system for staging technologies that can be applied in myriad ways based on the content of the spectacle, be it theatrical, sporting events or music. The same technology can translate to a Broadway stage, a Las Vegas production like Cirque de Soleil or even a sporting event such as the X Games, using a flipping motorcycle instead of a tap dance.”
Harari says that Nokia Bell Labs and Stevens will continue to build on the system they created, adding that their ongoing relationship harkens back to the kind of arts and technology collaboration that took place at Bell Labs in the 1960s.
Nokia Bell Labs reinforced this collaboration with the recent announcement that Jeff Thompson, assistant professor and director of the Visual Arts and Technology program at CAL, would become the first Artist in Residence at Bell Labs in Nokia’s recent revival of E.A.T. He is now located at the company’s Murray Hill, New Jersey, campus.
The continued success of Nokia Bell Labs’ collaborative research with Stevens, according to Harari, dispels the notion that artist and technologist have different functions and skill sets, and exemplifies Stevens’ philosophical approach to humanities and technology education.
“At Stevens, the pursuit of innovation, grounded both in scientific principles and the humanities, has been the cornerstone of the university
since its founding.”
* Human Digital Orchestra is a trademark of Nokia.