Let's Get "Phygital"

It doesn’t involve dodgeball, square dancing or capture the flag, but Stevens School of Business associate professor Aron Lindberg is working with undergraduates across the University to advance “phygital” education.

This past summer, Lindberg, the SSB Information Systems and Analytics Area Chair, collaborated with Pinnacle Scholars Lawrence Park and Isabelle Jorge on projects that explored the possibilities created by phygital innovation, which combines physical artifacts and digital technology.

Park’s project included starting construction on an autonomous drone inspector. When fully realized, the AI-powered drone would be able to take a command about exploring the condition of a structure, execute the prompt, analyze the data it collects and produce a report. Jorge’s project, Developing AI-Powered Interfaces for Open-Source 3D Printing Systems, began the work of creating a 3D printer you can talk to.

The concept of phygital innovation has been a long-standing interest of Lindberg's.

“Using artificial intelligence in combination with a physical artifact creates new forms of innovation that draw both from the digital arena, and the things that we've learned about how the digital world and digital innovation works, but also from the physical arena of engineering and other sciences,” he explained. “In these two particular projects that my students worked on, we wanted to create artifacts imbued with artificially intelligent capabilities that existed in the real physical world and did real physical things.”

The combination of the physical and the digital necessitates a cross-disciplinary approach, something Stevens is particularly well-positioned to execute by combining the expertise of the School of Business, the Charles V. Schaefer, Jr. School of Engineering and Science and the School of Humanities, Arts, and Social Sciences (HASS). Park is a mechanical engineering major, while Jorge is a HASS student, majoring in visual arts and technology with an information systems minor in the School of Business.

"I think it's critical that we engage in more interdisciplinary innovation,” Lindberg said. “What we're seeing today with the development in AI is that skills that are easy to codify or explicitly write down are the most vulnerable to being replaced by AI tools of various kinds. Where I see a role for our students is to integrate multiple technologies with multiple fields of knowledge, creating synergies between different forms of innovation. I believe that people who stay narrowly within a single domain are going to find it more and more difficult to compete with AI as we move forward, so interdisciplinarity is a crucial source of innovation and is a crucial source of human creativity.”

It's not merely a creative exercise or a means of making students more marketable to employers. The idea of phygital innovation can play a crucial role in accelerating the entrepreneurial process with the potential to generate new ideas and bring them to market in a more efficient and streamlined manner.

“What we see happening with these types of AI tools in the software space, but also in this phygital innovation space, is a greater role for people who are close to the business to engage in design and prototyping, creating minimum viable products at an early stage,” Linderg said. “Before, they had to hand over those processes to technical staff, engineering departments or IT departments. Today, with the use of AI tools, people can speed up innovation processes and the creation of entrepreneurial ventures because people without technical skills can generate that first prototype of a physical artifact that you may want to build a business around.”

“This experience was really a game-changer for me.”

Lawrence had a personal connection to the drone project. His father is an electrician with the Port Authority who works throughout the operation, including airports, bridges, tunnels, ports and mass transit depots.

“To work on the George Washington Bridge, you have to demonstrate that you can walk all the way up the cables, looking down at all the cars and everything from how high up you are, and then you have to come back down.”

Still in its early stages, the project's ultimate goal is to create a tool that reduces some of the human stress associated with potentially hazardous work, such as traversing cables above one of the nation’s busiest bridges.

“The user would set it down, turn it on and then talk into his interface with a command like, ‘Please get me a condition report of the bolts on the top of this bridge trunk,’” Lawrence said. “The drone flies up to the top of the bridge, flies over the bolts and gets a good 360-degree view. Then it performs AI image inspection to evaluate the conditions, like whether they're rusted or if the threads are sheared out. It would fly back and then deliver a report to the user about what it found.”

His work with Professor Lindberg was not his first undergraduate research experience, but it was by far his most hands-on project. The collaboration and exposure to some of the business principles involved helped hone some essential non-technical skills.

“I did research before, but it was all sort of data analysis,” he said. “This experience was really a game-changer for me, actually getting to work hands-on with things, getting to design mechanical parts, and getting to integrate that all with electronics and software. It also helped me learn about project management and communication because we had to build the whole thing from scratch. Especially with hardware projects like this, you can't just buy the first motor or the first camera you find because you have to make sure they all can integrate. You have to plan everything out beforehand and know how it's all going to go together before you buy it because if you buy it and it doesn't mix, you just wasted a whole bunch of resources.”

“Working with Professor Lindberg really did broaden my horizons because in my engineering classes, we go over some of these business concepts, like doing what the customer wants as opposed to what you want, but actually getting firsthand experience in these things, putting this theory to the test was really quite helpful.”

The more in-depth AI application was also new to Lawrence. Designing and building a phygital artifact with such high-stakes potential required more than “messing around with ChatGPT now and then.” While building models and exploring applications that would push the project toward success, Lindberg also stressed a critical factor when working with artificial intelligence.

“Professor Lindberg told me to focus on learning the certain limitations that it has,” Lawrence said. “There are certain things, especially in the coding aspect, where it's very good at certain things, but then it’s really kind of lacking in other fields. It's interesting to see where the balance lies.”

The physical part of constructing the drone advanced the most during the project, so Lawrence believes expanding the AI applications should be the focus for whoever carries on his work.

“We have a lot of the hardware, a lot of the electronics and a lot of the software, but because of the way the AI works, you need to expand it and train it on new applications,” he said. “If you want to actually use it in those applications, for instance, if you want it to understand what bolt conditions are, you have to train it with what those different bolt conditions actually are.”

“I gained a lot of skills, managing different AI and seeing what their limitations are.”

Isabelle has never seen an episode of Star Trek, but her AI-enabled printer functions along the lines of the show’s “Replicator” technology.

Experimenting with “vibe coding” platforms like Replit, she worked on building an application that would fully automate the 3D printing process, from user prompt to modeling, slicing and production.

“Say you wanted to 3D print a mug,” Isabelle said. “You would need to find a model online, download it, find a slicer, put the model in the slicer, hit slice, download the G-code it makes and then send that to your printer. I fully automated that process. There is no need for any of that.”

While starting with Replit, Isabelle quickly discovered that, although it offers non-coders previously unavailable opportunities to build and innovate, it was not an ideal match for this project. Similar to the drone application, she searched for and found some of AI’s current limitations.

“I think the biggest issue or challenge that I had with this project was finding a way to do the backend work, especially after realizing that Replit couldn't do a lot of the necessary things that it needed to do, such as generating the 3D model and slicing it. I did the research on finding a slicer that worked with my printer and was able to be controlled via code. It was about putting all of the pieces together, finding each individual piece of generating the model, slicing it and sending it to the 3D printer.”

“Professor Lindberg really stressed pushing the limitations, and I tried to do that, especially with Replit, but I had to tell him eventually that there was a limitation to what I could do. He pushed me to give my best, and that was really great. I had a really good experience with him.”

Isabelle was successful in creating objects, but the technology is not quite ready to be installed on the USS Enterprise.

“I would say the next steps for this project would be to refine the 3D model generation,” she said. “It has very minor inconsistencies throughout the model. It has bumps, and you could literally see that in the 3D model. You could see that it's not smooth. That's a major thing to work on with this project. Refine the models so they could be a bit smoother and also generate a bit better. That could require using a different AI and maybe testing a different 3D model with the generative AI. There is definitely a lot of work that could be done with it because it's not perfect, but then again, any AI isn't perfect.”

Working with the various artificial intelligence platforms and improving her coding skills are things that Isabelle will gladly carry forward in her remaining classwork and beyond.

“This project is definitely going to help me in my future classes and my future career,” she said. Everyone uses AI, and at the end of the day, I gained a lot of skills, managing different AI and seeing what their limitations are. I see myself working with UI and UX. I'm concentrating on creative programming, combining art and code in some way.”