After months of being limited to testing their revolutionary smart-glove+virtual-reality (VR) platform with healthy participants during the height of the COVID-19 pandemic, Stevens Institute of Technology biomedical engineering professor Raviraj “Ravi” Nataraj and his team are finally back on the field. They’re conducting clinical trials focused on improving hand reach and grasp at the Kessler Foundation with people who have sustained traumatic brain injuries, and at the James J. Peters Bronx VA Medical Center with people dealing with spinal cord injuries.
“It feels great to be back,” said Nataraj, whose Movement Control Rehabilitation (MOCORE) Lab works to optimize functional rehabilitation through a sense of agency (the feeling of control over actions), sensory-guided training using vision and touch, and computer simulations. “The goals of our research are to advance the quality of life and access of people, specifically those of us with a disability. Ultimately, testing on patients is what this work is all about.”
The lab’s smart-glove+virtual-reality platform uses force and flex sensors to provide inputs to a neural network that determines when the patient achieves secure grasp on an object. The platform then uses sounds, lights and vibrations on the glove or through synchronized VR environments to help the patient recognize and even improve progress.
“We’re investigating how performance feedback – through these sensory modalities activated by visual and audio alerts from the glove and from VR – can affect brain and muscle activation patterns to accelerate movement learning for persons with neural injury,” Nataraj explained. “This is exciting, because we will eventually see how the type or level of neural injury plays a role in processing sensory feedback intended to better engage persons in physical therapy practices.”
Testing at Kessler will continue for the rest of 2021, and testing at the Bronx VA will continue through 2022.
Developing the next generation to develop next-generation therapeutic solutions
The project also includes an educational component, teaching local high school students how to develop their own rehabilitation programs in a VR-based language called Unity. Nataraj and his Ph.D. students provide tutorial materials, and then, during bi-weekly Zoom meetings, supervise and guide the students’ progress in creating rehabilitation-based applications. The goal is two-fold: to help the students get interested in programming, and to help patients who can benefit from such rehabilitation innovations.
“First, we want to motivate students to learn how to program,” Nataraj said. “Programming is an increasingly crucial job skill, but stigmas still exist as to how accessible these types of jobs are. We hope to show how people can get started in programming pretty quickly and create applications they can immediately be proud of developing.”
The second objective involves inspiring the students to get further involved in helping rehabilitate people who have experienced severe physical traumas.
“We encourage the students to identify and investigate a clinical-level disability that can benefit from VR-based physical therapy,” Nataraj said. “Then they consider and develop design application features, typically in the form of a game that they think is fun and would be helpful to that clinical population. We also ask the students to do their own literature review so they can reference their own knowledge base in creating their VR applications. Through the Zoom meetings, we share anecdotes about how their applications can be further adapted to be practically deployed and to better address the needs of those clinical populations. Ultimately, through literature review, application development, and experienced supervision, we hope to broaden awareness and inspire greater involvement in the field of rehabilitation.”
Currently the MOCORE lab is working with students from Ossining High School in New York and Bergen County Technical, High Tech, and Wall high schools in New Jersey, and partnerships with the Liberty Science Center. The lab anticipates extending the program to students from additional under-represented regions in the area.
‘We’re at the tip of the iceberg’
The MOCORE Lab is also advancing its VR+brace system, which aims not only to help improve the muscle activation control of a virtual prosthetic arm, but also to better engage users in the process.
“Regaining muscle-level control is critical for persons who have lost accuracy and strength in their movements after injuries to the brain or spinal cord,” Nataraj noted. “The brace is position-adjustable to hold an arm in place, ideally at a position recommended by a physical therapist, and provides some resistance against muscular efforts. Using skin sensors that measure muscle activity, we translate those muscular efforts to actions observed on an avatar in a virtual environment. We then vary the guidance sensations either visually via VR or haptically via vibration motors to cognitively engage the user while accelerating learning of muscle-level control.”
After several rounds of refinements, the research-grade brace is now operable, safe, comfortable and ready to test. At the same time, undergraduate students are already developing design improvements to create an affordable, commercially accessible version that is more streamlined and self-contained, through onboard integration of muscle activity sensors and vibration motors built into the brace.
Beyond helping users perform better, the team is committed to ensuring that users are also engaged without being overly stressed.
“We are newly exploring how to maximize the person's well-being throughout the training process by looking at specific cognitive and physiological factors,” Nataraj said. “We will measure attention through eye-tracking using the VR headset, mental loading through electroencephalography, heart rate through a wrist monitor, and skin conductance through electrodermal activity sensors. We will observe fatigue mentally through electroencephalography and physically through the muscle activity sensors. In total, all these measurements will tell us how a person is fundamentally responding to variations in sensory feedback from our VR-based platforms. That way, they can achieve greater user engagement and agency – the perception of better control – without over exertion, which we believe supports more efficient rehabilitation.”
To that end, the team is committed to exploring options for personalization. Today, that means customization for disability types and levels. In the future, it could encompass the exciting vision of presenting feedback in specific ways literally tailored to each person.
“Our ultimate goal is the creation of control systems that will auto-identify and present optimal visual and haptic feedback that maximizes each person's performance and well-being during training,” Nataraj said. “This is a complex yet fascinating facet to explore. Beyond the disability, how people may respond to a training environment may depend on their personality type, cultural tendencies, personal experiences, and more. The challenge is how to effectively survey and identify the elements that can truly help each person perform better, and then implement them into the sensory-driven training system.”
One approach may be the development of flexible rehabilitation training platforms that allow the users to select their own training experiences, customizing sensory feedback to improve their engagement, ownership, and, most of all, progress.
“Empowering users to make decisions around avatars, colors, sounds and landscapes isn’t novel in computerized interfaces,” Nataraj said. “What’s novel in our approach is that we’re monitoring how persons undergoing movement training respond to those elements, and whether they’re getting the benefits they think they’re getting. On a more advanced level, we may need to develop methods to auto-identify for each user the complexity and intermittency of how sensory feedback is best provided, such that users are responding well during training. We’re at the tip of the iceberg in exploring sensory feedback and its effect on spectrums of physical and cognitive abilities, and this may be the key to facilitating better rehabilitation outcomes. In this way, we are hopefully forging new avenues of research that will inform any developers of assistive and rehabilitative technology how they may create devices to maximize both function and experience.”
“The goal with assistive and rehabilitative technologies is to make lives better,” said Raviraj “Ravi” Nataraj, Stevens professor of biomedical engineering, whose VR+brace system is one of the pioneering rehabilitation therapy devices his Movement Control Rehabilitation Lab is developing. “Understandably, much focus is on the functional outcomes, but we also have to consider how we improve the training process and how we can better integrate people with these instrumented wearables and other technologies. Ultimately, if people do not have a good experience with the technology, regardless of the functional benefit, there is greater risk of abandonment. What excites me about this project is that we are looking to help people at a deeper level. Not only do we want them to function better, but we also want to maximize their well-being and experience as they learn to function better.”
Learn more about biomedical engineering at Stevens: