ECE seminar series: Rehabilitation Learning in Virtual Reality with Robot Assistance

Wednesday, March 27, 2013 ( 3:00 pm to 4:00 pm )

Location: Babbio Center, Room 319

Prof. Yingying Chen ([email protected])


Rehabilitation Learning in Virtual Reality with Robot Assistance

BY Dr. Katherine G. August

University of Zurich and ETH Zurich, Institute of Neuroinformatics, Zurich, Switzerland


Many patients experience severe motor, proprioception and tactile sensory loss following stroke, spinal cord injury, Guillain-Barre syndrome, traumatic brain injury, or with Cerebral Palsy which disrupts body schema and sensory dependent skills. For many, there are no traditional therapies available and patients fail to use the hand and arm dramatically affecting their quality of life. A number of learning strategies can be employed for technology assisted rehabilitation, yet only a few, such as intense repetition, have been exploited. Our project investigates potential personalized user-focused neuro-rehabilitation protocols to help re-calibrate body perceptions and improve sensory dependent motor skills and to rapidly translate solutions to practice. We designed, built and tested a low cost easy to use system to provide technology assistance and value to a large variety of underserved patients, and therapists. The Sensory Motor Training Station (SMTS) accommodates the patient’s lost sensory and motor skills and is used to train cognitive, sensory, motor, and proprioception skills. The system uses a unique blended reality -- virtual reality (VR) with immersive virtual limbs and real objects to engage special dorsal and ventral visual stream and provide real tactile experiences, and a novel vibrotactile sensory stimulation that augments patient’s lost sensations. SMTS VR features include configurable action observation and imitation, personalized virtual teacher, virtual mirror, and other important protocols. Robot assistance as needed is provided to overcome patient weakness and promote practice plasticity. Our project investigates ideal conditions and feedback for the individual during training and potential to adjust protocols and predict robot, real object, vibrotactile, and other assistance needed during training. We conducted two arm location task healthy subject experiments with 12 and 11 subjects. Results demonstrated significant improvements in limb location accuracy when touch sensory real stimulus was present in robot assisted VR exercise. Touching the real object improved accuracy above virtual object exercise with VR and the robot. Exercise with the novel vibrotactile stimulation improved accuracy above virtual object exercise alone. Patients successfully exercise in the system. The low cost system can easily be adapted to accommodate left or right limb, heterogeneous patients, and individual cognitive, sensory and motor issues.

Another aspect of the research focuses on action observation, imagery, and imitation in VR to benefit human motor skills learning which might benefit rehabilitation. Utilizing personalized Virtual Teachers may further tap into human abilities to recognize human movement and preferentially recognize his or her own movement. VR conditions impart a sense of realness, and might be capable of facilitating voluntary movement. The SMTS system records the less-affected limb during stroke rehabilitation exercise, to create more realistic personalized sensorimotor training employing an Action Observation Virtual Teacher for mental imagery and imitation. Recordings permit measures of performance important for natural spatial, temporal, and joint coordination, for assessment, and goal setting. Recordings also provide a means to formulate and investigate control programs and training models for robot assistance. More able limb practice enables generation of components of the training system including the personalized Virtual Teacher, objective measures, and goal setting; it also provides an opportunity for a low stress training experience, to create motor memories, to promote generalized skills learning, and to acquaint the client with requirements of the task before the challenge of exercising the more affected limb in a novel robot training environment. A number of assessment techniques have been used including monitoring robot support needed within and across sessions, optimal sensory conditions to bolster client performance, emotion, and assessing heart asymmetry during VR training.


Kit’s recent research has been focused upon designing, developing, and evaluating mixed reality, haptic, and robotic systems that innovate human-robot collaborative experiences. Kit uses various experimental techniques to investigate the underlying neural mechanisms and behavioral effects of multi-sensorial experiences in robot assisted mixed reality task performance to improve the design of human robot interactions, for rehabilitation, and for assistant robot applications. She is interested in the application of engineering to improve smart connected devices and solutions in media, evidence based healthcare, robotics, and assistive devices.  Kit was a Visiting Research Fellow at Queens University of Belfast from 2011 to 2012, Whitaker International Scholar in Biomedical Engineering and Academic Guest at the Institute of Neuroinformatics at the University of Zurich/ ETH Zurich and Sensory Motor Systems Laboratory, Institute of Robotics and Intelligent Systems at ETH Zurich, and Balgrist University Hospital from 2009 to 2011, Research Assistant at New Jersey Institute of Technology Department of Biomedical Engineering 2003 to 2009 while working on the PhD. Kit was a Member of the Technical Staff at Bell Laboratories between 1991 and 2002 where she worked on New Service Concepts and Advanced Communications Technologies, earning 18 U.S. and more than 50 international patents for technology innovations in intelligent networks, systems, methods, devices, and applications including but not limited to: improvements in signal processing for wireless; intelligent adaptive sensors and antenna arrays; Virtual Assistant Coach; the Language Tutor; Intelligent Search Algorithms; Multiple Embedded Streams of Imperceptible Data in Media; technologies for Controlling Remotely Operated Devices; advanced intelligent wireless transactions; etc.  Kit has a Bachelor of Fine Arts in Communications Design from Parsons The New School for Design, a Master of Science in Computer Science Management Information Systems from Marist College, and a PhD in Biomedical Engineering from New Jersey Institute of Technology. She is a Senior Member of IEEE, Secretary of Women in Engineering, Secretary of IEEE Education Society, Member of IEEE Societies: EMBS, Robotics and Automation, Communications, Computational Intelligence; Swiss Neuroscience Society, Sigma Xi.