Knowledge Augmented Visual Learning

By Dr. Qiang Ji, Rensselaer Polytechnic Institute

Monday, October 29, 2012 2:00 pm

Location: Babbio Center, Room 110, Stevens Institute of Technology


Substantial progress has been made in the past decades in computer vision, in particular as a result of the application of machine learning methods. Despite these rapid developments, computer vision remains primitive as compared with human vision. One factor contributing to this is the data-driven nature of the machine learning methods as well as their inability to incorporate prior knowledge into the learning process. the data-driven approaches suffer when the required training data is inadequate in either quantity or quality.   This problem can be effectively alleviated with the incorporation of the prior knowledge. The prior knowledge, however, often exists in different and diverse formats, inaccessible to current machine learning methods.

    To address this challenge, we advocate a knowledge-augmented visual learning paradigm, whereby prior knowledge from various sources are systematically exploited, captured, and are principally integrated, along with the image data, into each stage of visual learning.   For this, we first systematically identify sources of knowledge and classify them into four groups:  permanent theoretical knowledge, circumstantial knowledge, subjective experiential knowledge, and the temporary knowledge. We then introduce mechanisms to capture the knowledge and to encode them into different stages of visual learning including target prior, feature learning, and classifier design.  Finally, visual understanding is performed by integrating the captured prior knowledge with the image measurements through a probabilistic inference.  To demonstrate the proposed knowledge-augmented visual learning, we apply it to different computer vision problems. The experiments demonstrate the advantages of the proposed learning method over the existing methods in terms of accuracy, robustness and generalization capabilities, in particular under limited or no training data.



Qiang Ji received his Ph.D degree in Electrical Engineering from the University of Washington. He is currently a Professor with the Department of Electrical, Computer, and Systems Engineering at Rensselaer Polytechnic Institute.  He recently served as a program director at the National Science Foundation (NSF), where he managed NSF’s computer vision and machine learning programs.  He also held teaching and research positions with the Beckman Institute at University of Illinois at Urbana-Champaign, the Robotics Institute at Carnegie Mellon University, the Dept. of Computer Science at University of Nevada at Reno, and the US Air Force Research Laboratory.  Prof. Ji currently serves as the director of the Intelligent Systems Laboratory (ISL) at RPI.

    Prof. Ji's research interests are in computer vision, probabilistic graphical models, information fusion, and their applications in various fields.  He has published over 150 papers in peer-reviewed journals and conferences. His research has been supported by major governmental agencies including NSF, NIH, DARPA, ONR, ARO, and AFOSR as well as by major companies including Honda and Boeing.  Prof.  Ji is an editor on several related IEEE and international journals and he has served as a general chair,  program chair, technical area chair, and program committee member in numerous international conferences/workshops.