Jianbo Shi, University of Pennsylvania

Imagine you are given a long video, possibly thousands of hours long, and you are asked to analyze the video to detect unusual events. This situation arises in applications such as video surveillance, health care monitoring, and biometric human identification. Humans can extract essential characteristics of video events relatively quickly and easily. How to quantify this process in a computable form is an open problem, and at the first glance is an ill defined problem. Any system that detects unusual events must sift through extremely large amount of minute statistical details to detect a few relevant bits.

Our research can be broadly summarized in two directions. In the first direction, we focus on the extraction of unusual video activities using a large set of simple image features. The work is concentrated on detecting a) unusual video segments of a fixed time length, and b) those of variable time length, varying potentially from a few seconds to a few days. Much of our work has been focused in this area, particularly on concurrent unsupervised feature selection and data classification. Motivated by a similar problem in document-keyword analysis, we have developed a graph spectral based method for finding patterns in video events. In the second direction, we focus on development of image/video features, tools for operator feedback and large scale visualization. For specific application domains, it is important to introduce features that can detect more precisely the action event of interests. Ultimately such a monitoring system will be used by human operators. For this system to be useful, one must develop intuitive feedback and visualization tools for large video sets.

Experimentally, we have tested our algorithm on a variety of videos ranging from nursing home monitoring, poker game cheating, to roadway surveillance.

This is joint work with Hua Zhong at CMU, and Mirko Visontai at U. Penn.

Sponsored by the Multimedia Vision and Visualization Group.