Video Distractor Detection with Applications

Abstract

In recent years, with the rapid development of digital technology, the public can capture and produce high-quality visual media conten. However, the current visual media processing techniques cannot satisfy the increasing demands of automatically organize, understand and enhance the media data according to the human intentions and potentially billions of personalized editing. So we focus on how to analysis the images and videos in a perception-oriented way. Personal videos often contain visual distractors, i.e. objects that are accidentally captured that can distract viewers from focusing on the main subjects. We propose methods to automatically detect and localize these distractors. A rule-based method is firstly proposed to detect common distractors. Then a learning based framework is proposed with a manually-labeled dataset. To achieve spatially- and temporally-coherent detection, we propose to extract features at the Temporal-Superpixel (TSP) level in a traditional SVM-based learning framework. We have also experimented with end-to-end learning with Convolutional Neural Networks (CNNs), which achieve slightly higher performance. The classification result is further refined in a post-processing step based on graph-cut optimization. Experimental results show that our method achieves an accuracy of 81% and a recall of 86%. We demonstrate several ways to remove the detected distractors to improve the video quality, including video hole filling; video frame replacement; and camera path re-planning.

Biography

Shi-Min Hu is currently a professor in the department of Computer Science and Technology, Tsinghua University, Beijing. He received the PhD degree from Zhejiang University in 1996. His research interests include digital geometry processing, video processing, rendering, computer animation, and computer-aided geometric design. He has published more than 100 papers in journals and refereed conference. He is Editor-in-Chief of Computational Visual media (Springer), and on editorial board of several journals, including IEEE Transactions on Visualization and Computer Graphics, Computer Aided Design (Elsevier) and Computer & Graphics (Elsevier).