Learning to Build a New Reality

Abstract

There have been tremendous advances on applying deep learning techniques for 2d image understanding. In contrast, very little work has focused on employing deep learning for modeling datasets beyond 2D such as 3D geometry and 4D light fields. In this talk, I present several latest works from our group on in this exciting new arena, with a focus on their applications to virtual and augmented reality and computational photography. I first present a novel deep surface light field (DSLF) technique. A surface light field represents the radiance of rays originating from any points on the surface in any directions. Traditional approaches require ultra-dense sampling to ensure the rendering quality. Our DSLF works on sparse data and automatically filling in the missing data by leveraging different sampling patterns across the vertices and at the same time eliminates redundancies due to the network’s prediction capability. For real data, we address the image registration problem as well as conduct texture-aware remeshing for aligning texture edges with vertices to avoid blurring. Next, I present an end-to-end deep learning scheme to establish dense shape correspondences and subsequently compress 3d dynamic human bodies. Our approach uses sparse set of “panoramic” depth maps or PDMs, each emulating an inward-viewing concentric mosaics (CM). We then develop a learning-based technique to learn pixel-wise feature descriptors on PDMs. The results are fed into an autoencoder-based network to achieve ultra-high compression ratio.

Biography

Jingyi Yu is currently a Full Professor and Associate Dean of the School of Information Science and Technology at ShanghaiTech University. He is also affiliated with the Department of Computer and Information Sciences at University of Delaware. He received B.S. from Caltech in 2000 and Ph.D. from MIT in 2005. He has published over 120 papers at highly refereed conferences and journals, and holds over 10 international patents on computational imaging. His research interests span a range of topics in computer vision and computer graphics, especially on computational photography and non-conventional optics and camera designs. He is a recipient of the NSF CAREER Award and the AFOSR YIP Award, and has served as an area chair of many international conferences including CVPR, ICCV, ECCV, ICCP and NIPS. He is currently an Associate Editor of IEEE TPAMI, IEEE TIP, and Elsevier CVIU, and will be program chair of ICPR 2020 and IEEE CVPR 2021.