Plenoptic layer-based modeling for image based rendering

Abstract

Image based rendering is an attractive alternative to model based rendering for generating novel views due to its lower complexity and potential for photo-realistic results. In order to reduce the number of images necessary for alias-free rendering, some geometric information for the 3D scene is normally necessary. Because the assumptions underlying Plenoptic theory are not fully met in practice, some aliasing is always present in real world examples. We will describe how we can mitigate these errors and achieve the performance predicted in plenoptic theory on real world data. We will present a fast unsupervised layer-based method for synthesising arbitrary new view of a scene from a set of existing views. Our algorithm takes advantage of the knowledge of the typical structure of multiview data in order to perform occlusionaware layer extraction. Moreover, the number of depth layers used to approximate the geometry of the scene is chosen using Plenoptic sampling theory. We further generalise this theory to allow the use of angled layers and multiple camera planes. The rendering is achieved by using a probabilistic interpolation approach and by extracting the depth layer information on a small number of key images. Simulation results show that our method is only 0.25 dB away from the ideal performance achieved when having access to the ground truth pixel based geometric information of the scene and comparisons are also made to alternative methods. These results demonstrates the effectiveness of our method and the validity of the layer-based model.