JPEG PLENO: Going Deep the Plenoptic Way

Prof. Dr. Fernando Pereira
Instituto Superior Técnico (Universidade de Lisboa) - Instituto de Telecomunicações

Abstract

It is undisputable that light plays a vital role in our daily lives while communicating with the world around us. But while the world is made of objects, these objects do not communicate their properties directly to the observers; they rather fill the space around them with a pattern of light rays that is perceived and interpreted by the human visual system. Over the past decades, tremendous progress has been achieved in the way consumers and professionals capture, represent, code, store, distribute, display and ultimately use images and video. This motivated an ever-growing acceleration in the creation and usage of image and video in a multitude of sectors, applications, products and services providing the users adapted, powerful experiences. To make these experiences possible, visual information has to be acquired and represented according to some models and formats which allow replicating the light patterns that constitute the visual world. This process is simultaneously driven and conditioned by the available sensors, transmission and storage channels, displays, and the human visual system. In recent years, 3D experiences have become more popular, acknowledging that a faithful, transparent and immersive representation of the world requires more than 2D video. This has asked for 3D video acquisition, representation and coding models, notably stereo pairs to provide depth impressions (stereo parallax) and multiview video to offer some immersive 3D navigation capabilities (both stereo and motion parallaxes). Currently, an explosion of new sensors and displays is happening which go beyond the traditional representation model to provide more realistic, adaptive and immersive visual experiences. These less conventional developments led to revisiting the fundamentals of the light capture and display processes, notably the structure of the information in the light impinging on an observer of a scene, in a deep quest to find more powerful and universal visual representation models. In this context, it is only natural to consider the so-called plenoptic function which represents the intensity of light seen from any viewpoint or 3D spatial position, any angular viewing direction, over time, and for each wavelength. While new and denser samplings of the plenoptic function as performed by the emerging cameras/sensors may provide richer visual representations using new imaging modalities, and thus additional user functionalities, they also imply more visual data, sometimes even a huge amount of data. As a consequence, the new plenoptic imaging modalities need appropriate representation models with associated efficient coding solutions. Naturally, when speaking about coding, interoperability is a must and thus standardization bodies become major players. In this context, the main objective of this talk is to start from the recent developments in visual cameras/sensors and displays to discuss new visual information representation models, new user functionalities, and finally relevant emerging technological challenges, notably in modeling, coding, rendering and quality assessment. In this context, JPEG PLENO will be presented as the first standardization project addressing new 3D visual representations based on a plenoptic function based understanding of the light phenomenon.