Parallel Light Fields: A Perspective and A Framework

Fei-Yue Wang,,, and Yu Shen

Dear Editor,

Light fields give relatively complete description of scenes from perspective of angles and positions of rays.At present time, most of the computer vision algorithms take 2D images as input which are simplified expression of light fields with depth information discarded.In theory, computer vision tasks may achieve better performance as long as complete light fields are acquired.Light field data enjoy a natural advantage over images or videos in 3D reconstruction, and are in great demand in applications such as virtual reality(VR) and augmented reality (AR).However, the high cost hardware and complicated synchronization issues in their array deployment severely hindered development of light field cameras.As a consequence, available light field data are much smaller than traditional images and videos in volume, and a lot of deep learning based computer vision algorithms that take light field images as input are difficult to be deployed.

To overcome difficulties in light field data acquisition, in this letter, we propose a novel method called parallel light fields [1] in a real and virtual interaction manner based on Parallel Intelligence.As shown in Fig.1, by constructing artificial systems or digital twins of light sources, light field cameras and scenarios in virtual environments, we can set up various computational experiment scenarios.For example, we can create scenes with strong sunshine in the afternoon, this is common in autonomous driving scenario, and troublesome for visual perception system in autonomous vehicles.Combining with the real scene small data, the generated virtual big data will constitute diverse big data which will be used to conduct computational experiments in virtual spaces.The abstracted knowledge together with evaluation results will guide the data acquisition in real world, in contrast with those purely rendering methods and pure data collection methods, our parallel light field systems worked in an interactive and evolving way so that we can take full advantage of the diversified real and virtual data.

Framework of parallel light fields: The research of parallel light fields mainly focuses on three components: light sources, light field cameras and scenes.The light sources play an important role in light fields research.Rays emitted from different light sources will render diverse scenarios, by analyzing light distribution, we can control the virtual lights more flexibly and generate the desired scenes.As a kind of sensing instrument, light fields camera is the most direct and reliable device to acquire the light fields.The purpose of light fields research is to generate diverse scenes.Whether in VR, AR or stage lighting arrangement, they all need real or nearly real contents.After obtaining the distribution of light fields, we can constantly optimize the virtual scene.However, it often costs enormous human and financial investment, and the practical effect is not satisfactory.

To achieve all this in the real world, this letter proposes to apply the parallel intelligent technology featured as virtual real interaction and parallel execution to the research of light field problems from the three perspectives of virtual light sources, virtual cameras and virtual scenes.Parallel intelligence was first proposed by Fei-Yue Wang in 2004 to tackle management and control of complex systems, due to its advanced concept and domain adaptation.Recently, parallel intelligence has been used in areas from light fields [2], artistic creation[3], computer vision [4], smart sensors [5] to intelligent transportation [6].The core idea of parallel intelligence is the ACP theory, that A is for: artificial systems for modeling, C is for: computational experiment for analysis and P is for: parallel execution for control.Since the capture and control of rays are very complicated and significant, we consult to ACP theory and try to find a solution.The proposed parallel light fields framework formulate light fields as three parts, that is virtual light source digital twins, virtual cameras digital twins and virtual light fields digital twin and these three components construct the artificial systems.Then, some experiments such as different scenes with different lighting can be created by manipulating light intensities and ray directions to verify the effectiveness.Lastly,according to the previous experiment results, we obtain optimal strategy to modify the artificial systems and the whole system evolve iteratively till convergence, an implemetation pipeline is depicted in Fig.2.

Fig.1.Framework of parallel light fields.

Virtual light source digital twins: The purpose of light fields research is to study the distribution of directions and positions of the rays in the air, so that it is convenient to manipulate the virtual light sources to generate the virtual scene layouts under different lighting conditions by setting different parameters.For example, in a stage play, the stage lighting of the theater often relies on human experiences for manual control.It is difficult to accurately reproduce the scenes with particularly amazing effects, which cause serious effects on viewing experiences of the audience.Aiming at the precise and flexible control of light sources, based on the theory of virtual real interaction of parallel intelligence, we propose to build virtual digital twin of light sources, and finally achieve the goal of customizing the scenes.Although there are some rendering software platforms that can achieve very close to the real light rendering effects in the simulation environment, they still surfer from reality gap because they do not completely retain the characteristics of light propagation in space.

Fig.2.An Implementation of parallel light fields.

In the automatic driving scene, an important factor that affects the camera’s environmental perception is illumination.For example, in the afternoon, the scene objects in front of the vehicle transmits the ambient light into the lens in a manner approximately perpendicular to the camera lens.However, the intensity of the inclined sunlights are far greater than the ambient lights around the vehicle, such scenarios are likely to cause the visual perception completely ineffective.A feasible solution is to use the light fields camera to obtain the angle information and intensity information of the inclined sunlight in the scene.Because the illumination is time-varying, it is necessary to design a controllable specific filter to filter the sunlight, so that the ambient lights around the vehicle can be obtained by the camera lens,which can effectively reduce the interference of sunlight and improve the camera’s ability to perceive the environment.However, the illumination information of different seasons, different time periods and different scenes in the real environment are different, and the cost of collecting relevant data is also huge.Therefore, through the digital twin of solar light sources in the virtual environment, we can simulate the complex and diverse illumination scenarios in various environments.At the same time, we can carry out calculation experiments for different illumination in the virtual environment, design corresponding light filtering schemes, and form a knowledge system to Guide lighting filtering strategies in the real world.

Construction of virtual camera arrays: At present, one of the main factors hindering the research of light fields camera is the high cost of hardware and the complexity in camera array construction.In order to solve the problem in light field data acquisition, we propose a solution of parallel light fields camera, by constructing virtual digital twin of light fields camera in virtual world.The current technical scheme of light fields camera is mainly divided into: light fields acquisition system based on microlens array [7], camera array [8] and encoded mask [9].Their common features are that the hardware platform construction require high precision and can only passively collect rays in the environment.Taking the array camera [10] solution as an example, the parallel light fields camera in the virtual world can adjust the numbers, angles, arrangements and distances between cameras freely and flexibly, which can meet the requirements of different scenes complexity, scene angles and spatial resolutions.It is an active and controllable way in data acquisition.The existing light source digital twins irradiate the rendering from different angles and in different modes, thus forming different scenes.The virtual multiarray camera can deal with various occlusion phenomenas more efficiently, so as to obtain more diverse light field data.

Light fields scene synthesis-A case study: In this part, to verify the feasibility of our scheme, we conduct an experiment as a case study to construct light fields in an synthetic manner by using the popular rendering tool blender [11].We start with the virtual light source digital twins, in our settings, a spot light was choosen as our virtual light source since it is more similar to artificial light source in the real world.It is very convenient for us to reproduce diverse lighting with shadows to represent direction of rays by adjusting intensities, locations and views, etc.The following step is construction of virtual light fields, as mentioned above, virtual sensors are much more flexible to deploy and robust to environmental noises.We take the scheme of light field camera arrays by arrange cameras in a plane with predefined distances within ajacent cameras, all cameras with focal lengthes of 50 mm are shifted towards a common focus plane while keeping the optical axes parallel.For our scene, we import two different object 3D model termed as elephant and car to create a simple scenario, these two objects were arranged in the same plane with occlusions.Although it is very simple scene, necessary components like light, sensors and scene for light fields analysis have been satisfied, the final light field images captured by virtual camera arrays are in a resolution of 7×7×1024×1024×3, where the number 7 is the angular resolution and 1024 is the spatial resolution of each angular image with 3 channels.

As shown in Fig.3, on the left is the rendered light fields images,due to space limitation, we only display 2×2 four views of the light field images and on the right is the surface normal which has the same direction with lights, as we can see, light was emitted from up right, the rendered scene looks like sunlight effect in the afternoon,textures and shadows are obtained which look very natural.

Fig.3.Rendered light fields images.

A very important property of light field images is that images from different perspectives represent the cameras focusing at different depths [12].Due to space limitations, we only show light field images of four depths, as shown in Fig.4.With the continuous change of viewing angles, the focus of the camera also moves from the elephant’s legs to the nose, and the depth and location difference of the vehicle and the elephant has also been fully expressed, as shown in the red bounding box in the figure.The results demonstrate that our experimental scheme can reasonably simulate the imaging process of light fields camera.

Fig.4.Rendered light fields images with different depth.

On the other hand, to tackle 3D motion estimation problems especially the scene flow [13], our scheme can provide a better solution comparing with current 2D image methods.The traditional scene flow [14] calculation depends on the optical flow and depth of the 2D image, this method is not a complete scene flow.With the development of lidar technology, the change of point clouds, which can accurately express the three-dimensional spatial structural information,has become a new direction of scene flow research.However, the research of scene flow based on point clouds only includes the change of spatial structures, and can not express more abundant scene changes from the perspective of textures and colors, this scheme is also incomplete as well.Compared with two-dimensional images, the light fields retain the structural information of scenes.

We take the HCI dataset to make an example, the HCI dataset was a fully synthetic light field dataset with angular resolutions by9×9 and spatial resolutions by 512×512.As depicted in Fig.5, the red lines with arrows are the vector fields and when scene changes with time, the same ray from objects will enter camera lens with a different entry angle, the pixels will be recorded in different locations in image plane.Specifically, light fields camera can also capture changes in angular domain, the combination of angular and spatial variations will lead to a relatively accurate scene flow [15] and [16]which can promote improvements in object detection and tracking.As we know, light field images have relatively complete scene structural information, by moving our virtual camera arrays in a predefined route and speed, we can get time varying 4D scene images,since scene structures and camera poses are known in virtual space,scene flow annations can naturally be obtained.

Conclusion: Light field images contain more complicate information of scenes compared with 2D images which lost depth, depth plays a vital role in a lot of scene understanding tasks, such as object detection, 3D reconstruction and target tracking.Besides, rich scenarios brought by light fields are especially important in virtual reality and augmentation reality.In this paper, we promote the concept of parallel light fields to find a solution to difficulties in acquisition of light field images.By constructing interactive virtual and real worlds,a closed loop from light field data acquisition to data generation and then to model update is formed.Small data in the real world are expanded into big data, then the virtually big data with authenticity are thus used to train perception models, abstract small knowledge for specific tasks, and the knowledge database is constantly updated,iteratively evolved.The combination of software defined light fields camera and real light fields camera effectively solves the problem of difficulties in light field data acquisition caused by hardware problems, and provides a better solution for improving the performance of downstream tasks.

Fig.5.Light fields scene flow.We take the HCI dataset as an example.From Left to right are sub-aperture array flow in two consecutive time and scene flow in two plane representation respectively.