Exploitation and recovery of reflectance properties and lighting conditions in (dynamic) three-dimensional vision problems

PhD thesis

Project:

In this project, we propose to design three-dimensional reconstruction algorithms which exploit and recover the reflectance properties and the lighting of the scene. In addition to be able to generate photo-realistic renderings of the reconstructed scenes, we hope to improve 3D reconstructions of non-Lambertian scenes.

We are also interested in theoretical aspects of the problems, since the use and the recovery of reflectance/lighting are two complementary and strongly related problems which both are based on the understanding of some ambiguities which yield to fundamental theoretical questions in this field.

If in the first time we will focus on static scenes, the final goal of the project is explicitly to deal with dynamic ones.

Motivations and Applications:

The state of the art in 3D reconstruction from images is limited by the poor modelizations typically considered and used in this field. Generally, in most works it is assumed that the scene is ideally Lambertian ("rough") meaning that lighting effects are completely neglected. As a consequence, the actual stereovision and multi-view reconstruction algorithms return quite imprecise and disappointing 3D shapes in specular ("shiny") areas of the scene.

In the above example, the wrong reconstruction of the forehead is due to the absence of texture (and of fine geometric structures) on this part of the bust and to the specular highlights visible in the images. By taking into account and possibly explicitly modeling non-Lambertian reflectance properties of surfaces as well as the surrounding lighting conditions, the quality of the geometric 3D reconstruction can be improved.

In other respects, recovering and separating the reflectance properties and lighting conditions are fundamental stages for generating photorealistic renderings of reconstructed scenes from any viewpoint. This challenge has important applications in augmented or virtual reality for example. Addition or elimination of shadows can be eased by the knowledge of the illumination conditions. Addition of reflectance effects significantly improves the realism of the augmented and rendered scenes (especially when the real or virtual camera move) and has proven to be fundamental in many applications. Relighting of scenes, i.e. rendering them using a virtual illumination is also an important application of these issues. The applications in cinematographic postproduction are numerous. As an example, most multi-view reconstruction algorithms only recover the three-dimensional geometry of the scene (shape). Static textures are then mapped on the shape by directly projecting the original images on it. This practice involves strongly visible and displeasing artifacts, as shown in the below illustration. For example, if the input images contain shadows, these will be attached rigidly to the 3D model, prohibiting a realistic impression when displacing the 3D model in an augmented scene for example. The assignment and the reconstruction of reflectance properties to each point of the surface would allow to remove these problems.

Relevant publications of the PERCEPTION group and others:
  N. Birkbeck, Cobzas, P. Sturm and M. Jägersand. Variational Shape and Reflectance Estimation under Changing Light and Viewpoints, ECCV — European Conference on Computer Vision, May 2006 — [ PDF].
  P. Gargallo and P. Sturm. Bayesian 3D Modeling from Images using Multiple Depth Maps. CVPR — IEEE Conference on Computer Vision and Pattern Recognition, 2005 — [ PDF].
  H. Lensch, M. Goesele, J. Kautz, W. Heidrich, H. Seidel, Image-based reconstruction of spatially varying materials, Eurographics Workshop on Rendering Techniques, 2001 — [PDF].
  H. Jin, D. Cremers, A. Yezzi and S. Soatto. Shedding light on stereoscopic segmentation. CVPR — IEEE Conference on Computer Vision and Pattern Recognition, 2004 — [ PDF].

People and Colaborations:

  The project will be carried out in the PERCEPTION group at INRIA Rhône-Alpes, under the supervision of Dr. Emmanuel Prados [ HTML] and Pr. Peter Sturm [HTML].

  Colaborations: This project will be done in tight colaboration with the CERTIS, in particular R. Keriven, J.-P. Pons and F. Segonne. Several stays in Paris are foreseeable.

  GrImage platform: The GrImage platform of INRIA Rhône-Alpes offers an idea experimental setup for dealing with this problem. Also, the method is going to be developed and experimented in this framework. Generation and distribution of data would be an integral part of the project.

Funding: ANR MDCA (French Research Funding Agency) Flamenco Project, 2007-2010. Coordinator of the Flamenco Project: Emmanuel Prados. Salary: 1 893 € / months.

Application / Contact:

For applying contact Dr. Emmanuel Prados [ HTML] and Pr. Peter Sturm [HTML] :
  Peter.Sturm@inrialpes.fr, Tel: +33 476 61 52 32,
  Emmanuel.Prados@inrialpes.fr, Tel: +33 476 61 52 27.

Send a letter or a pdf file containing a complete CV, your publication list, and a list of two references (with telephone numbers and postal and e-mail addresses).

Start date: As soon as possible!

Deadline: June 1st, 2007.

Start date: 1 June 2007

Contact person: Emmanuel PRADOS

Deadline: 1 June 2007