3D shape and volumetric radiance estimation of non-opaque objects from multiview images

PhD thesis

Multi-view stereovision consists in recovering the three-dimensional shape of an object or a scene from several images of it. Nowadays, multi-view stereovision algorithms allow more or less to recover opaque objects. Despite the recent advances in opaque surface modeling, the modeling of translucent, semi-transparent or transparent objects has not yet received a very widespread attention. Also, successful methods aiming for opaque surface recovery definitively fail to deal with translucent/transparent surfaces. Only recently, some prospective techniques for modeling transparent surfaces have emerged.

The most significative work has been done inside the Computer Graphics community. Goesele, Lensch et al. [GLLFS:04] propose a method for recovering the scattering behavior of translucent objects by using a laser. Matusik, Pfister et al. [MPZNM:02] propose a first complete acquisition and rendering system for transparent and refractive 3D objects from arbitrary viewpoints under novel illumination. Let us underline that in both, the goal is rendering and not to solve the inversion problem. In [MPZNM:02], the recovered geometry is just the visual hull (which is a very rough approximation of the real 3D shape of the object). Also, for the geometry acquisition, [GLLFS:04] covers the test object with a layer of white dust in order to make the object surface opaque and Lambertian (matte). In fact, very few works really try to solve the inverse problem, recovering shape from images of the object as it is. Kutulakos et al. [KS:08] develops a theory for recovering the 3D shape of (non-diffusive) refractive and specular objects by light-path triangulation [KS:08]; this allows them to recover for example the shape of some diamonds. Bonfort et al. [BSG:06] and Fanany et al. [FKK:05] also propose such a geometric method. Finally, Hasinoff and Kutulakos [HK:07] have considered and proposed a first solution to the problem of reconstructing 3D models of semitrasparent scenes such as fire.

This thesis will concern the area of 3D modelling of non-opaque objects. Several goals are considered:
  an overall theoretical goal is to study minimal requirements for modelling non-opaque objects. In most previous works, dedicated equipment, e.g. the use of a reference object or a laser, is required. We would like to free ourselves from this, studying the feasibility of 3D modelling of non-opaque objects from images taken in every-day conditions.
  development of practical methods for the actual 3D modelling.
  segmentation of scenes into different types of objects: most of the above mentioned approaches require to know in advance the specific type of object to model (transparent, translucent, specular, ...) and most approaches work for only one of these types each. When modelling real-world scenes, e.g. urban scenes, one is usually confronted with a mix of different types. Simple example are buildings, which usually consist of opaque (walls) and semi-transparent objects (windows allow to look through but also reflect the rest of the scene). For achieving a truly generic real-world 3D modelling system, it seems crucial to be able to automatically segment the scene into objects of different types and of course to identify the appropriate types. This can then be followed up by launching the suitable specific 3D modelling method for each segmented object. Such a segmentation will be based on a geometric analysis of the input images but also on machine learning or appearance-based classification techniques.

[BSG:06] T.Bonfort, P.Sturm, P.Gargallo. General Specular Surface Triangulation. ACCV, 2006, 872-881.

[HK:07] S.Hasinoff, K.Kutulakos, Photo-Consistent Reconstruction of Semitransparent Scenes by Density-Sheet Decomposition PAMI, 2007, 29 (5), 870-885

[M:02] S.Hasinoff Three-Dimensional Reconstruction of Fire from Images PhD thesis 2002, Univ.of Toronto

[GLLFS:04] M.Goesele, H.Lensch, J.Lang, C.Fuchs, H.Seidel "DISCO: acquisition of translucent objects" ACM Trans. Graph.,2004,23(3),835-844

[KS:08] K.Kutulakos, E. Steger "A Theory of Refractive and Specular 3D Shape by Light-Path Triangulation" IJCV, 2008,76(1),13-29

[MPZNM:02] W.Matusik, H.Pfister, R.Ziegler, A.Ngan, L.McMillan "Acquisition and rendering of transparent and refractive objects" Eurographics workshop on Rendering, 2002, 267-278

[FKK:05] M.Fanany, I.Kumazawa, K.Kobayashi "A neural network scheme for transparent surface modelling" GRAPHITE ’05, 433-437

Location and Contact

The project will be carried out in the PERCEPTION group at INRIA Grenoble Rhône-Alpes, under the supervision of Dr. Peter Sturm and Dr. Emmanuel Prados:
  Peter.Sturm@inrialpes.fr, Tel: +33 476 61 52 32.
  Emmanuel.Prados@inrialpes.fr, Tel: +33 476 61 52 27,

Related links
  PERCEPTION Lab,
  INRIA Grenoble homepage,

Application Forms and INRIA official web site for CORDI-S applications:
  INRIA official web site !.
  Application Forms !.

Eligibility: Basic knowledge in Computer Graphics, Computer Vision and Applied Mathematics is required. A strong expertise in at least two of these fields is highly desirable. Skills in C++ programming are also mandatory.

Start date: 1 September 2008

Contact person: Peter STURM

Deadline: 14 May 2008