Scene Flow and Threedimensional Geometry by Global Optimization Methods
Post-doctoral fellow
Introduction / State of the Art:
Scene flow was introduced by Vedula et al. [1] as the 3D vector field, defined on each point on every surface in the 3D scene, which represents the motion of these points between two time frames. Since the optical flow is simply a projection of the scene flow onto a camera image plane, the most obvious way to compute scene flow is to reconstruct it from the optical flow measured in one [1] or more cameras [1,2], possibly helped by a dense stereo reconstruction [3]. In these cases, the difficulty consists in constructing a scene flow which is compatible with several observed optical flows which may bring contradictory information.
Another approach is to work in the scene domain, and to track 3D scene points or surface elements (surfels) instead of 2D image points. The most notable work in this area is perhaps the one of Carceroni and Kutulakos [4]. They model the scene as a set of surfels, each surfel being described by its shape (an oriented planar patch), reflectance (a texture for the albedo and the two specular coefficients of a Phong model), bump map (which models local surface curvature), and motion (modeled as a 3D affine transform). Pons et al. [5] propose a two-step approach, in which they solve alternatively for 3D reconstruction and scene flow using a variational method.
Project:
We propose to work on a global method that would solve simultaneously for the scene geometry and for the scene flow using multiple static cameras. The method could be an extension of the work of Pons et al., but it could also use a completely different scene representation, such as a
surface mesh or surfels. We will consider three situations to solve this problem: we can use the preceding and the current frame from every camera, all the frames from the first one to the current frame, or all the frames from the first one to the last one. As a first approximation, we will make the assumption that the scene is nearly Lambertian: specularities will be detected and rejected, but no explicit reflectance model will be used. The method will later be extended to non-Lambertian scenes, where a reflectance model will be optimized together with geometry and motion.
The GrImage platform of INRIA Rhône-Alpes offers an idea experimental setup for dealing with this problem. Also, the method is going to be developed and experimented in this framework.
Example of reconstruction:
``Yiannis’’ sequence — P. Baker and J. Neumann (University of Maryland); see 3D Photography Challenge.
Estimated shape and 3D motion at corresponding times by Pons, Keriven Et al. [5].
References:
[1] S. Vedula, S. Baker, P. Rander, R. Collins and T. Kanade. Three-dimensional scene flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3):475-480, 2005 — PDF.
[2] Y. Zhang and C. Kambhamettu. Integrated 3D scene flow and structure recovery from multiview image sequences. In Proceedings of CVPR’00, 2674-2681, 2000 — PDF.
[3] R. Li and S. Sclaroff. Multi-scale 3D scene flow from binocular stereo sequences. In Proceedings of WACV/MOTION, 147-153, 2005 — PDF.
[4] R.L. Carceroni and K.N. Kutulakos. Multi-view scene capture by surfel sampling: From video streams to
non-rigid 3D motion, shape and reflectance. IJCV, 49(2-3):175—214, 2002 — PDF.
[5] J.-P. Pons, R. Keriven, and O. Faugeras. Modelling dynamic scenes by registering multi-view image sequences. In Proceedings of CVPR’05, 2005 — PDF.
Contact
The project will be carried out in the PERCEPTION group at INRIA Rhône-Alpes, under the supervision of
Dr. Emmanuel Prados [ HTML], Dr. Frédéric Devernay [ HTML] and Dr. Peter Sturm [HTML]:
Emmanuel.Prados@inrialpes.fr, Tel: +33 476 61 52 27,
Frederic.Devernay@inrialpes.fr, Tel: +33 476 61 52 58,
Peter.Sturm@inrialpes.fr, Tel: +33 476 61 52 32.
Dates
Application deadline: April 16, 2006,
Positions start: as early as possible in summer 2006,
Duration: one-year positions.
Conditions for applicants
An ideal candidate should have a PhD in computer science or applied mathematics with some previous experience on variational methods and/or computer vision. Programming skills with C/C++/Matlab is desired.
See also the official INRIA conditions.
Related links
PERCEPTION Lab,
INRIA Grenoble homepage,
INRIA post-doc recruitment.
Application Forms and INRIA-RA official web site for post-doc applications:
click here !.
Start date: 1 June 2006
Contact person: Emmanuel PRADOS
Deadline: 16 April 2006
