Very High-Resolution Image-Based Rendering with Motion Extraction

Steve Matuszek

GOAL

Michael Naimark, who has recently visited UNC, has taken very high-resolution (35mm ASA 50 film), very well-registered stereoscopic, panoramic footage of historical locations around the world. I would like to apply image-based rendering techniques to this data (soon to be ours in the original film form, from which we can obtain digital copies), to result in a three-dimensional environment.

What I hope to contribute on top of existing techniques is:

Improved attention to the parallelizable aspects of preprocessing, so that code can be written to better utilize evans' power
A method of detecting those pixels in the data which represent moving figures, rather than stationary objects, so that this information can be removed, leaving only the environment, or rendered independently of their background.

MOTIVATION

This is one of the best-registered and highest-resolution stereoscopic data sets extant, and since UNC has in evans one of the biggest number crunchers being applied in research, an environment of unmatched realism could ideally result. Furthermore, these are no "example" locations such as someone's graphics lab; these are UNESCO-designated "in danger" World Heritage Sites, such as Jerusalem, Dubrovnik (Croatia), Timbuktu (Mali), and Angkor (Cambodia).

PERSONAL INTEREST

Working for Dr. Henry Fuchs, I am involved with applications that use image-based rendering, but I am not myself that conversant with the techniques. I'd like to get experience using them, and possibly extending them. Also in that project, I have tried with limited success to understand and modify our existing display code. Hopefully, I can start from first principles to create code that makes more sense (to myself, anyway).

PREVIOUS WORK

Michael Naimark's "Be Now Here" page can be found at http://web.interval.com/projects/be_now_here/. [Note: Interval folded a couple months into this project. This page be now not here.]
M. Naimark, "Field Cinematography for Virtual Reality Applications," VSMM '98, 4th International Conference on Virtual Systems and Multimedia, Gifu, Japan.

Additional references:

D. McAllister et al., "Real-Time Rendering of Real World Environments", Rendering Techniques '99, Proceedings of the Eurographics Workshop on Rendering, (Granada, Spain), June 21-23, 1999.
Levoy, M. and P. Hanrahan. Light field rendering. Proc. of SIGGRAPH Õ96, New Orleans, LA, August, 1996.
Levoy, M. and T. Whitted. The Use of Points as a Display Primitive. Univ. of North Carolina, Computer Science, Technical Report 85-022, 1985.

DEMONSTRATION

The demonstration will be a run of the application that displays the resulting 3-D environment, possibly on the head-mounted display, but more likely just with mouse interaction.