We will develop new algorithms for comparing examples of similar gestures or activities, and organize those examples into a fully segmented and annotated library of actions, as performed by different actors, in different settings and in different styles. Examples will be taken from everyday gestures, athletic actions and dance, recorded with multiple, synchronized cameras. The libraries will store the acquired videos and synchronized annotation, as well as results of geometric reconstructions and recovered textures/appearances.

The annotated database will be used to query and retrieve representative examples of both primitive and composite events. Thus, it will be important to design efficient indexing and retrieval schemes for them. Related work includes the annotated video library of human movements being built by Ben-Arie and coworkers at the University of Illinois or the recent work of Ramanan and Forsyth at Berkeley, both of which provides querying and retrieval mechanisms, but remain limited to single-views of human movements. In contrast to the above work, we will use multiple-camera video sequence and associated geometric reconstructions (including textures) rather than a combination of single-camera video and synthetic animation. Thus, we hope to preserve interesting details and intra-class variations, and build more realistic action models.