CAVA Computational Audio-Visual Analysis

• Participants: One speaker.
• Development: One speaker, walking while speaking continuously though the whole scene. The speaker moves in front of the camera and passes behind. He reappears from the right, and turns to the cameras.

• Participants: One speaker.
• Development: One speaker, walking. The speaker moves while speaking in front of the camera and passes behind it from the left. As soon as he gets out of the field of view, the actor become silent. Only on reappearing from the right, does he start speaking again and turns to the cameras.

• Participants: One speaker.
• Development:Same scenario as CT1OS again with one walking speaker. The main distinction is that, when reappearing, the actor has changed appearance (taken off jacket, put on glasses).

• Participants: One speaker - Two actors.
• Development: Two actors, only one seen and heard at a time. The first speaker moves toward the camera then disappears from the field of view and stops talking. The second speaker enters the field of view while speaking and faces the cameras.

• Participants: One speaker, music source (M), wall, additional light source (L), clicker-chirper (C).
• Development: One speaker, walking. The actor walks behind a wall and returns to his initial position, always speaking. Various audio noises like clicks and music are present. The lighting condition is intentionally modified.

• Participants: Four speakers.
• Development: A more complex tracking scenario than the single speaker TTOS. Four actors are initially in the scene. As they start speaking (and go on speaking throughout the test), they move around; one person exits the scene, walks behind the camera while talking, and reappears.

• Participants: Four speakers.
• Development: A more complex clustering test scenario than the single speaker CTOS. Here four actors are initially in the scene. As they start speaking and moving around, two people exit the scene, stop talking, reappear and start talking again.

• Participants: Five speakers.
• Development: Five actors in total. Initially there are two speakers, then a third joins, one leaves, and later on a fifth joins. Then another two leaves. All actors speak while in the scene and move around.

• Participants: Two speakers, music source (M), wall, additional light source (L), clicker-chirper (C).
• Development: Similar to the one speaker noise test, NTOS. Two speakers are talking, occasionally walking behind a screen.Meanwhile music and clicks are heard in the background.

• Participants: Dummy on swirvel chair.
• Development: A single speaker static while speaking, or a speaker playing white noise, is standing at 0° azimuth. The perceiver starts facing the sound source, and then is moving with periodic left-right movements. The purpose of this sequence is to measure the effects of the perceiver's head movements on binaural cues for localisation.

• Participants: Dummy on swirvel chair. Various noise and speech placed at different locations around the room.
• Development: Two loud speakers playing white noise or speaking people are positioned at -15° and 45° azimuth respectively. The dummy is being moved in a “random” fashion from side to side. This scenario provides interesting data for investigating to what degree it is possible to do ego-location using audio, video and AV cures.

• Participants: Dummy on swirvel chair.
• Development: One moving speaker is walking in the scene and may disappear from the field of view. Two additional static sound sources playing white noise are placed at -15° and 45° azimuth. The perceiver is static during the first part of the sequence and then is panning randomly. The purpose of this sequence is to study whether the head movement can be inferred from binaural dynamics, and to highlight complementarity between audio and video modalities.

• Participants: Blindfolded, seated (on swirwel chair) listener is asked to stay facing toward moving target speaker. Speaker talks while slowly moving around room following an unpredictable pattern.
• Development:
  • AH: One speaker talking while slowly moving around room following an unpredictable pattern. Blindfolded, seated on a swivel chair, the listener is asked to stay facing towards the moving speaker. The research question that may be addressed with this sequence concerns a behavioural study, i.e. how does the perceiver move in order to track the sound source?
  • AHN: The same scenario as AH, with one moving speaker. An additional static sound source produces stationary noise. The research purpose is similar to the previous scenario, here in a challenging noisy environment.

• Participants: Seven speakers.
• Development: 7 actors in total, 6 in scene and one to the left of the fixed perceiver. Two groups of conversation (one immediately in front of and one further away from the dummy head) are formed. People are seated and generally not moving a lot. At some point one speaker from the furthest away group gets up and joins the conversation of the front group. This setup makes for a very challenging auditory and visual scene.

• Participants: Perceiver is moving around/mingling in cocktail party situation.
• Development: Small groups of people chatting are placed around the person wearing the acquisition helmet. He turns around to join in different conversations through the scenario. This is a very challenging scene with lots of people talking from all directions and with people moving in and out of field of view.

M3

M1

• Participants: Five speakers.
• Development:
  • M1: Five actors are seated around a table, three are visible to the fixed perceiver (dummy head); one is to the left and one is to the right of the dummy. Initially all join into the same conversation and later on two subgroups of conversations are formed.
  • M3: Five speakers are sitting around a table having a conversation with normal turn taking. The sequence starts out with each speaker in turn saying their name and affiliation. The perceiver person is instructed to move naturally in terms of who to face during the conversation.

Circ1

P1

• Participants: Seated (on swirwel chair) listener is asked to stay facing towards person speaker. Speakers sit around table “meeting-style” and have a conversation taking turns. The listener is asked to look at dominant speaker. Speech is non-informational.
• Development:
  • P1: This scenario is to mimic a person listening into a conversation with non-overlapping speech and looking at the person speaking. Five people are positioned around a table and taking it in turn to count out loud.
  • Circ1: Several speakers positioned in a circle around the perceiver. Each person takes it in turn to speak and the person wearing the acquisition helmet attempts to look at them as soon as possible after they've started speaking.