How do we associate visual and auditory perception within a coherent interpretation framework? How do we associate the voice of a person with his/her visual apperance? For example, stereoscopic vision provides depth information. Similarly, binaural cue detection may be used for spatial sound localization. Nevertheless, sound and visual stimuli come under different physical formats that cannot be expressed in the same mathematical space. Therefore, crossmodal integration necessitates a higher level association process that integrates decision taking. We believe that crossmodal integration must be cast into a probabilistic framework such as Bayesian decision theory.

First, we will study stimuli individually, second we will propose theoretical models for spatial fusion and for temporal fusion, third we will devise a statistical method for sensory integration, and fourth we will study the biological mechanisms of sensory integration. Regarding possible biological mechanisms, we will test the hypothesis that coherence of neural signals can change during the formation of crossmodal conjunctions or during crossmodal interference in human subjects. We assume that changes of neural synchrony between areas processing different stimulus modalities will occur in instances of cross-modal feature binding and, moreover, that temporal patterning will be influences by cross-modal shifts of attention.