Article
Neurosciences
Song Zhao, Chongzhi Wang, Chengzhi Feng, Yijun Wang, Wenfeng Feng
Summary: This study investigated the cross-modal interaction and semantic congruency effect in the attentional blink using event-related potentials. The results showed that when the sound synchronized with T2, there was a significant improvement in T2 discrimination and further semantic modulation. These findings suggest that audiovisual interactions and temporal synchrony play a crucial role in the cross-modal boost of T2 discrimination and semantic modulation.
HUMAN BRAIN MAPPING
(2022)
Article
Neurosciences
Chenyang Lin, Maggie Yeh, Ladan Shams
Summary: Human perception is multisensory, and the interaction between different sensory modalities influences our perceptual pleasure, even subconsciously. The study found that temporal congruency between video and audio enhances perceptual pleasure.
NEUROSCIENCE LETTERS
(2022)
Article
Neuroimaging
Natalie Layer, Anna Weglage, Verena Mueller, Hartmut Meister, Ruth Lang-Roth, Martin Walger, Micah M. Murray, Pascale Sandmann
Summary: This study compared auditory, visual, and audiovisual speech perception in cochlear implant users and normal-hearing listeners using electroencephalography (EEG) and a redundant target paradigm. The results showed that both groups experienced multisensory integration, but the cochlear implant users exhibited distinct visual impact on auditory speech processing and differences in N1/P2 event-related potentials (ERPs) compared to normal-hearing listeners. These findings suggest that cochlear implant users use a compensatory strategy to overcome their limited auditory input.
NEUROIMAGE-CLINICAL
(2022)
Article
Computer Science, Information Systems
Fiseha B. Tesema, Jason Gu, Wei Song, Hong Wu, Shiqiang Zhu, Zheyuan Lin
Summary: Active speaker detection (ASD) is about identifying the person speaking in a video among visible human instances. This study proposes an efficient audiovisual fusion (AVF) approach that captures correlations between facial regions and sound signals, focusing on discriminative facial features and associating them with corresponding audio features, resulting in improved detection accuracy.
Article
Neurosciences
G. Karthik, John Plass, Adriene M. Beltz, Zhongming Liu, Marcia Grabowecky, Satoru Suzuki, William C. Stacey, Vibhangini S. Wasade, Vernon L. Towle, James X. Tao, Shasha Wu, Naoum P. Issa, David Brang
Summary: The study found that visual speech modulates auditory processes in the superior temporal gyrus in multiple ways, eliciting temporally and spatially distinct patterns of activity that differed across frequency bands. This indicates the presence of multiple mechanisms supporting audiovisual speech perception.
EUROPEAN JOURNAL OF NEUROSCIENCE
(2021)
Article
Multidisciplinary Sciences
Patrycja Delong, Uta Noppeney
Summary: This study evaluated the impact of spatial and semantic correspondences on audiovisual binding by combining forward-backward masking with spatial ventriloquism. The results showed that semantic congruency enhances visual biases on perceived sound location only when the picture is within observers' awareness.
SCIENTIFIC REPORTS
(2021)
Article
Computer Science, Artificial Intelligence
Jiazhong Chen, Qingqing Li, Hefei Ling, Dakai Ren, Ping Duan
Summary: This paper introduces a multisensory framework for video saliency prediction using audio and visual signals, which includes four modules to be implemented through a deep learning network architecture. The method shows a significant improvement over existing saliency models that do not take into account audio stimuli, according to numerical and visual results.
Article
Neurosciences
Erfan Ghaneirad, Ellyn Saenger, Gregor R. Szycik, Anja Cus, Laura Moede, Christopher Sinke, Daniel Wiswede, Stefan Bleich, Anna Borgolte
Summary: In daily verbal communication, background noise often disrupts auditory speech perception. Additional visual articulatory information, particularly in disadvantageous hearing conditions, can help improve speech comprehension. Patients with schizophrenia have difficulty integrating visual and auditory information during speech perception. Research on the neural mechanisms behind this deficit has produced inconsistent findings. This study examined the early sensory processing of speech perception in individuals with schizophrenia and found reduced speech comprehension under noisy conditions. The results also indicated disturbances in multimodal speech perception at an early stage of processing, potentially due to deficits in auditory speech perception. Notably, a positive correlation was observed between fronto-central N1 amplitudes and the positive symptom subscale of the Positive and Negative Syndrome Scale (PANSS).
Article
Neurosciences
Yi Yuan, Yasneli Lleo, Rebecca Daniel, Alexandra White, Yonghee Oh
Summary: The study aimed to determine the significant impact of visual presentations of the acoustic amplitude envelopes on speech perception under certain signal-to-noise ratio (SNR) conditions, with the most improvement observed at -3 and -1 dB SNRs.
FRONTIERS IN NEUROSCIENCE
(2021)
Article
Neurosciences
Song Zhao, Chengzhi Feng, Xinyin Huang, Yijun Wang, Wenfeng Feng
Summary: The study explored the temporal dynamics of cross-modal boost on attentional blink by recording ERPs during a visual object-recognition task. It found that both semantically congruent and incongruent sounds improved T2 discrimination, but the enhancement was larger for congruent sounds. The results suggest that the cross-modal boost on attentional blink is hierarchical and depends on the stage of visual object recognition processing.
Review
Neurosciences
Jessica L. Pepper, Helen E. Nuttall
Summary: Multisensory integration is crucial for rapid and accurate perception of the environment, especially in tasks like speech perception. Research highlights the importance of bottom-up and top-down contributions to multisensory integration and how they change with age. Factors such as the temporal binding window, attention, and inhibition play a fundamental role in integrating visual and auditory information, and these processes may decline as we age. Understanding age-related changes in multisensory integration is essential for the well-being of the aging population, as it can impact communication and mobility; further investigation is necessary.
Article
Psychology, Multidisciplinary
Pamela Trudeau-Fisette, Laureline Arnaud, Lucie Menard
Summary: The audiovisual interaction in speech perception has been well defined in adults, but there is limited research on children. This study examined the effect of visual input on auditory perception of phonemes in preschoolers. The results showed that while all participants experienced visual influence on auditory perception, there were significant individual differences in the 5-6-year-old group. This suggests that the auditory and visual systems are developing at that age, and multisensory phonological categorization occurs only in children with mature sensory systems and sensorimotor representations.
FRONTIERS IN PSYCHOLOGY
(2022)
Article
Zoology
Clementine Mitoyen, Cliodhna Quigley, Virginie Canoine, Silvia Colombo, Simon Woelfl, Leonida Fusani
Summary: The study investigates the effect of multimodal timing on female response during male courtship in ring doves. The findings suggest that the temporal association between different sensory modalities plays a role in female behavior and that playback exposure can efficiently study sexual communication in doves.
INTEGRATIVE ZOOLOGY
(2023)
Article
Psychology, Developmental
Evelyne Mercure, Peter Bright, Isabel Quiroz, Roberto Filippi
Summary: Infants growing up in a multilingual environment tend to follow early milestones of language development. The way bilingual infants process visual and auditory speech may be influenced by the increased variability in their linguistic environment.
JOURNAL OF EXPERIMENTAL CHILD PSYCHOLOGY
(2022)
Article
Gerontology
James W. Dias, Carolyn M. McClaskey, Kelly C. Harris
Summary: This study investigates the impact of multisensory input on auditory, visual, and audiovisual speech identification, and found that with age, auditory-visual superadditivity improves, potentially compensating for declines in auditory and visual processing to preserve audiovisual speech identification in older adults.
PSYCHOLOGY AND AGING
(2021)
Article
Linguistics
Yi-Chuan Chen, Su-Ling Yeh
JOURNAL OF MEMORY AND LANGUAGE
(2015)
Article
Multidisciplinary Sciences
Yi-Chuan Chen, Pi-Chun Huang, Andy Woods, Charles Spence
SCIENTIFIC REPORTS
(2016)
Article
Audiology & Speech-Language Pathology
Yi-Chuan Chen, Su-Ling Yeh
LANGUAGE COGNITION AND NEUROSCIENCE
(2017)
Article
Psychology
Yi-Chuan Chen, Charles Spence
JOURNAL OF EXPERIMENTAL PSYCHOLOGY-HUMAN PERCEPTION AND PERFORMANCE
(2018)
Article
Multidisciplinary Sciences
Yi-Chuan Chen, Gert Westermann
SCIENTIFIC REPORTS
(2018)
Article
Psychology, Developmental
Brendan M. Stanley, Yi-Chuan Chen, Terri L. Lewis, Daphne Maurer, David I. Shore
JOURNAL OF EXPERIMENTAL CHILD PSYCHOLOGY
(2019)
Article
Psychology, Mathematical
Yi-Chuan Chen, Pi-Chun Huang, Andy Woods, Charles Spence
PSYCHONOMIC BULLETIN & REVIEW
(2019)
Article
Psychology
Nora Turoman, Carlos Velasco, Yi-Chuan Chen, Pi-Chun Huang, Charles Spence
ATTENTION PERCEPTION & PSYCHOPHYSICS
(2018)
Article
Psychology, Developmental
Yi-Chuan Chen, Terri L. Lewis, David I. Shore, Charles Spence, Daphne Maurer
JOURNAL OF EXPERIMENTAL CHILD PSYCHOLOGY
(2018)
Article
Psychology, Mathematical
Yi-Chuan Chen, Charles Spence
PSYCHONOMIC BULLETIN & REVIEW
(2018)
Article
Psychology
Yi-Chuan Chen, Daphne Maurer, Terri L. Lewis, Charles Spence, David I. Shore
ATTENTION PERCEPTION & PSYCHOPHYSICS
(2017)
Review
Psychology, Mathematical
Yi-Chuan Chen, Charles Spence
PSYCHONOMIC BULLETIN & REVIEW
(2017)
Review
Psychology, Multidisciplinary
Yi-Chuan Chen, Charles Spence
FRONTIERS IN PSYCHOLOGY
(2017)
Article
Biochemistry & Molecular Biology
Yi-Chuan Chen, Terri L. Lewis, David I. Shore, Daphne Maurer
Article
Psychology, Developmental
Yi-Chuan Chen, David I. Shore, Terri L. Lewis, Daphne Maurer
JOURNAL OF EXPERIMENTAL CHILD PSYCHOLOGY
(2016)