Article
Biology
Sean A. Rands, Heather M. Whitney, Natalie Hempel de Ibarra
Summary: Flowers convey information to insects through multiple sensory modalities. Recent research has focused on the importance of 'invisible' information and how information presented in multiple modalities affects the interaction between flowers and their insect visitors. This review highlights the influence of multimodality on behavior, with a specific focus on bumblebees, which are capable of learning and using multiple sensory modes to differentiate between flowers. The review examines the evidence for bumblebees using humidity, electrical potential, surface texture, and temperature as additional modalities, and explores how multimodality enhances their performance. The review also discusses mechanisms, such as cross-modal transfer of learning, where bees can transfer patterns learned in one modality to another without additional learning.
CURRENT OPINION IN INSECT SCIENCE
(2023)
Article
Chemistry, Multidisciplinary
Theodoros Psallidas, Panagiotis Koromilas, Theodoros Giannakopoulos, Evaggelos Spyrou
Summary: This approach utilizes both aural and visual features to create dynamic video summaries from user-generated videos, training a classifier to recognize important parts of the videos. Additionally, a novel dataset with videos from various categories has been introduced to evaluate the approach, showing its potential in video summarization.
APPLIED SCIENCES-BASEL
(2021)
Article
Multidisciplinary Sciences
Megan Nicole Williams, Coren Lee Apicella
Summary: Human mate value is assessed based on various factors such as reproductive potential and disease resistance. These factors have been associated with attractiveness judgments in terms of physical appearance, vocal quality, and body odor. While some researchers argue that attractiveness judgments across different sensory modalities reflect the same underlying factor(s), others propose that judgments made in different modalities reflect different factors. Previous research suggests that attractiveness judgments of faces, bodies, and voices are interconnected, supporting the redundancy hypothesis. However, less is known about the correlation between body odor attractiveness and other modalities. In the present study, the correlation between different modalities of attractiveness is examined in a large sample of men and women, with findings indicating weak correlations between odor, face, and voice attractiveness in women but no correlations in men, providing some supporting evidence for the redundancy hypothesis.
Article
Biology
A. Onishi
Summary: This study looked at the impact of multimodal stimuli on RSVP and RSMP BCIs, finding that audiovisual stimuli can enhance the performance of RSMP BCIs, suggesting the development of better gaze-independent BCI systems.
COMPUTERS IN BIOLOGY AND MEDICINE
(2021)
Article
Computer Science, Information Systems
Carlos de la Fuente, Francisco J. Castellanos, Jose J. Valero-Mas, Jorge Calvo-Zaragoza
Summary: This research presents a new approach to detect frustration in game-play scenarios by automatically extracting meaningful descriptors from individual audio and video sources of information using Deep Neural Networks (DNN). The multimodal proposals introduced in this study outperform other state-of-the-art approaches, achieving error rate improvements of between 40% and 90%.
MULTIMEDIA TOOLS AND APPLICATIONS
(2023)
Article
Computer Science, Artificial Intelligence
Helena Liz-Lopez, Mamadou Keita, Abdelmalik Taleb-Ahmed, Abdenour Hadid, Javier Huertas-Tato, David Camacho
Summary: Generative deep learning techniques have been widely discussed in the public, but the slow progress in applying these techniques to counter disinformation is concerning. With the ease and credibility of manipulating multimedia content, developing effective forensic techniques becomes invaluable. This survey comprehensively describes modern manipulation and forensic techniques, focusing on their applications in video, audio, and multimodal fusion. The classification of manipulation techniques and the generation of datasets using generative techniques are provided for forensic purposes. The review and comparative analysis of forensic techniques from 2018 to 2023, as well as the comparison of end-to-end forensic tools for end-users, are presented. Clear trends and challenges, such as multilinguality, multimodality, and improving data quality, are identified for future research in an ever-changing adversarial environment.
INFORMATION FUSION
(2024)
Article
Computer Science, Artificial Intelligence
Gokhan Ince, Rabia Yorganci, Ahmet Ozkul, Taha Berkay Duman, Hatice Kose
Summary: This study focuses on an audiovisual interface-based drumming system for multimodal human-robot interaction. It evaluates robot/virtual avatar tutors, tangible interaction devices, and mobile multimedia devices within a simple drumming-based interactive music tutoring game scenario. The research shows that the physical robot tutor, feedback, and training mechanisms have a positive impact on user performance.
JOURNAL ON MULTIMODAL USER INTERFACES
(2021)
Editorial Material
Biochemistry & Molecular Biology
Federico Pedraja, Nathaniel B. Sawtell
Summary: The new study on social communication behavior in weakly electric fish has identified neural mechanisms that may explain the significance of silent pauses in communication.
Article
Public, Environmental & Occupational Health
Michelle McDowell, Astrid Kause
Summary: Communicating uncertainties in scientific evidence is crucial for accurately reflecting scientific knowledge and increasing public understanding. This study investigated how different types of uncertainty influence responses to medical evidence communication and found that communicating uncertainty does not have detrimental effects, and may even lead to more favorable responses compared to communicating a lack of evidence. Additionally, the study suggests that the format of communication may not significantly impact outcomes, if well-designed.
Article
Psychology, Social
Yanbing Hu, Zhen Mou, Xiaoming Jiang
Summary: Physical attractiveness plays an important role in interpersonal communication, and this study explores how individuals perceive attractiveness from multimodal information. The results show that attention significantly influences the perception of facial attractiveness in multimodal stimuli, while vocal attractiveness is suppressed by attention. These findings provide important insights into the cognitive mechanisms underlying multimodal communication.
JOURNAL OF NONVERBAL BEHAVIOR
(2023)
Article
Computer Science, Cybernetics
Jingyu Wu, Shi Chen, Wei Xiang, Lingyun Sun, Hongzeng Zhang, Zhengyu Zhang, Yanxu Li
Summary: Impressive progress has been made in developing companion SIAs that reduce loneliness, but recent works neglect the analysis of multimodal feedback in the questioning part and focus mainly on English-based research, which poses a challenge for Chinese SIAs due to cultural differences. To address this, a Chinese multimodal database called CNAMD is introduced, providing videos and annotations across eight modalities for multi-person interaction. Through the analysis of vocal-verbal, audio, behavioral, and multimodal combinations during questioning, the performance of baseline models is evaluated and improvements for processing daily Chinese data are proposed. These findings contribute to the design of culturally appropriate Chinese SIAs that consider Chinese customs and language.
INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION
(2023)
Article
Optics
Xingyu Bao, Junxing Pan, Zetao Cai, Jingwen Li, Xia Huang, Renzhang Chen, Junbin Fang
Summary: The study presents a real-time visible light communication system based on LED displays and smartphones, addressing issues of frame rate mismatch and symbol unsynchronization, with fast image processing algorithms designed to reduce computational complexity.
Article
Computer Science, Artificial Intelligence
Mingyue Niu, Jianhua Tao, Bin Liu, Jian Huang, Zheng Lian
Summary: Physiological studies indicate differences in speech and facial activities between depressive and healthy individuals. To predict the individual depression level, this study proposes a spatio-temporal attention network and a multimodal attention feature fusion strategy. The approach integrates spatial and temporal information and emphasizes audio/video frames related to depression detection. Experimental results demonstrate the effectiveness of the proposed method on depression databases.
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
(2023)
Article
Acoustics
Noora Al Roken, Gerassimos Barlas
Summary: Emotion recognition is a complex and significant area in human-computer interaction, with researchers employing various methods and combining multiple modalities to improve classification accuracy. The paper introduces a new Arabic audio-visual natural-emotion dataset, evaluates existing classifiers, and proposes a new classifier trained on the Arabic dataset.
SPEECH COMMUNICATION
(2023)
Review
Biology
Wim Pouw, Shannon Proksch, Linda Drijvers, Marco Gamba, Judith Holler, Christopher Kello, Rebecca S. Schaefer, Geraint A. Wiggins
Summary: Animal communication is mainly conducted through different modalities like sound and visual cues. The various fields studying multimodal communication are diverse and somewhat disconnected, highlighting the challenge of connecting processes at different levels. Multimodal communication involves unique contributions from neural, bodily, and social interaction levels to the complex rhythms in human and non-human animal communication.
PHILOSOPHICAL TRANSACTIONS OF THE ROYAL SOCIETY B-BIOLOGICAL SCIENCES
(2021)
Review
Agriculture, Dairy & Animal Science
Nicky N. E. Kim-McCormack, Carolynn L. Smith, Alison M. Behie
APPLIED ANIMAL BEHAVIOUR SCIENCE
(2016)
Article
Agriculture, Dairy & Animal Science
Hannah Larsen, Greg M. Cronin, Sabine G. Gebhardt-Henrich, Carolynn L. Smith, Paul H. Hemsworth, Jean-Loup Rault
Article
Agriculture, Dairy & Animal Science
H. Larsen, P. H. Hemsworth, G. M. Cronin, S. G. Gebhardt-Henrich, C. L. Smith, J-L Rault
Article
Medicine, General & Internal
Yvonne Zurynski, James Ansell, Louise A. Ellis, Chiara Pomare, Carolynn L. Smith, Joanna Holt, Jo Root, James Gillespie, Leanne Wells, Jeffrey Braithwaite
Summary: Health consumers in Australia, especially those with chronic conditions, identified significant cost barriers to accessing healthcare. They often cannot afford needed care due to cost issues, indicating that healthcare reform needs to ensure equitable access to healthcare.
INTERNAL MEDICINE JOURNAL
(2021)
Review
Medicine, General & Internal
Yvonne Zurynski, Carolynn Smith, Joyce Siette, Brona Nic Giolla Easpaig, Mary Simons, Gilbert Thomas Knaggs
Summary: Education and training, effective interdisciplinary communication, and influential advocates are key enablers for health professionals to refer patients to lifestyle modification programs, while barriers to patient uptake and completion include poor accessibility and lack of support.
Editorial Material
Public, Environmental & Occupational Health
Yvonne Zurynski, Carolynn L. Smith, Gilbert Knaggs, Isabelle Meulenbroeks, Jeffrey Braithwaite
AUSTRALIAN AND NEW ZEALAND JOURNAL OF PUBLIC HEALTH
(2021)
Review
Medical Informatics
Louise A. Ellis, Mitchell Sarkies, Kate Churruca, Genevieve Dammery, Isabelle Meulenbroeks, Carolynn L. Smith, Chiara Pomare, Zeyad Mahmoud, Yvonne Zurynski, Jeffrey Braithwaite
Summary: This scoping review examines the empirical research conducted in the field of learning health systems (LHSs) over the past 5 years. The study finds that there has been considerable growth in empirical applications of LHSs; however, high-quality studies are still limited. The review suggests improving implementation and evaluation efforts by using implementation determinant and outcome frameworks to enhance assessment and reporting, as well as facilitate comparison and identification of trends across studies.
JMIR MEDICAL INFORMATICS
(2022)
Article
Computer Science, Information Systems
Carolynn L. Smith, Yvonne Zurynski, Jeffrey Braithwaite
Summary: Climate change, human health, and healthcare systems are interconnected. The warming climate leads to increased demand for healthcare due to extreme weather events, while healthcare itself contributes to climate change. In order to reduce healthcare's carbon footprint, it is important to measure the greenhouse gas emissions from healthcare systems and practices. Informatics solutions can help monitor and support climate-change decision-making for clinicians and healthcare policy makers, but there are challenges to the sustainability of monitoring efforts.
JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION
(2022)
Correction
Medical Informatics
Louise A. Ellis, Mitchell Sarkies, Kate Churruca, Genevieve Dammery, Isabelle Meulenbroeks, Carolynn L. Smith, Chiara Pomare, Zeyad Mahmoud, Yvonne Zurynski, Jeffrey Braithwaite
JMIR MEDICAL INFORMATICS
(2022)
Article
Psychiatry
Louise A. Ellis, Genevieve Dammery, Leanne Wells, James Ansell, Carolynn L. Smith, Yvonne Tran, Jeffrey Braithwaite, Yvonne Zurynski
Summary: This study investigates the mental health impact of the COVID-19 pandemic on Australians and the utilization of digital health services. The results show that a significant proportion of the respondents experienced serious psychological distress in 2021, with certain demographic groups being at higher risk. The use of digital health technologies has increased compared to 2018, particularly among those with psychological distress, who also reported high satisfaction with the care received via digital health services.
FRONTIERS IN PSYCHIATRY
(2022)
Article
Health Care Sciences & Services
Genevieve Dammery, Kathryn Vitangcol, James Ansell, Louise A. Ellis, Carolynn L. Smith, Ann Carrigan, Jeffrey Braithwaite, Yvonne Zurynski
Summary: Preventative healthcare is crucial for improving patient outcomes, and the effectiveness of prevention programs is enhanced by activated populations. This study examined the level of activation among Australian adults using the Patient Activation Measure (PAM). The results showed that Australian adults demonstrated high levels of patient activation, but lower activation was associated with lower income, younger age, and psychological distress.
HEALTH EXPECTATIONS
(2023)
Review
Health Care Sciences & Services
Yvonne Zurynski, Kristiana Ludlow, Luke Testa, Hanna Augustsson, Jessica Herkes-Deane, Karen Hutchinson, Gina Lamprell, Elise McPherson, Ann Carrigan, Louise A. Ellis, Putu Novi Arfirsta Dharmayani, Carolynn L. Smith, Lieke Richardson, Genevieve Dammery, Nehal Singh, Jeffrey Braithwaite
Summary: This study conducted a comprehensive analysis of barriers and facilitators associated with the sustainability of healthcare programs, emphasizing the importance of considering both internal and external factors when sustaining healthcare projects, and the need for long-term program evaluations to strengthen evidence in this area.
IMPLEMENTATION SCIENCE
(2023)
Review
Health Policy & Services
Chiara Pomare, Zeyad Mahmoud, Alex Vedovi, Louise A. Ellis, Gilbert Knaggs, Carolynn L. Smith, Yvonne Zurynski, Jeffrey Braithwaite
Summary: This study conducted a scoping review and bibliometric analysis of literature on Learning Health Systems (LHS) from 2016 to 2020. The majority of included publications were non-empirical and focused on specific topic areas, while the definitions of LHS are converging.
LEARNING HEALTH SYSTEMS
(2022)
Article
Veterinary Sciences
H. Larsen, G. Cronin, C. L. Smith, P. Hemsworth, J-L Rault
Article
Behavioral Sciences
Carolynn L. Smith
