Article
Computer Science, Artificial Intelligence
Junjie Ye, Jie Zhou, Junfeng Tian, Rui Wang, Jingyi Zhou, Tao Gui, Qi Zhang, Xuanjing Huang
Summary: This study proposes a sentiment-aware multimodal pre-training (SMP) framework for multimodal sentiment analysis. It captures fine-grained sentiment information through a cross-modal contrastive learning module and additional sentiment-aware pre-training objectives, achieving superior performance in multimodal sentiment classification tasks.
KNOWLEDGE-BASED SYSTEMS
(2022)
Article
Computer Science, Artificial Intelligence
Jianfei Yu, Kai Chen, Rui Xia
Summary: The study proposes a general Hierarchical Interactive Multimodal Transformer (HIMT) model to address the shortcomings in aspect-based multimodal sentiment analysis (ABMSA) and achieves significant improvements in experimental results.
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
(2023)
Article
Computer Science, Theory & Methods
Ringki Das, Thoudam Doren Singh
Summary: Sentiment analysis has evolved from unimodality to multimodality, incorporating text, audio, and video data. Complex deep neural network architectures, such as transformer-based models, have shown significant success in improving sentiment analysis performance. This comprehensive study highlights the changing trends in sentiment analysis and emphasizes its tremendous potential.
ACM COMPUTING SURVEYS
(2023)
Article
Automation & Control Systems
Jieyu An, Wan Mohd Nazmee Wan Zainon
Summary: Multimodal sentiment analysis is an important research area, especially in social media where emotions are expressed through text and images. This paper proposes a novel model called ICCI, which integrates color cues to improve sentiment analysis accuracy. The model extracts semantic and color features, and utilizes a cross-attention mechanism for feature interaction. Experimental results on benchmark datasets demonstrate the effectiveness of ICCI, outperforming existing methods with higher accuracy.
ENGINEERING APPLICATIONS OF ARTIFICIAL INTELLIGENCE
(2023)
Article
Computer Science, Artificial Intelligence
Sarah A. Abdu, Ahmed H. Yousef, Ashraf Salem
Summary: This research provides a comprehensive overview of the latest updates in the field of video sentiment analysis, categorizing thirty-five state-of-the-art models based on the architecture used in each model. It concludes that the most powerful architecture in multimodal sentiment analysis task is the Multi-Modal Multi-Utterance based architecture.
INFORMATION FUSION
(2021)
Article
Computer Science, Artificial Intelligence
Yun Liu, Zhoujun Li, Ke Zhou, Leilei Zhang, Lang Li, Peng Tian, Shixun Shen
Summary: The rise of social networks has allowed people to showcase their lives and emotions through multimodal forms such as images and descriptive texts. Analyzing the emotions within these multimodal content poses research challenges and practical values. We propose the Scanning, Attention, and Reasoning (SAR) model for multimodal sentiment analysis, which includes a scanning model to perceive image and text content, an attention model to understand their complementary features, and a reasoning model to capture network communication in social networks for sentiment predictions. Our model outperforms state-of-the-art methods in extensive experiments conducted on three benchmark datasets.
KNOWLEDGE-BASED SYSTEMS
(2023)
Article
Computer Science, Information Systems
Zemin Tang, Qi Xiao, Xu Zhou, Yangfan Li, Cen Chen, Kenli Li
Summary: This paper proposes a multimodal sentiment analysis method for learning discriminative multi-relation representations, where the core units are modal-utterance-temporal attention (MUTA) and multimodal sentiment loss (MMSL). The method incorporates utterance-level feature vectors into the interactions of different modalities to extract useful relationships and enhances the discriminative power of feature representations through MMSL. Experimental results show that this method outperforms previous baselines on four public multimodal datasets.
INFORMATION SCIENCES
(2023)
Review
Computer Science, Artificial Intelligence
Ganesh Chandrasekaran, Tu N. Nguyen, D. Jude Hemanth
Summary: Sentiment analysis is crucial for identifying and classifying opinions on products or services, with traditional text-based methods no longer meeting the needs of analyzing multimodal data effectively.
WILEY INTERDISCIPLINARY REVIEWS-DATA MINING AND KNOWLEDGE DISCOVERY
(2021)
Article
Computer Science, Artificial Intelligence
Qiuchi Li, Dimitris Gkoumas, Christina Lioma, Massimo Melucci
Summary: The study addresses the challenge of feature fusion for multimodal sentiment analysis by proposing a new quantum-inspired framework. Utilizing a complex-valued neural network, the model achieves comparable results to state-of-the-art systems in benchmarking video sentiment analysis datasets.
INFORMATION FUSION
(2021)
Review
Computer Science, Hardware & Architecture
Songning Lai, Xifeng Hu, Haoxuan Xu, Zhaoxia Ren, Zhi Liu
Summary: This review provides a comprehensive overview of multimodal sentiment analysis, discussing its definition, historical context, and evolutionary trajectory. It explores recent datasets and state-of-the-art models, highlighting challenges and future prospects. By offering constructive suggestions for research directions and model development, this review aims to guide researchers in this dynamic field.
Article
Physics, Multidisciplinary
Kia Dashtipour, Mandar Gogate, Ahsan Adeel, Hadi Larijani, Amir Hussain
Summary: This paper introduces a novel Persian sentiment analysis approach using deep learning to automatically classify movie reviews as having positive or negative sentiments. The study found that the LSTM algorithm outperformed other deep learning algorithms and manual-feature-engineering-based methods in performance.
Article
Computer Science, Artificial Intelligence
Jie Zhou, Jiabao Zhao, Jimmy Xiangji Huang, Qinmin Vivian Hu, Liang He
Summary: Aspect-based sentiment analysis has achieved great success in recent years, however, integrating text and image in multimodal content for sentiment analysis presents challenges and requires further research and improvement in model performance.
Article
Computer Science, Artificial Intelligence
Bo Yang, Bo Shao, Lijun Wu, Xiaola Lin
Summary: This paper introduces a multimodal translation framework for sentiment analysis, improving the quality of visual and audio features by translating them to text features extracted by BERT. Experimental results on two benchmark datasets demonstrate that the proposed model outperforms state-of-the-art methods in terms of all metrics, showing the effectiveness of the approach.
Article
Computer Science, Artificial Intelligence
Ankita Gandhi, Kinjal Adhvaryu, Soujanya Poria, Erik Cambria, Amir Hussain
Summary: This survey paper explores the importance and recent advancements in sentiment analysis and multimodal sentiment analysis in the fields of artificial intelligence and natural language processing. It compares various fusion architectures in terms of MSA categories and presents interdisciplinary applications and future research directions.
INFORMATION FUSION
(2023)
Article
Computer Science, Artificial Intelligence
Linan Zhu, Zhechao Zhu, Chenwei Zhang, Yifei Xu, Xiangjie Kong
Summary: Sentiment analysis is a new technology that explores people's attitudes towards an entity. It has wide applications in various fields and scenarios including product review analysis, public opinion analysis, psychological disease analysis, and risk assessment analysis. Traditional sentiment analysis only focuses on text, but multimodal sentiment analysis incorporates visual and acoustic information to accurately infer sentiment polarity. This article discusses the framework, characteristics, and challenges of different fusion methods in multimodal sentiment analysis, along with the development status, popular datasets, feature extraction algorithms, application areas, and existing challenges.
INFORMATION FUSION
(2023)
Article
Physics, Multidisciplinary
Giuseppe Varone, Cosimo Ieracitano, Aybike Ozyuksel Ciftcioglu, Tassadaq Hussain, Mandar Gogate, Kia Dashtipour, Bassam Naji Al-Tamimi, Hani Almoamari, Iskender Akkurt, Amir Hussain
Summary: The development of reinforced polymer composite materials has significantly influenced the problem of shielding high-energy photons in industrial and healthcare facilities. Machine learning approaches can be used to assess the gamma-ray shielding behavior of composites and provide an alternative to theoretical calculations. In this study, a dataset was developed and machine learning models were used to replicate the gamma-ray shielding characteristics of concrete.
Article
Engineering, Biomedical
Yao Ge, Ahmad Taha, Syed Aziz Shah, Kia Dashtipour, Shuyuan Zhu, Jonathan Cooper, Qammer H. Abbasi, Muhammad Ali Imran
Summary: WiFi sensing has gained significant attention as a potential mechanism for remote monitoring of the aging population without deploying devices on users' bodies. It has the potential to detect critical events such as falls, sleep disturbances, and respiratory disorders. Unlike other sensing methods, WiFi technology is easy to implement and unobtrusive. This paper reviews the current state-of-the-art research on WiFi-based sensing and discusses its healthcare applications and open research challenges.
IEEE REVIEWS IN BIOMEDICAL ENGINEERING
(2023)
Article
Computer Science, Artificial Intelligence
Seham Basabain, Erik Cambria, Khalid Alomar, Amir Hussain
Summary: An increasing number of studies are using pre-trained language models to tackle few/zero-shot text classification problems. However, most of these studies fail to consider the semantic information embedded in the natural language class labels. This work demonstrates how label information can be leveraged to enhance feature representation in input texts, particularly in scenarios with scarce data resources and short texts lacking semantic information like tweets. The study also shows the effectiveness of zero-shot implementation in predicting new classes across different domains, achieving high accuracy in Arabic sarcasm detection.
Article
Computer Science, Artificial Intelligence
Kai He, Yucheng Huang, Rui Mao, Tieliang Gong, Chen Li, Erik Cambria
Summary: This paper proposes a virtual prompt pre-training method that incorporates the virtual prompt into PLM parameters to achieve entity-relation-aware pre-training. The proposed method provides robust initialization for prompt encoding and avoids the labor-intensive and subjective issues in label word mapping and prompt template engineering.
EXPERT SYSTEMS WITH APPLICATIONS
(2023)
Article
Computer Science, Artificial Intelligence
Mostafa Amin, Erik W. Cambria, Bjorn Schuller
Summary: ChatGPT demonstrates the potential of general artificial intelligence capabilities and performs well across various natural language processing tasks. This study evaluates ChatGPT's text classification abilities for affective computing problems including personality prediction, sentiment analysis, and suicide tendency detection. Results show that task-specific RoBERTa models generally outperform other baselines, while ChatGPT performs decently and is comparable to Word2Vec and BoW baselines. ChatGPT exhibits robustness against noisy data, outperforming Word2Vec in such scenarios. The study concludes that ChatGPT is a good generalist model but not as specialized as task-specific models for optimal performance.
IEEE INTELLIGENT SYSTEMS
(2023)
Article
Chemistry, Analytical
Heqing Huang, Bing Zhao, Fei Gao, Penghui Chen, Jun Wang, Amir Hussain
Summary: This paper proposes a novel unsupervised learning framework for video anomaly detection in smart city surveillance applications, using a training model based on the Cloze Test strategy. By encoding motion and appearance information at an object level, the proposed method improves the accuracy of anomaly perception. Comparative experiments on benchmark datasets demonstrate high AUROC scores through the proposed method.
Article
Computer Science, Artificial Intelligence
Mostafa M. Amin, Erik Cambria, Bjoern W. Schuller
Summary: The employment of foundation models is expanding and ChatGPT has the potential to enhance existing NLP techniques with its novel knowledge.
IEEE INTELLIGENT SYSTEMS
(2023)
Article
Computer Science, Artificial Intelligence
Wei Li, Yang Li, Vlad Pandelea, Mengshi Ge, Luyao Zhu, Erik Cambria
Summary: The paper introduces a new task called emotion-cause pair extraction in conversations (ECPEC), which aims to extract pairs of emotional utterances and corresponding cause utterances in conversations. The utterance-level ECPEC task is more challenging as the distance between emotion and cause utterances is greater. The experimental results on the proposed ConvECPE dataset demonstrate the feasibility of the ECPEC task and the effectiveness of the framework.
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
(2023)
Article
Computer Science, Artificial Intelligence
Dazhi Jiang, Runguo Wei, Jintao Wen, Geng Tu, Erik Cambria
Summary: Emotion recognition in conversations has wide applications in various fields. We propose an AutoML strategy based on emotion congruent effect to select suitable knowledge and models, and effectively capture context information and enhance external knowledge in conversations.
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
(2023)
Editorial Material
Computer Science, Artificial Intelligence
Frank Xing, Bjoern Schuller, Iti Chaturvedi, Erik Cambria, Amir Hussain
Summary: Neural network-based methods, such as word2vec and GPT-based models, have achieved significant progress in AI research, especially in handling large datasets. However, these methods lack in-depth understanding of the internal features and representations of the data, leading to various problems and concerns.
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
(2023)
Article
Computer Science, Artificial Intelligence
Kai He, Rui Mao, Tieliang Gong, Chen Li, Erik Cambria
Summary: The study proposes a meta-based self-training method for aspect-based sentiment analysis (ABSA). By generating pseudo-labels and controlling convergence rates, the method improves model performance and accuracy in fine-grained sentiment analysis.
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
(2023)
Article
Computer Science, Artificial Intelligence
Tan Yue, Rui Mao, Heng Wang, Zonghai Hu, Erik Cambria
Summary: Sarcasm detection is a challenging task in natural language processing, especially in the context of social media where sarcasm is prevalent. This paper proposes a novel model called KnowleNet that incorporates prior knowledge and cross-modal semantic contrast for multimodal sarcasm detection. By leveraging the ConceptNet knowledge base and utilizing contrastive learning, the model achieves state-of-the-art performance on benchmark datasets.
INFORMATION FUSION
(2023)
Proceedings Paper
Computer Science, Information Systems
Maher Heal, Kia Dashtipour, Mandar Gogate
Summary: The P vs. NP problem is a major problem in computer science that has been open for almost 50 years. Its solution would have a tremendous impact on various fields including mathematics, cryptography, algorithm research, artificial intelligence, game theory, multimedia processing, philosophy, and economics. This work attempts to solve the maximum independent set problem in a polynomial time by transforming any graph into a perfect graph.
ADVANCES IN INFORMATION AND COMMUNICATION, FICC, VOL 2
(2023)
Article
Computer Science, Artificial Intelligence
Deeksha Varshney, Asif Ekbal, Erik Cambria
Summary: This paper focuses on the neural-based interactive dialogue system that aims to engage and retain humans in long-lasting conversations. It proposes a new neural generative model that combines step-wise co-attention, self-attention-based transformer network, and an emotion classifier to control emotion and knowledge transfer during response generation. The results from quantitative, qualitative, and human evaluation show that the proposed models can generate natural and coherent sentences, capturing essential facts with significant improvement over emotional content.
KNOWLEDGE-BASED SYSTEMS
(2024)
Article
Computer Science, Artificial Intelligence
Jialun Wu, Kai He, Rui Mao, Chen Li, Erik Cambria
Summary: Predicting a patient's future health condition is a trending topic in the intelligent medical field. This paper proposes a knowledge-guided predictive framework called MEGACare, which leverages multi-faceted medical knowledge and multi-view learning to enhance clinical prediction accuracy.
INFORMATION FUSION
(2023)
Article
Computer Science, Artificial Intelligence
Rui Lv, Dingheng Wang, Jiangbin Zheng, Zhao-Xu Yang
Summary: In this paper, the authors investigate tensor decomposition for neural network compression. They analyze the convergence and precision of tensor mapping theory, validate the rationality of tensor mapping and its superiority over traditional tensor approximation based on the Lottery Ticket Hypothesis. They propose an efficient method called 3D-KCPNet to compress 3D convolutional neural networks using the Kronecker canonical polyadic (KCP) tensor decomposition. Experimental results show that 3D-KCPNet achieves higher accuracy compared to the original baseline model and the corresponding tensor approximation model.
Article
Computer Science, Artificial Intelligence
Xiangkun He, Zhongxu Hu, Haohan Yang, Chen Lv
Summary: In this paper, a novel constrained multi-objective reinforcement learning algorithm is proposed for personalized end-to-end robotic control with continuous actions. The approach trains a single model using constraint design and a comprehensive index to achieve optimal policies based on user-specified preferences.
Article
Computer Science, Artificial Intelligence
Zhijian Zhuo, Bilian Chen, Shenbao Yu, Langcai Cao
Summary: In this paper, a novel method called Expansion with Contraction Method for Overlapping Community Detection (ECOCD) is proposed, which utilizes non-negative matrix factorization to obtain disjoint communities and applies expansion and contraction processes to adjust the degree of overlap. ECOCD is applicable to various networks with different properties and achieves high-quality overlapping community detection.
Article
Computer Science, Artificial Intelligence
Yizhe Zhu, Chunhui Zhang, Jialin Gao, Xin Sun, Zihan Rui, Xi Zhou
Summary: In this work, the authors propose a Contrastive Spatio-Temporal Distilling (CSTD) approach to improve the detection of high-compressed deepfake videos. The approach leverages spatial-frequency cues and temporal-contrastive alignment to fully exploit spatiotemporal inconsistency information.
Review
Computer Science, Artificial Intelligence
Laijin Meng, Xinghao Jiang, Tanfeng Sun
Summary: This paper provides a review of coverless steganographic algorithms, including the development process, known contributions, and general issues in image and video algorithms. It also discusses the security of coverless steganography from theoretical analysis to actual investigation for the first time.
Article
Computer Science, Artificial Intelligence
Yajie Bao, Tianwei Xing, Xun Chen
Summary: Visual question answering requires processing multi-modal information and effective reasoning. Neural-symbolic learning is a promising method, but current approaches lack uncertainty handling and can only provide a single answer. To address this, we propose a confidence based neural-symbolic approach that evaluates NN inferences and conducts reasoning based on confidence.
Article
Computer Science, Artificial Intelligence
Anh H. Vo, Bao T. Nguyen
Summary: Interior style classification is an interesting problem with potential applications in both commercial and academic domains. This project proposes a method named ISC-DeIT, which combines data-efficient image transformer architectures and knowledge distillation, to address the interior style classification problem. Experimental results demonstrate a significant improvement in predictive accuracy compared to other state-of-the-art methods.
Article
Computer Science, Artificial Intelligence
Shashank Kotyan, Danilo Vasconcellos Vargas
Summary: This article introduces a novel augmentation technique called Dynamic Scanning Augmentation to improve the accuracy and robustness of Vision Transformer (ViT). The technique leverages dynamic input sequences to adaptively focus on different patches, resulting in significant changes in ViT's attention mechanism. Experimental results demonstrate that Dynamic Scanning Augmentation outperforms ViT in terms of both robustness to adversarial attacks and accuracy against natural images.
Article
Computer Science, Artificial Intelligence
Hiba Alqasir, Damien Muselet, Christophe Ducottet
Summary: The article proposes a solution to improve the learning process of a classification network by providing shape priors, reducing the need for annotated data. The solution is tested on cross-domain digit classification tasks and a video surveillance application.
Article
Computer Science, Artificial Intelligence
Dexiu Ma, Mei Liu, Mingsheng Shang
Summary: This paper proposes a method using neural dynamics solvers to solve infinity-norm optimization problems. Two improved solvers are constructed and their effectiveness and superiority are demonstrated through theoretical analysis and simulation experiments.
Article
Computer Science, Artificial Intelligence
Francesco Gregoretti, Giovanni Pezzulo, Domenico Maisto
Summary: Active Inference is a computational framework that uses probabilistic inference and variational free energy minimization to describe perception, planning, and action. cpp-AIF is a header-only C++ library that provides a powerful tool for implementing Active Inference for Partially Observable Markov Decision Processes through multi-core computing. It is cross-platform and improves performance, memory management, and usability compared to existing software.
Article
Computer Science, Artificial Intelligence
Zelin Ying, Dawei Cheng, Cen Chen, Xiang Li, Peng Zhu, Yifeng Luo, Yuqi Liang
Summary: This paper proposes a novel stock market trends prediction framework called SMART, which includes a self-supervised stock technical data sequence embedding model S3E. By training with multiple self-supervised auxiliary tasks, the model encodes stock technical data sequences into embeddings and uses the learned sequence embeddings for predicting stock market trends. Extensive experiments on China A-Shares market and NASDAQ market prove the high effectiveness of our model in stock market trends prediction, and its effectiveness is further validated in real-world applications in a leading financial service provider in China.
Article
Computer Science, Artificial Intelligence
Hao Li, Hao Jiang, Dongsheng Ye, Qiang Wang, Liang Du, Yuanyuan Zeng, Liu Yuan, Yingxue Wang, C. Chen
Summary: DHGAT1, a dynamic hyperbolic graph attention network, utilizes hyperbolic metric properties to embed dynamic graphs. It employs a spatiotemporal self-attention mechanism and weighted node representations, resulting in excellent performance in link prediction tasks.
Article
Computer Science, Artificial Intelligence
Jiehui Huang, Zhenchao Tang, Xuedong He, Jun Zhou, Defeng Zhou, Calvin Yu-Chian Chen
Summary: This study proposes a progressive learning multi-scale feature blending model for image deraining tasks. The model utilizes detail dilation and texture extraction to improve the restoration of rainy images. Experimental results show that the model achieves near state-of-the-art performance in rain removal tasks and exhibits better rain removal realism.
Article
Computer Science, Artificial Intelligence
Lizhi Liu, Zilin Gao, Yinhe Wang, Yongfu Li
Summary: This paper proposes a novel discrete-time interconnected model for depicting complex dynamical networks. The model consists of nodes and edges subsystems, which consider the dynamic characteristic of both nodes and edges. By designing control strategies and coupling modes, the stabilization and synchronization of the network are achieved. Simulation results demonstrate the effectiveness of the proposed methods.
