Journal
MULTIMEDIA TOOLS AND APPLICATIONS
Volume 78, Issue 8, Pages 10035-10056Publisher
SPRINGER
DOI: 10.1007/s11042-018-6565-5
Keywords
Sign language recognition; Multimodal learning; Convolutional neural networks; Kinect; Leap motion
Categories
Funding
- Protect NanoSTIMA: Macro-to-Nano Human Sensing: Towards Integrated Multimodal Health Monitoring and Analytics - North Portugal Regional Operational Programme (NORTE 2020), under PORTUGAL 2020 Partnership Agreement [NORTE010145-FEDER000016]
- European Regional Development FUND (ERDF)
- FundacAo para a Ciencia e a Tecnologia (FCT) [SFRH/BD/102177/2014, SFRH/BPD/101439/2014]
- Fundação para a Ciência e a Tecnologia [SFRH/BD/102177/2014] Funding Source: FCT
Ask authors/readers for more resources
Sign Language Recognition (SLR) has become one of the most important research areas in the field of human computer interaction. SLR systems are meant to automatically translate sign language into text or speech, in order to reduce the communicational gap between deaf and hearing people. The aim of this paper is to exploit multimodal learning techniques for an accurate SLR, making use of data provided by Kinect and Leap Motion. In this regard, single-modality approaches as well as different multimodal methods, mainly based on convolutional neural networks, are proposed. Our main contribution is a novel multimodal end-to-end neural network that explicitly models private feature representations that are specific to each modality and shared feature representations that are similar between modalities. By imposing such regularization in the learning process, the underlying idea is to increase the discriminative ability of the learned features and, hence, improve the generalization capability of the model. Experimental results demonstrate that multimodal learning yields an overall improvement in the sign recognition performance. In particular, the novel neural network architecture outperforms the current state-of-the-art methods for the SLR task.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available