Visual question answering model based on the fusion of multimodal features by a two-wav co-attention mechanism

Scene Text Visual Question Answering (VQA) needs to understand both the visual contents and the texts in an image to predict an answer for the image-related question. Existing Scene Text VQA models predict an answer by choosing a word from a fixed vocabulary or the extracted text tokens. In this paper, we have strengthened the representation power of the text tokens by using Fast Text embedding, appearance, bounding box and PHOC features for text tokens. Our model employs two-way co-attention by using self-attention and guided attention mechanisms to obtain the discriminative image features. We compute the text token position and combine this information with the predicted answer embedding for final answer generation. We have achieved an accuracy of 51.27% and 52.09% on the TextVQA validation set and test set. For the ST-VQA dataset, our model predicted an ANLS score of 0.698 on validation set and 0.686 on test set.

Automated summary

This paper proposes a Scene Text VQA model that strengthens the representation power of text tokens by using Fast Text embedding and other features. By employing a two-way co-attention mechanism to obtain discriminative image features, combining text token position information to predict the final answer, the model achieved high accuracy on both TextVQA and ST-VQA datasets.