Classification of multi-modal remote sensing images based on knowledge graph

With the development of remote sensing (RS) technology, single-modal data alone has gradually become difficult to meet the requirement for high accuracy of RS image classification. As a result, multi-modal RS image classification has become a hot research topic. However, multi-modal RS image classification faces challenges in effectively utilizing advanced semantic information regarding the relationships between land cover classes and extracting discriminative features from the data. Based on this, a multi-modal RS image classification method based on knowledge graph (KG) is proposed. Firstly, graph topology is used to align the hyperspectral image (HSI) and multispectral image (MSI) features extracted by graph convolution networks. Constraint alignment is performed on different feature graphs to reduce the difficulty of fusion and the false recognition rate. Then, we use self-attention and cross-attention to purposefully fuse HSI and MSI to obtain discriminative features rich in two modal information and achieve feature weighted fusion. Finally, a KG based on object spatial relationships is constructed to obtain spatial relationships between different classes to assist in multi-modal RS image classification. The experimental results on the Houston and Ausburg datasets demonstrate that the proposed method achieves overall accuracy of 90.40% and 90.85%, respectively, both of which are more than 3% higher than existing classification methods. The results indicate that our method has better classification performance and can provide a useful reference for RS image classification research.

Automated summary

With the development of remote sensing technology, the demand for high accuracy in image classification cannot be met by single-modal data alone, leading to the emergence of multi-modal remote sensing image classification as a hot research topic. However, effectively utilizing semantic information and extracting discriminative features pose challenges in this field. To address this, a multi-modal remote sensing image classification method based on knowledge graph is proposed, which aligns the features extracted from hyperspectral and multispectral images using graph topology, fuses these features purposefully with self-attention and cross-attention, and constructs a knowledge graph based on object spatial relationships to assist in classification. Experimental results show that the proposed method outperforms existing methods, with overall accuracies of 90.40% and 90.85% on the Houston and Ausburg datasets, respectively.