CNN-G: Convolutional Neural Network Combined With Graph for Image Segmentation With Theoretical Analysis

Deep convolutional neural network (CNN), although recognized to be considerably successful in its application to semantic segmentation, is inadequate for extracting the overall structure information, for its representing images with the data in the Euclidean space. To improve this inadequacy, a new model in the graph domain that transforms semantic segmentation into graph node classification is proposed for semantic segmentation. In this model, the image is represented by a graph, with its nodes initialized by the feature map obtained by a CNN, and its edges reflecting the relationships of the nodes. The node relationships that are taken into consideration include distance-based ones and semantic ones, respectively, calculated with the Gauss kernel function and attention mechanism. The graph neural network is also introduced in this model for the classification of graph nodes, which can expand the receptive field without the loss of location information and combine the structure with the feature extraction. Most importantly, it is theoretically concluded that the proposed graph model takes the same role as a Laplace regularization term in image segmentation, which has been proven by multiple comparative experiments that show the effectiveness of the model in image semantic segmentation. The learned attention is visualized by the heatmap to validate the structure learning ability of our model. The results of these experiments show the importance of structural information in image segmentation. Hence, an idea of deep learning combined with graph structural information is provided in theory and method.

Automated summary

The article introduces a novel model that transforms semantic segmentation into graph node classification to address the inadequacy of deep convolutional neural networks in extracting overall structure information. The model represents the image as a graph, with nodes initialized by feature maps obtained by CNN and edges reflecting relationships between nodes. By introducing a graph neural network for node classification, the model effectively expands the receptive field.