Joint architecture and knowledge distillation in CNN for Chinese text recognition

The distillation technique helps transform cumbersome neural networks into compact networks so that models can be deployed on alternative hardware devices. The main advantage of distillation-based approaches include a simple training process, supported by most off-the-shelf deep learning software and no special hardware requirements. In this paper, we propose a guideline for distilling the architecture and knowledge of pretrained standard CNNs. The proposed algorithm is first verified on a large-scale task: offline handwritten Chinese text recognition (HCTR). Compared with the CNN in the state-of-the-art system, the reconstructed compact CNN can reduce the computational cost by > 10x and the model size by > 8x with negligible accuracy loss. Then, by conducting experiments on two additional classification task datasets: Chinese Text in the Wild (CTW) and MNIST, we demonstrate that the proposed approach can also be successfully applied on mainstream backbone networks. (C) 2020 Elsevier Ltd. All rights reserved.

Automated summary

The proposed distillation technique helps transform cumbersome neural networks into compact networks for deployment on alternative hardware devices. Experimental results show that the approach is simple yet efficient in reducing computational cost and model size for various tasks.