Article
Computer Science, Artificial Intelligence
Jianping Gou, Baosheng Yu, Stephen J. Maybank, Dacheng Tao
Summary: This paper provides a comprehensive survey of knowledge distillation, covering knowledge categories, training schemes, teacher-student architecture, distillation algorithms, performance comparison, and applications. It also briefly reviews challenges in knowledge distillation and discusses future research directions.
INTERNATIONAL JOURNAL OF COMPUTER VISION
(2021)
Article
Computer Science, Artificial Intelligence
Seunghyun Lee, Byung Cheol Song
Summary: Filter pruning is a representative technique for lightweighting CNNs. To increase the usability of CNNs, filter pruning itself needs to be lightweighted. Thus, a coarse-to-fine NAS algorithm and a fine-tuning structure based on CKT are proposed.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Computer Science, Artificial Intelligence
Seunghyun Lee, Byung Cheol Song
Summary: Knowledge distillation is a method to improve the performance of a student network by transferring knowledge from a teacher network. The proposed method transfers knowledge independently of the spatial shape of the teacher's feature map using singular value decomposition. Additionally, a multitask learning method is presented to effectively adjust the teacher's constraints to the student's learning speed. Experimental results show significant improvements on different datasets.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2022)
Article
Energy & Fuels
Hui Song, Nameer Al Khafaf, Ammar Kamoona, Samaneh Sadat Sajjadi, Ali Moradi Amani, Mahdi Jalili, Xinghuo Yu, Peter McTaggart
Summary: With the increasing importance of renewable energy, predicting photovoltaic (PV) power generation becomes crucial for power management and optimization. This paper proposes a multitasking prediction approach using recurrent neural networks (RNNs) to improve the accuracy of PV power generation prediction across different customer categories. The proposed multitasking RNN (MT-RNN) framework transfers knowledge among tasks, achieving superior performance compared to individual deep neural network (DNN) models.
Article
Computer Science, Information Systems
Chao Tan, Jie Liu
Summary: Knowledge distillation is a method to train a lightweight network by transferring class probability knowledge from a cumbersome teacher network. Several approaches have been proposed to transfer the teacher's knowledge at the feature map level.
INFORMATION SCIENCES
(2023)
Article
Computer Science, Information Systems
Chao Tan, Jie Liu
Summary: Knowledge distillation is effective for transferring knowledge, but the existing training strategy for online knowledge distillation may limit diversity among peer networks. A new strategy called KDEP is introduced to address this issue and improve the overall performance of online knowledge distillation.
INFORMATION SCIENCES
(2022)
Article
Computer Science, Artificial Intelligence
Elizabeth Irenne Yuwono, Dian Tjondonegoro, Golam Sorwar, Alireza Alaei
Summary: This paper investigates the scalability of incremental deep learning for visual recognition, specifically for fast object detection. The experimental results show that incremental learning with knowledge transfer and distillation can save storage requirements compared to training-at-once, but it increases computational time. Adjusting key parameters plays an important role in balancing the accuracy of new and old classes and reducing computational cost.
APPLIED SOFT COMPUTING
(2022)
Article
Computer Science, Artificial Intelligence
Chao Tan, Jie Liu, Xiang Zhang
Summary: Knowledge distillation is a network compression technique where a teacher network guides a student network to mimic its behavior. This study explores how to train as a good teacher, proposing inter-class correlation regularization. Experimental results show that this method achieves good performance in image classification tasks.
KNOWLEDGE-BASED SYSTEMS
(2021)
Article
Computer Science, Artificial Intelligence
Ke You, Lieyun Ding, Quanli Dou, Yutian Jiang, Zhangang Wu, Cheng Zhou
Summary: Bulldozers are crucial in earthwork construction, and improving their intelligence is significant for the industry. This study proposes a hybrid method that imitates expert knowledge using modified deep convolutional neural networks and observation dataset. The method successfully solves the observation-based expert knowledge imitation problem.
ADVANCED ENGINEERING INFORMATICS
(2022)
Article
Multidisciplinary Sciences
Catherine F. Higham, Adrian Bedford
Summary: We demonstrate the feasibility of using a classically learned deep neural network as an energy based model on a quantum annealer to exploit fast sampling times. We propose solutions for the challenges of high resolution image classification on a quantum processing unit (QPU): the required number of model states and the binary nature of these states. By transferring a pretrained convolutional neural network to the QPU, we show the potential for classification speedup of at least one order of magnitude.
SCIENTIFIC REPORTS
(2023)
Article
Biology
Yuzhen Qin, Li Sun, Hui Chen, Wenming Yang, Wei-Qiang Zhang, Jintao Fei, Guijin Wang
Summary: The aim of this study is to improve the diagnostic capabilities of single-lead ECG for multi-label disease classification by transferring disease knowledge from multi-lead ECG to a single-lead ECG interpretation model using a teacher-student approach. The study presents a new method called Contrastive Lead-information Transferring (CLT) and modifies Knowledge Distillation into Multi-label disease Knowledge Distillation (MKD) to facilitate the transfer of disease information between different views of ECG. The experiments demonstrate significant improvements in diagnostic performance for single-lead ECG.
COMPUTERS IN BIOLOGY AND MEDICINE
(2023)
Article
Business
Kwok Tai Chui, Varsha Arya, Shahab S. Band, Mobeen Alhalabi, Ryan Wen Liu, Hao Ran Chi
Summary: Open datasets provide researchers with authentic data for conducting research. Transfer learning algorithms enable the extraction of innovation and knowledge from homogeneous datasets of different domains, facilitating the use of machine learning models. This study proposes a multiple incremental transfer learning approach to achieve optimal results in the target model.
JOURNAL OF INNOVATION & KNOWLEDGE
(2023)
Article
Computer Science, Artificial Intelligence
Chao Tan, Jie Liu
Summary: Knowledge distillation (KD) is a method to transfer knowledge from a complex network to a lightweight network, by selecting teachers based on the standard deviation of secondary soft probabilities, and using pretraining under dual supervision and an asymmetrical transformation function to enhance the dispersion of teachers' secondary soft probabilities.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2022)
Article
Computer Science, Information Systems
Zeyi Tao, Qi Xia, Songqing Cheng, Qun Li
Summary: In recent years, deep neural networks have excelled in practical learning tasks, but deploying them on resource-limited devices is challenging. Knowledge distillation transfers model knowledge from a well-trained model to a smaller one, reducing computational cost. A novel neuron manifold distillation method is proposed to improve accuracy-speed trade-offs and a confident prediction mechanism is introduced to enhance the reliability of cloud-based learning systems.
IEEE TRANSACTIONS ON CLOUD COMPUTING
(2023)
Article
Computer Science, Artificial Intelligence
Dimitrios Boursinos, Xenofon Koutsoukos
Summary: This paper presents a method for computing probability intervals in real-time, which assigns pseudo-labels to unlabeled input data to improve efficiency. Empirical evaluation shows that the proposed method improves accuracy and calibration in image classification and botnet attack detection in IoT applications.
PATTERN RECOGNITION
(2023)