Article
Engineering, Electrical & Electronic
Tongtong Su, Qiyu Liang, Jinsong Zhang, Zhaoyang Yu, Ziyue Xu, Gang Wang, Xiaoguang Liu
Summary: Recent online knowledge distillation methods have focused on capturing rich intermediate information through multi-layer feature learning. However, these methods only consider intermediate layer feature maps within the same layers, neglecting valuable information across layers. In this work, we propose a Deep Cross-layer Collaborative Learning network (DCCL) that efficiently harnesses knowledge from peer student models through appropriate cross-layer supervision. Our experiments demonstrate that DCCL achieves superior performance compared to mainstream OKD methods. The code is available at https://github.com/nanxiaotong/DCCL.
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
(2023)
Article
Computer Science, Artificial Intelligence
Jianping Gou, Yue Hu, Liyuan Sun, Zhi Wang, Hongxing Ma
Summary: In this paper, a new collaborative knowledge distillation method is proposed, which utilizes the strategy of Filter Knowledge Transfer (FKT) to extract valuable filter information from the teacher model. The experimental results on various datasets demonstrate the superiority of the proposed method.
EXPERT SYSTEMS WITH APPLICATIONS
(2024)
Article
Computer Science, Artificial Intelligence
Chuanxiu Li, Guangli Li, Hongbin Zhang, Donghong Ji
Summary: Knowledge distillation is an effective method to obtain small networks for hardware-constrained devices. We propose an innovative online distillation method called embedded mutual learning (EML) that surpasses the state-of-the-art methods on image classification task. The embedded implementation makes EML highly flexible.
APPLIED INTELLIGENCE
(2023)
Article
Computer Science, Information Systems
Chao Tan, Jie Liu
Summary: Knowledge distillation is effective for transferring knowledge, but the existing training strategy for online knowledge distillation may limit diversity among peer networks. A new strategy called KDEP is introduced to address this issue and improve the overall performance of online knowledge distillation.
INFORMATION SCIENCES
(2022)
Article
Computer Science, Artificial Intelligence
Mengya Gao, Yujun Wang, Liang Wan
Summary: Knowledge distillation is a popular method for model compression, transferring knowledge from a deep teacher model to a shallower student model. The RKD method introduces an assistant model to further distill knowledge, achieving appealing results on popular classification datasets.
Article
Computer Science, Artificial Intelligence
Hao Liu, Mang Ye, Yan Wang, Sanyuan Zhao, Ping Li, Jianbing Shen
Summary: This article presents a new adaptive metric distillation approach that significantly improves the backbone features of student networks and achieves better classification results. Previous knowledge distillation methods focused on transferring knowledge across classifier logits or feature structure, neglecting the excessive sample relations in the feature space. The proposed collaborative adaptive metric distillation (CAMD) optimizes the relationship between key pairs, adapts the metric for student embeddings using teacher embeddings as supervision, and employs a collaborative scheme for knowledge aggregation. Extensive experiments show that CAMD outperforms other cutting-edge distillers and sets a new state-of-the-art in both classification and retrieval tasks.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Computer Science, Artificial Intelligence
Jianping Gou, Xiangshuo Xiong, Baosheng Yu, Lan Du, Yibing Zhan, Dacheng Tao
Summary: Knowledge distillation is an effective technique for compressing deep models by transferring knowledge from a large teacher model to a small student model. Existing methods mainly focus on unidirectional knowledge transfer, overlooking the effectiveness of students' self-reflection in real-world education scenarios. To address this, we propose a new framework called MTKD-SSR that enhances the teacher's ability to transfer knowledge and improves the student's capacity to absorb knowledge through self-reflection.
INTERNATIONAL JOURNAL OF COMPUTER VISION
(2023)
Article
Computer Science, Information Systems
Yu-e Lin, Xingzhu Liang, Gan Hu, Xianjin Fang
Summary: Researchers propose a smarter mutual learning method called Smarter Peer Learning (SPL) for online knowledge distillation. By constructing a virtual teacher and a novel online distillation framework, this method allows student models to learn from better performing peers and achieves excellent results on multiple datasets.
MULTIMEDIA SYSTEMS
(2022)
Article
Computer Science, Information Systems
Zhiwei Hao, Yong Luo, Zhi Wang, Han Hu, Jianping An
Summary: In this paper, a framework named CDFKD-MFS is proposed for compressing multiple models into a small model without using original data. The framework utilizes multi-level feature sharing, asymmetric adversarial data-free knowledge distillation, and attention-based aggregation to achieve model compression and training. Experimental results demonstrate that the proposed framework achieves higher accuracy compared to competitive alternatives on three computer visual datasets.
IEEE TRANSACTIONS ON MULTIMEDIA
(2022)
Article
Computer Science, Artificial Intelligence
Renrong Shao, Wei Zhang, Jun Wang
Summary: Data-free knowledge distillation (DFKD) is an effective method to compress models, overcome transmission restrictions and protect privacy. However, current methods have limitations such as inability to distinguish sample distributions and optimize category-wise diversity samples. This paper proposes a new learning paradigm called CPSC-DFKD to address these limitations and achieve better performance.
PATTERN RECOGNITION
(2023)
Article
Multidisciplinary Sciences
Kazi Rafat, Sadia Islam, Abdullah Al Mahfug, Md. Ismail Hossain, Fuad Rahman, Sifat Momen, Shafin Rahman, Nabeel Mohammed
Summary: Deep learning techniques have achieved remarkable success in various domains, but they often come with substantial energy costs and carbon footprint emissions. This article focuses on the environmental costs of deep learning models and proposes a method for reducing carbon footprints using knowledge distillation. By selecting an appropriate hyperparameter and applying a stochastic approach, the proposed method significantly reduces energy consumption and CO2 emissions without sacrificing performance.
Article
Computer Science, Artificial Intelligence
Maria Tzelepi, Nikolaos Passalis, Anastasios Tefas
Summary: This paper introduces a novel single-stage self knowledge distillation method called Online Subclass Knowledge Distillation (OSKD), aiming to reveal similarities inside classes to enhance the performance of deep neural models. Experimental evaluation on five datasets demonstrates that the proposed method improves classification performance, outperforming existing online distillation methods.
EXPERT SYSTEMS WITH APPLICATIONS
(2021)
Article
Computer Science, Hardware & Architecture
Jiao Xie, Shaohui Lin, Yichen Zhang, Linkai Luo
Summary: This paper proposes a method to improve the performance of compact student networks with cheap convolutions using knowledge distillation. By online constructing a teacher network with standard convolutions and conducting mutual learning between the teacher and student networks, the proposed approach achieves superior performance in reducing memory and computation overhead compared to previous CNN compression and acceleration methods on different datasets.
Article
Computer Science, Artificial Intelligence
Xiaotong Lu, Weisheng Dong, Xin Li, Jinjian Wu, Leida Li, Guangming Shi
Summary: Both network pruning and neural architecture search can be automated to design and optimize artificial neural networks. This paper challenges the conventional training before pruning by proposing a joint search-and-training approach from scratch. The proposed method achieves a better balance in efficiency and accuracy with notable advantages over current pruning methods.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
(2023)
Article
Computer Science, Artificial Intelligence
Abdolmaged Alkhulaifi, Fahad Alsahli, Irfan Ahmad
Summary: This paper presents an outlook on knowledge distillation techniques applied to deep learning models, introducing a new metric called distillation metric for comparing performances of different solutions. Interesting conclusions drawn from the survey, along with current challenges and possible research directions, are discussed in the paper.
PEERJ COMPUTER SCIENCE
(2021)
Article
Computer Science, Artificial Intelligence
Chenglong Li, Xiaobin Yang, Guohao Wang, Aihua Zheng, Chang Tan, Jin Tang
Summary: License plate recognition is crucial in various practical applications, however, recognizing license plates of large vehicles is challenging due to low resolution, contamination, low illumination, and occlusion. To address this problem, a novel data generation framework based on the Disentangled Generation Network is proposed to ensure the generation diversity and integrity for robust enlarged license plate recognition.
COMPUTER VISION AND IMAGE UNDERSTANDING
(2024)
Article
Computer Science, Artificial Intelligence
Sara Casao, Alvaro Serra-Gomez, Ana C. Murillo, Wendelin Bohmer, Javier Alonso-Mora, Eduardo Montijano
Summary: This paper presents a hybrid camera system that combines static and mobile cameras, exploiting the cooperation between tracking and control modules to achieve high-level scene understanding. The static camera network provides global awareness, while the mobile cameras enhance the information about the people on the scene.
COMPUTER VISION AND IMAGE UNDERSTANDING
(2024)
Article
Computer Science, Artificial Intelligence
Anh-Dzung Doan, Bach Long Nguyen, Surabhi Gupta, Ian Reid, Markus Wagner, Tat-Jun Chin
Summary: To ensure reliable object detection in autonomous systems, the detector needs to adapt to changes in appearance caused by environmental factors. We propose a selective adaptation approach using domain gap as a criterion to improve the efficiency of the detector's operation.
COMPUTER VISION AND IMAGE UNDERSTANDING
(2024)
Article
Computer Science, Artificial Intelligence
Tianhong Dai, Wei Li, Xilei Cao, Jianzhuang Liu, Xu Jia, Ales Leonardis, Youliang Yan, Shanxin Yuan
Summary: This study proposes a novel frequency-guided deep neural network (FHDRNet) for high dynamic range (HDR) imaging from multiple low dynamic range (LDR) images, aiming to address ghosting artifacts. By conducting HDR fusion in the frequency domain, the network utilizes low-frequency signals to remove specific ghosting artifacts and high-frequency signals to preserve details. Extensive experiments demonstrate that this approach achieves state-of-the-art performance.
COMPUTER VISION AND IMAGE UNDERSTANDING
(2024)
Article
Computer Science, Artificial Intelligence
Guobin Li, Reyer Zwiggelaar
Summary: Breast cancer is the most commonly diagnosed female malignancy worldwide. Recent developments in deep convolutional neural networks have shown promising performance for breast cancer detection and classification. However, biased features can be learned due to variations in appearance and small datasets. To address this issue, a densely connected convolutional network (DenseNet) was trained using texture features representing different physical morphological representations as inputs. The use of connectivity estimation and nearest neighbors improved the network's unbiased prediction. The approach achieved higher diagnostic accuracy and provided visual explanations for model predictions.
COMPUTER VISION AND IMAGE UNDERSTANDING
(2024)
Article
Computer Science, Artificial Intelligence
Yuezun Li, Cong Zhang, Honggang Qi, Siwei Lyu
Summary: Deep Neural Networks (DNNs) are vulnerable to adversarial perturbations, limiting their applicability in safe-critical scenarios. To address this, a new method called AdaNI is proposed to increase feature randomness through adaptive noise injection, improving adversarial robustness. Extensive experiments demonstrate the efficacy of AdaNI against various white-box and black-box attacks, as well as its applicability in DeepFake detection.
COMPUTER VISION AND IMAGE UNDERSTANDING
(2024)
Article
Computer Science, Artificial Intelligence
Chengyin Hu, Weiwen Shi, Ling Tian, Wen Li
Summary: In this study, we introduce a pioneering black-box light-based physical attack called Adversarial Neon Beam (AdvNB). Our method excels in attack modeling, efficient attack simulation, and robust optimization, striking a balance between robustness and efficiency. Through rigorous evaluation, we achieve impressive attack success rates in both digital and real-world scenarios. AdvNB demonstrates its stealthiness through comparisons with baseline samples and consistently achieves high success rates when targeting advanced DNN models.
COMPUTER VISION AND IMAGE UNDERSTANDING
(2024)
Article
Computer Science, Artificial Intelligence
Hang Wang, Zhenyu Ding, Cheng Cheng, Yuhai Li, Hongbin Sun
Summary: Learning-based super resolution has made remarkable progress in improving image quality, but the performance decreases when the degradation kernel changes. Blind SR networks can estimate the degradation kernel and adapt well in realistic scenarios, improving performance and runtime. This paper proposes a design that imposes constraints for the kernel estimation network in both the image domain and kernel domain, resulting in high-quality images and efficient runtime.
COMPUTER VISION AND IMAGE UNDERSTANDING
(2024)
Article
Computer Science, Artificial Intelligence
Yuantao Chen, Runlong Xia, Kai Yang, Ke Zou
Summary: This paper proposes an improved image inpainting network using a multi-scale feature module and improved attention module. The network addresses issues in deep learning-based image inpainting algorithms, such as information loss in deep level features and the neglect of semantic features. The proposed network generates better inpainting results by reducing information loss and enhancing the ability to restore texture and semantic features.
COMPUTER VISION AND IMAGE UNDERSTANDING
(2024)
Article
Computer Science, Artificial Intelligence
Yi-Tung Chan
Summary: This study proposes a novel maritime background subtraction method based on ensemble learning theory to address the challenges posed by dynamic marine environments and noise, improving the detection accuracy and enhancing maritime transportation security for autonomous ships in open waters.
COMPUTER VISION AND IMAGE UNDERSTANDING
(2024)
