Article
Computer Science, Information Systems
Xueying Wang, Yudong Guo, Zhongqi Yang, Juyong Zhang
Summary: In this paper, a prior-guided implicit neural rendering network is proposed to recover a high-fidelity 3D head model using a few multi-view portrait images as input. By utilizing human head priors including facial prior knowledge, head semantic segmentation information, and 2D hair orientation maps, the proposed method achieves improved reconstruction accuracy and robustness.
IEEE TRANSACTIONS ON MULTIMEDIA
(2022)
Article
Computer Science, Software Engineering
Julien Philip, Sebastien Morgenthaler, Michael Gharbi, George Drettakis
Summary: The neural relighting algorithm introduced in this study enables interactive free-viewpoint navigation in captured indoors scenes, allowing synthetic changes in illumination while maintaining coherent rendering of shadows and glossy materials. By utilizing both image-based and physically based rendering elements, along with a three-dimensional mesh obtained through multiview stereo reconstruction, the method facilitates learning of an implicit representation of scene materials and illumination.
ACM TRANSACTIONS ON GRAPHICS
(2021)
Article
Geography, Physical
Anzhu Yu, Wenyue Guo, Bing Liu, Xin Chen, Xin Wang, Xuefeng Cao, Bingchuan Jiang
Summary: Our proposed approach introduces a coarse-to-fine depth inference strategy to achieve high resolution depth maps, and experimental results on multiple datasets show that our method outperforms most existing methods.
ISPRS JOURNAL OF PHOTOGRAMMETRY AND REMOTE SENSING
(2021)
Article
Computer Science, Software Engineering
Sebastian Weiss, Ruediger Westermann
Summary: This paper presents a differentiable volume rendering solution that allows for differentiation of all parameters of the rendering process. The approach is tailored for volume rendering and facilitates automatic optimization of parameters and volumetric density field. The effectiveness of the method is demonstrated through experiments and comparisons with other techniques.
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS
(2022)
Article
Computer Science, Software Engineering
Beibei Sun, Ping Jiang, Dali Kong, Ting Shen
Summary: Researchers propose a voxel-based network, IV-Net, for single-view 3D reconstruction. This network combines features from images and recovered volumes, and utilizes multi-scale convolutional blocks and an IV refiner to improve the accuracy of shape and detail reconstruction.
Article
Computer Science, Artificial Intelligence
Kai-En Lin, Guowei Yang, Lei Xiao, Feng Liu, Ravi Ramamoorthi
Summary: Image view synthesis has been successful in reconstructing realistic visuals, but view synthesis of dynamic scenes presents challenges due to lack of high-quality training datasets and a time dimension for videos. Researchers have introduced a multi-view video dataset and a new algorithm that enables stable view extrapolation from dynamic scene videos captured by static cameras. Their method operates in 3D space and demonstrates better temporal stability and visual effects compared to traditional methods.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
(2023)
Article
Computer Science, Artificial Intelligence
Jinyu Zhao, Yusuke Monno, Masatoshi Okutomi
Summary: This paper proposes a novel 3D reconstruction method that effectively utilizes geometric, photometric, and polarimetric cues extracted from input multi-view color-polarization images. The method estimates camera poses and an initial 3D model using standard structure-from-motion and multi-view stereo techniques, and then refines the model by optimizing photometric rendering errors and polarimetric errors. Experimental results show that the proposed method can reconstruct detailed 3D shapes without assuming specific surface materials and lighting conditions.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
(2023)
Article
Computer Science, Software Engineering
S. M. Kim, C. Choi, H. Heo, Y. M. Kim
Summary: This paper proposes a universal color transform module that maximizes the utilization of captured evidence for neural networks. Experimental results demonstrate that the learned color space significantly improves reconstruction quality, particularly in low-light and low-textured regions.
COMPUTER GRAPHICS FORUM
(2023)
Article
Chemistry, Analytical
Ashraf Siddique, Seungkyu Lee
Summary: In this paper, we propose Sym3DNet for single-view 3D reconstruction using a three-dimensional reflection symmetry structure prior. The method achieves superior efficiency and accuracy compared to state-of-the-art approaches on synthetic and real-world datasets, and demonstrates good generalization ability.
Article
Computer Science, Artificial Intelligence
Lei Han, Dawei Zhong, Lin Li, Kai Zheng, Lu Fang
Summary: This paper proposes a novel view synthesis system based on SRN, which improves the clarity and visual effects of synthesized results by learning residual color instead of radiance color.
IEEE TRANSACTIONS ON IMAGE PROCESSING
(2022)
Article
Computer Science, Artificial Intelligence
Jinzhi Zhang, Mengqi Ji, Guangyu Wang, Zhiwei Xue, Shengjin Wang, Lu Fang
Summary: This paper proposes SurRF, an unsupervised multi-view stereopsis pipeline that learns Surface Radiance Field. By defining the radiance field on a continuous and explicit 2D surface, SurRF provides a compact representation while maintaining complete shape and realistic texture, leading to competitive results for large-scale complex scenes.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
(2022)
Article
Computer Science, Artificial Intelligence
Xuancheng Zhang, Rui Ma, Changqing Zou, Minghao Zhang, Xibin Zhao, Yue Gao
Summary: Reconstructing 3D shape from a single-view image using deep learning has gained popularity, but existing methods suffer from the lack of explicit structure modeling and loss of view information. In this paper, we propose VGSNet, an encoder-decoder architecture that jointly learns the feature representation of 2D image and 3D shape to achieve geometry and structure reconstruction from a single-view image.
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
(2022)
Article
Computer Science, Software Engineering
Sebastian Weiss, Mustafa Isk, Justus Thies, Rudiger Westermann
Summary: This article introduces a novel neural rendering pipeline that predicts the sampling positions for data visualization through learning the correspondence between data, sampling patterns, and generated images. By leveraging differentiable sampling and reconstruction stages, it facilitates joint learning of relevant structure selection and image reconstruction, enabling adaptive sampling and high-resolution image generation.
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS
(2022)
Article
Computer Science, Information Systems
Daixian Zhu, Haoran Kong, Qiang Qiu, Xiaoman Ruan, Shulin Liu
Summary: This paper introduces a learning-based multi-view stereo (MVS) algorithm based on attention mechanism and neural volume rendering. The algorithm addresses the issue of incorrect feature matching and incomplete scene reconstruction by employing multi-scale feature extraction and neural volume rendering. Experimental results demonstrate superior performance and generalization capability of the proposed algorithm.
Article
Automation & Control Systems
Rongshan Chen, Yuancheng Yang, Chao Tong
Summary: In this paper, the authors propose a graph-based implicit function G2IFu, which successfully reconstructs a highly detailed 3D object mesh from a single image. By mapping graphs to implicit values, G2IFu improves the prediction accuracy of implicit functions compared to traditional point-based methods. Additionally, the authors introduce prior boundary loss and self-attention module to enhance the performance of G2IFu.
ENGINEERING APPLICATIONS OF ARTIFICIAL INTELLIGENCE
(2023)
Article
Computer Science, Information Systems
Chuanhao Zhang, Emil Jovanov, Hongen Liao, Yuan-Ting Zhang, Benny Lo, Yuan Zhang, Cuntai Guan
Summary: With the development of modern cameras, physiological signals can now be obtained from portable devices like smartphones. In this paper, a framework called cocktail causal container is proposed to fuse multiple physiological representations and reconstruct the correlation between frequency and temporal information for blood pressure and blood glucose classification. The framework utilizes a token feature fusion block and a causal net to extract discriminative features and disentangle latent factors. A pair-wise temporal frequency map is also developed for extracting PPG information. Extensive comparisons using clinical data and PPG-BP benchmark show promising results with low error rate and high precision.
IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS
(2023)
Article
Computer Science, Information Systems
Yamei Li, Shengqiong Luo, Haibo Zhang, Yinkai Zhang, Yuan Zhang, Benny Lo
Summary: The demand for automatic pediatric sleep staging has increased due to the rising incidence and recognition of children's sleep disorders. The existing supervised sleep stage recognition algorithms face challenges such as limited availability of pediatric sleep physicians and data heterogeneity. To address this, we propose a multi-task contrastive learning strategy that combines semi-supervised learning and self-supervised contrastive learning, named MtCLSS. By applying signal-adapted transformations and an extended contrastive loss function, MtCLSS learns task-specific and general features from limited labeled data, improving the robustness of the model for EEG based automatic pediatric sleep staging in limited data scenarios.
IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS
(2023)
Article
Computer Science, Information Systems
Xianmin Song, Yingnan Sun, Haitao Li, Bo Liu, Yuxuan Cao
Summary: This paper proposes an Ecological Control Unit-Cooperative Adaptive Control (ECU-CACC) system to reduce energy consumption in mixed and power-heterogeneous traffic flow. A bi-level control framework is designed to improve traffic efficiency and reduce energy consumption. Numerical experiments verify the effectiveness of the system and analyze the energy-saving effect under different vehicle mixing situations.
Article
Medicine, General & Internal
Peilun Shi, Jianing Qiu, Sai Mu Dalike Abaxi, Hao Wei, Frank P. -W. Lo, Wu Yuan
Summary: Medical image analysis is crucial in clinical diagnosis. This paper evaluates the Segment Anything Model (SAM) on various medical image segmentation benchmarks and finds that while SAM performs well on general domain images, its zero-shot segmentation ability is limited for medical images. Inconsistent performance is observed across different medical domains, with complete failure in segmentation of certain structured targets. However, fine-tuning SAM with a small amount of data leads to significant improvement, showing the potential of achieving accurate medical image segmentation for precision diagnostics.
Article
Robotics
Xuhui Zhou, Ziqi Yang, Yunxiao Ren, Weibang Bai, Benny Lo, Eric M. M. Yeatman
Summary: The presence of computation delay, transmission delay, and mechanical delay in robotic teleoperation systems is a major factor in system degradation. A neural network-based open-loop approach called BAEM has been proposed to compensate for transmission delay by sending predicted trajectories as commands. A modified version of BAEM (m-BAEM) is proposed to explicitly compensate for all three types of delay, and a real-time robotic teleoperation system based on the ROS 2 framework is built to evaluate its performance.
IEEE ROBOTICS AND AUTOMATION LETTERS
(2023)
Article
Robotics
Ruiyang Zhang, Junhong Chen, Zeyu Wang, Ziqi Yang, Yunxiao Ren, Peilun Shi, James Calo, Kyle Lam, Sanjay Purkayastha, Benny Lo
Summary: Robot-Assisted Minimally Invasive Surgery (RAMIS) has gained popularity worldwide due to its precision, ergonomics, and intuitive control. With advancements in AI and surgical robot technologies, the cognitive load on surgeons can be reduced, while improving the precision and safety of robot operations. However, research is still focused on task autonomy due to operation complexity and limited clinical data. This paper proposes a method for conditional autonomy in robotic appendectomy, utilizing demonstrated data to carry out the procedure semi-automatically.
IEEE ROBOTICS AND AUTOMATION LETTERS
(2023)
Article
Computer Science, Information Systems
James Calo, Benny Lo
Summary: This paper proposes a method using blockchain and federated learning to effectively train neural networks on IoT devices. It addresses issues of data scarcity and privacy concerns, and enables distributed training across multiple devices.
Article
Nutrition & Dietetics
Modou L. Jobarteh, Megan A. Mccrory, Benny Lo, Konstantinos K. Triantafyllidis, Jianing Qiu, Jennifer P. Griffin, Edward Sazonov, Mingui Sun, Wenyan Jia, Tom Baranowski, Alex K. Anderson, Kathryn Maitland, Gary Frost
Summary: This study validated an objective, passive image-based method for assessing dietary intake in London, UK and demonstrated its potential applicability in low- and middle-income countries (LMICs). The findings showed good agreement between the image-based method and weighed food records, indicating that this method can provide a comparable assessment of nutritional intake.
Article
Automation & Control Systems
Shuo Jiang, Zach Strout, Bin He, Daiyan Peng, Peter B. B. Shull, Benny P. L. Lo
Summary: This article proposes an IoT-based solution that utilizes shared bikes to intelligently detect road surface conditions and riding events for travel efficiency and rider safety in cities. The proposed dual stream meta learning approach solves the reliability problem with different bike types and the self-adaptive problem when classifying new classes without retraining the model. The results demonstrate high accuracy in road surface condition and riding event detection.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2023)
Article
Nutrition & Dietetics
Tonmoy Ghosh, Megan A. McCrory, Tyson Marden, Janine Higgins, Alex Kojo Anderson, Christabel Ampong Domfe, Wenyan Jia, Benny Lo, Gary Frost, Matilda Steiner-Asiedu, Tom Baranowski, Mingui Sun, Edward Sazonov
Summary: This paper presents a semi-automatic dietary assessment tool called Image to Nutrients (I2N), which uses wearable sensors to process eating events and food images for nutritional analysis. The tool provides access to multiple food databases and estimates energy intake and nutrient content.
FRONTIERS IN NUTRITION
(2023)
Article
Gastroenterology & Hepatology
Amirhosein Alian, Emilia Zari, Zeyu Wang, Enrico Franco, James P. Avery, Mark Runciman, Benny Lo, Ferdinando Rodriguez y Baena, George Mylonas
Summary: In the past four decades, the incidence of early-onset gastrointestinal cancer has increased. Mass screening colonoscopy is the most effective prevention strategy for early-stage cancer detection, but conventional endoscopy is a painful and technically challenging procedure. To overcome these limitations, technological innovation is needed in colonoscopy.
TECHNIQUES AND INNOVATIONS IN GASTROINTESTINAL ENDOSCOPY
(2023)
Article
Automation & Control Systems
Jianing Qiu, Frank P. -W. Lo, Xiao Gu, Modou L. Jobarteh, Wenyan Jia, Tom Baranowski, Matilda Steiner-Asiedu, Alex K. Anderson, Megan A. McCrory, Edward Sazonov, Mingui Sun, Gary Frost, Benny Lo
Summary: Camera-based passive dietary intake monitoring captures eating episodes, recording visual information on food type, volume, and eating behavior. However, no method incorporates these clues to provide a comprehensive dietary context. Privacy is a concern with wearable cameras. This paper proposes a privacy-preserved solution for dietary assessment, using egocentric image captioning to convert images into text descriptions and reduce privacy risks. A dataset is built for egocentric dietary image captioning, and a transformer-based architecture is designed and evaluated for effectiveness.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Computer Science, Information Systems
Frank Po Wen Lo, Yao Guo, Yingnan Sun, Jianing Qiu, Benny Lo
Summary: Dietary assessment is effective for evaluating the dietary intake of patients with diabetes and obesity. However, traditional methods have limitations, so researchers proposed an intelligent nutritional assessment approach using weakly-supervised point cloud completion. This method shows promising results in estimating food volume and can be implemented using wearable and handheld cameras.
IEEE TRANSACTIONS ON MULTIMEDIA
(2023)
Proceedings Paper
Automation & Control Systems
Xiao Gu, Jinpei Han, Guang-Zhong Yang, Benny Lo
Summary: This paper proposes a method for human movement intention recognition using motor imagery electroencephalogram. Two networks are developed to handle the heterogeneity of inter-subject and inter-dataset, and an online knowledge co-distillation framework is used for collaborative learning, achieving better generalization performance.
2023 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION (ICRA 2023)
(2023)