Article
Biology
Kangshun Li, Can Chen, Wuteng Cao, Hui Wang, Shuai Han, Renjie Wang, Zaisheng Ye, Zhijie Wu, Wenxiang Wang, Leng Cai, Deyu Ding, Zixu Yuan
Summary: This study proposes a framework called DeAF, which separates the training of multimodal deep learning models into two stages: unsupervised representation learning with modality adaptation (MA) module, and supervised learning with self-attention fusion (SAF) module. The DeAF framework achieves significant improvement in disease prediction tasks compared to previous methods, and extensive experiments demonstrate its rationality and effectiveness.
COMPUTERS IN BIOLOGY AND MEDICINE
(2023)
Article
Biology
Xiuquan Du, Yuefan Zhao
Summary: With the increasing incidence of breast cancer, accurate prognosis prediction plays a significant role in cancer research, psychological rehabilitation, and clinical decision-making for patients. Integrating data from different modalities has shown greater success in prognostic prediction compared to using only one modality. However, existing approaches often fail to reduce the modality gap, highlighting the need for a method that effectively integrates multimodal data. This study proposes a multimodal data adversarial representation framework (MDAR) to reduce modality heterogeneity and improve prognostic performance by aligning distributions. Experimental results on the METABRIC dataset demonstrate enhanced prognostic prediction of breast cancer patients using the proposed method.
COMPUTERS IN BIOLOGY AND MEDICINE
(2023)
Article
Oncology
Stefan Schulz, Ann-Christin Woerl, Florian Jungmann, Christina Glasner, Philipp Stenzel, Stephanie Strobl, Aurelie Fernandez, Daniel-Christoph Wagner, Axel Haferkamp, Peter Mildenberger, Wilfried Roth, Sebastian Foersch
Summary: This study developed and evaluated a multimodal deep learning model (MMDLM) for prognosis prediction in ccRCC. The model showed great performance in predicting the prognosis of ccRCC patients and could potentially improve clinical management.
FRONTIERS IN ONCOLOGY
(2021)
Article
Biochemical Research Methods
Ruiqing Li, Xingqi Wu, Ao Li, Minghui Wang
Summary: This study proposes a novel hierarchical multimodal fusion approach, which addresses the limitations of existing methods in terms of computational cost and mining complex relations from multimodal data. The proposed method, named HFBSurv, employs factorized bilinear model to progressively fuse genomic and image features, leading to a more specialized fusion procedure and expressive multimodal representation. By designing modality-specific and cross-modality attentional factorized bilinear modules, the method effectively captures and quantifies the complex relations in multimodal data, achieving consistently better performance for survival prediction.
Article
Multidisciplinary Sciences
Luis A. Vale-Silva, Karl Rohr
Summary: MultiSurv is a multimodal deep learning method for long-term pan-cancer survival prediction that can handle missing data and achieve accurate results, as well as provide insights on cancer characteristics and heterogeneity through visualizations of learned multimodal representations.
SCIENTIFIC REPORTS
(2021)
Article
Computer Science, Artificial Intelligence
Miguel Vasco, Hang Yin, Francisco S. Melo, Ana Paiva
Summary: This work presents a novel hierarchical generative model NEXUS for cross-modality inference and introduces a benchmark dataset MHD for evaluation. Results show that NEXUS outperforms current state-of-the-art multimodal generative models in cross-modality inference capabilities.
Article
Engineering, Industrial
Wei Zhang, Xiang Li, Hui Ma, Zhong Luo, Xu Li
Summary: This paper proposes a transfer learning method for remaining useful life predictions using deep representation regularization. By aligning the life-cycle data of different machine entities across different operating conditions, prognostic knowledge transfer is achieved.
RELIABILITY ENGINEERING & SYSTEM SAFETY
(2021)
Article
Computer Science, Artificial Intelligence
Divya Saxena, Jiannong Cao
Summary: This article proposes a stochastic spatio-temporal generative model called D-GAN, which accurately predicts spatio-temporal data with multiple time steps. D-GAN adopts a GANs-based structure to capture the multimodal nature of future contents and supports fusion of external factors, improving the model learning.
ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY
(2022)
Article
Biochemistry & Molecular Biology
Yuzhi Guo, Jiaxiang Wu, Hehuan Ma, Sheng Wang, Junzhou Huang
Summary: The significance of protein secondary structure, successful models in protein sequence study area, and novel methods like CondGCNN and ASP network were discussed in this paper. Experimental results showed that the proposed method achieved higher performance in protein secondary structure prediction tasks.
Article
Multidisciplinary Sciences
Alexander S. Garruss, Katherine M. Collins, George M. Church
Summary: Recent advances in DNA synthesis and sequencing have allowed for systematic studies of protein function at a large scale. A deep learning approach was used to predict the impact of protein variants on transcriptional repression, with promising results suggesting the potential for improving predictions of other important protein properties.
PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA
(2021)
Review
Biochemical Research Methods
Soren Richard Stahlschmidt, Benjamin Ulfenborg, Jane Synnergren
Summary: Biomedical data are increasingly multimodal, and deep learning-based data fusion strategies are effective in capturing their complex relationships, especially joint representation learning which models the interactions between different levels of biological organization.
BRIEFINGS IN BIOINFORMATICS
(2022)
Article
Biochemical Research Methods
Linlin Zhang, Chunping Ouyang, Yongbin Liu, Yiming Liao, Zheng Gao
Summary: This paper introduces a multimodal information fusion model (FMDTA) for predicting the strength of drug-target binding. The model utilizes the information from different modalities of drugs and targets and balances the feature representations using contrastive learning. Experimental results demonstrate that FMDTA outperforms the state-of-the-art model.
Article
Computer Science, Artificial Intelligence
Surupendu Gangopadhyay, Prasenjit Majumder
Summary: This paper explores the relationship between news articles and the direction of close price movement in the stock market. It proposes a parallel CNN model that combines text and price representation to accurately predict the close price movement. The results show strong correlation between the predicted and actual time series, indicating the effective utilization of economic events in news articles for predicting stock market movement.
EXPERT SYSTEMS WITH APPLICATIONS
(2023)
Article
Computer Science, Artificial Intelligence
Daheng Wang, Tong Zhao, Wenhao Yu, Nitesh Chawla, Meng Jiang
Summary: Complementarity is crucial in complex data objects, but learning complementarity in multimodal data poses challenges. Existing metrics fail to capture complementarity adequately. In this research, a novel deep architecture is proposed to systematically learn complementarity from multimodal multi-item data. The model consists of three modules: unimodal aggregation, cross-modal fusion, and interactive aggregation. Experimental results show superior performance compared to state-of-the-art methods in object classification and item prediction tasks.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Geochemistry & Geophysics
Mengru Ma, Wenping Ma, Licheng Jiao, Xu Liu, Fang Liu, Lingling Li, Shuyuan Yang, Biao Hou
Summary: In this article, a transfer representation learning fusion network (TRLF-Net) is proposed for multisource remote sensing images collaborative classification. It enhances the representation ability and accuracy of image features through the design of a dual-branch attention sparse transfer module, a deep dual-scale decomposition module, and a representation fusion of the global and local features module.
IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING
(2022)
Article
Oncology
Almudena Espin-Perez, Kevin Brennan, Asiri Saumya Ediriwickrema, Olivier Gevaert, Izidore S. Lossos, Andrew J. Gentles
Summary: The lack of accurate early detection methods for lymphoma limits the ability to cure patients. A DNA methylation-based prediction tool for NHL was developed, which showed high accuracy in identifying patients at risk of developing future NHL and detecting active NHL and healthy status.
NPJ PRECISION ONCOLOGY
(2022)
Article
Engineering, Biomedical
Xianghao Zhan, Yuzhe Liu, Nicholas J. Cecchi, Olivier Gevaert, Michael M. Zeineh, Gerald A. Grant, David B. Camarillo
Summary: The study utilized principal component analysis (PCA) to analyze the spatial co-variation of injury metrics in four types of head impacts, aiding in the improvement of the machine learning head model (MLHM). PCA-MLHM reduced model parameters by 74% with comparable MPS estimation accuracy.
IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING
(2022)
Article
Computer Science, Information Systems
Xianghao Zhan, Fanjin Wang, Olivier Gevaert
Summary: Drug-induced liver injury refers to the adverse effects of drugs on the liver, and it is important to assess new drug candidates. This study developed a model using natural language processing techniques to rapidly filter literature and find relevant information about liver injury induced by medications. The ensemble model and TF-IDF model achieved satisfactory classification results.
IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS
(2022)
Article
Multidisciplinary Sciences
Sabrina H. Rossi, Izzy Newsham, Sara Pita, Kevin Brennan, Gahee Park, Christopher G. Smith, Radoslaw P. Lach, Thomas Mitchell, Junfan Huang, Anne Babbage, Anne Y. Warren, John T. Leppert, Grant D. Stewart, Olivier Gevaert, Charles E. Massie, Shamith A. Samarajiwa
Summary: Current diagnostic strategies are unable to differentiate between benign and malignant small renal masses accurately, leading to unnecessary surgery in 20% of patients. The MethylBoostER machine learning model, utilizing DNA methylation data, can classify pathological subtypes of renal tumors and provide a more confident presurgical diagnosis, potentially improving treatment decision-making.
Article
Biology
Lydia J. Wilson, Frederico C. Kiffer, Daniel C. Berrios, Abigail Bryce-Atkinson, Sylvain V. Costes, Olivier Gevaert, Bruno F. E. Matarese, Jack Miller, Pritam Mukherjee, Kristen Peach, Paul N. Schofield, Luke T. Slater, Britta Langen
Summary: The era of high-throughput techniques has generated large amounts of data in the medical and research fields. Machine intelligence (MI) approaches are being used to overcome limitations in processing, analyzing, and interpreting these massive data sets. The 67th Annual Meeting of the Radiation Research Society featured a symposium on MI approaches, highlighting recent advancements in radiation sciences and their clinical applications. This article summarizes three presentations on metadata processing and ontological formalization, data mining for radiation outcomes in pediatric oncology, and imaging in lung cancer.
INTERNATIONAL JOURNAL OF RADIATION BIOLOGY
(2023)
Article
Oncology
Chen Chen, June Ho Shin, Zhuoqing Fang, Kevin Brennan, Nina B. Horowitz, Kathleen L. Pfaff, Emma L. Welsh, Scott J. Rodig, Olivier Gevaert, Or Gozani, Ravindra Uppaluri, John B. Sunwoo
Summary: In head and neck squamous cell carcinoma (HNSCC), inactivating mutations in the histone methyltransferase NSD1 disproportionately contribute to tumor development and immune exclusion. Understanding the NSD1-mediated mechanism and targeting the histone-modifying enzyme KDM2A could enhance T-cell infiltration and suppress tumor growth in HNSCC.
Article
Biochemistry & Molecular Biology
Alexander H. H. Thieme, Yuanning Zheng, Gautam Machiraju, Chris Sadee, Mirja Mittermaier, Maximilian Gertler, Jorge L. Salinas, Krithika Srinivasan, Prashnna Gyawali, Francisco Carrillo-Perez, Angelo Capodici, Maximilian Uhlig, Daniel Habenicht, Anastassia Loeser, Maja Kohler, Maximilian Schuessler, David Kaul, Johannes Gollrad, Jackie Ma, Christoph Lippert, Kendall Billick, Isaac Bogoch, Tina Hernandez-Boussard, Pascal Geldsetzer, Olivier Gevaert
Summary: A deep-learning algorithm, MPXV-CNN, was developed to identify skin lesions caused by the mpox virus for early detection and mitigation. It demonstrated a sensitivity of 0.83-0.91 and a specificity of 0.965-0.898 across different datasets. The algorithm was robust in classifying lesions on various skin tones and body regions, and a web-based app was developed for patient guidance.
Article
Multidisciplinary Sciences
Kexin Ding, Mu Zhou, He Wang, Olivier Gevaert, Dimitris Metaxas, Shaoting Zhang
Summary: To enhance computational pathology, we introduce a large-scale synthetic pathological image dataset paired with nucleus annotations, called SNOW. By applying off-the-shelf image generator and nuclei annotator, SNOW offers a cost-effective means to improve model performance. Results show that models trained on synthetic data are competitive and expand the use of synthetic images for data-driven clinical tasks.
Article
Computer Science, Artificial Intelligence
Sandra Steyaert, Marija Pizurica, Divya Nagaraj, Priya Khandelwal, Tina Hernandez-Boussard, Andrew J. Gentles, Olivier Gevaert
Summary: Cancer diagnosis and treatment decisions often focus on a single data source. However, there is a need for effective multimodal fusion approaches to integrate complementary data types. The current technological advances and introduction of deep learning have the potential to address the challenges of data integration in cancer research.
NATURE MACHINE INTELLIGENCE
(2023)
Article
Dermatology
Michelle Phung, Vijaytha Muralidharan, Veronica Rotemberg, Roberto Andres Novoa, Albert Sean Chiou, Christoph Y. Sadee, Bailie Rapaport, Kiana Yekrang, Jared Bitz, Olivier Gevaert, Justin Meng Ko, Roxana Daneshjou
Summary: Recent developments in artificial intelligence research have led to the increased use of algorithms for detecting malignancies in clinical and dermoscopic images of skin diseases. Gathering training and testing data is crucial for these methods. This paper explores the best practices and challenges in collecting skin images and data for translational artificial intelligence research, including ethics, image acquisition, labeling, curation, and storage. The aim is to enhance malignancy detection using artificial intelligence by facilitating intentional data collection and collaboration between dermatologists and data scientists.
JOURNAL OF INVESTIGATIVE DERMATOLOGY
(2023)
Article
Multidisciplinary Sciences
Yuanning Zheng, Francisco Carrillo-Perez, Marija Pizurica, Dieter Henrik Heiland, Olivier Gevaert
Summary: In this study, two deep learning models were used to predict the transcriptional subtypes and prognosis of glioblastoma (GBM) cells from histology images. The results showed consistent associations between spatial cellular organization and patient prognosis. The study also confirmed that transcriptional heterogeneity and cell-state plasticity are key factors in the development of therapeutic resistance in GBM.
NATURE COMMUNICATIONS
(2023)
Article
Biochemical Research Methods
Yuanning Zheng, John Jun, Kevin Brennan, Olivier Gevaert
Summary: DNA methylation is an important epigenetic factor that affects gene expression, and alterations in it can lead to cancer and immunological and cardiovascular diseases. Recent advancements in technology have made it possible to analyze DNA methylation on a genome-wide scale in large human cohorts. This study presents an analytical framework called EpiMix, which provides higher sensitivity in detecting abnormal DNA methylation patterns in small subsets of patients compared to existing methods. The researchers also used EpiMix to analyze cis-regulatory elements, enhancers, and genes encoding microRNAs and long non-coding RNAs, and discovered epigenetic mechanisms underlying childhood food allergy and survival-associated, methylation-driven ncRNAs in non-small cell lung cancer.
CELL REPORTS METHODS
(2023)
Article
Medicine, Research & Experimental
Sandra Steyaert, Yeping Lina Qiu, Yuanning Zheng, Pritam Mukherjee, Hannes Vogel, Olivier Gevaert
Summary: Steyaert, Qiu et al. developed a deep learning framework for multimodal data fusion in brain tumors. Combining histopathology imaging and gene expression data, the multimodal data models outperformed single data models in predicting prognosis.
COMMUNICATIONS MEDICINE
(2023)
Article
Biochemical Research Methods
Shaimaa Bakr, Kevin Brennan, Pritam Mukherjee, Josepmaria Argemi, Mikel Hernaez, Olivier Gevaert
Summary: In this study, we propose SparseGMM, a statistical approach that uses latent variable modeling with sparsity constraints to learn Gaussian mixtures from multiomic data, aiming to improve our understanding of diseases with genetic underpinnings. By combining coexpression patterns with a Bayesian framework, SparseGMM quantitatively measures confidence in regulators and uncertainty in target gene assignment by computing gene entropy. We apply SparseGMM to liver cancer and normal liver tissue data and identify regulators of angiogenesis, immune response, and blood coagulation in cancer. Furthermore, we show that high-entropy genes in cancer include key multifunctional components shared by critical pathways.
CELL REPORTS METHODS
(2023)
Meeting Abstract
Oncology
Ravi Parikh, Petr Jordan, Rita Ciaravino, Ryan Beasley, Arpan Patel, Dwight Owen, Arya Amini, Brendan Curti, Ray Page, Aurelie Swalduz, Jean-Paul Beregi, Jan Chrusciel, Eric Snyder, Pritam Mukherjee, Heather Selby, Soohee Lee, Roshanthi Weerasinghe, Shwetha Pindikuri, Jakob Weiss, Andrew Wentland, Anish Kirpalani, An Liu, Olivier Gevaert, George Simon, Hugo Aerts
JOURNAL FOR IMMUNOTHERAPY OF CANCER
(2022)