4.6 Article

VenusAI: An artificial intelligence platform for scientific discovery on supercomputers

Journal

JOURNAL OF SYSTEMS ARCHITECTURE
Volume 128, Issue -, Pages -

Publisher

ELSEVIER
DOI: 10.1016/j.sysarc.2022.102550

Keywords

AI platform; Scientific discovery; Large-scale computation; Supercomputer; AI application

Funding

  1. Strategic Priority Research Program of the Chinese Academy of Sciences [XDA27000000]
  2. Beijing Natural Science Foundation-Haidian Original Innovation Joint Founda-tion, China [L182053]

Ask authors/readers for more resources

The machine learning platform has been widely used in the industrial and commercial internet fields, providing one-stop solutions for artificial intelligence applications. This paper proposes the VenusAI platform based on a heterogeneous resource scheduling framework for large-scale computing in scientific research. The platform demonstrates its advantages and performance in solving practical scientific problems.
Since the machine learning platform can provide one-stop artificial intelligence (AI) application solutions, it has been widely used in the industrial and commercial internet fields in recent years. Based on the heterogeneous accelerator cards, scientific discovery using large-scale computation and massive data is a significant tendency in the future. However, building a platform for scientific discovery remains challenging, including large-scale heterogeneous resource scheduling and support for massive multi-source data. To free researchers from tedious resource management and environmental configuration, we propose a VenusAI platform for large-scale computing scenarios in scientific research, based on heterogeneous resources scheduling framework. This paper firstly illustrates the VenusAI platform architecture design scheme based on the supercomputers and elaborates on the virtualization and containerization of the underlying hardware resources. Next, a technical framework for heterogeneous resource aggregation and scheduling is proposed. A unified resource interface in the application service layer is introduced. Considering the core three parts of the AI scenario: data, model, and computing power, modularized service decoupling is carried out. Furthermore, three types of experiments are evaluated on the supercomputers and show that the performance of the scheduling framework on virtual clusters is better than that on common clusters. Finally, three scientific discovery applications deployed on VenusAI, i.e., new energy forecasting, materials design, and unmanned aerial vehicle planning, demonstrate the advantages of the platform in solving practical scientific problems.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.6
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

Article Imaging Science & Photographic Technology

GPU Computing based fast discrete wavelet transform for l1-regularized SPIRiT reconstruction

Tiechui Yao, Li Xiao, Di Zhao, Yuzhong Sun

IMAGING SCIENCE JOURNAL (2018)

Article Energy & Fuels

A historical weather forecast dataset from the European Centre for Medium-Range Weather Forecasts (ECMWF) for energy forecasting

Dazhi Yang, Wenting Wang, Tao Hong

Summary: Weather is a crucial factor for power generation and energy consumption, and energy forecasting models often rely on numerical weather prediction. This article offers an NWP forecast dataset from ECMWF for the energy forecasting community, along with case studies on post-processing of solar forecasts.

SOLAR ENERGY (2022)

Article Energy & Fuels

A photovoltaic power output dataset: Multi-source photovoltaic power output dataset with Python toolkit

Tiechui Yao, Jue Wang, Haoyan Wu, Pei Zhang, Shigang Li, Yangang Wang, Xuebin Chi, Min Shi

Summary: The power output of PV systems is influenced by climate and weather conditions. The scarcity of datasets combining power and weather data hinders progress in solar PV research. The PVOD dataset provides multi-source high-quality data for solar energy research.

SOLAR ENERGY (2021)

Article Green & Sustainable Science & Technology

Intra-Hour Photovoltaic Generation Forecasting Based on Multi-Source Data and Deep Learning Methods

Tiechui Yao, Jue Wang, Haoyan Wu, Pei Zhang, Shigang Li, Ke Xu, Xiaoyan Liu, Xuebin Chi

Summary: This paper proposes a data-driven forecasting framework based on deep learning to enhance the accuracy of PV generation prediction by integrating multiple data sources such as time series records and satellite images.

IEEE TRANSACTIONS ON SUSTAINABLE ENERGY (2022)

Article Chemistry, Multidisciplinary

Dual-Encoder Transformer for Short-Term Photovoltaic Power Prediction Using Satellite Remote-Sensing Data

Haizhou Cao, Jing Yang, Xuemeng Zhao, Tiechui Yao, Jue Wang, Hui He, Yangang Wang

Summary: The penetration of photovoltaic (PV) energy has significantly increased in recent years due to its sustainable and clean characteristics. However, accurate short-term prediction of PV power is challenging due to the uncertainty caused by variable weather. Existing methods focus on utilizing deep neural networks to extract features from satellite images and ground measurements, but a flexible predictive framework that can handle both data structures is lacking. Therefore, this study proposes a novel dual-encoder transformer (DualET) for short-term PV power prediction, which utilizes wavelet transform and series decomposition blocks to extract informative features from image and sequence data, respectively. Additionally, a cross-domain attention module is introduced to learn the correlation between temporal features and cloud information, and attention modules are modified using sparse form and Fourier transform to improve performance. Experimental results on real-world datasets demonstrate that the proposed model outperforms other models in short-term PV power prediction.

APPLIED SCIENCES-BASEL (2023)

Proceedings Paper Computer Science, Information Systems

Secure Shell Remote Access for Virtualized Computing Environment

He Li, Rongqiang Cao, Hanwen Xiu, Meng Wan, Kai Li, Xiaoguang Wang, Yangang Wang, Jue Wang

Summary: SSHRA is a secure shell remote access information system for virtualized computing environment, which provides more convenient remote login and enhances the security through certificate-based authentication.

SMART COMPUTING AND COMMUNICATION (2022)

Proceedings Paper Computer Science, Information Systems

Sci-Base: A Resource Aggregation and Sharing Ecology for Software on Discovery Science

Meng Wan, Jiaheng Wang, Jue Wang, Rongqiang Cao, Yangang Wang, He Li

Summary: This paper discusses the importance of scientific software in the process of open science and Open research, and proposes a design scheme for a safe and controllable support platform for open-source scientific software. China has made breakthroughs in theory and technology, support platform, ecosystem, and operation system, and created a proven scientific software ecosystem. The progress of research transparency and maturity of scientific software contribute to the collaboration between community developers and the cultivation of scientific research talents.

SMART COMPUTING AND COMMUNICATION (2022)

Proceedings Paper Computer Science, Information Systems

Defects Detection System of Medical Gloves Based on Deep Learning

Jing Wang, Meng Wan, Jue Wang, Xiaoguang Wang, Yangang Wang, Fang Liu, Weixiao Min, He Lei, Lihua Wang

Summary: This paper presents a surface defect detection system for medical gloves based on deep learning, achieving high efficiency and accuracy through improved real-time performance, a dual model detection strategy, and the use of auxiliary models.

SMART COMPUTING AND COMMUNICATION (2022)

No Data Available