Article
Computer Science, Artificial Intelligence
Zhan Shi, Zhanshan Wang
Summary: This paper investigates an adaptive output-feedback optimal control problem for a class of continuous time (CT) linear systems with dynamic uncertainties. An algorithm based on adaptive dynamic programming (ADP) technique is proposed for data-based controller design, which only uses measured input and output information to learn optimal control gain without requiring exact system knowledge. The adaptive controllers learned by the algorithm exhibit robustness to dynamic uncertainties, as demonstrated through three examples.
Article
Automation & Control Systems
Shan Xue, Biao Luo, Derong Liu, Ying Gao
Summary: The article introduces an event-triggered constrained optimal tracking control algorithm using integral reinforcement learning (IRL) which transforms the problem into an optimal regulation one and solves the Hamilton-Jacobi-Bellman equation. The event-triggering mechanism helps ease the pressure on data transmission, making it suitable for control systems with limited computational and communication resources.
INTERNATIONAL JOURNAL OF ROBUST AND NONLINEAR CONTROL
(2021)
Article
Automation & Control Systems
Derong Liu, Shan Xue, Bo Zhao, Biao Luo, Qinglai Wei
Summary: This article reviews the recent development of adaptive dynamic programming (ADP) with applications in control, highlighting efficient algorithms and future research directions. ADP is applied in optimization, game theory, and large-scale systems, showing great potential in the era of artificial intelligence.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2021)
Article
Computer Science, Artificial Intelligence
Jingliang Duan, Zhengyu Liu, Shengbo Eben Li, Qi Sun, Zhenzhong Jia, Bo Cheng
Summary: This paper presents a constrained adaptive dynamic programming algorithm that can directly handle state-constrained nonlinear nonaffine optimal control problems. By transforming the traditional policy improvement process into a constrained policy optimization problem and approximating the policy and value functions with multi-layer neural networks, the algorithm linearizes the constrained optimization problem and obtains optimal updates.
Editorial Material
Computer Science, Artificial Intelligence
Jie Lu, Joao Gama, Xin Yao, Leandro Minku
Summary: In recent years, stream learning has significantly developed in both conceptual and application levels, becoming a hot research direction in machine learning and data science. Advancements include detecting concept drift, adapting to drifts, and utilizing online, active, incremental, and reinforcement learning in data streaming scenarios.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Computer Science, Artificial Intelligence
Fuyu Zhao, Weinan Gao, Zhong-Ping Jiang, Tengfei Liu
Summary: This article introduces an event-triggered output-feedback adaptive optimal control method for continuous-time linear systems. It reconstructs unmeasurable states and reduces controller updates through an event-based feedback strategy. The iterative solution to the discrete-time algebraic Riccati equation is carried out using event-triggered adaptive dynamic programming, with convergence and closed-loop stability verified using Lyapunov techniques.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2021)
Article
Computer Science, Information Systems
Zhongyang Wang, Yunjun Yu
Summary: This paper introduces a data-driven adaptive optimal control approach for CVCF inverters, which effectively handles dynamic uncertainties and designs robust state-feedback controllers. The simulation results show that the proposed controller exhibits inherent robustness and does not require re-adjustment in different applications.
Article
Computer Science, Information Systems
Zongsheng Huang, Weiwei Bai, Tieshan Li, Yue Long, C. L. Philip Chen, Hongjing Liang, Hanqing Yang
Summary: This paper considers the reinforcement learning-based prescribed performance optimal tracking control problem for a class of strict-feedback nonlinear systems. The unknown nonlinearities and cost function are approximated by radial-basis-function (RBF) neural networks. The overall controller consists of an adaptive controller and an optimal compensation term, designed using backstepping control method and policy iteration, respectively. The prescribed performance control ensures that the tracking error is limited within the prescribed area, guaranteeing convergence to a bound with prescribed performance while minimizing the cost function. Stability analysis shows that all signals in the closed-loop system are bounded. Simulation examples illustrate the effectiveness and advantages of the designed control strategy.
INFORMATION SCIENCES
(2023)
Article
Computer Science, Artificial Intelligence
Qinglai Wei, Tao Li
Summary: This research focuses on optimal control problems (OCPs) with constrained cost for discrete-time nonlinear systems. A novel value iteration with constrained cost (VICC) method is developed to solve the optimal control law with the constrained cost functions. The VICC method is initialized through a value function constructed by a feasible control law. It is proven that the iterative value function is nonincreasing and converges to the solution of the Bellman equation with constrained cost. The feasibility of the iterative control law is proven, and the method to find the initial feasible control law is given. Implementation using neural networks (NNs) is introduced, and the convergence is proven by considering the approximation error. Finally, the property of the present VICC method is shown by two simulation examples.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Automation & Control Systems
Zhongyang Wang, Yunjun Yu, Weinan Gao, Masoud Davari, Chao Deng
Summary: This article proposes an adaptive, optimal, data-driven control approach based on reinforcement learning and adaptive dynamic programming for the three-phase grid-connected inverter employed in virtual synchronous generators (VSGs). Comparative simulations and experimental results validate the effectiveness and practicality of the proposed method.
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS
(2022)
Article
Automation & Control Systems
Yuhong Tang, Xiong Yang, Na Dong
Summary: This article presents a decentralized neuro-control scheme for solving the control problem of a class of continuous-time nonlinear systems with mismatched interconnections through an optimal control method. The decentralized control consisting of optimal control policies of auxiliary subsystems can stabilize the entire interconnected system. An adaptive dynamic programming algorithm is used to solve the Hamilton-Jacobi-Bellman equations, and critic neural networks are employed to approximate the value functions of auxiliary subsystems.
OPTIMAL CONTROL APPLICATIONS & METHODS
(2022)
Article
Computer Science, Information Systems
Qi Zhang, Yang Yang, Xiaoran Xie, Chunming Xu, Han Yang
Summary: In this paper, the consensus optimal control problem for linear multi-agent systems with directed communication networks is studied using adaptive dynamic programming. By utilizing a dynamic event-triggered control law and a novel adaptive dynamic programming approach, the limitations of agent processing capability and actuator lifetime are effectively overcome.
Article
Automation & Control Systems
Ruyue Yang, Ding Wang, Junfei Qiao
Summary: In this article, a new method is proposed to improve the control performance of wastewater treatment plants and reduce the need for system modeling.
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS
(2022)
Article
Automation & Control Systems
Bo Zhao, Guang Shi, Derong Liu
Summary: This article investigates local control problems for nonlinear interconnected systems by using adaptive dynamic programming (ADP) with particle swarm optimization (PSO). It constructs a proper local value function and employs a local critic neural network to solve the local Hamilton-Jacobi-Bellman equation. The event-triggering mechanism is introduced to determine the sampling time instants and ensure asymptotic stability through Lyapunov stability analysis.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2023)
Article
Automation & Control Systems
Qi Zhang, Yang Yang, Xue Song, Xiaoran Xie, Naibo Zhu, Zhi Liu
Summary: The purpose of this article is to use adaptive dynamic programming to solve the optimal consensus problem for double-integrator multiagent systems with completely unknown dynamics. Flocking algorithms that neglect agents' inertial effect in double-integrator multiagent systems can cause unstable group behavior. Despite the existence of an inertia-independent protocol, its control law is determined by dynamics and inertia. However, accurately measuring inertia in reality is difficult, so adaptive dynamic programming is developed to enable the consensus of agents in the presence of entirely unknown dynamics.
OPTIMAL CONTROL APPLICATIONS & METHODS
(2023)
Article
Automation & Control Systems
Shunchao Zhang, Bo Zhao, Derong Liu, Cesare Alippi, Yongwei Zhang
Summary: In this article, an event-triggered robust control (ETRC) method is investigated for multi-player nonzero-sum games of continuous-time input constrained nonlinear systems with mismatched uncertainties. The method transforms the robust control problem into an optimal regulation problem by constructing an auxiliary system and designing an appropriate value function. A critic neural network (NN) is used to approximate the value function of each player and obtain control laws. The method reduces computational burden and communication bandwidth by updating the control laws when events occur. The effectiveness of the developed ETRC method is demonstrated through theoretical analysis and examples.
INTERNATIONAL JOURNAL OF ROBUST AND NONLINEAR CONTROL
(2023)
Article
Computer Science, Artificial Intelligence
Qiuye Wu, Bo Zhao, Derong Liu, Marios M. Polycarpou
Summary: This paper proposes an event-triggered adaptive dynamic programming method to solve the decentralized tracking control problem for input constrained unknown nonlinear interconnected systems. A neural-network-based local observer is established to reconstruct the system dynamics using local input-output data and desired trajectories. The DTC problem is transformed into an optimal control problem using a nonquadratic value function. The DTC policy is obtained by solving the local Hamilton-Jacobi-Bellman equation through the observer-critic architecture, with weights tuned by the experience replay technique. Simulation examples demonstrate the effectiveness of the proposed scheme.
Article
Computer Science, Artificial Intelligence
Mingduo Lin, Bo Zhao, Derong Liu
Summary: A novel policy gradient (PG) adaptive dynamic programming method is proposed for nonlinear discrete-time zero-sum games with unknown dynamics. A policy iteration algorithm is used to approximate the Q-function and the control and disturbance policies using neural network approximators. The control and disturbance policies are then updated using the PG method based on the iterative Q-function. The experience replay technique is applied to improve training stability and data usage efficiency. Simulation results show the effectiveness of the proposed method.
Article
Computer Science, Artificial Intelligence
Mingming Liang, Derong Liu
Summary: This article presents a novel neural-network-based optimal event-triggered impulsive control method. The proposed method utilizes a general-event-based impulsive transition matrix (GITM) to represent the evolving characteristics of all system states across impulsive actions. Through the developed event-triggered impulsive adaptive dynamic programming (ETIADP) algorithm and its high-efficiency version (HEIADP), the optimization problems for stochastic systems with event-triggered impulsive controls are addressed. The results show that the proposed methods can reduce computational and communication burdens and fulfill the desired goals.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Automation & Control Systems
Mingming Ha, Ding Wang, Derong Liu
Summary: Inspired by the successive relaxation method, a novel discounted iterative adaptive dynamic programming framework is developed, which possesses an adjustable convergence rate for the iterative value function sequence. The convergence properties of the value function sequence and the stability of the closed-loop systems under the new discounted value iteration (VI) are investigated. An accelerated learning algorithm with convergence guarantee is presented based on the properties of the given VI scheme. Amidst the implementation of the new VI scheme and its accelerated learning design, value function approximation and policy improvement are involved. The performance of the developed approaches is verified using a nonlinear fourth-order ball-and-beam balancing plant, showing significant acceleration of the convergence rate of the value function and reduction in computational cost compared to traditional VI.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Automation & Control Systems
Yongwei Zhang, Bo Zhao, Derong Liu, Shunchao Zhang
Summary: In this article, the event-triggered robust control problem of unknown multiplayer nonlinear systems with constrained inputs and uncertainties is investigated using adaptive dynamic programming. A neural network-based identifier is constructed to relax the requirement of system dynamics. By designing a nonquadratic value function, the stabilization problem is converted into a constrained optimal control problem. The approximate solution of the event-triggered Hamilton-Jacobi equation is obtained using a critic network with a novel weight updating law, and the Lyapunov stability theorem ensures that the multiplayer system is uniformly ultimately bounded.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Automation & Control Systems
Mingming Liang, Yonghua Wang, Derong Liu
Summary: In this study, a novel general impulsive transition matrix is defined to reveal the transition dynamics and probability distribution evolution patterns between impulsive events. Based on this matrix, policy iteration-based impulsive adaptive dynamic programming algorithms are developed to solve optimal impulsive control problems. The algorithms demonstrate convergence to the optimal impulsive performance index function and allow for optimization on computing devices with low memory spaces.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Proceedings Paper
Automation & Control Systems
Jinquan Lin, Bo Zhao, Derong Liu
Summary: In this paper, an integral reinforcement learning (IRL)-based approximate optimal control (AOC) method is developed for unknown nonaffine systems using dynamic feedback. The optimal control policy for nonaffine systems cannot be explicitly expressed due to the unknown input gain matrix. Thus, a dynamic feedback signal is introduced to transform the nonaffine system into an augmented affine system. The AOC for unknown nonaffine systems is formulated by designing an appropriate value function for the augmented affine system, and the IRL method is adopted to derive the approximate solution of the Hamilton-Jacobi-Bellman equation.
2023 IEEE 12TH DATA DRIVEN CONTROL AND LEARNING SYSTEMS CONFERENCE, DDCLS
(2023)
Article
Automation & Control Systems
Bo Zhao, Guang Shi, Derong Liu
Summary: This article investigates local control problems for nonlinear interconnected systems by using adaptive dynamic programming (ADP) with particle swarm optimization (PSO). It constructs a proper local value function and employs a local critic neural network to solve the local Hamilton-Jacobi-Bellman equation. The event-triggering mechanism is introduced to determine the sampling time instants and ensure asymptotic stability through Lyapunov stability analysis.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2023)
Article
Automation & Control Systems
Hui Wang, Zhigang Liu, Zhiwei Han, Yanbo Wu, Derong Liu
Summary: Active pantograph control is a promising technique for improving train's current collection quality. Existing solutions have limitations in handling various operating conditions and lack of adaptability. In this study, a context-based deep meta-reinforcement learning algorithm is proposed to alleviate these problems. Experimental results show that the proposed algorithm can quickly adapt to new conditions and reduce contact force fluctuations.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Computer Science, Artificial Intelligence
Zhanyu Yang, Bo Zhao, Derong Liu
Summary: In this article, a novel pinning control method that requires only partial node information is developed to synchronize drive-response memristor-based neural networks with time delay. An improved mathematical model of the networks is established to accurately describe their dynamic behaviors. Unlike previous literature that requires information from all nodes, the proposed method only relies on local information to achieve synchronization of delayed networks, reducing communication and calculation burdens. Sufficient conditions for synchronization are provided, and numerical simulation and comparative experiments validate the effectiveness and superiority of the proposed method.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Automation & Control Systems
Mingduo Lin, Bo Zhao, Derong Liu
Summary: In this article, an event-triggered robust adaptive dynamic programming (ETRADP) algorithm is proposed to solve multiplayer Stackelberg-Nash games (MSNGs) for uncertain nonlinear continuous-time systems. The hierarchical decision-making process considering different roles of players is described, transforming the robust control problem into an optimal regulation problem. An online policy iteration algorithm is used to solve the derived Hamilton-Jacobi equation with an event-triggered mechanism to reduce computational and communication burdens. Critic neural networks (NNs) are constructed to obtain the event-triggered approximate optimal control policies for all players, ensuring the stability of the closed-loop system.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Automation & Control Systems
Mingming Liang, Derong Liu
Summary: This article focuses on designing the optimal impulsive controller (IMC) of continuous-time nonlinear systems and proposes a new adaptive dynamic programming algorithm with high generality and feasibility. The introduced policy-improving mechanism makes the algorithm more flexible for memory-limited computing devices.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2023)
Article
Automation & Control Systems
Ke Wang, Chaoxu Mu, Zhen Ni, Derong Liu
Summary: This paper presents a novel composite obstacle avoidance control method that generates safe motion trajectories for autonomous systems in an adaptive manner. The method combines model-based policy iteration and state-following-based approximation in a safe reinforcement learning framework. The proposed learning-based controller achieves stable reaching of target points while maintaining a safe distance from obstacles. The effectiveness of the method is demonstrated through simulations and comparisons with other avoidance control methods.
IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING
(2023)
Article
Computer Science, Artificial Intelligence
Bo Zhao, Yongwei Zhang, Derong Liu
Summary: This article presents a cooperative motion/force control scheme for modular reconfigurable manipulators (MRMs) based on adaptive dynamic programming (ADP). The dynamic model of the entire MRM system is treated as a set of joint modules interconnected by coupling torque, and the Jacobian matrix is mapped into each joint. A neural network is used as a robust decentralized observer, and an improved local value function is constructed for each joint module. The control scheme is achieved by using force feedback compensation and is proven to be uniformly ultimately bounded through Lyapunov stability analysis.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
