Article
Automation & Control Systems
Yongwei Zhang, Bo Zhao, Derong Liu, Shunchao Zhang
Summary: In this article, the event-triggered robust control problem of unknown multiplayer nonlinear systems with constrained inputs and uncertainties is investigated using adaptive dynamic programming. A neural network-based identifier is constructed to relax the requirement of system dynamics. By designing a nonquadratic value function, the stabilization problem is converted into a constrained optimal control problem. The approximate solution of the event-triggered Hamilton-Jacobi equation is obtained using a critic network with a novel weight updating law, and the Lyapunov stability theorem ensures that the multiplayer system is uniformly ultimately bounded.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Computer Science, Artificial Intelligence
Qinglai Wei, Liao Zhu, Ruizhuo Song, Pinjia Zhang, Derong Liu, Jun Xiao
Summary: In this article, an online adaptive optimal control algorithm based on adaptive dynamic programming is developed to solve the multiplayer nonzero-sum game (MP-NZSG) problem for discrete-time unknown nonlinear systems. The algorithm utilizes a model-free structure and online adaptive learning to approximate the value functions and optimal policies for all players.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2022)
Article
Computer Science, Information Systems
Menghua Li, Ding Wang, Mingming Zhao, Junfei Qiao
Summary: This article investigates the optimal control problem of nonlinear continuous-time multiplayer nonzero-sum games with asymmetric control constraints by exploiting the event-triggered neural critic control scheme. The author proposes an appropriate nonquadratic function to satisfy the asymmetric constraints and deduces the time-based coupled Hamilton-Jacobi (HJ) equations and the optimal control policies. To improve control efficiency, an event-driven mechanism is innovated. The article establishes a novel triggering condition, builds critic neural networks to evaluate the optimal cost functions, and obtains the event-triggered near-optimal controls.
INFORMATION SCIENCES
(2023)
Article
Automation & Control Systems
Jiayue Sun, Huaguang Zhang, Ying Yan, Shun Xu, Xiaoxi Fan
Summary: The article investigates the optimal control strategy problem for nonzero-sum games of the immune system based on adaptive dynamic programming. It aims to approximate a Nash equilibrium between tumor cells and the immune cell population by using chemotherapy drugs and immunoagents. A novel intelligent nonzero-sum games-based ADP method is proposed to reduce the growth rate of tumor cells and minimize the usage of chemotherapy drugs and immunotherapy drugs. The feasibility of this approach is proven through convergence analysis and an iterative ADP algorithm. Simulation examples are provided to demonstrate the availability and effectiveness of the research methodology.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Computer Science, Artificial Intelligence
Qingtao Zhao, Jian Sun, Gang Wang, Jie Chen
Summary: This article proposes an event-triggered ADP method for NZS games of nonlinear systems, which can effectively approximate the Nash equilibrium solution while reducing resource occupation. The system stability and weights' convergence are guaranteed, and Zeno behavior is excluded by proving the existence of a minimum inter-event time.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2022)
Article
Computer Science, Artificial Intelligence
Chunbin Qin, Zhongwei Zhang, Ziyang Shang, Jishi Zhang, Dehua Zhang
Summary: In this paper, an optimal safety tracking control scheme based on adaptive dynamic programming (ADP) is proposed for multiplayer mixed zero-sum games, with the introduction of a control barrier function to ensure safe operation of the system. The original tracking problem is transformed into a state tracking error problem, and an augmented Hamilton-Jacobi-Bellman (HJB) equation is derived. Unlike traditional methods, a single critic neural network (NN) is used to approximate the Nash equilibrium solution, and a concurrent learning technique is introduced to relax the continuous excitation condition. The stability of the system is analyzed using Lyapunov theory, and simulation examples are provided to verify the effectiveness of the proposed scheme.
APPLIED INTELLIGENCE
(2023)
Article
Automation & Control Systems
Chunbin Qin, Xiaopeng Qiao, Jinguang Wang, Dehua Zhang, Yandong Hou, Shaolin Hu
Summary: In this article, an adaptive robust stabilization scheme based on the control barrier function (CBF) is proposed for the nonzero-sum (NZS) differential games problem of uncertain nonlinear systems with state constraints, considering random disturbances and control input matrix uncertainty. The nominal system of the original system is adopted to deal with the impact of uncertainty, converting the robust regulation problem of multiplayer differential games into an optimal regulation problem. Each player only needs a critic neural network (NN) to approach the corresponding cost function, and the combination of the cost function and the CBF ensures the evolution of system states in the safe area. Under the influence of random disturbances and state constraints, the state and critic NN weights of the closed-loop system are guaranteed to be uniformly ultimately bounded (UUB) by combining with the Lyapunov stability theory. Two simulation examples are provided to verify the validity of the proposed scheme.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2023)
Article
Automation & Control Systems
Francois Dufour, Tomas Prieto-Rumeau
Summary: This study examines a nonzero-sum Markov game on an abstract measurable state space with compact metric action spaces, focusing on maximizing each player's respective discounted payoff function under certain constraints. The existence of a constrained stationary Markov Nash equilibrium is established under appropriate conditions.
SIAM JOURNAL ON CONTROL AND OPTIMIZATION
(2022)
Article
Computer Science, Artificial Intelligence
Ankush Chakrabarty, Devesh K. Jha, Gregery T. Buzzard, Yebin Wang, Kyriakos G. Vamvoudakis
Summary: A method is developed for obtaining safe initial policies for uncertain systems using ADP techniques and kernelized Lipschitz estimation. The multiplier matrices learned are used in semidefinite programming frameworks to compute admissible initial control policies with provably high probability, enabling safe initialization and constraint enforcement while ensuring exponential stability of the closed-loop system equilibrium.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2021)
Article
Computer Science, Information Systems
Jingang Zhao
Summary: In this paper, a new adaptive dynamic programming (ADP) scheme is proposed to solve the optimal control problem of multi-player systems with unknown dynamics from the perspective of nonzero-sum (NZS) games. The scheme is able to learn the control policy and value function for each player without identifying the system dynamics. The use of neural network-based function approximation techniques overcomes the difficulty of unknown system dynamics. A new non-model-based ADP algorithm is developed based on the given iterative equation and neural network-based function approximation techniques, and its convergence is rigorously analyzed and proved. Two numerical simulation examples are provided to demonstrate the performance of the developed non-model-based ADP algorithm.
Article
Automation & Control Systems
Xiong Yang, Mengmeng Xu, Qinglai Wei
Summary: We study the dynamic event-driven Hop constrained control problem through approximate dynamic programming (ADP). Differing from the existing literature considering systems with either symmetric constraints or asymmetric constraints, we consider the two different constraints simultaneously. Initially, by constructing a generalized nonquadratic value function, we transform the H-8 constrained control problem into an unconstrained two-player zero-sum game. Then, we present an event-driven Hamilton-Jacobi-Isaacs equation (ED-HJIE) corresponding to the zero-sum game for lowering down the computational load. To solve the ED-HJIE, we propose a dynamic triggering mechanism together with a sole critic neural network (CNN) being built under the ADP framework. The CNN's weights are tuned via the gradient descent approach. After that, we prove uniform ultimate boundedness of the closed-loop system and the CNN's weight estimation error via Lyapunov's method. Finally, we separately use an F16 aircraft plant and an inverted pendulum system to validate the present theoretical claims.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2023)
Article
Computer Science, Artificial Intelligence
Hongbing Xia, Bo Zhao, Ping Guo
Summary: In this paper, a synergetic learning structure-based neuro-optimal fault tolerant control method is proposed for unknown nonlinear continuous-time systems with actuator failures. The optimal control input and the actuator failure are treated as two subsystems under the framework of synergetic learning structure. The fault tolerant control problem is then formulated as a two-player zero-sum differential game. A radial basis function neural network-based identifier is constructed to identify the completely unknown system dynamics. The Hamilton-Jacobi-Isaacs equation is solved using an asymptotically stable critic neural network, and the stability of the closed-loop system is guaranteed by Lyapunov stability analysis.
Article
Automation & Control Systems
Chun Li, Jinliang Ding, Frank L. Lewis, Tianyou Chai
Summary: This paper introduces a novel formulation of the value function to address the optimal tracking problem of nonlinear discrete-time systems, successfully applied in adaptive dynamic programming algorithms. The optimal control policy can be deduced through this value function, demonstrating the optimality of the obtained control policy.
Article
Mathematics, Applied
Zhongyang Ming, Huaguang Zhang, Yuling Liang, Hanguang Su
Summary: In this paper, a single network adaptive dynamic programming (ADP) control method is proposed for the non-zero sum (NZS) differential game problem of the autonomous nonlinear system. The Osgood condition is introduced to ensure the existence and uniqueness of the solution of the dynamic nonlinear systems and to weaken the limited conditions of nonlinear dynamic functions. The proposed method achieves real-time approximations of the optimal value and the non-zero sum Nash-equilibrium, while ensuring the uniform ultimate epsilon-boundedness of the closed-loop system. The effectiveness of the method is verified through a simulation example.
APPLIED MATHEMATICS AND COMPUTATION
(2022)
Article
Computer Science, Artificial Intelligence
Ziyu Lin, Jingliang Duan, Shengbo Eben Li, Haitong Ma, Jie Li, Jianyu Chen, Bo Cheng, Jun Ma
Summary: The research addresses the challenge of solving the finite-horizon HJB equation, proposes a new algorithm, and validates its effectiveness through simulations.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Computer Science, Artificial Intelligence
Mingming Ha, Ding Wang, Derong Liu
Summary: In this article, a novel value iteration scheme is proposed, which introduces a relaxation factor and combines with other methods to accelerate and guarantee the convergence. The theoretical results and numerical examples demonstrate its fast convergence speed and stability.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Automation & Control Systems
Shunchao Zhang, Bo Zhao, Derong Liu, Cesare Alippi, Yongwei Zhang
Summary: In this article, an event-triggered robust control (ETRC) method is investigated for multi-player nonzero-sum games of continuous-time input constrained nonlinear systems with mismatched uncertainties. The method transforms the robust control problem into an optimal regulation problem by constructing an auxiliary system and designing an appropriate value function. A critic neural network (NN) is used to approximate the value function of each player and obtain control laws. The method reduces computational burden and communication bandwidth by updating the control laws when events occur. The effectiveness of the developed ETRC method is demonstrated through theoretical analysis and examples.
INTERNATIONAL JOURNAL OF ROBUST AND NONLINEAR CONTROL
(2023)
Article
Computer Science, Artificial Intelligence
Qiuye Wu, Bo Zhao, Derong Liu, Marios M. Polycarpou
Summary: This paper proposes an event-triggered adaptive dynamic programming method to solve the decentralized tracking control problem for input constrained unknown nonlinear interconnected systems. A neural-network-based local observer is established to reconstruct the system dynamics using local input-output data and desired trajectories. The DTC problem is transformed into an optimal control problem using a nonquadratic value function. The DTC policy is obtained by solving the local Hamilton-Jacobi-Bellman equation through the observer-critic architecture, with weights tuned by the experience replay technique. Simulation examples demonstrate the effectiveness of the proposed scheme.
Article
Computer Science, Artificial Intelligence
Mingduo Lin, Bo Zhao, Derong Liu
Summary: A novel policy gradient (PG) adaptive dynamic programming method is proposed for nonlinear discrete-time zero-sum games with unknown dynamics. A policy iteration algorithm is used to approximate the Q-function and the control and disturbance policies using neural network approximators. The control and disturbance policies are then updated using the PG method based on the iterative Q-function. The experience replay technique is applied to improve training stability and data usage efficiency. Simulation results show the effectiveness of the proposed method.
Article
Computer Science, Artificial Intelligence
Mingming Liang, Derong Liu
Summary: This article presents a novel neural-network-based optimal event-triggered impulsive control method. The proposed method utilizes a general-event-based impulsive transition matrix (GITM) to represent the evolving characteristics of all system states across impulsive actions. Through the developed event-triggered impulsive adaptive dynamic programming (ETIADP) algorithm and its high-efficiency version (HEIADP), the optimization problems for stochastic systems with event-triggered impulsive controls are addressed. The results show that the proposed methods can reduce computational and communication burdens and fulfill the desired goals.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Automation & Control Systems
Runqi Chai, Derong Liu, Tianhao Liu, Antonios Tsourdos, Yuanqing Xia, Senchun Chai
Summary: This paper presents an integrated real-time trajectory planning and tracking control framework for autonomous ground vehicles (AGV) parking maneuver problems, utilizing deep neural networks and recurrent network structures. Two transfer learning strategies are applied to adapt the motion planner for different AGV types. Experimental studies demonstrate enhanced performance in fulfilling parking missions.
IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING
(2023)
Article
Computer Science, Artificial Intelligence
Danyu Lin, Shan Xue, Derong Liu, Mingming Liang, Yonghua Wang
Summary: In this paper, a problem of multiplayer hierarchical decision-making for non-affine systems is solved using adaptive dynamic programming. The control dynamics are obtained and combined with the original system dynamics, transforming the non-affine multiplayer system into a general affine form. The hierarchical decision problem is modeled as a Stackelberg game, and a neural network is used to reconstruct the augmented system and approximate the value function. The feasibility and effectiveness of the algorithm are confirmed through simulation.
Article
Computer Science, Artificial Intelligence
Chujian Zeng, Bo Zhao, Derong Liu
Summary: This paper proposes a neuro-dynamic programming-based fault tolerant control scheme for a class of nonlinear systems, considering the occurrence of both actuator and sensor faults simultaneously. The scheme combines a descriptor observer with an adaptive observer to estimate system states and multiple faults. By employing a critic neural network, the approximate optimal control policy is obtained for the fault-free system. An FTC law is developed to suppress the influence of actuator faults by combining the estimations of actuator faults with the approximate optimal control policy. The stability of the closed-loop nonlinear system is analyzed using the Lyapunov stability theorem.
Article
Automation & Control Systems
Yongwei Zhang, Bo Zhao, Derong Liu, Shunchao Zhang
Summary: In this article, the event-triggered robust control problem of unknown multiplayer nonlinear systems with constrained inputs and uncertainties is investigated using adaptive dynamic programming. A neural network-based identifier is constructed to relax the requirement of system dynamics. By designing a nonquadratic value function, the stabilization problem is converted into a constrained optimal control problem. The approximate solution of the event-triggered Hamilton-Jacobi equation is obtained using a critic network with a novel weight updating law, and the Lyapunov stability theorem ensures that the multiplayer system is uniformly ultimately bounded.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Automation & Control Systems
Mingming Liang, Yonghua Wang, Derong Liu
Summary: In this study, a novel general impulsive transition matrix is defined to reveal the transition dynamics and probability distribution evolution patterns between impulsive events. Based on this matrix, policy iteration-based impulsive adaptive dynamic programming algorithms are developed to solve optimal impulsive control problems. The algorithms demonstrate convergence to the optimal impulsive performance index function and allow for optimization on computing devices with low memory spaces.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Proceedings Paper
Automation & Control Systems
Jinquan Lin, Bo Zhao, Derong Liu
Summary: In this paper, an integral reinforcement learning (IRL)-based approximate optimal control (AOC) method is developed for unknown nonaffine systems using dynamic feedback. The optimal control policy for nonaffine systems cannot be explicitly expressed due to the unknown input gain matrix. Thus, a dynamic feedback signal is introduced to transform the nonaffine system into an augmented affine system. The AOC for unknown nonaffine systems is formulated by designing an appropriate value function for the augmented affine system, and the IRL method is adopted to derive the approximate solution of the Hamilton-Jacobi-Bellman equation.
2023 IEEE 12TH DATA DRIVEN CONTROL AND LEARNING SYSTEMS CONFERENCE, DDCLS
(2023)
Article
Automation & Control Systems
Bo Zhao, Guang Shi, Derong Liu
Summary: This article investigates local control problems for nonlinear interconnected systems by using adaptive dynamic programming (ADP) with particle swarm optimization (PSO). It constructs a proper local value function and employs a local critic neural network to solve the local Hamilton-Jacobi-Bellman equation. The event-triggering mechanism is introduced to determine the sampling time instants and ensure asymptotic stability through Lyapunov stability analysis.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2023)
Article
Automation & Control Systems
Mingming Liang, Derong Liu
Summary: This article focuses on designing the optimal impulsive controller (IMC) of continuous-time nonlinear systems and proposes a new adaptive dynamic programming algorithm with high generality and feasibility. The introduced policy-improving mechanism makes the algorithm more flexible for memory-limited computing devices.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2023)
Article
Automation & Control Systems
Ke Wang, Chaoxu Mu, Zhen Ni, Derong Liu
Summary: This paper presents a novel composite obstacle avoidance control method that generates safe motion trajectories for autonomous systems in an adaptive manner. The method combines model-based policy iteration and state-following-based approximation in a safe reinforcement learning framework. The proposed learning-based controller achieves stable reaching of target points while maintaining a safe distance from obstacles. The effectiveness of the method is demonstrated through simulations and comparisons with other avoidance control methods.
IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING
(2023)
Article
Computer Science, Artificial Intelligence
Bo Zhao, Yongwei Zhang, Derong Liu
Summary: This article presents a cooperative motion/force control scheme for modular reconfigurable manipulators (MRMs) based on adaptive dynamic programming (ADP). The dynamic model of the entire MRM system is treated as a set of joint modules interconnected by coupling torque, and the Jacobian matrix is mapped into each joint. A neural network is used as a robust decentralized observer, and an improved local value function is constructed for each joint module. The control scheme is achieved by using force feedback compensation and is proven to be uniformly ultimately bounded through Lyapunov stability analysis.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
