Article
Automation & Control Systems
Bo Pang, Zhong-Ping Jiang
Summary: This article studies the infinite-horizon adaptive optimal control of continuous-time linear periodic systems and proposes a novel value iteration-based off-policy adaptive dynamic programming algorithm for a general class of systems. The algorithm is proven to uniformly converge to optimal solutions in both model-based and model-free cases, without assuming knowledge of an initial stabilizing controller. Application to a triple inverted pendulum demonstrates the feasibility and effectiveness of the proposed method.
IEEE TRANSACTIONS ON AUTOMATIC CONTROL
(2021)
Article
Computer Science, Artificial Intelligence
Tao Bian, Zhong-Ping Jiang
Summary: This article studies the adaptive optimal control problem for continuous-time nonlinear systems described by differential equations and proposes a new continuous-time value iteration method to address the limitations of existing methods. Adaptive optimal controllers for systems with unknown dynamics are obtained through this method, along with a learning-based control algorithm.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2022)
Article
Automation & Control Systems
Jorge R. Chavez-Fuentes, Eduardo F. Costa, Marco H. Terra, Kaio D. T. Rocha
Summary: This paper addresses the linear quadratic optimal control problem for discrete-time Markov jump linear singular systems, obtaining results under conditions that bring additional structure to the considered systems. The approach involves base transformations and control action restrictions to ensure regularity of the closed-loop system. The results are evaluated using an example.
Article
Computer Science, Artificial Intelligence
Lingzhi Zhang, Lei Xie, Yi Jiang, Zhishan Li, Xueqin Liu, Hongye Su
Summary: This article proposes a constrained optimal control approach for discrete-time nonlinear systems based on safe reinforcement learning. By introducing a barrier function, the constrained optimization problem is transformed into an unconstrained one, and a constrained policy iteration algorithm is developed to ensure optimal control and constraint satisfaction.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Acoustics
Mohammad Shirazian
Summary: This paper proposes an improved approximate solution method for optimal control of linear time-varying systems. The optimality conditions are derived and the well-known variational iteration method, interpolated by B-spline functions, is applied to solve these conditions. The method is accelerated through redundant calculation elimination and does not require solving system equations or optimization problems. The convergence of the proposed method is proved and its efficiency is illustrated through several examples.
JOURNAL OF VIBRATION AND CONTROL
(2023)
Article
Computer Science, Artificial Intelligence
Ding Wang, Jin Ren, Mingming Ha, Junfei Qiao
Summary: This article discusses the impact of the discount factor on the stabilization of control strategies. It presents methods to judge the stability of the controlled system and select appropriate discount factors. The practical rule for selecting discount factors is constructed based on the undiscounted optimal control problem.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Automation & Control Systems
Longyan Hao, Chaoli Wang, Guang Zhang, Chonglin Jing, Yibo Shi
Summary: This paper studies the optimal tracking problem for discrete-time linear systems with multiple delays without system dynamics. A new data-driven value iteration algorithm is proposed, considering past control inputs, system outputs, and external reference trajectories. The algorithm transforms the original system according to the characteristics of the time-delay system, derives a novel data-driven state equation, and solves the optimal control of multi-delay systems. Results demonstrate the convergence of the algorithm and the asymptotic stability of the tracking error. Simulations show the effectiveness of the controller.
INTERNATIONAL JOURNAL OF SYSTEMS SCIENCE
(2022)
Article
Automation & Control Systems
Huaiyuan Jiang, Bin Zhou, Guang-Ren Duan
Summary: This article studies the general policy iteration (GPI) method for optimal control of discrete-time linear systems. The existing result on the GPI method is recalled and some new properties are proposed. A model-based modified GPI algorithm is proposed based on these new properties, with its convergence proof provided. In addition, a data-driven implementation for the proposed method is introduced, which does not require the use of system matrices. The proposed algorithm further relaxes the condition to initiate the GPI based algorithm compared to existing results. The effectiveness of the proposed modified GPI based algorithm is verified through a simulation example.
INTERNATIONAL JOURNAL OF ROBUST AND NONLINEAR CONTROL
(2022)
Article
Automation & Control Systems
Guangyu Zhu, Xiaolu Li, Ranran Sun, Yiyuan Yang, Peng Zhang
Summary: In this paper, a new iterative adaptive dynamic programming algorithm called discrete-time time-varying policy iteration (DTTV) algorithm is developed for infinite horizon optimal control problems of discrete time-varying nonlinear systems. The algorithm updates the iterative value function to approximate the index function of optimal performance. The admissibility and convergence properties of the iterative control law are analyzed.
IEEE-CAA JOURNAL OF AUTOMATICA SINICA
(2023)
Article
Computer Science, Artificial Intelligence
Min Lin, Zhongqi Sun, Yuanqing Xia, Jinhui Zhang
Summary: This article proposes a novel reinforcement learning-based model predictive control (RLMPC) scheme that integrates model predictive control (MPC) and reinforcement learning (RL) through policy iteration (PI). The scheme improves the generated policy by using the obtained value function as the terminal cost of MPC, eliminating the need for the offline design paradigm of traditional MPC. RLMPC enables a more flexible choice of prediction horizon and shows superiority over traditional MPC for nonlinear systems.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Mathematics, Interdisciplinary Applications
Xue Yang, Shujun Liu
Summary: This paper investigates the optimal control problem with a long run average cost for unknown linear discrete-time systems with additive noise. The authors propose a value iteration-based stochastic adaptive dynamic programming (VI-based SADP) algorithm to obtain the optimal controller. Unlike existing work, this algorithm does not require estimation of the expectation and variance of states or other relevant variables, and its convergence can be rigorously proven. A simulation example is provided to verify the effectiveness of the proposed approach.
JOURNAL OF SYSTEMS SCIENCE & COMPLEXITY
(2023)
Article
Automation & Control Systems
Jin Zhu, Qingkun Zhang
Summary: This paper investigates the discrete-time Markovian jump linear systems (MJLSs) whose mode transition probability matrix (MTPM) can be adjusted by decisions. A decision strategy is proposed for stabilisation and optimisation of such MJLSs, considering the decision cost. The paper gives the feasible domain of decision for stable and unstable MJLSs with initial MTPM, introduces a generalised performance index, and presents a value iteration algorithm for optimisation.
INTERNATIONAL JOURNAL OF SYSTEMS SCIENCE
(2023)
Article
Automation & Control Systems
Teng Song
Summary: In this paper, the indefinite stochastic optimal control problems of discrete-time Markov jump linear systems are considered. A new stochastic maximum principle is established, and the necessary and sufficient solvability condition of the indefinite control problem with non-discounted cost is derived. The optimal control is designed using coupled generalized Riccati difference equations with Markov jump and linear recursive equations with Markov jump. An example of a defined-benefit pension fund with regime switching is provided to illustrate the validity of the obtained results.
OPTIMAL CONTROL APPLICATIONS & METHODS
(2023)
Article
Automation & Control Systems
Chen Liu, Xiaoe Ruan, Dong Shen, Hao Jiang
Summary: This article investigates an intermittent optimal learning control scheme that considers partially available information to address the issue of varying operational lengths in rehabilitation training. The proposed scheme achieves optimal learning gain by adopting the latest captured historical timewise input and tracking error.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Correction
Automation & Control Systems
Mohammad Hejri
Summary: This passage points out the need for corrections in the statements of Lemmas 2 and 4, as well as in the proofs of Lemma 2 and Prop. 3.
IEEE-CAA JOURNAL OF AUTOMATICA SINICA
(2022)
Article
Automation & Control Systems
Shunchao Zhang, Bo Zhao, Derong Liu, Cesare Alippi, Yongwei Zhang
Summary: In this article, an event-triggered robust control (ETRC) method is investigated for multi-player nonzero-sum games of continuous-time input constrained nonlinear systems with mismatched uncertainties. The method transforms the robust control problem into an optimal regulation problem by constructing an auxiliary system and designing an appropriate value function. A critic neural network (NN) is used to approximate the value function of each player and obtain control laws. The method reduces computational burden and communication bandwidth by updating the control laws when events occur. The effectiveness of the developed ETRC method is demonstrated through theoretical analysis and examples.
INTERNATIONAL JOURNAL OF ROBUST AND NONLINEAR CONTROL
(2023)
Article
Computer Science, Artificial Intelligence
Qiuye Wu, Bo Zhao, Derong Liu, Marios M. Polycarpou
Summary: This paper proposes an event-triggered adaptive dynamic programming method to solve the decentralized tracking control problem for input constrained unknown nonlinear interconnected systems. A neural-network-based local observer is established to reconstruct the system dynamics using local input-output data and desired trajectories. The DTC problem is transformed into an optimal control problem using a nonquadratic value function. The DTC policy is obtained by solving the local Hamilton-Jacobi-Bellman equation through the observer-critic architecture, with weights tuned by the experience replay technique. Simulation examples demonstrate the effectiveness of the proposed scheme.
Article
Computer Science, Artificial Intelligence
Mingduo Lin, Bo Zhao, Derong Liu
Summary: A novel policy gradient (PG) adaptive dynamic programming method is proposed for nonlinear discrete-time zero-sum games with unknown dynamics. A policy iteration algorithm is used to approximate the Q-function and the control and disturbance policies using neural network approximators. The control and disturbance policies are then updated using the PG method based on the iterative Q-function. The experience replay technique is applied to improve training stability and data usage efficiency. Simulation results show the effectiveness of the proposed method.
Article
Computer Science, Artificial Intelligence
Mingming Liang, Derong Liu
Summary: This article presents a novel neural-network-based optimal event-triggered impulsive control method. The proposed method utilizes a general-event-based impulsive transition matrix (GITM) to represent the evolving characteristics of all system states across impulsive actions. Through the developed event-triggered impulsive adaptive dynamic programming (ETIADP) algorithm and its high-efficiency version (HEIADP), the optimization problems for stochastic systems with event-triggered impulsive controls are addressed. The results show that the proposed methods can reduce computational and communication burdens and fulfill the desired goals.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Automation & Control Systems
Mingming Ha, Ding Wang, Derong Liu
Summary: Inspired by the successive relaxation method, a novel discounted iterative adaptive dynamic programming framework is developed, which possesses an adjustable convergence rate for the iterative value function sequence. The convergence properties of the value function sequence and the stability of the closed-loop systems under the new discounted value iteration (VI) are investigated. An accelerated learning algorithm with convergence guarantee is presented based on the properties of the given VI scheme. Amidst the implementation of the new VI scheme and its accelerated learning design, value function approximation and policy improvement are involved. The performance of the developed approaches is verified using a nonlinear fourth-order ball-and-beam balancing plant, showing significant acceleration of the convergence rate of the value function and reduction in computational cost compared to traditional VI.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Automation & Control Systems
Yongwei Zhang, Bo Zhao, Derong Liu, Shunchao Zhang
Summary: In this article, the event-triggered robust control problem of unknown multiplayer nonlinear systems with constrained inputs and uncertainties is investigated using adaptive dynamic programming. A neural network-based identifier is constructed to relax the requirement of system dynamics. By designing a nonquadratic value function, the stabilization problem is converted into a constrained optimal control problem. The approximate solution of the event-triggered Hamilton-Jacobi equation is obtained using a critic network with a novel weight updating law, and the Lyapunov stability theorem ensures that the multiplayer system is uniformly ultimately bounded.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Automation & Control Systems
Mingming Liang, Yonghua Wang, Derong Liu
Summary: In this study, a novel general impulsive transition matrix is defined to reveal the transition dynamics and probability distribution evolution patterns between impulsive events. Based on this matrix, policy iteration-based impulsive adaptive dynamic programming algorithms are developed to solve optimal impulsive control problems. The algorithms demonstrate convergence to the optimal impulsive performance index function and allow for optimization on computing devices with low memory spaces.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Proceedings Paper
Automation & Control Systems
Jinquan Lin, Bo Zhao, Derong Liu
Summary: In this paper, an integral reinforcement learning (IRL)-based approximate optimal control (AOC) method is developed for unknown nonaffine systems using dynamic feedback. The optimal control policy for nonaffine systems cannot be explicitly expressed due to the unknown input gain matrix. Thus, a dynamic feedback signal is introduced to transform the nonaffine system into an augmented affine system. The AOC for unknown nonaffine systems is formulated by designing an appropriate value function for the augmented affine system, and the IRL method is adopted to derive the approximate solution of the Hamilton-Jacobi-Bellman equation.
2023 IEEE 12TH DATA DRIVEN CONTROL AND LEARNING SYSTEMS CONFERENCE, DDCLS
(2023)
Article
Automation & Control Systems
Bo Zhao, Guang Shi, Derong Liu
Summary: This article investigates local control problems for nonlinear interconnected systems by using adaptive dynamic programming (ADP) with particle swarm optimization (PSO). It constructs a proper local value function and employs a local critic neural network to solve the local Hamilton-Jacobi-Bellman equation. The event-triggering mechanism is introduced to determine the sampling time instants and ensure asymptotic stability through Lyapunov stability analysis.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2023)
Article
Automation & Control Systems
Hui Wang, Zhigang Liu, Zhiwei Han, Yanbo Wu, Derong Liu
Summary: Active pantograph control is a promising technique for improving train's current collection quality. Existing solutions have limitations in handling various operating conditions and lack of adaptability. In this study, a context-based deep meta-reinforcement learning algorithm is proposed to alleviate these problems. Experimental results show that the proposed algorithm can quickly adapt to new conditions and reduce contact force fluctuations.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Computer Science, Artificial Intelligence
Zhanyu Yang, Bo Zhao, Derong Liu
Summary: In this article, a novel pinning control method that requires only partial node information is developed to synchronize drive-response memristor-based neural networks with time delay. An improved mathematical model of the networks is established to accurately describe their dynamic behaviors. Unlike previous literature that requires information from all nodes, the proposed method only relies on local information to achieve synchronization of delayed networks, reducing communication and calculation burdens. Sufficient conditions for synchronization are provided, and numerical simulation and comparative experiments validate the effectiveness and superiority of the proposed method.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)
Article
Automation & Control Systems
Mingduo Lin, Bo Zhao, Derong Liu
Summary: In this article, an event-triggered robust adaptive dynamic programming (ETRADP) algorithm is proposed to solve multiplayer Stackelberg-Nash games (MSNGs) for uncertain nonlinear continuous-time systems. The hierarchical decision-making process considering different roles of players is described, transforming the robust control problem into an optimal regulation problem. An online policy iteration algorithm is used to solve the derived Hamilton-Jacobi equation with an event-triggered mechanism to reduce computational and communication burdens. Critic neural networks (NNs) are constructed to obtain the event-triggered approximate optimal control policies for all players, ensuring the stability of the closed-loop system.
IEEE TRANSACTIONS ON CYBERNETICS
(2023)
Article
Automation & Control Systems
Mingming Liang, Derong Liu
Summary: This article focuses on designing the optimal impulsive controller (IMC) of continuous-time nonlinear systems and proposes a new adaptive dynamic programming algorithm with high generality and feasibility. The introduced policy-improving mechanism makes the algorithm more flexible for memory-limited computing devices.
IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS
(2023)
Article
Automation & Control Systems
Ke Wang, Chaoxu Mu, Zhen Ni, Derong Liu
Summary: This paper presents a novel composite obstacle avoidance control method that generates safe motion trajectories for autonomous systems in an adaptive manner. The method combines model-based policy iteration and state-following-based approximation in a safe reinforcement learning framework. The proposed learning-based controller achieves stable reaching of target points while maintaining a safe distance from obstacles. The effectiveness of the method is demonstrated through simulations and comparisons with other avoidance control methods.
IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING
(2023)
Article
Computer Science, Artificial Intelligence
Bo Zhao, Yongwei Zhang, Derong Liu
Summary: This article presents a cooperative motion/force control scheme for modular reconfigurable manipulators (MRMs) based on adaptive dynamic programming (ADP). The dynamic model of the entire MRM system is treated as a set of joint modules interconnected by coupling torque, and the Jacobian matrix is mapped into each joint. A neural network is used as a robust decentralized observer, and an improved local value function is constructed for each joint module. The control scheme is achieved by using force feedback compensation and is proven to be uniformly ultimately bounded through Lyapunov stability analysis.
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
(2023)