Journal
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
Volume 34, Issue 11, Pages 8894-8908Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TNNLS.2022.3153985
Keywords
Neurons; Biology; Training; Heuristic algorithms; Approximation algorithms; Biological neural networks; Backpropagation; Backpropagation; backpropagation through time (BPTT); online learning; real-time recurrent learning (RTRL); spiking neurons
Ask authors/readers for more resources
This paper presents an online learning algorithm framework called online spatio-temporal learning (OSTL) for deep recurrent neural networks (RNNs) and spiking neural networks (SNNs). The framework, based on insights from biology, separates spatial and temporal gradient components, allowing for online training with equivalent gradients to offline computation. Results on various tasks show that OSTL performs on par with traditional methods.
Biological neural networks are equipped with an inherent capability to continuously adapt through online learning. This aspect remains in stark contrast to learning with error backpropagation through time (BPTT) that involves offline computation of the gradients due to the need to unroll the network through time. Here, we present an alternative online learning algorithm framework for deep recurrent neural networks (RNNs) and spiking neural networks (SNNs), called online spatio-temporal learning (OSTL). It is based on insights from biology and proposes the clear separation of spatial and temporal gradient components. For shallow SNNs, OSTL is gradient equivalent to BPTT enabling for the first time online training of SNNs with BPTT-equivalent gradients. In addition, the proposed formulation unveils a class of SNN architectures trainable online at low time complexity. Moreover, we extend OSTL to a generic form, applicable to a wide range of network architectures, including networks comprising long short-term memory (LSTM) and gated recurrent units (GRUs). We demonstrate the operation of our algorithm framework on various tasks from language modeling to speech recognition and obtain results on par with the BPTT baselines.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available