Gradient Descent Learning With Floats

The gradient learning descent method is the main workhorse of training tasks in artificial intelligence and machine-learning research. Current theoretical studies of gradient descent only use the continuous domains, which is unreal since electronic computers use the float point numbers to store and deal with data. Although existing results are sufficient for the extremely tiny errors in high-precision machines, they need to be improved for low-precision cases. This article presents an understanding of the learning algorithm in computers with floats. The performances of three gradient descents with the floating domain are investigated when the objective function is smooth. When the function is assumed to have the PL condition, the convergence speed can be improved. We proved that for floating gradient descent to obtain an error with is an element of, the iteration is O(1/is an element of) for the general smooth case, and O(ln(1/is an element of)) for the PL case. But is an element of should be larger than the s-bit machine epsilon delta(s) in the deterministic case, that is, is an element of >= Omega (delta(s)), while is an element of >= Omega (root delta(s)) for the stochastic case. Floating stochastic and sign gradient descents can both output an is an element of noised result in O(1/is an element of(2)) iterations.

Automated summary

This article investigates the learning algorithm of gradient descent with floating point numbers in computers, and explores the performance of three gradient descent methods for smooth objective functions. The research shows that the convergence speed can be improved under certain conditions of the objective function.