Deep Belief Network Training Improvement Using Elite Samples Minimizing Free Energy

Nowadays, it is very popular to use deep architectures in machine learning. Deep Belief Networks (DBNs) are deep architectures that use stack of Restricted Boltzmann Machines (RBMs) to create a powerful generative model using the training data. In this paper, we present an improvement in a common method that is usually used in training RBMs. The new method uses free energy as a criterion to obtain elite samples from generative model. We argue that these samples can compute gradient of log probability of training data more accurately. According to the results, an error rate of 0.99% was achieved on MNIST test set. This result shows that the proposed method outperforms the method presented in the first paper introducing DBN (1.25% error rate) and general classification methods such as SVM (1.4% error rate) and KNN (1.6% error rate). In another test using ISOLET dataset, letter classification error dropped to 3.59% compared to 5.59% error rate achieved in the papers using this dataset.