Improving extreme search with natural gradient descent using dirichlet distribution

Abstract

Natural gradient descent is an optimization algorithm, which is proposed to replace stochastic gradient descent and its modifications. The most precious ability of this algorithm is to reach the extreme with little number of iterations and required accuracy, which has high value in machine learning and statistics. The goal of this article is to propose a natural gradient descent algorithm with the Dirichlet distribution, which includes step-size adaptation. We will prove experimentally advantage of natural gradient descent over stochastic gradient descent and Adam algorithm. Additionally, the calculating of the Fisher information matrix of Dirichlet distribution will be shown.