Transformer operator network with fast difference gradient positive–negative momentum for solving Navier–Stokes equations

Abstract

Modern machine learning approaches solve many problems in human activity. Recent models of neural networks find applications in solving equations of mathematical physics. Along with traditional numerical methods, physics-informed learning builds the approximate solution with a relatively small L2 and mean squared errors. In this paper, we propose a fast difference-gradient positive–negative momentum optimizer that achieves a global minimum of the loss function with a convergence rate O(logT) and stability O(T). This optimization algorithm solves loss function minimization problems such as gradient discontinuity, vanishing gradient and local minimum traversal. Analysis on the set of test functions confirms the superiority of the proposed optimizer over state-of-the-art methods. Through the modified positive–negative moment estimation, the proposed optimizer gives more appropriate weight updates in physics-informed, deep operator, and transformer operator neural networks in Navier–Stokes equation solving. In particular, the proposed optimizer minimizes the loss function better than known analogs in solving 2d-Kovasznay, (2d+t)-Taylor–Green, (3d+t)-Beltrami, and (2d+t) circular cylindrical flows. Fast difference gradient positive–negative moment estimation allows the physics-informed model to reduce the L2 and mean squared errors solution by 14.6−53.9 percentage points. The proposed fast difference gradient positive–negative momentum can increase the approximate solutions of partial differential equations in physics-informed neural, deep operator, and transformer operator networks.