A proof of convergence and equivalence for 1D finite element methods and ReLU neural networks

Abstract

 This paper investigates the convergence and equivalence properties of the Finite Element Method (FEM) and Rectified Linear Unit Neural Networks (ReLU NNs) in solving differential equations. We provide a detailed comparison of the two approaches, highlighting their mutual capabilities in function space approximation. Our analysis proves the subset and superset inclusions that establish the equivalence between FEM and ReLU NNs for approximating piecewise linear functions. Furthermore, a comprehensive numerical evaluation is presented, demonstrating the error convergence behavior of ReLU NNs as the number of neurons per basis function varies. Our results show that while increasing the number of neurons improves approximation accuracy, this benefit diminishes beyond a certain threshold. The maximum observed error between FEM and ReLU NNs is 10^-4, reflecting excellent accuracy in solving partial differential equations (PDEs). These findings lay the groundwork for integrating FEM and ReLU NNs, with important implications for computational mathematics and engineering applications.