PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks

Sifan Wang, Bowen Li, Yuhan Chen, Paris Perdikaris

Research output: Journal Publications and ReviewsRGC 21 - Publication in refereed journalpeer-review

5 Downloads (CityUHK Scholars)

Abstract

While physics-informed neural networks (PINNs) have become a popular deep learning framework for tackling forward and inverse problems governed by partial differential equations (PDEs), their performance is known to degrade when larger and deeper neural network architectures are employed. Our study identifies that the root of this counter-intuitive behavior lies in the use of multi-layer perceptron (MLP) architectures with non-suitable initialization schemes, which result in poor trainablity for the network derivatives, and ultimately lead to an unstable minimization of the PDE residual loss. To address this, we introduce Physics-Informed Residual Adaptive Networks (PirateNets), a novel architecture that is designed to facilitate stable and efficient training of deep PINN models. PirateNets leverage a novel adaptive residual connection, which allows the networks to be initialized as shallow networks that progressively deepen during training. We also show that the proposed initialization scheme allows us to encode appropriate inductive biases corresponding to a given PDE system into the network architecture. We provide comprehensive empirical evidence showing that PirateNets are easier to optimize and can gain accuracy fromconsiderably increased depth, ultimately achieving state-of-the-art results across various benchmarks. All code and data accompanying this manuscript will be made publicly available at https://github.com/PredictiveIntelligenceLab/jaxpi/tree/pirate.
©2024 Sifan Wang, Bowen Li, Yuhan Chen and Paris Perdikaris.
Original languageEnglish
Article number402
JournalJournal of Machine Learning Research
Volume25
Online publishedDec 2024
Publication statusPublished - 2025

Funding

We would like to acknowledge support from the US Department of Energy under the Advanced Scientific Computing Research program (grant DE-SC0024563). We also thank the developers of the software that enabled our research, including JAX (Bradbury et al., 2018), Matplotlib (Hunter, 2007), and NumPy (Harris et al., 2020).

Research Keywords

  • Deep learning
  • Physics-informed neural networks
  • Partial differential equations
  • Computational physics
  • Neural solvers

Publisher's Copyright Statement

  • This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

Fingerprint

Dive into the research topics of 'PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks'. Together they form a unique fingerprint.

Cite this