Abstract : Many interesting phenomena in science and engineering involve operators mapping functions to functions. The application of data-driven tools from machine learning to scientific computing has thus given rise to the rapidly emerging field of operator learning. Despite encouraging practical successes, our understanding of these methods is still in its infancy, leaving important open questions to be addressed, including approximation guarantees, learning in data-scarce regimes, and understanding the limitations of current approaches and overcoming them. This minisymposium brings together researchers at the intersection of machine learning, approximation theory and PDEs to discuss theoretical foundations and recent algorithmic developments in this field.
[03323] BelNet: basis enhanced learning, a mesh-free neural operator
Format : Online Talk on Zoom
Author(s) :
zecheng zhang (Carnegie Mellon University)
Wing Tat Leung (City University Hong Kong)
Hayden Schaeffer (UCLA)
Abstract : Operator learning trains a neural network to map functions to functions. An ideal operator learning framework should be mesh-free in the sense that the training does not require a particular choice of discretization for the input functions, allows for the input and output functions to be on different domains, and is able to have different grids between samples. We propose a mesh-free neural operator for solving parametric partial differential equations. The basis enhanced learning network (BelNet) projects the input function into a latent space and reconstructs the output functions. In particular, we construct part of the network to learn the ``basis'' functions in the training process. This generalized the networks proposed in Chen and Chen's universal approximation theory for the nonlinear operators to account for differences in input and output meshes. Through several challenging high-contrast and multiscale problems, we show that our approach outperforms other operator learning methods for these tasks and allows for more freedom in the sampling and/or discretization process.
[01354] The curse of dimensionality in operator learning
Format : Talk at Waseda University
Author(s) :
Samuel Lanthaler (California Institute of Technology)
Abstract : Neural operator architectures employ neural networks to approximate operators between Banach spaces of functions. We show that for general classes of operators, which are characterized only by their Lipschitz- or $C^r$-regularity, operator learning with neural operators suffers from a curse of dimensionality related to the infinite-dimensional input and output function spaces. This curse, made rigorous in this work, is characterized by an exponential lower complexity bound: in order to achieve approximation accuracy $\epsilon$, the number of tunable parameters generally has to scale exponentially in $\epsilon^{-1}$. This negative result is applicable to a wide variety of existing neural operators, including DeepONet, the Fourier neural operator and PCA-Net. It is then shown that the general curse of dimensionality can be overcome for operators possessing additional structure, going beyond regularity. This is illustrated for the solution operator of the Hamilton-Jacobi equation.
[05247] Score-based Diffusion Models in Function Space
Format : Talk at Waseda University
Author(s) :
Nikola Kovachki (NVIDIA)
Abstract : We present a generalization of score-based diffusion models to function space by perturbing functional data via a Gaussian process at multiple scales. We obtain an appropriate notion of score by defining densities with respect to Guassian measures and generalize denoising score matching. We then define the generative process by integrating a function-valued Langevin dynamic. We show that the corresponding discretized algorithm generates accurate samples at a fixed cost that is independent of the data discretization.
[04771] Deep Learning Theories for Problems with Low–Dimensional Structures
Format : Talk at Waseda University
Author(s) :
Hao Liu (Hong Kong Baptist University)
Minshuo Chen (Princeton University)
Siawpeng Er (Georgia Institute of Technology)
Haizhao Yang (University of Maryland College Park)
Tong Zhang (he Hong Kong University of Science and Technology)
Tuo Zhao (Georgia Institute of Technology)
Wenjing Liao (Georgia Institute of Technology)
Abstract : Deep neural networks have demonstrated a great success on many applications, especially on problems with high-dimensional data sets. However, most existing theories are cursed by data dimension. To mitigate the curse of dimensionality, we exploit the low-dimensional structures of data set and establish theoretical guarantees with a fast rate that is only cursed by the intrinsic dimension of the data set. This presentation addresses our recent work on function approximation and operator learning.
HRUSHIKESH N MHASKAR (Claremont Graduate University)
Abstract : We study the question of approximation of an operator $\mathfrak{F}$ from one metric space $X$ to another, $Y$. The input $f\in X$ and $\mathfrak{F}(f)$ are encoded in terms of a point on a sphere $S^d$ ($S^D$), Local approximation techniques are developed to achieve approximation of properly defined smooth function in a tractable manner.
[04141] Neural operator surrogates for Gaussian inputs
Format : Talk at Waseda University
Author(s) :
Jakob Zech (Universität Heidelberg)
Abstract : In this talk we discuss the use of operator surrogates to approximate smooth maps between infinite-dimensional Hilbert spaces. Such surrogates have a wide range of applications in uncertainty quantification and parameter estimation problems. The error is measured in the $L^2$-sense with respect to a Gaussian measure on the input space. Under suitable assumptions, we show that algebraic and dimension-independent convergence rates can be achieved.
[03497] Derivative-Informed Neural Operators for Scalable and Efficient UQ
Format : Talk at Waseda University
Author(s) :
Thomas O'Leary-Roseberry (The University of Texas at Austin)
Omar Ghattas (The University of Texas at Austin)
Peng Chen (Georgia Tech)
Umberto Villa (The University of Texas at Austin)
Dingcheng Luo (The University of Texas at Austin)
Lianghao Cao (The University of Texas at Austin)
Abstract : We present a novel operator learning methodology "derivative-informed neural operators" (DINOs) that can accurately represent both operator maps and their derivatives in function spaces. DINOs are built using advanced adjoint methods and dimension reduction techniques, resulting in efficient computation of derivative quantities that can facilitate fast and scalable UQ. We showcase the potential of DINOs in two applications: Bayesian inversion using data from the 2011 Tōhoku earthquake, and optimal control of PDEs under uncertainty.
[05169] Deep Operator Network Approximation Rates for Lipschitz Operators
Format : Online Talk on Zoom
Author(s) :
Christoph Schwab (ETH Zurich)
Abstract : We establish expression rate bounds for neural Deep Operator Networks (DON) emulating Lipschitz continuous maps G between (suitable subsets of) separable Hilbert spaces X and Y. The DON architecture uses linear encoders E and decoders D via Riesz bases of X, Y, and an approximator network of a parametric coordinate map that is Lipschitz continuous on the sequence space. The present results require mere Lipschitz (or Holder) continuity of G.
[02675] Transfer Learning Enhanced Physics-informed DeepONets for Long-time Prediction
Format : Talk at Waseda University
Author(s) :
wuzhe xu (University of Massachusetts Amherst)
Abstract : Deep operator network (DeepONet) has demonstrated great success in various learning tasks, including learning solution operators of partial differential equations. In particular, it provides an efficient approach to predict the evolution equations in a finite time horizon. Nevertheless, the vanilla DeepONet suffers from the issue of stability degradation in the long-time prediction. This paper proposes a transfer-learning aided DeepONet to enhance the stability. Our idea is to use transfer learning to sequentially update the DeepONets as the surrogates for propagators learned in different time frames. The evolving DeepONets can better track the varying complexities of the evolution equations, while only need to be updated by efficient training of a tiny fraction of the operator networks. Through systematic experiments, we show that the proposed method not only improves the long-time accuracy of DeepONet while maintaining similar computational cost but also substantially reduces the sample size of the training set.
[03360] Generic bounds on the approximation error for physics-informed (and) operator learning
Format : Online Talk on Zoom
Author(s) :
Tim De Ryck (ETH Zürich)
Siddhartha Mishra (ETH Zürich)
Abstract : We propose a very general framework for deriving rigorous bounds on the approximation error for physics-informed neural networks (PINNs) and operator learning architectures such as DeepONets and FNOs as well as for physics-informed operator learning. These bounds guarantee that PINNs and (physics-informed) DeepONets or FNOs will efficiently approximate the underlying solution or solution operator of generic PDEs.
[03124] Overcoming Fundamental Limitations of Current AI Approaches: From Digital to Analog Hardware
Format : Online Talk on Zoom
Author(s) :
Gitta Kutyniok (LMU Munich)
Holger Boche (TU Munich)
Adalbert Fono (LMU Munich)
Aras Bacho (LMU Munich)
Yunseok Lee (LMU Munich)
Abstract : Artificial intelligence is currently leading to one breakthrough after the other. However, one current major drawback is the lack of reliability of such methodologies. In this talk, we will discuss fundamental limitations, showing that there do exist severe problems in terms of computability on any type of digital hardware, which seriously affects their reliability. At the same time, we also show that analog hardware such as neuromorphic computing or quantum computing could achieve true reliability.
