Abstract : In recent years, supervised and unsupervised learning models based on deep neural networks play an increasingly important role in many directions, such as approximating continuous functions, constructing image models, and solving inverse problems. Meanwhile, many theoretical results are carried out to study the approximation and training properties of these approaches. This mini-symposium will bring together researchers in different areas to discuss recent advances in training algorithms and model applications, as well as relevant theoretical analysis. The aim is to assemble new understandings of the efficiency and limitation of deep learning models through the intersection discussion.
02386 (1/2) : 4E @E817 A715 [Chair: Yongqiang Cai]
[03440] Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems
Format : Talk at Waseda University
Author(s) :
Yongqiang Cai (Beijing Normal University)
Abstract : Deep learning has made significant progress in the fields of data science and natural science. Some studies have linked deep neural networks to dynamical systems, but the network structure is restricted to a residual network. It is known that residual networks can be regarded as a numerical discretization of dynamical systems. In this talk, we consider the traditional network structure and prove that vanilla feedforward networks can also be used for the numerical discretization of dynamical systems, where the width of the network is equal to the dimensions of the input and output. Our proof is based on the properties of the leaky-ReLU function and the numerical technique of the splitting method for solving differential equations. Our results could provide a new perspective for understanding the approximation properties of feedforward neural networks.
[03441] Phase Diagram of Initial Condensation for Two-layer Neural Networks
Format : Talk at Waseda University
Author(s) :
Zhengan Chen (Shanghai Jiao Tong University)
Yuqing Li (Shanghai Jiao Tong University)
Tao Luo (Shanghai Jiao Tong University)
Zhangchen Zhou (Shanghai Jiao Tong University)
Zhiqin Xu (Shanghai Jiao Tong University)
Abstract : The phenomenon of distinct behaviors exhibited by neural networks under varying scales of initialization remains an enigma in deep learning research. In this paper, based on the earlier work by Luo et al.~\cite{luo2021phase}, we present a phase diagram of initial condensation for two-layer neural networks. Condensation is a phenomenon wherein the weight vectors of neural networks concentrate on isolated orientations during the training process, and it is a feature in non-linear learning process that enables neural networks to possess better generalization abilities. Our phase diagram serves to provide a comprehensive understanding of the dynamical regimes of neural networks and their dependence on the choice of hyperparameters related to initialization. Furthermore, we demonstrate in detail the underlying mechanisms by which small initialization leads to condensation at the initial training stage.
[03386] Robust Full Waveform Inversion: A Source Wavelet Manipulation Perspective
Format : Talk at Waseda University
Author(s) :
Chenglong Bao (Tsinghua University)
Lingyun Qiu (Tsinghua University)
Rongqian Wang (Tsinghua University)
Abstract : Full-waveform inversion (FWI) is a powerful tool for high-resolution subsurface parameter reconstruction. Due to the existence of local minimum traps, the success of the inversion process usually requires a good initial model. Our study primarily focuses on understanding the impact of source wavelets on the landscape of the corresponding optimization problem. We thus introduce a decomposition scheme that divides the inverse problem into two parts. The first step transforms the measured data into data associated with the desired source wavelet. Here, we consider inversions with known and unknown sources to mimic real scenarios. The second sub-problem is the conventional full waveform inversion, which is much less dependent on an accurate initial model since the previous step improves the misfit landscape. A regularized deconvolution method and a convolutional neural network are employed to solve the source transformation problem. Numerical experiments on the benchmark models demonstrate that our approach improves the gradient's quality in the subsequent FWI and provides a better inversion performance.
[02945] Learning robust imaging model with unpaired data
Format : Talk at Waseda University
Author(s) :
Chenglong Bao (Tsinghua University)
Abstract : In this talk, in the unpaired data regime, we discuss our recent progress in building AI-aided robust models and their applications in image processing. Leveraging the Bayesian inference framework, our model combines classical mathematical modeling and deep neural networks to improve interpretability. Experimental results on various real datasets validate the advantages of the proposed methods.
[03329] Generative Models Based Statistical Priors for Compressive Sensing and Medical Imaging
Format : Talk at Waseda University
Author(s) :
Jiulong Liu (Academy of Mathematics and Systems Science, Chinese Academy of Sciences)
Abstract : Sparsity is a mathematically elegant tool for reducing the sampling rate for compressive sensing reconstruction and thereby its applications are also extended to many underdetermined imaging systems, such as MRI and CT. However, with the development of deep learning, there are many methods proposed to learn data representation and they are shown to be more efficient in signal and image processing. In order to efficiently and stably solve the under-determined and ill-conditioned inverse problems with fewer measurements, we established compressive sensing reconstruction methods using generative priors which are shown much more efficient than the traditional priors or some other data-driven priors. In this talk, I will introduce some of these methods and present our recent results for MRI reconstruction, phase retrieval, and some other nonlinear inverse problems.
[03483] Normalizing-flows based design of experiments for failure probability estimation
Format : Talk at Waseda University
Author(s) :
Hongqiao Wang (Central South University)
Abstract : Failure probability estimation problem is an crucial task in engineering. In this work we consider this problem in the situation that the underlying computer models are extremely expensive, which often arises in the practice, and in this setting, reducing the calls of computer model is of essential importance. We formulate the problem of estimating the failure probability with expensive computer models as an sequential experimental design for the limit state (i.e., the failure boundary) and propose a series of efficient adaptive design criteria to solve the design of experiment (DOE). In particular, the proposed method employs the deep neural network (DNN) as the surrogate of limit state function for efficiently reducing the calls of expensive computer experiment. A map from the Gaussian distribution to the posterior approximation of the limit state is learned by the normalizing flows for the ease of experimental design. Three normalizing-flows-based design criteria are proposed in this work for deciding the design locations based on different assumption of generalization error. The accuracy and performance of the proposed method is demonstrated by both theory and practical examples.
[03477] Unsupervised learning driven by Langevin dynamics and its applications to inverse problems
Format : Talk at Waseda University
Author(s) :
Ji Li (Capital Normal University)
Abstract : From the Bayesian view, the key component of image restoration is to estimate the posterior distribution. Generally, the sampling from posterior distribution is intractable. To this end, there have been some variational approaches to approximate the posterior distribution using a proxy distribution. In this talk, we first review the Langevin dynamics as an effective sampler for a given distribution. Then we apply it or embed it to the unsupervised learning solution to two image restoration problems with slight modifications.
[03450] Self-supervised Deep learning Methods in Imaging
Format : Online Talk on Zoom
Author(s) :
Tongyao Pang (National University of Singapore)
Abstract : In this talk, I will share our recent research on using self-supervised deep learning techniques for image reconstruction. Deep learning has recently become a powerful tool in image restoration but it requires a large amount of paired training data. Our proposed self-supervised methods alleviate this requirement while still achieving comparable performance to supervised learning. Our methods are designed to find the minimum mean-squared error (MMSE) solution from a Bayesian inference perspective.
