Abstract : Geometric deep learning has important applications in the fields of quantum computing, 3D perception, molecular designs, and the discovery of mathematical theorems. It takes account of properties such as invariance and equivariance. Many existing structure-aware deep networks lack rigorous theoretical foundations of desired properties in modeling, such as network stability, interpretability, and efficient computation. This workshop will gather researchers from mathematics and computer sciences to provide a forum to establish diverse mathematical theories for geometric deep learning, such as harmonic analysis, algebraic topology, algebraic geometry, combinatorics, differential geometry, differential equations, graph theory, approximation theory, statistics, and theoretical computer science.
Organizer(s) : Yuguang Wang, Bingxin Zhou, Yuelin Wang
[03411] On the stability of spectral graph filters and beyond
Format : Talk at Waseda University
Author(s) :
Xiaowen Dong (University of Oxford)
Abstract : Data collected in network domains, hence supported by an (irregular) graph rather than a (regular) grid-like structure, are becoming pervasive. Typical examples include gene expression data associated with a protein-protein interaction graph, or behaviours of a group of individuals in a social network. Graph-based signal processing and machine learning are recent techniques that have been developed to handle such graph-structured data and have seen applications in such diverse fields as drug discovery, fake news detection, and traffic prediction. However, a theoretical understanding of the robustness of these models against perturbation to the input graph domain has been lacking. In this talk, I will present our results on the stability bounds of spectral graph filters as well as other recent work on the robustness of graph machine learning models, which together will contribute to the deployment of these models in real-world scenarios.
[01565] Spherical Framelets with Directionality for Spherical Neural Networks
Format : Talk at Waseda University
Author(s) :
Jianfei Li (City University of Hong Kong)
Han Feng (City University of Hong Kong)
Xiaosheng Zhuang (City University of Hong Kong)
Abstract : In this talk, we shall focus on the constructions and applications of directional framelets beyond the Euclidean domain, i.e., on the 2-sphere. We shall discuss their characterizations in terms of the affine systems. Fast algorithmic framelet transforms associated with the underlying filter banks or multiscale structures will be investigated. Moreover, based on our spherical framelets with directionality, we shall consider the development of spherical convolutional neural network (SNN) model for deep learning tasks.
[01568] Some Applications of Hyperplane Arrangements in Deep Learning
Format : Talk at Waseda University
Author(s) :
Huan Xiong (HIT and MBZUAI)
Abstract : In this talk, we build some connections between hyperplane arrangements and Piecewise Linear Convolutional Neural Networks (PLCNNs), and use them to derive maximal and average numbers of linear regions for one-layer PLCNNs. Furthermore, we obtain upper and lower bounds for the number of linear regions of multi-layer PLCNNs. Our results suggest that deeper ReLU CNNs have more powerful expressivity than their shallow counterparts, while ReLU CNNs have more expressivity than fully-connected ReLU NNs per parameter.
[02341] Generalization Capabilities of Graph Neural Networks
Format : Talk at Waseda University
Author(s) :
Gitta Kutyniok (LMU Munich)
Sohir Maskey (LMU Munich)
Ron Levie (Technion)
Yunseok Lee (LMU Munich)
Abstract : The tremendous importance of graph structured data due to recommender systems, social networks, or biological applications led to the introduction of graph neural networks. One key question in machine learning is the ability of a learnt model to generalize to unknown data sets. In this talk, we will present several results on the generalization capabilities of graph neural networks, focussing on both message passing and spectral graph neural networks.
[01769] Machine Learning in Banach Spaces: A Black-box or White-box Method?
Format : Talk at Waseda University
Author(s) :
Qi Ye (South China Normal University)
Abstract : In this talk, we study the whole theory of regularized learning for generalized data in Banach spaces including representer theorems, approximation theorems, and convergence theorems. Specially, we combine the data-driven and model-driven methods to study the new algorithms and theorems of the regularized learning. Usually the data-driven and model-driven methods are used to analyze the black-box and white-box models, respectively. With the same thought of the Tai Chi diagram, we use the discrete local information of the black-box and white-box models to construct the global approximate solutions by the regularized learning. Our original ideas are inspired by the eastern philosophy such as the golden mean. The work of the regularized learning for generalized data provides another road to study the algorithms of machine learning including: 1. the interpretability in approximation theory, 2. the nonconvexity and nonsmoothness in optimization theory, 3. the generalization and overfitting in regularization theory. Moreover, based on the theory of the regularized learning, we will construct the composite algorithms combining the model‑driven and data‑driven methods for our current research projects of the big data analytics in education and medicine.
[01857] Stable Hyperbolic Neural Networks for Graph Generation and Classification
Format : Online Talk on Zoom
Author(s) :
Eric Qu (Duke Kunshan University)
Dongmian Zou (Duke Kunshan University)
Abstract : The past few years have witnessed successful development of hyperbolic neural networks. However, they are known to suffer from instability in training. In this talk, we present two recent works that build novel hyperbolic layers for generation and classification tasks on tree-like and hierarchical-structured data. The first defines a stable hybrid AE-GAN model; the second defines a hyperbolic convolutional layer built upon pre-defined kernel points. We illustrate their competitiveness by showing extensive numerical results.
[02110] Spectral-Inspired Graph Neural Networks
Format : Talk at Waseda University
Author(s) :
Teresa Huang (Johns Hopkins University)
Abstract : Message Passing Neural Networks (MPNNs) can suffer from expressivity issues like over-smoothing and over-squashing. To mitigate such issues, we propose PowerEmbed -- a simple layer-wise normalization technique to boost MPNNs. We show PowerEmbed can provably express the top-k leading eigenvectors of the graph operator, which prevents over-smoothing and is agnostic to the graph topology; meanwhile, it produces a list of representations ranging from local features to global signals, which avoids over-squashing.
[01563] Negative sampling for graph neural networks based on determinantal point processes
Format : Talk at Waseda University
Author(s) :
Junyu Xuan (University of Technology Sydney)
Abstract : Graph neural networks (GNNs) have become the de facto standard of a variety of graph-based applications. Most GNNs are built on a message-passing mechanism and only aggregate information from the first-order neighbours (positive samples), which may lead to over-smoothing, limited expressive power and over-squashing. However, beyond these neighbouring nodes, graphs have a large, dark, all-but forgotten world in which we find the non-neighbouring nodes (negative samples) that are helpful for representation learning.
Abstract : Graph neural network (GNN) with message Passing scheme provides an elegant way to process graph data. However, the stability, oversmoothing and oversquashing problems commonly exist in current GNN models. We propose the message passing scheme with scattering transform which is proved theoretically to solve the above problems. In the numerical experiments, the scattering message passing has been validated to be effective compared to SOTA GNN models.
[05231] Ridgelet Transforms of Neural Network on Manifolds and Hilbert Spaces
Format : Talk at Waseda University
Author(s) :
Sho Sonoda (RIKEN AIP)
Abstract : Ridgelet transform is a pseudo-inverse operator of neural networks. Namely, given a function $f \in L^2(\mathbb{R}^m)$, the ridgelet transform $R[f]$ describes how the network parameters should be distributed for the network to represent $f$. In this talk, I will explain a systematic scheme to derive the ridgelet transform by turning the network into a Fourier expression. As applications, we extend the scheme to networks on manifolds $G/K$ and Hilbert spaces $H$, and derive their associated ridgelet transforms.
[03588] Geometric Diffusion Generative model for protein sequence design
Format : Talk at Waseda University
Author(s) :
Kai Yi (UNSW)
Abstract : Propose a new approach to protein sequence design using a diffusion generative model in geometric. This approach has the potential to improve protein properties and has implications for biotechnology and medicine. The method outperforms state-of-the-art methods on benchmark datasets.
[04999] On oversquashing and expressivity: can GNNs mix variables?
Format : Online Talk on Zoom
Author(s) :
Francesco Di Giovanni (University of Cambridge)
Abstract : I discuss how Message Passing Neural Networks (MPNNs) model mixing among features in a graph. As a consequence of this approach, I show that MPNNs may need as many layers as the (largest) commute time to model strong mixing of distant nodes in a graph. This allows to derive a measure for over-squashing and to clarify how the latter limits the expressivity of MPNNs to learn functions with long-range interactions.
[05109] Geometric Deep Learning from a Topological Viewpoint
Format : Online Talk on Zoom
Author(s) :
Cristian Bodnar (Microsoft Research)
Abstract : The multitude of applications where data is attached to spaces with non-Euclidean structure has driven the rise of the field of Geometric Deep Learning (GDL). Nonetheless, from many points of view, geometry does not always provide the right level of abstraction to study all the spaces that commonly emerge in such settings. For instance, graphs, by far the most prevalent type of space in GDL, do not even have a geometrical structure in the strict sense. In this talk, I will explore how we can take a (more general) topological perspective of the field with a focus on understanding and developing Graph Neural Network models.
[03727] FoSR: First-order spectral rewiring for addressing oversquashing in GNNs
Format : Talk at Waseda University
Author(s) :
Guido Montufar (UCLA and MPI MiS)
Abstract : Graph neural networks (GNNs) are able to leverage the structure of graph data by passing messages along the edges of the graph. While this allows GNNs to learn features depending on the graph structure, for certain graph topologies it leads to inefficient information propagation and a problem known as oversquashing. This has recently been linked with the curvature and spectral gap of the graph. On the other hand, adding edges to the message-passing graph can lead to increasingly similar node representations and a problem known as oversmoothing. We propose a computationally efficient algorithm that prevents oversquashing by systematically adding edges to the graph based on spectral expansion. We combine this with a relational architecture, which lets the GNN preserve the original graph structure and provably prevents oversmoothing. We find experimentally that our algorithm outperforms existing graph rewiring methods in several graph classification tasks. This is work with Kedar Karhadkar and Pradeep Kr. Banerjee.
[04482] DynG2G: An Efficient Stochastic Graph Embedding Method for Temporal Graphs
Format : Online Talk on Zoom
Author(s) :
Mengjia Xu (Brown University & MIT)
Apoorva Vikram Singh (National Institute of Technology)
George Em Karniadakis (Brown University)
Abstract : Dynamic graph embedding has gained great attention due to its capability of learning low-dimensional graph embeddings for complex temporal graphs with high accuracy. However, recent advances mostly focus on learning node embeddings as deterministic ``vectors'' for static graphs, hence disregarding the key graph temporal dynamics and the evolving uncertainty associated with node embedding in the latent space. We propose an efficient stochastic dynamic graph embedding method (DynG2G) that applies an inductive feed-forward encoder trained with node triplet energy-based ranking loss. Every node per timestamp is encoded as a time-dependent probabilistic multivariate Gaussian distribution in the latent space. We adopted eight benchmarks of different sizes and evolving dynamics (from slowly changing dynamics to rapidly varying multi-rate dynamics). Our experiments indicate that DynG2G achieves new state-of-the-art performance in capturing the temporal node embeddings and simultaneously predicting the evolving node embedding uncertainty, which plays a crucial role in quantifying the intrinsic dimensionality of the dynamical system over time. We also obtain a “universal” relation of the optimal embedding dimension $L$ versus the effective dimensionality of uncertainty ($D$). The $L$ - $D$ correlation provides a clear path for selecting the optimum embedding size adaptively per timestamp by $L \ge D$.
[05074] Applied harmonic analysis and particle dynamics for designing neural message passing on graphs
Format : Talk at Waseda University
Author(s) :
Yuguang Wang (Shanghai Jiao Tong University)
Abstract : Graph representation learning has broad applications applications from recommendation systems to drug and protein designs. In this talk, I will talk about using harmonic analysis and particle systems to design useful neural message passing with theoretically guaranteed separability and efficient computation. These message passings are proved to have strictly positive lower bounded Dirichlet energy and thus to circumvent the oversmoothing problem appearing in many spatial GNNs, when the node features are indistinguishable as the network deepens.
