Registered Data

Requests to Author
Please respond to the following request regarding your contributed talk.
(Deadline : August 1, 2023)

  1. Please specify the presentation format.
  2. Please note that a speaker must make registration by July 20.
  3. If you would like to make any corrections to the contents, please submit your request here (Deadline : August 1, 2023). However, it will take some time to reflect the changes.

[00141] Multiscale Perturbed Gradient Descent: Chaotic Regularization and Heavy-Tailed Limits

  • Session Time & Room : 5D (Aug.25, 15:30-17:10) @E711
  • Type : Contributed Talk
  • Submitted : 2022-10-11
  • Abstract : Recent studies have shown that gradient descent (GD) can achieve improved generalization when its dynamics exhibits a chaotic behavior. However, to obtain the desired effect, the step-size should be chosen sufficiently large, a task which is problem dependent and can be difficult in practice. In this talk, we introduce multiscale perturbed GD (MPGD), a novel optimization framework where the GD recursion is augmented with chaotic perturbations that evolve via an independent dynamical system. We analyze MPGD from three different angles: (i) By building up on recent advances in rough paths theory, we show that, under appropriate assumptions, as the step-size decreases, the MPGD recursion converges weakly to a stochastic differential equation (SDE) driven by a heavy-tailed Lévy-stable process. (ii) By making connections to recently developed generalization bounds for heavy-tailed processes, we derive a generalization bound for the limiting SDE and relate the worst-case generalization error over the trajectories of the process to the parameters of MPGD. (iii) We analyze the implicit regularization effect brought by the dynamical regularization and show that, in the weak perturbation regime, MPGD introduces terms that penalize the Hessian of the loss function. Empirical results are provided to demonstrate the advantages of MPGD.
  • Classification : 68T07, Machine learning, optimization, stochastic differential equations
  • Format : Online Talk on Zoom [Change to Onsite]
  • Author(s) :
    • Soon Hoe Lim (Nordita, KTH Royal Institute of Technology and Stockholm University)