Course
-
Recommended 18.408 Theoretical Foundations for Deep Learning, Spring 202
– Universal Approximation and the Fourier Transform, NTK - GitHub - joanbruna/MathsDL-spring19: Mathematics of Deep Learning, Courant Insititute, Spring 19
- https://stats385.github.io
- https://mjt.cs.illinois.edu/courses/dlt-f21/
- ECE236C - Optimization Methods for Large-Scale Systems
- 18.409 Algorithmic Aspects of Machine Learning, Spring 2015
- Deep learning theory lecture notes
- DL Theory class 2020
- Syllabus for Statistical Learning Theory
- Machine Learning Theory (CS 6783) Course Webpage
- 18.408 Algorithmic Aspects of Machine Learning, Spring 2023
-
CMSC 828W: Foundations of Deep Learning
– let 7 NTK - Recommended Book [2106.10165] The Principles of Deep Learning Theory
- Book Neural Network Learning: Theoretical Foundations. Martin Anthony and Peter L. Bartlett
- Book A Probabilistic Theory of Pattern Recognition | SpringerLink
Survey, Tutorial
- GitHub - WeiHuang05/Awesome-Feature-Learning-in-Deep-Learning-Thoery: Welcome to the Awesome Feature Learning in Deep Learning Thoery Reading Group! This repository serves as a collaborative platform for scholars, enthusiasts, and anyone interested in delving into the fascinating world of feature learning within deep learning theory.
- Ju Sun | Provable Nonconvex Methods/Algorithms
- [2012.10931] Recent advances in deep learning theory
- IJCAI 21 survey Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities | IJCAI
- ICML 19 workshop https://icml.cc/Conferences/2019/ScheduleMultitrack?event=3531
- Blog Deep Learning Optimization Theory — Introduction | Towards Data Science
- Blog Advancing AI theory with a first-principles understanding of deep neural networks
- Theoretical issues in deep networks - PubMed
- [1610.01145] Error bounds for approximations with deep ReLU networks
- 深度学习理论研究之路 - 知乎
Conference
Paper
-
https://dl.acm.org/doi/10.5555/3491440.3491833
– spectral pruning -
https://www.pnas.org/doi/10.1073/pnas.1907369117
– gradient flow -
[2105.02375] A Geometric Analysis of Neural Collapse with Unconstrained Features
– optimization landscape -
Deep Neural Network Approximation Theory | IEEE Journals & Magazine | IEEE Xplore
– approximation theory - A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
– NTK -
Information Geometry of Orthogonal Initializations and Training | OpenReview
– info geometry, Fisher information matrix -
[1806.01316] Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
– info geometry, Fisher information matrix
dynamical isometry
- 深度学习平均场理论第三讲：tanh的复兴和dynamical isometry - 知乎
- https://www.annualreviews.org/doi/pdf/10.1146/annurev-conmatphys-031119-050745
- https://arxiv.org/pdf/1711.04735.pdf
- http://proceedings.mlr.press/v84/pennington18a/pennington18a.pdf
- Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
- The emergence of spectral universality in deep networks