NAVER LABS Europe seminars are open to the public. This seminar is virtual and requires registration.
Date: 8th December 2020, 3pm (GMT +01.00)
Expressivity and learnability: linear regions in deep ReLU networks
Speaker: David Rolnick is assistant professor in the School of Computer Science at McGill University and at the Mila Quebec AI Institute. His work focuses on deep learning theory and on applications of machine learning to climate change. He is co-founder and chair of Climate Change AI and serves as scientific co-director of Sustainability in the Digital Age. Dr. Rolnick has also worked at Google and DeepMind, and is a former NSF Mathematical Sciences Postdoctoral Research Fellow, NSF Graduate Research Fellow, and Fulbright Scholar. He received his Ph.D. in Applied Mathematics from MIT.
Abstract: In this talk, we show that there is a large gap between the maximum complexity of the functions that a neural network can express and the expected complexity of the functions that it learns in practice. Deep ReLU networks are piecewise linear functions, and the number of distinct linear regions is a natural measure of their expressivity. It is well-known that the maximum number of linear regions grows exponentially with the depth of the network, and this has often been used to explain the success of deeper networks. We show that the expected number of linear regions in fact grows polynomially in the size of the network, far below the exponential upper bound and independent of the depth of the network. This statement holds true both at initialization and after training, under natural assumptions for gradient-based learning algorithms. We also show that the linear regions of a ReLU network reveal information about the network’s parameters. In particular, it is possible to reverse-engineer the weights and architecture of an unknown deep ReLU network merely by querying it.