Statistical theory for deep neural networks with ReLU activation function  - VVSOR - VVSOR

Netherlands Society for Statistics and Operations Research | Dutch

Statistical theory for deep neural networks with ReLU activation function 

We cordially invite you to the next meeting of the Thematic Statistics Seminar with current focus on Machine Learning from a statistical perspective.

Johannes Schmidt-Hieber (MI Leiden):  Statistical theory for deep neural networks with ReLU activation function 

Time: 15:00-16:00, April 6, 2018

Location: UvA, Science Park 904, room D1.115

Abstract: The universal approximation theorem states that neural networks are capable of approximating any continuous function up to a small error that depends on the size of the network. The expressive power of a network does, however, not guarantee that deep networks perform well on data. For that, control of the statistical estimation risk is needed. In the talk, we derive statistical theory for fitting deep neural networks to data generated from the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to logarithmic factors) under a general composition assumption on the regression function. The framework includes many well-studied structural constraints such as (generalized) additive models. While there is a lot of flexibility in the network architecture, the tuning parameter is the sparsity of the network. Specifically, we consider large networks with number of potential parameters being much bigger than the sample size. Interestingly, the depth (number of layers) of the neural network architectures plays an important role and our theory suggests that scaling the network depth with the logarithm of the sample size is natural.

For the list of upcoming talks and further information about the seminar please visit the seminar webpage: StructuresSeminar/