|
7 Nov 2011 - Cambridge MA
New paper about hyper-parameter optimization in deep models, to appear at NIPS 2011. [pdf] Also, check out the software related to the paper. You can use it to optimize hyper-parameters in your own work.
Update: I’ll also present this and the hyperopt software at the Workshop on Bayesian Optimization, Experimental Design and Bandits at NIPS 2011 (Dec 16).
Title:
Algorithms for Hyper-parameter Optimization
Authors:
J. Bergstra, R. Bardenet, Y. Bengio, and B. Kegl.
Abstract:
Several recent advances to the state of the art in image classification benchmarks have come from better configurations of existing techniques rather than novel approaches to feature learning. Traditionally, hyper-parameter optimization has been the job of humans because they can be very efficient in regimes where only a few trials are possible. Presently, computer clusters and GPU processors make it possible to run more trials and we show that algorithmic approaches can find better results. We present hyper-parameter optimization results on tasks of training neural networks and deep belief networks (DBNs). We optimize hyper-parameters using random search and two new greedy sequential methods based on the expected improvement criterion. Random search has been shown to be sufficiently efficient for learning neural networks for several datasets, but we show it is unreliable for training DBNs. The sequential algorithms are applied to the most difficult DBN learning problems from [Larochelle et. al, 2007] and find significantly better results than the best previously reported. This work contributes novel techniques for making response surface models P(y|x) in which many elements of hyper-parameter assignment (x) are known to be irrelevant given particular values of other elements.