Deep learning neural networks are trained using the stochastic gradient descent optimization algorithm.<\/p>\n
The learning rate is a hyperparameter that controls how much to change the model in response to the estimated error each time the model weights are updated. Choosing the learning rate is challenging as a value too small may result in a long training process that could get stuck, whereas a value too large may result in learning a sub-optimal set of weights too fast or an unstable training process.<\/p>\n
The learning rate may be the most important hyperparameter when configuring your neural network. Therefore it is vital to know how to investigate the effects of the learning rate on model performance and to build an intuition about the dynamics of the learning rate on model behavior.<\/p>\n
In this tutorial, you will discover the effects of the learning rate, learning rate schedules, and adaptive learning rates on model performance.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- How large learning rates result in unstable training and tiny rates result in a failure to train.<\/li>\n
- Momentum can accelerate training and learning rate schedules can help to converge the optimization process.<\/li>\n
- Adaptive learning rates can accelerate training and alleviate some of the pressure of choosing a learning rate and learning rate schedule.<\/li>\n<\/ul>\n
Let\u2019s get started.<\/p>\n
![\"Understand](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/01\/Understand-the-Dynamics-of-Learning-Rate-on-Model-Performance-With-Deep-Learning-Neural-Networks.jpg\")
<\/p>\nUnderstand the Dynamics of Learning Rate on Model Performance With Deep Learning Neural Networks
Photo by Abdul Rahman<\/a> some rights reserved<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into six parts; they are:<\/p>\n
- \n
- Learning Rate and Gradient Descent<\/li>\n
- Configure the Learning Rate in Keras<\/li>\n
- Multi-Class Classification Problem<\/li>\n
- Effect of Learning Rate and Momentum<\/li>\n
- Effect of Learning Rate Schedules<\/li>\n
- Effect of Adaptive Learning Rates<\/li>\n<\/ol>\n
Learning Rate and Gradient Descent<\/h2>\n
Deep learning neural networks are trained using the stochastic gradient descent algorithm.<\/p>\n
Stochastic gradient descent is an optimization algorithm that estimates the error gradient for the current state of the model using examples from the training dataset, then updates the weights of the model using the back-propagation of errors algorithm, referred to as simply backpropagation.<\/p>\n
The amount that the weights are updated during training is referred to as the step size or the \u201clearning rate<\/em>.\u201d<\/p>\n
Specifically, the learning rate is a configurable hyperparameter used in the training of neural networks that has a small positive value, often in the range between 0.0 and 1.0.<\/p>\n
The learning rate controls how quickly the model is adapted to the problem. Smaller learning rates require more training epochs given the smaller changes made to the weights each update, whereas larger learning rates result in rapid changes and require fewer training epochs.<\/p>\n
A learning rate that is too large can cause the model to converge too quickly to a suboptimal solution, whereas a learning rate that is too small can cause the process to get stuck.<\/p>\n
The challenge of training deep learning neural networks involves carefully selecting the learning rate. It may be the most important hyperparameter for the model.<\/p>\n
\n
The learning rate is perhaps the most important hyperparameter. If you have time to tune only one hyperparameter, tune the learning rate.<\/p>\n<\/blockquote>\n