Neural networks are trained using gradient descent where the estimate of the error used to update the weights is calculated based on a subset of the training dataset.<\/p>\n
The number of examples from the training dataset used in the estimate of the error gradient is called the batch size and is an important hyperparameter that influences the dynamics of the learning algorithm.<\/p>\n
It is important to explore the dynamics of your model to ensure that you\u2019re getting the most out of it.<\/p>\n
In this tutorial, you will discover three different flavors of gradient descent and how to explore and diagnose the effect of batch size on the learning process.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- Batch size controls the accuracy of the estimate of the error gradient when training neural networks.<\/li>\n
- Batch, Stochastic, and Minibatch gradient descent are the three main flavors of the learning algorithm.<\/li>\n
- There is a tension between batch size and the speed and stability of the learning process.<\/li>\n<\/ul>\n
Let\u2019s get started.<\/p>\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/01\/How-to-Control-the-Speed-and-Stability-of-Training-Neural-Networks-With-Gradient-Descent-Batch-Size.jpg\")
<\/p>\nHow to Control the Speed and Stability of Training Neural Networks With Gradient Descent Batch Size
Photo by Adrian Scottow<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into seven parts; they are:<\/p>\n
- \n
- Batch Size and Gradient Descent<\/li>\n
- Stochastic, Batch, and Minibatch Gradient Descent in Keras<\/li>\n
- Multi-Class Classification Problem<\/li>\n
- MLP Fit With Batch Gradient Descent<\/li>\n
- MLP Fit With Stochastic Gradient Descent<\/li>\n
- MLP Fit With Minibatch Gradient Descent<\/li>\n
- Effect of Batch Size on Model Behavior<\/li>\n<\/ol>\n
Batch Size and Gradient Descent<\/h2>\n
Neural networks are trained using the stochastic gradient descent optimization algorithm.<\/p>\n
This involves using the current state of the model to make a prediction, comparing the prediction to the expected values, and using the difference as an estimate of the error gradient. This error gradient is then used to update the model weights and the process is repeated.<\/p>\n
The error gradient is a statistical estimate. The more training examples used in the estimate, the more accurate this estimate will be and the more likely that the weights of the network will be adjusted in a way that will improve the performance of the model. The improved estimate of the error gradient comes at the cost of having to use the model to make many more predictions before the estimate can be calculated, and in turn, the weights updated.<\/p>\n
\n
Optimization algorithms that use the entire training set are called batch or deterministic gradient methods, because they process all of the training examples simultaneously in a large batch.<\/p>\n<\/blockquote>\n
\u2014 Page 278, Deep Learning<\/a>, 2016.<\/p>\n
Alternately, using fewer examples results in a less accurate estimate of the error gradient that is highly dependent on the specific training examples used.<\/p>\n
This results in a noisy estimate that, in turn, results in noisy updates to the model weights, e.g. many updates with perhaps quite different estimates of the error gradient. Nevertheless, these noisy updates can result in faster learning and sometimes a more robust model.<\/p>\n
\n
Optimization algorithms that use only a single example at a time are sometimes called stochastic or sometimes online methods. The term online is usually reserved for the case where the examples are drawn from a stream of continually created examples rather than from a fixed-size training set over which several passes are made.<\/p>\n<\/blockquote>\n
\u2014 Page 278, Deep Learning<\/a>, 2016.<\/p>\n
The number of training examples used in the estimate of the error gradient is a hyperparameter for the learning algorithm called the \u201cbatch size<\/em>,\u201d or simply the \u201cbatch<\/em>.\u201d<\/p>\n
A batch size of 32 means that 32 samples from the training dataset will be used to estimate the error gradient before the model weights are updated. One training epoch means that the learning algorithm has made one pass through the training dataset, where examples were separated into randomly selected \u201cbatch size<\/em>\u201d groups.<\/p>\n
Historically, a training algorithm where the batch size is set to the total number of training examples is called \u201cbatch gradient descent<\/em>\u201d and a training algorithm where the batch size is set to 1 training example is called \u201cstochastic gradient descent<\/em>\u201d or \u201conline gradient descent<\/em>.\u201d<\/p>\n
A configuration of the batch size anywhere in between (e.g. more than 1 example and less than the number of examples in the training dataset) is called \u201cminibatch gradient descent<\/em>.\u201d<\/p>\n
- \n
- Batch Gradient Descent<\/strong>. Batch size is set to the total number of examples in the training dataset.<\/li>\n
- Stochastic Gradient Descent<\/strong>. Batch size is set to one.<\/li>\n
- Minibatch Gradient Descent<\/strong>. Batch size is set to more than one and less than the total number of examples in the training dataset.<\/li>\n<\/ul>\n
For shorthand, the algorithm is often referred to as stochastic gradient descent regardless of the batch size. Given that very large datasets are often used to train deep learning neural networks, the batch size is rarely set to the size of the training dataset.<\/p>\n
Smaller batch sizes are used for two main reasons:<\/p>\n
- \n
- Smaller batch sizes are noisy, offering a regularizing effect and lower generalization error.<\/li>\n
- Smaller batch sizes make it easier to fit one batch worth of training data in memory (i.e. when using a GPU).<\/li>\n<\/ul>\n
A third reason is that the batch size is often set at something small, such as 32 examples, and is not tuned by the practitioner. Small batch sizes such as 32 do work well generally.<\/p>\n
\n
\u2026 [batch size] is typically chosen between 1 and a few hundreds, e.g. [batch size] = 32 is a good default value<\/p>\n<\/blockquote>\n
\u2014 Practical recommendations for gradient-based training of deep architectures<\/a>, 2012.<\/p>\n
\n
The presented results confirm that using small batch sizes achieves the best training stability and generalization performance, for a given computational cost, across a wide range of experiments. In all cases the best results have been obtained with batch sizes m = 32 or smaller, often as small as m = 2 or m = 4.<\/p>\n<\/blockquote>\n
- Stochastic Gradient Descent<\/strong>. Batch size is set to one.<\/li>\n
- Batch Gradient Descent<\/strong>. Batch size is set to the total number of examples in the training dataset.<\/li>\n