The training process of neural networks is a challenging optimization process that can often fail to converge.<\/p>\n
This can mean that the model at the end of training may not be a stable or best-performing set of weights to use as a final model.<\/p>\n
One approach to address this problem is to use an average of the weights from multiple models seen toward the end of the training run. This is called Polyak-Ruppert averaging and can be further improved by using a linearly or exponentially decreasing weighted average of the model weights. In addition to resulting in a more stable model, the performance of the averaged model weights can also result in better performance.<\/p>\n
In this tutorial, you will discover how to combine the weights from multiple different models into a single model for making predictions.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- The stochastic and challenging nature of training neural networks can mean that the optimization process does not converge.<\/li>\n
- Creating a model with the average of the weights from models observed towards the end of a training run can result in a more stable and sometimes better-performing solution.<\/li>\n
- How to develop final models created with the equal, linearly, and exponentially weighted average of model parameters from multiple saved models.<\/li>\n<\/ul>\n
Let\u2019s get started.<\/p>\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/01\/How-to-Create-an-Equally-Linearly-and-Exponentially-Weighted-Average-of-Neural-Network-Model-Weights-in-Keras.jpg\")
<\/p>\nHow to Create an Equally, Linearly, and Exponentially Weighted Average of Neural Network Model Weights in Keras
Photo by netselesoobrazno<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into seven parts; they are:<\/p>\n
- \n
- Average Model Weight Ensemble<\/li>\n
- Multi-Class Classification Problem<\/li>\n
- Multilayer Perceptron Model<\/li>\n
- Save Multiple Models to File<\/li>\n
- New Model With Average Model Weights<\/li>\n
- Predicting With an Average Model Weight Ensemble<\/li>\n
- Linearly and Exponentially Decreasing Weighted Average<\/li>\n<\/ol>\n
Average Model Weight Ensemble<\/h2>\n
Learning the weights for a deep neural network model requires solving a high-dimensional non-convex optimization problem.<\/p>\n
A challenge with solving this optimization problem is that there are many \u201cgood<\/em>\u201d solutions and it is possible for the learning algorithm to bounce around and fail to settle in on one. In the area of stochastic optimization, this is referred to as problems with the convergence of the optimization algorithm on a solution, where a solution is defined by a set of specific weight values.<\/p>\n
A symptom you may see if you have a problem with the convergence of your model is train and\/or test loss value that shows higher than expected variance, e.g. it thrashes or bounces up and down over training epochs.<\/p>\n
One approach to address this problem is to combine the weights collected towards the end of the training process. Generally, this might be referred to as temporal averaging and is known as Polyak Averaging or Polyak-Ruppert averaging, named for the original developers of the method.<\/p>\n
\n
Polyak averaging consists of averaging together several points in the trajectory through parameter space visited by an optimization algorithm.<\/p>\n<\/blockquote>\n
\u2014 Page 322, Deep Learning<\/a>, 2016.<\/p>\n
Averaging multiple noisy sets of weights during the learning process may paradoxically sound less desirable than tuning the optimization process itself, but may prove a desirable solution, especially for very large neural networks that may take days, weeks, or even months to train.<\/p>\n
\n
The essential advancement was reached on the basis of the paradoxical idea: a slow algorithm having less than optimal convergence rate must be averaged.<\/p>\n<\/blockquote>\n
\u2014 Acceleration of Stochastic Approximation by Averaging<\/a>, 1992.<\/p>\n
Averaging the weights of multiple models from a single training run has the effect of calming down the noisy optimization process that may be noisy because of the choice of learning hyperparameters (e.g. learning rate) or the shape of the mapping function that is being learned. The result is a final model or set of weights that may offer a more stable, and perhaps more accurate result.<\/p>\n
\n
The basic idea is that the optimization algorithm may leap back and forth across a valley several times without ever visiting a point near the bottom of the valley. The average of all of the locations on either side should be close to the bottom of the valley though.<\/p>\n<\/blockquote>\n
\u2014 Page 322, Deep Learning<\/a>, 2016.<\/p>\n
The simplest implementation of Polyak-Ruppert averaging involves calculating the average of the weights of the models over the last few training epochs.<\/p>\n
This can be improved by calculating a weighted average, where more weight is applied to more recent models, which is linearly decreased through prior epochs. An alternative and more widely used approach is to use an exponential decay in the weighted average.<\/p>\n
\n
Polyak-Ruppert averaging has been shown to improve the convergence of standard SGD [\u2026] . Alternatively, an exponential moving average over the parameters can be used, giving higher weight to more recent parameter value.<\/p>\n<\/blockquote>\n
\u2014 Adam: A Method for Stochastic Optimization<\/a>, 2014.<\/p>\n
Using an average or weighted average of model weights in the final model is a common technique in practice for ensuring the very best results are achieved from the training run. The approach is one of many \u201ctricks<\/em>\u201d used in the Google Inception V2 and V3 deep convolutional neural network models for photo classification, a milestone in the field of deep learning.<\/p>\n
\n
Model evaluations are performed using a running average of the parameters computed over time.<\/p>\n<\/blockquote>\n