Training deep neural networks was traditionally challenging as the vanishing gradient meant that weights in layers close to the input layer were not updated in response to errors calculated on the training dataset.<\/p>\n
An innovation and important milestone in the field of deep learning was greedy layer-wise pretraining that allowed very deep neural networks to be successfully trained, achieving then state-of-the-art performance.<\/p>\n
In this tutorial, you will discover greedy layer-wise pretraining as a technique for developing deep multi-layered neural network models.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- Greedy layer-wise pretraining provides a way to develop deep multi-layered neural networks whilst only ever training shallow networks.<\/li>\n
- Pretraining can be used to iteratively deepen a supervised model or an unsupervised model that can be repurposed as a supervised model.<\/li>\n
- Pretraining may be useful for problems with small amounts labeled data and large amounts of unlabeled data.<\/li>\n<\/ul>\n
Let\u2019s get started.<\/p>\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/02\/How-to-Develop-Deep-Neural-Networks-With-Greedy-Layer-Wise-Pretraining.jpg\")
<\/p>\nHow to Develop Deep Neural Networks With Greedy Layer-Wise Pretraining
Photo by Marco Verch<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into four parts; they are:<\/p>\n
- \n
- Greedy Layer-Wise Pretraining<\/li>\n
- Multi-Class Classification Problem<\/li>\n
- Supervised Greedy Layer-Wise Pretraining<\/li>\n
- Unsupervised Greedy Layer-Wise Pretraining<\/li>\n<\/ol>\n
Greedy Layer-Wise Pretraining<\/h2>\n
Traditionally, training deep neural networks with many layers was challenging.<\/p>\n
As the number of hidden layers is increased, the amount of error information propagated back to earlier layers is dramatically reduced. This means that weights in hidden layers close to the output layer are updated normally, whereas weights in hidden layers close to the input layer are updated minimally or not at all. Generally, this problem prevented the training of very deep neural networks and was referred to as the vanishing gradient problem<\/em>.<\/p>\n
An important milestone in the resurgence of neural networking that initially allowed the development of deeper neural network models was the technique of greedy layer-wise pretraining, often simply referred to as \u201cpretraining<\/em>.\u201d<\/p>\n
\n
The deep learning renaissance of 2006 began with the discovery that this greedy learning procedure could be used to find a good initialization for a joint learning procedure over all the layers, and that this approach could be used to successfully train even fully connected architectures.<\/p>\n<\/blockquote>\n
\u2014 Page 528, Deep Learning<\/a>, 2016.<\/p>\n
Pretraining involves successively adding a new hidden layer to a model and refitting, allowing the newly added model to learn the inputs from the existing hidden layer, often while keeping the weights for the existing hidden layers fixed. This gives the technique the name \u201clayer-wise<\/em>\u201d as the model is trained one layer at a time.<\/p>\n
The technique is referred to as \u201cgreedy<\/em>\u201d because the piecewise or layer-wise approach to solving the harder problem of training a deep network. As an optimization process, dividing the training process into a succession of layer-wise training processes is seen as a greedy shortcut that likely leads to an aggregate of locally optimal solutions, a shortcut to a good enough global solution.<\/p>\n
\n
Greedy algorithms break a problem into many components, then solve for the optimal version of each component in isolation. Unfortunately, combining the individually optimal components is not guaranteed to yield an optimal complete solution.<\/p>\n<\/blockquote>\n
\u2014 Page 323, Deep Learning<\/a>, 2016.<\/p>\n
Pretraining is based on the assumption that it is easier to train a shallow network instead of a deep network and contrives a layer-wise training process that we are always only ever fitting a shallow model.<\/p>\n
\n
\u2026 builds on the premise that training a shallow network is easier than training a deep one, which seems to have been validated in several contexts.<\/p>\n<\/blockquote>\n
\u2014 Page 529, Deep Learning<\/a>, 2016.<\/p>\n
The key benefits of pretraining are:<\/p>\n
- \n
- Simplified training process.<\/li>\n
- Facilitates the development of deeper networks.<\/li>\n
- Useful as a weight initialization scheme.<\/li>\n
- Perhaps lower generalization error.<\/li>\n<\/ul>\n
\n
In general, pretraining may help both in terms of optimization and in terms of generalization.<\/p>\n<\/blockquote>\n
\u2014 Page 325, Deep Learning<\/a>, 2016.<\/p>\n
There are two main approaches to pretraining; they are:<\/p>\n
- \n
- Supervised greedy layer-wise pretraining.<\/li>\n
- Unsupervised greedy layer-wise pretraining.<\/li>\n<\/ul>\n
Broadly, supervised pretraining involves successively adding hidden layers to a model trained on a supervised learning task. Unsupervised pretraining involves using the greedy layer-wise process to build up an unsupervised autoencoder model, to which a supervised output layer is later added.<\/p>\n
\n
It is common to use the word \u201cpretraining\u201d to refer not only to the pretraining stage itself but to the entire two phase protocol that combines the pretraining phase and a supervised learning phase. The supervised learning phase may involve training a simple classifier on top of the features learned in the pretraining phase, or it may involve supervised fine-tuning of the entire network learned in the pretraining phase.<\/p>\n<\/blockquote>\n
\u2014 Page 529, Deep Learning<\/a>, 2016.<\/p>\n
Unsupervised pretraining may be appropriate when you have a significantly larger number of unlabeled examples that can be used to initialize a model prior to using a much smaller number of examples to fine tune the model weights for a supervised task.<\/p>\n
\n
\u2026. we can expect unsupervised pretraining to be most helpful when the number of labeled examples is very small. Because the source of information added by unsupervised pretraining is the unlabeled data, we may also expect unsupervised pretraining to perform best when the number of unlabeled examples is very large.<\/p>\n<\/blockquote>\n
\u2014 Page 532, Deep Learning<\/a>, 2016.<\/p>\n
Although the weights in prior layers are held constant, it is common to fine tune all weights in the network at the end after the addition of the final layer. As such, this allows pretraining to be considered a type of weight initialization method.<\/p>\n
\n
\u2026 it makes use of the idea that the choice of initial parameters for a deep neural network can have a significant regularizing effect on the model (and, to a lesser extent, that it can improve optimization).<\/p>\n<\/blockquote>\n
\u2014 Page 530-531, Deep Learning<\/a>, 2016.<\/p>\n
Greedy layer-wise pretraining is an important milestone in the history of deep learning, that allowed the early development of networks with more hidden layers than was previously possible. The approach can be useful on some problems; for example, it is best practice to use unsupervised pretraining for text data in order to provide a richer distributed representation of words and their interrelationships via word2vec<\/a>.<\/p>\n
\n
Today, unsupervised pretraining has been largely abandoned, except in the field of natural language processing [\u2026] the advantage of pretraining is that one can pretrain once on a huge unlabeled set (for example with a corpus containing billions of words), learn a good representation (typically of words, but also of sentences), and then use this representation or fine-tune it for a supervised task for which the training set contains substantially fewer examples.<\/p>\n<\/blockquote>\n
\u2014 Page 535, Deep Learning<\/a>, 2016.<\/p>\n
Nevertheless, it is likely better performance may be achieved using modern methods such as better activation functions, weight initialization, variants of gradient descent, and regularization methods.<\/p>\n
\n
Today, we now know that greedy layer-wise pretraining is not required to train fully connected deep architectures, but the unsupervised pretraining approach was the first method to succeed.<\/p>\n<\/blockquote>\n