An interesting benefit of deep learning neural networks is that they can be reused on related problems.<\/p>\n
Transfer learning refers to a technique for predictive modeling on a different but somehow similar problem that can then be reused partly or wholly to accelerate the training and improve the performance of a model on the problem of interest.<\/p>\n
In deep learning, this means reusing the weights in one or more layers from a pre-trained network model in a new model and either keeping the weights fixed, fine tuning them, or adapting the weights entirely when training the model.<\/p>\n
In this tutorial, you will discover how to use transfer learning to improve the performance deep learning neural networks in Python with Keras.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- Transfer learning is a method for reusing a model trained on a related predictive modeling problem.<\/li>\n
- Transfer learning can be used to accelerate the training of neural networks as either a weight initialization scheme or feature extraction method.<\/li>\n
- How to use transfer learning to improve the performance of an MLP for a multiclass classification problem.<\/li>\n<\/ul>\n
Let\u2019s get started.<\/p>\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/02\/How-to-Improve-Performance-With-Transfer-Learning-for-Deep-Learning-Neural-Networks.jpg\")
<\/p>\nHow to Improve Performance With Transfer Learning for Deep Learning Neural Networks
Photo by Damian Gadal<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into six parts; they are:<\/p>\n
- \n
- What Is Transfer Learning?<\/li>\n
- Blobs Multi-Class Classification Problem<\/li>\n
- Multilayer Perceptron Model for Problem 1<\/li>\n
- Standalone MLP Model for Problem 2<\/li>\n
- MLP With Transfer Learning for Problem 2<\/li>\n
- Comparison of Models on Problem 2<\/li>\n<\/ol>\n
What Is Transfer Learning?<\/h2>\n
Transfer learning generally refers to a process where a model trained on one problem is used in some way on a second related problem.<\/p>\n
\n
Transfer learning and domain adaptation refer to the situation where what has been learned in one setting (i.e., distribution P1) is exploited to improve generalization in another setting (say distribution P2).<\/p>\n<\/blockquote>\n
\u2014 Page 536, Deep Learning<\/a>, 2016.<\/p>\n
In deep learning, transfer learning is a technique whereby a neural network model is first trained on a problem similar to the problem that is being solved. One or more layers from the trained model are then used in a new model trained on the problem of interest.<\/p>\n
\n
This is typically understood in a supervised learning context, where the input is the same but the target may be of a different nature. For example, we may learn about one set of visual categories, such as cats and dogs, in the first setting, then learn about a different set of visual categories, such as ants and wasps, in the second setting.<\/p>\n<\/blockquote>\n
\u2014 Page 536, Deep Learning<\/a>, 2016.<\/p>\n
Transfer learning has the benefit of decreasing the training time for a neural network model and resulting in lower generalization error.<\/p>\n
There are two main approaches to implementing transfer learning; they are:<\/p>\n
- \n
- Weight Initialization.<\/li>\n
- Feature Extraction.<\/li>\n<\/ul>\n
The weights in re-used layers may be used as the starting point for the training process and adapted in response to the new problem. This usage treats transfer learning as a type of weight initialization scheme. This may be useful when the first related problem has a lot more labeled data than the problem of interest and the similarity in the structure of the problem may be useful in both contexts.<\/p>\n
\n
\u2026 the objective is to take advantage of data from the first setting to extract information that may be useful when learning or even when directly making predictions in the second setting.<\/p>\n<\/blockquote>\n
\u2014 Page 538, Deep Learning<\/a>, 2016.<\/p>\n
Alternately, the weights of the network may not be adapted in response to the new problem, and only new layers after the reused layers may be trained to interpret their output. This usage treats transfer learning as a type of feature extraction scheme. An example of this approach is the re-use of deep convolutional neural network models trained for photo classification as feature extractors when developing photo captioning models<\/a>.<\/p>\n
Variations on these usages may involve not training the weights of the model on the new problem initially, but later fine tuning all weights of the learned model with a small learning rate.<\/p>\n
<\/div>\n<\/p>\n Want Better Results with Deep Learning?<\/h3>\n
Take my free 7-day email crash course now (with sample code).<\/p>\n
Click to sign-up and also get a free PDF Ebook version of the course.<\/p>\n