{"id":3375,"date":"2020-04-24T06:33:48","date_gmt":"2020-04-24T06:33:48","guid":{"rendered":"https:\/\/www.aiproblog.com\/index.php\/2020\/04\/24\/deep-learning-leverage-transfer-learning\/"},"modified":"2020-04-24T06:33:48","modified_gmt":"2020-04-24T06:33:48","slug":"deep-learning-leverage-transfer-learning","status":"publish","type":"post","link":"https:\/\/www.aiproblog.com\/index.php\/2020\/04\/24\/deep-learning-leverage-transfer-learning\/","title":{"rendered":"Deep Learning: Leverage Transfer Learning"},"content":{"rendered":"

Author: Ramesh<\/p>\n

\n
\n
\n
\"\"<\/a><\/div>\n
<\/div>\n

<\/p>\n

For my current project, I was reaching about how to leverage the transfer learning techniques to improve model accuracy. In this blog, I am sharing my experience of how to leverage transfer learning in deep learning models.<\/span><\/div>\n
In the recent development in deep learning space, we can develop a complex neural network model which trained on a very large dataset. However, the main challenge is the limitation of resources and time to train the model. Even a simple image classification model with 1000 images gets hours of training time and GPU resources. In addition, due to limited training data getting good accuracy is challenges. The transfer learning technique is used to help limited resource and time challenges.<\/span><\/div>\n
Simply put, Transfer learning is leveraging a deep learning model which trained on millions of data on a different problem can be used to similar problem without full training. In general, deep learning models are highly re-purposeful. Since the original already training on a large dataset, it has the benefit of less training time and can result in lower generalization error. In addition, the weights with the convolution filters are well estimated due to original large training data.<\/span><\/div>\n
<\/div>\n
Popular Pre-Training models available for Image Classification<\/span><\/b><\/div>\n
There are many pre-trained models available for image classification problem which can be used for prediction, feature extraction, and fine-tuning. Model weights will be downloaded automatically when instantiating a Keras model. Here is the list of image classification models, which weights were trained on ImageNet. Full list available in <\/span><\/span>Keras Applications page<\/span><\/a><\/span>:<\/span><\/span><\/div>\n
•         Xception<\/span><\/div>\n
•          VGG16<\/span><\/div>\n
•          VGG19<\/span><\/div>\n
•          ResNet50<\/span><\/div>\n
<\/div>\n
There are many ways we can fine-tune the original models and leverage transfer learning. Depends on the problem nature, we can use the model as it is, or train only part of the model, or add or remove the layers of the model. I take image classification as an example for this article and let us see each option in detail.<\/span><\/div>\n
<\/div>\n
Pre-Trained Model as Feature Extractor<\/span><\/b><\/div>\n
An image classification model is classifying the image based on the detected features. Also features detected in the bottom layer are general features of the image and go towards the top it gets more narrowed that features specific to the image class. This means the features from bottom layers features can be reused to similar images for a classification problem. Thus, we can use some or all of the layers in a pre-trained model as a feature extraction component of a new model directly.<\/span><\/div>\n
In the VGG16 model, by removing just the top layer, which classifies image to one from 1000 classes, we can use the entire neutral network as a fixed feature extractor for the new data set. To use the pre-trained VGG16 model as feature extraction, Initiate the VGG16 model with “include_top” parameter as “False”, and define the new dataset image shape as additional parameters.<\/span><\/div>\n
<\/div>\n
num_class = 4<\/span><\/b><\/pre>\n
image_size =300<\/span><\/b><\/pre>\n
vgg = VGG16(include_top=False,pooling='avg',weights='imagenet',input_shape=(image_size,image_size,3))<\/span><\/b><\/pre>\n
vgg.summary()<\/span><\/b><\/pre>\n
<\/div>\n
Model: \"vgg16\"<\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
Layer (type)                 Output Shape              Param #   <\/span><\/pre>\n
=================================================================<\/span><\/pre>\n
input_3 (InputLayer)         (None, 300, 300, 3)       0         <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block1_conv1 (Conv2D)        (None, 300, 300, 64)      1792      <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block1_conv2 (Conv2D)        (None, 300, 300, 64)      36928     <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block1_pool (MaxPooling2D)   (None, 150, 150, 64)      0         <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block2_conv1 (Conv2D)        (None, 150, 150, 128)     73856     <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block2_conv2 (Conv2D)        (None, 150, 150, 128)     147584    <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block2_pool (MaxPooling2D)   (None, 75, 75, 128)       0         <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block3_conv1 (Conv2D)        (None, 75, 75, 256)       295168    <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block3_conv2 (Conv2D)        (None, 75, 75, 256)       590080    <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block3_conv3 (Conv2D)        (None, 75, 75, 256)       590080    <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block3_pool (MaxPooling2D)   (None, 37, 37, 256)       0         <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block4_conv1 (Conv2D)        (None, 37, 37, 512)       1180160   <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block4_conv2 (Conv2D)        (None, 37, 37, 512)       2359808   <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block4_conv3 (Conv2D)        (None, 37, 37, 512)       2359808   <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block4_pool (MaxPooling2D)   (None, 18, 18, 512)       0         <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block5_conv1 (Conv2D)        (None, 18, 18, 512)       2359808   <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block5_conv2 (Conv2D)        (None, 18, 18, 512)       2359808   <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block5_conv3 (Conv2D)        (None, 18, 18, 512)       2359808   <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block5_pool (MaxPooling2D)   (None, 9, 9, 512)         0         <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
global_average_pooling2d_2 ( (None, 512)               0         <\/span><\/pre>\n
=================================================================<\/span><\/pre>\n
Total params: 14,714,688<\/span><\/pre>\n
Trainable params: 14,714,688<\/span><\/pre>\n
Non-trainable params: 0<\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
<\/div>\n
Custom model with some layers of a pre-train model<\/span><\/b><\/div>\n
For the problems that predict new classes that are not in the original VGG16 model, and with small datasets available for training, we can use some layers of pretrained model with new additional layers specific to the new problem what we trying to solve.<\/span><\/span><\/div>\n
To deploy this model, replace the last few layers with a new neural network of fully connected layers. What we can do is keep the weights of initial layers of the model frozen while we retrain only the higher layers. We can try and test as to how many layers to be frozen and how many to be trained. In my experience with other projects, this approach has given me good accuracy with even small datasets.<\/span><\/span><\/div>\n
The weights in re-used layers may be used as the starting point for the training process and adapted in response to the new problem. This usage treats transfer learning as a type of weight initialization scheme.<\/span><\/span><\/div>\n
Here is how we can do the new model using only a few layers from the original model and save the computational time and improve the accuracy.<\/span><\/span><\/div>\n
First, load the VGG16 pre-trained model without the final prediction layer, and input the image shape as below.<\/span><\/span><\/div>\n
<\/div>\n
num_class = 4<\/span><\/b><\/pre>\n
image_size =300<\/span><\/b><\/pre>\n
vgg = VGG16(include_top=False,pooling='avg',weights='imagenet',input_shape=(image_size,image_size,3))<\/span><\/b><\/pre>\n
<\/div>\n
Here, I create a new model (my_model) from input layer to block1_pool layer from the VGG16 model. You can see the total trainable parameters reduce to 38k from instead of original model 14 million trainable parameters reduce to 38k<\/span><\/div>\n
<\/div>\n
layer_name ='block1_pool'<\/span><\/b><\/pre>\n
my_model = Model(inputs=vgg.input,outputs=vgg.get_layer(layer_name).output)<\/span><\/b><\/pre>\n
my_model.summary()<\/span><\/b><\/pre>\n
<\/div>\n
Model: \"model_1\"<\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
Layer (type)                 Output Shape              Param #   <\/span><\/pre>\n
=================================================================<\/span><\/pre>\n
input_3 (InputLayer)         (None, 300, 300, 3)       0         <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block1_conv1 (Conv2D)        (None, 300, 300, 64)      1792      <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block1_conv2 (Conv2D)        (None, 300, 300, 64)      36928     <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
block1_pool (MaxPooling2D)   (None, 150, 150, 64)      0         <\/span><\/pre>\n
=================================================================<\/span><\/pre>\n
Total params: 38,720<\/span><\/pre>\n
Trainable params: 38,720<\/span><\/pre>\n
Non-trainable params: 0<\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
<\/div>\n
Using the same technique, this time we’ll add additional convolutional blocks and two fully connected layers. Now we built our problem-specific layers on top of the pre-trained models. our custom model has the top layer with has the softmax activation function to predict the new image class. Finally, we set the trainable flag to false to freeze the layers from the first block before training.<\/span><\/div>\n
from keras.layers import Dense,Conv2D,MaxPooling2D,BatchNormalization,GlobalAveragePooling2D<\/span><\/b><\/pre>\n
model = models.Sequential()<\/span><\/b><\/pre>\n
model.add(my_model)<\/span><\/b><\/pre>\n
model.add(Conv2D(128,(3,3),activation='relu',padding='same'))<\/span><\/b><\/pre>\n
model.add(MaxPooling2D((2,2),padding='same'))<\/span><\/b><\/pre>\n
model.add(Conv2D(256,(3,3),activation='relu',padding='same'))<\/span><\/b><\/pre>\n
model.add(MaxPooling2D((2,2),padding='same'))<\/span><\/b><\/pre>\n
model.add(GlobalAveragePooling2D())<\/span><\/b><\/pre>\n
model.add(Dense(64,activation='relu'))<\/span><\/b><\/pre>\n
model.add(BatchNormalization())<\/span><\/b><\/pre>\n
model.add(Dense(num_class,activation='softmax'))<\/span><\/b><\/pre>\n
model.layers[0].trainable=False<\/span><\/b><\/pre>\n
model.summary()<\/span><\/b><\/pre>\n
 <\/span><\/pre>\n
The final architecture for our custom model with “model.summary()”. The complete model has only 400k trainable parameters as compared to 14 million trainable parameters as in original VGG19.<\/span><\/div>\n
<\/div>\n
Model: \"sequential_7\"<\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
Layer (type)                 Output Shape              Param #   <\/span><\/pre>\n
=================================================================<\/span><\/pre>\n
model_1 (Model)              (None, 150, 150, 64)      38720     <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
conv2d_4 (Conv2D)            (None, 150, 150, 128)     73856     <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
max_pooling2d_3 (MaxPooling2 (None, 75, 75, 128)       0         <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
conv2d_5 (Conv2D)            (None, 75, 75, 256)       295168    <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
max_pooling2d_4 (MaxPooling2 (None, 38, 38, 256)       0         <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
global_average_pooling2d_4 ( (None, 256)               0         <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
dense_5 (Dense)              (None, 64)                16448     <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
batch_normalization_2 (Batch (None, 64)                256       <\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
dense_6 (Dense)              (None, 4)                 260       <\/span><\/pre>\n
=================================================================<\/span><\/pre>\n
Total params: 424,708<\/span><\/pre>\n
Trainable params: 385,860<\/span><\/pre>\n
Non-trainable params: 38,848<\/span><\/pre>\n
_________________________________________________________________<\/span><\/pre>\n
<\/div>\n
Using Architecture of the pre-trained model<\/span><\/b><\/div>\n
Where we have a large image dataset and the classes are completely different. In this scenario, we can use the architecture of the VGG19 model and train the model with new data. Even though we need huge computational power to train the model, but it saves the time research time of model building efforts and it is a good starting point for the new problem. In my current project, I am using one of the fasterRCNN pre-trained models based on ImageNet. It helped me a lot as a starting point and easy to customize the layer after the initial training phase. On NLP space, I did use the pre-trained model as it is for embedding vector-based language models. I have used the question-answering model using the BioSentVec<\/a> model that trained on the medical dataset to use COVID19 related project.<\/span><\/span><\/div>\n
Here is the sample code snippet<\/span><\/span><\/div>\n
#### loading VGG16 and set to untrainable<\/span><\/b><\/pre>\n
conv_base  = applications.VGG16(weights='imagenet', include_top=False, input_shape=[150, 150, 3]) <\/b><\/pre>\n
conv_base.trainable = False<\/b><\/pre>\n
#### predict class with new image<\/span><\/b><\/pre>\n
ret1 = conv_base.predict(np.ones([1, 150, 150, 3]))<\/b><\/pre>\n
<\/div>\n
Conclusion<\/span><\/b><\/div>\n
<\/div>\n
There are many ways we can leverage the pre-trained models. There are various architectures people have tried on different types of data sets and I strongly encourage you to go through pre-trained models’ architectures and apply them on your own problem statements. The best approach is depending on the problem what you trying to solve. I would recommend to start the evaluation of the pre-trained model with a small sample dataset and see how the pre-trained model performs. Try to leverage as many layers of pre-training model to start with and add custom layers to improve the accuracy. Especially when you have limited training data, using pre-trained model layers with custom layers is the wise approach.<\/span><\/div>\n
<\/div>\n
Thanks!<\/span><\/div>\n

Reference:<\/span><\/div>\n
·     <\/span><\/span> https:\/\/blog.keras.io\/building-powerful-image-classification-models-using-very-little-data.html<\/a><\/span><\/div>\n
·     <\/span><\/span> https:\/\/opendatascience.com\/leverage-pre-trained-layers-in-image-classification\/<\/a><\/span><\/div>\n
·     <\/span><\/span> Book:<\/span> Deep Learning (Adaptive Computation and Machine Learning series) by Ian Goodfellow<\/a><\/span><\/span><\/h1>\n<\/div>\n<\/div>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Author: Ramesh For my current project, I was reaching about how to leverage the transfer learning techniques to improve model accuracy. In this blog, I […] Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":3376,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[26],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/3375"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=3375"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/3375\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/3376"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=3375"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=3375"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=3375"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}