{"id":3720,"date":"2020-07-31T06:34:31","date_gmt":"2020-07-31T06:34:31","guid":{"rendered":"https:\/\/www.aiproblog.com\/index.php\/2020\/07\/31\/introduction-to-dropout-to-regularize-deep-neural-network\/"},"modified":"2020-07-31T06:34:31","modified_gmt":"2020-07-31T06:34:31","slug":"introduction-to-dropout-to-regularize-deep-neural-network","status":"publish","type":"post","link":"https:\/\/www.aiproblog.com\/index.php\/2020\/07\/31\/introduction-to-dropout-to-regularize-deep-neural-network\/","title":{"rendered":"Introduction to Dropout to regularize Deep Neural Network"},"content":{"rendered":"

Author: saurav singla<\/p>\n

\n

Dropout <\/strong>means to drop out units which are covered up and noticeable in a neural network. Dropout is a staggeringly in vogue method to overcome overfitting in neural networks.<\/span><\/p><\/blockquote>\n

Deep Learning framework<\/strong> is now getting further and more profound. With these bigger networks, we can accomplish better prediction exactness. However, this was not the case a few years ago. Deep Learning was having overfitting issue. At that point, around the year 2012, the idea of Dropout by Hinton in their paper by randomly excluding subsets of features at each iteration of a training procedure. The concept revolutionized Deep Learning. A significant part of the achievement that we have with Deep Learning is ascribed to Dropout.<\/span><\/p>\n<\/p>\n

\"No<\/span><\/div>\n<\/p>\n

Preceding Dropout, a significant research area was in regularization. Introduction of regularization methods in neural networks, for example, L1 and L2<\/strong> weight penalties, began from the mid-2000s. Notwithstanding, these regularizations didn’t totally tackle the overfitting issue.<\/span><\/p>\n<\/p>\n

Wager et al. in their paper 2013, dropout regularization was better than L2-regularization<\/strong> for learning weights for features.<\/span><\/p>\n<\/p>\n

Dropout is a method where randomly selected neurons are dropped during training. They are “dropped-out” arbitrarily. This infers that their contribution to the activation of downstream neurons is transiently evacuated on the forward pass and any weight refreshes are not applied to the neuron on the backward pass. <\/span><\/p>\n<\/p>\n

You can envision that if neurons are haphazardly dropped out of the network during training, that other neuron will have to step in and handle the portrayal required to make predictions for the missing neurons. This is believed to bring about various independent internal representations being learned by the network.<\/span><\/p>\n<\/p>\n

In spite of the fact that dropout has ended up being an exceptionally successful technique, the reasons for its success are not yet well understood at a theoretic level.<\/span><\/p>\n<\/p>\n

\"No<\/span><\/div>\n<\/p>\n

We can see standard feedforward pass: weights multiply inputs, add bias, and pass it to the activation function. The second arrangement of equations clarify how it would look like in the event that we put in dropout:<\/span><\/p>\n<\/p>\n