Guest blog post by\u00a0<\/span>Zied HY<\/a>. Zied is\u00a0Senior Data Scientist at Capgemini Consulting. He is<\/span>\u00a0specialized in building predictive models utilizing both traditional statistical methods (Generalized Linear Models, Mixed Effects Models, Ridge, Lasso, etc.) and modern machine learning techniques (XGBoost, Random Forests, Kernel Methods, neural networks, etc.). Zied<\/span>\u00a0run some workshops for university students (ESSEC, HEC, Ecole polytechnique) interested in Data Science and its applications, and he is\u00a0<\/span>the co-founder of Global International Trading (GIT), a central purchasing office based in Paris.<\/span><\/em><\/p>\n Context<\/strong><\/span><\/p>\n In the previous course\u00a0<\/span>Introduction to Deep Learning<\/a>, we saw how to use Neural Networks to model a dataset of many examples. The good news is that the basic architecture of Neural Networks is quite generic whatever the application: a stacking of several perceptrons to compose complex hierarchical models and their optimization using gradient descent and backpropagation.<\/p>\n Inspite of this, you have probably heard about Multilayer Perceptrons (MLPs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), LSTM, Auto-Encoders, etc. These deep learning models are different from each other. Each model is known to be particulary performant in some specific tasks, even though, fundamentally, they all share the same basic architecture.<\/p>\n What makes the difference between them is their ability to be more suited for some data structures: text processing could be different from image processing, which in turn could be different from signal processing.<\/p>\n In the context of this post, we will focus on modeling\u00a0<\/span>sequences<\/strong>\u00a0<\/span>as a well-known data structure and will study its\u00a0<\/span>specific learning framework<\/strong>.<\/p>\n Applications of sequence modeling are plentiful in day-to-day business practice. Some of them emerged to meet today\u2019s challenges in terms of quality of service and customer engagement. Here some examples:<\/p>\n In the auto industry, self-parking is also a sequence modeling task. In fact, parking could be seen as a sequence of mouvements where the next movement depends on the previous ones.<\/p>\n Other applications cover text classification, translating videos to natural language, image caption generation, hand writing recognition\/generation, anomaly detection, and many more in the future\u2026which none of us can think (or aware) at the moment.<\/p>\n However, before we go any further in the applications of Sequence Modeling, let us understand what we are dealing with when we talk about sequences.<\/p>\n<\/div>\n Sequences are a data structure where each example could be seen as a series of data points. This sentence: \u201cI am currently reading an article about sequence modeling with Neural Networks\u201d is an example that consists of multiple words and words depend on each other. The same applies to medical records. One single medical record consists in many measurments across time. It is the same for speech waveforms.<\/p>\n So why we need a different learning framework to model sequences and what are the special features that we are looking for in this framework?<\/em><\/strong><\/p>\n For illustration purposes and with no loss of generality, let us focus on text as a sequence of words to motivate this need for a different learning framework.<\/p>\n In fact, machine learning algorithms typically require the text input to be represented as a\u00a0<\/span>fixed-length<\/strong>vector. Many operations needed to train the model (network) can be expressed through algebraic operations on the matrix of input feature values and the matrix of weights (think about a n-by-p design matrix, where n is the number of samples observed, and p is the number of variables measured in all samples).<\/p>\n Perhaps the most common fixed-length vector representation for texts is the\u00a0<\/span>bag-of-words<\/strong>\u00a0<\/span>or bag-of-n-grams due to its simplicity, efficiency and often surprising accuracy. However, the bag-of-words (BOW) representation has many disadvantages:<\/p>\n First, the word order is lost, and thus different sentences can have exactly the same representation, as long as the same words are used. Example: \u201cThe food was good, not bad at all.\u201d vs \u201cThe food was bad, not good at all.\u201d. Even though bag-of-n-grams considers the word order in short context, it suffers from data sparsity and high dimensionality.<\/p>\n<\/li>\n In addition, Bag-of-words and bag-of-n-grams have very little knowledge about the semantics of the words or more formally the distances between the words. This means that words \u201cpowerful\u201d, \u201cstrong\u201d and \u201cParis\u201d are equally distant despite the fact that semantically, \u201cpowerful\u201d should be closer to \u201cstrong\u201d than \u201cParis\u201d.<\/p>\n<\/li>\n Humans don\u2019t start their thinking from scratch every second. As you read this article,\u00a0<\/span>you understand each word based on your understanding of previous words<\/strong>. Traditional neural networks can\u2019t do this, and it seems like a major shortcoming. Bag-of-words and bag-of-n-grams as text representations do not allow to keep track of long-term dependencies inside the same sentence or paragraph.<\/p>\n<\/li>\n Another disadvantage of modeling sequences with traditional Neural Networks (e.g.\u00a0Feedforward Neural Networks) is the fact of not sharing parameters across time. Let us take for example these two sentences : \u201cOn Monday, it was snowing\u201d and \u201cIt was snowing on Monday\u201d. These sentences mean the same thing, though the details are in different parts of the sequence. Actually, when we feed these two sentences into a Feedforward Neural Network for a prediction task, the model will assign different weights to \u201cOn Monday\u201d at each moment in time.\u00a0<\/span>Things we learn about the sequence won\u2019t transfer if they appear at different points in the sequence.<\/strong>\u00a0<\/span>Sharing parameters gives the network the ability to look for a given feature everywhere in the sequence, rather than in just a certain area.<\/p>\n<\/li>\n<\/ul>\n Thus, to model sequences, we need a specific learning framework able to:<\/p>\n Recurrent neural networks (RNNs) could address this issue. They are networks with loops in them, allowing information to persist.<\/p>\n So, let us find out more about RNNs!<\/p>\n<\/div>\n A Recurrent Neural Network is architected in the same way as a \u201ctraditional\u201d Neural Network. We have some inputs, we have some hidden layers and we have some outputs.<\/p>\n The only difference is that each hidden unit is doing a slightly different function. So, let\u2019s explore how this hidden unit works.<\/p>\n A recurrent hidden unit computes a function of an input and its own previous output, also known as the cell state. For textual data, an input could be a vector representing a word\u00a0<\/span>x(i)<\/em>\u00a0<\/span>in a sentence of\u00a0<\/span>n<\/em>\u00a0<\/span>words (also known as word embedding).<\/p>\n W<\/em>\u00a0<\/span>and\u00a0<\/span>U<\/em>\u00a0<\/span>are weight matrices and\u00a0<\/span>tanh<\/em>\u00a0<\/span>is the hyperbolic tangent function.<\/p>\n Similarly, at the next step, it computes a function of the new input and its previous cell state:\u00a0<\/span>s2<\/em>\u00a0<\/span>=\u00a0<\/span>tanh<\/em>(Wx1<\/em>+\u00a0<\/span>Us1<\/em>)<\/strong>. This behavior is similar to a hidden unit in a feed-forward Network. The difference, proper to sequences, is that we are adding an additional term to incorporate its own previous state.<\/p>\n A common way of viewing recurrent neural networks is by unfolding them across time. We can notice that\u00a0<\/span>we are using the same weight matrices\u00a0<\/span>W<\/em>\u00a0<\/span>and\u00a0<\/span>U<\/em>\u00a0<\/span>throughout the sequence. This solves our problem of parameter sharing<\/strong>. We don\u2019t have new parameters for every point of the sequence. Thus, once we learn something, it can apply at any point in the sequence.<\/p>\n The fact of not having new parameters for every point of the sequence also helps us\u00a0<\/span>deal with variable-length sequences<\/strong>. In case of a sequence that has a length of 4, we could unroll this RNN to four timesteps. In other cases, we can unroll it to ten timesteps since the length of the sequence is not prespecified in the algorithm. By unrolling we simply mean that we write out the network for the complete sequence. For example, if the sequence we care about is a sentence of 5 words, the network would be unrolled into a 5-layer neural network, one layer for each word.<\/p>\n Now that we understand how a single hidden unit works, we need to figure out how to train an entire Recurrent Neural Network made up of many hidden units and even many layers of many hidden units.<\/p>\n<\/div>\n<\/div>\n Let\u2019s consider the following task: for a set of speeches in English, we need the model to automatically convert the spoken language into text i.e.\u00a0at each timestep, the model produces a prediction of a transcript (an output) based on the part of speech at this timestep (the new input) and the previous transcript (the previous cell state).<\/p>\n Naturally, because we have an output at every timestep, we can have a loss at every timestep. This loss reflects how much the predicted transcripts are close to the \u201cofficial\u201d transcripts.<\/p>\n The total loss is just the sum of the losses at every timestep.<\/span><\/p>\n Since the loss is a function of the network weights, our task it to find the set of weights\u00a0<\/span>theta<\/em>\u00a0<\/span>that achieve the lowest loss. For that, as explained in the first article \u201cIntroduction to Deep Learning\u201d, we we can apply\u00a0<\/span>the gradient descent algorithm with backpropagation (chain rule) at every timestep<\/strong>, thus taking into account the additional time dimension.<\/p>\n W<\/em>\u00a0<\/span>and\u00a0<\/span>U<\/em>\u00a0<\/span>are our two weight matrices. Let us try it out for\u00a0<\/span>W<\/em>.<\/p>\n Knowing that the total loss is the sum of the losses at every timestep, the total gradient is just the sum of the gradients at every timestep:<\/p>\n And now, we can focus on a single timestep to calculate the derivative of the loss with respect to\u00a0<\/span>W<\/em>.<\/span><\/p>\n Easy to handle: we just use backpropagation.<\/span><\/p>\n We remember that\u00a0<\/span>s2\u00a0<\/span>=\u00a0<\/span>tanh(Wx1\u00a0<\/span>+\u00a0<\/span>Us1)<\/em><\/strong>\u00a0<\/span>so s2 also depends on s1 and s1 also depends on W.\u00a0<\/span>This actually means that we can not just leave the derivative of\u00a0<\/span>s2<\/em>\u00a0<\/span>with respect to\u00a0<\/span>W<\/em>\u00a0<\/span>as a constant. We have to expand it out farther.<\/strong><\/p>\n So how does\u00a0<\/span>s2<\/em>\u00a0<\/span>depend on\u00a0<\/span>W<\/em>?<\/p>\n It depends directly on W because it feeds right in (c.f. above formula of\u00a0<\/span>s2<\/em>). We also know that\u00a0<\/span>s2<\/em>\u00a0<\/span>depends on\u00a0<\/span>s1<\/em>\u00a0<\/span>which depends on W. And we can also see that\u00a0<\/span>s2<\/em>\u00a0<\/span>depends on\u00a0<\/span>s0<\/em>\u00a0<\/span>which also depends on W.<\/p>\n Thus, the derivative of the loss with respect to\u00a0<\/span>W<\/em>\u00a0could be written as follows:<\/span><\/p>\n We can see that the last two terms are basically summing the contributions of\u00a0<\/span>W<\/em>\u00a0<\/span>in previous timesteps to the error at timestep\u00a0<\/span>t<\/em>. This is key to understand how we model long-term dependencies. From one iteration to another, the gradient descent algorithm allows to shift network parameters such that they include contributions to the error from past timesteps.<\/p>\n For any timestep\u00a0<\/span>t<\/em>, the derivative of the loss with respect to\u00a0<\/span>W<\/em>\u00a0<\/span>could be written as follows:<\/p>\n So to train the model i.e.\u00a0to estimate the weights of the network, we apply this same process of backpropagation through time for every weight (parameter) and then we use it in the process of gradient descent.<\/p>\n<\/div>\n In practice RNNs are a bit difficult to train. To understand why, let\u2019s take a closer look at the gradient we calculated above:<\/p>\n We can see that as the gap between timesteps gets bigger, the product of the gradients gets longer and longer. But, what are each of these terms?<\/span><\/p>\n Each term is basically a product of two terms: transposed\u00a0<\/span>W<\/em>\u00a0<\/span>and a second one that depends on f\u2019, the derivative of the activation function.<\/p>\n Initial weights\u00a0<\/span>W<\/em>\u00a0<\/span>are usually sampled from standard normal distribution and then mostly < 1.<\/p>\n<\/li>\n It turns out (I won\u2019t prove it here but\u00a0<\/span>this paper<\/a>\u00a0<\/span>goes into detail) that the second term is a Jacobian matrix because we are taking the derivative of a vector function with respect to a vector and its 2-norm, which you can think of it as an absolute value,\u00a0<\/span>has an upper bound of 1<\/strong>. This makes intuitive sense because our tanh (or sigmoid) activation function maps all values into a range between -1 and 1, and the derivative f\u2019 is bounded by 1 (1\/4 in the case of sigmoid).<\/p>\n<\/li>\n<\/ul>\n Thus, with small values in the matrix and multiple matrix multiplications, the\u00a0<\/span>gradient values are shrinking exponentially fast, eventually vanishing completely after a few time steps<\/strong>. Gradient contributions from \u201cfar away\u201d steps become zero, and the state at those steps doesn\u2019t contribute to what you are learning: you end up not learning long-range dependencies.<\/p>\n Vanishing gradients aren\u2019t exclusive to RNNs. They also happen in deep Feedforward Neural Networks. It\u2019s just that RNNs tend to be very deep (as deep as the sentence length in our case), which makes the problem a lot more common.<\/p>\n Fortunately, there are a few ways to combat the vanishing gradient problem.\u00a0<\/span>Proper initialization of the\u00a0<\/span>W<\/em>matrix<\/strong>\u00a0<\/span>can reduce the effect of vanishing gradients. So can regularization. A more preferred solution is to use\u00a0<\/span>ReLU<\/em><\/strong>\u00a0<\/span>instead of tanh or sigmoid activation functions. The ReLU derivative is a constant of either 0 or 1, so it isn\u2019t as likely to suffer from vanishing gradients.<\/p>\n An even more popular solution is to use Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU) architectures. LSTMs were first proposed in 1997 and are perhaps the most widely used models in NLP today. GRUs, first proposed in 2014, are simplified versions of LSTMs. Both of these RNN architectures were explicitly designed to deal with vanishing gradients and efficiently learn long-range dependencies. We\u2019ll cover them in the next part of this article.<\/p>\n It will come soon!<\/p>\n Originally posted here<\/a>.<\/em><\/p>\n<\/div>\n\n
Introduction to Sequence Modeling<\/h2>\n
\n
\n
Recurrent Neural Networks<\/h2>\n
How a Recurrent Neural Network works?<\/h3>\n
![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5RATRvUsIYhtJ2nr4pZMJ6-Nxrjauc8ZKaSp5Zy7KfVFxfBVcNXfWoId5nt1XPYgsKM8QaFIzFqPiymACUsCfBt\/Capture.PNG\")
<\/a><\/p>\n![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5SiFOYwgVXNnWoLwYj6isjskcB7xIiF3EwmKlKFJ8AFIXOUmpfzhyoj346HD0D8ih8TfusvAX2SjaUepqrja1w2\/Capture.PNG\")
<\/a><\/p>\n![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5Qh0K3mxHHTWmnZJW1*ivzaWnAXU7JUsUMe4Yi4PRXVS0IP9fj9jnrXE9-7MuSS0LK3vzju0eGKrB-4NFiIvAEq\/Capture.PNG\")
<\/a><\/p>\nNB:<\/h4>\n
\n
How do we train a Recurrent Neural Network?<\/h3>\n
![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5RlMKN1nxc*FaGoVKSk*UYNcL30eODKhV4Rt7OSEK3k7D1FnQXH-rdkZOjr-V0tpcDQ392aSI7ljFpt1nUbeEhT\/c1.PNG\")
<\/a><\/p>\n![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5To5Awzw3QOH*p*0siHpXUOL*ZRRD00nl7m1pufmc4uEkfmpDDKtCHP1S-wY8zeSE0f5i6ZOKOENwLZEZCLqMX1\/C2.PNG\")
<\/a><\/span><\/p>\n![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5RzdfFKF9uG52oaEJmiG8MzpXIUI8ZSRRz0J2NGmMpIe-3UNHmxEMDaR3aGY3QeAq8qqDeONV97GE51PHPBW8vu\/C3.PNG\")
<\/a><\/p>\n![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5RV5WtWD-7TiRJbpySXyjJgj7FW4Z-ChPcGaGZ0ZbMdjI4Qh0fayyioQzYwm7G5vOwCAhxExc6bZ7e7udsMGbcV\/c.PNG\")
<\/a><\/span><\/p>\n![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5S6Ao03aj*5wX-zWR0LjSnRk5hRFHOeIVRPPl2U*MskILtR2hWNMNSBudFSlo*XQcMieGi7WRNMI0Yz7E89gqQq\/c.PNG\")
<\/a><\/span><\/p>\n![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5TahhazU4OQC4h5o-EczTlWx1pAu*1dMbIlVFPXU0esQQ9A*GCn5tOMZumi-3kyP4TBUyCa2F63e2KJMOU2BH8p\/Capture.PNG\")
<\/a><\/p>\n![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5QLDtVNbonwPu1uYOupfGC0*UnTQKZNjU2MNkUYAuzROQaeZQRfJSJfKuFYvirsGmfuGBa9IPIm4Pv-ke3Vi-B*\/c1.PNG\")
<\/a><\/span><\/p>\n![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5RP1twYaZGjNw18WX4G4yKJnGRequyt7miF0OYEdm478FLlREBMPHSoG4OP7KBnpHv6bGPG*uiXOPbHFtioO6nr\/C2.PNG\")
<\/a><\/p>\nWhy are Recurrent Neural Networks hard to train?<\/h3>\n
![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5RAPwG2x20fGg2IszGmtoGPIA*oBO5UIo6cfq2M8*glqUJAsTAaMJfDC68TZ-uPXP9QAlupcz8KK7cC2OtD75bs\/C3.PNG\")
<\/a><\/p>\n![](\"http:\/\/api.ning.com\/files\/VizWPXcDR5SiFk6mg0cz8DdyjwNyPvEWrI9hILWJ8f8mzsGa-RPeCinrHqf6RDOwYEi0npHBPgXSz39YJpAFS3e0bugG8J6N\/c4.PNG\")
<\/a><\/span><\/p>\n\n