The chain rule is an important derivative rule that allows us to work with composite functions. It is essential in understanding the workings of the backpropagation algorithm, which applies the chain rule extensively in order to calculate the error gradient of the loss function with respect to each weight of a neural network. We will be building on our earlier introduction to the chain rule, by tackling more challenging functions.\u00a0<\/span><\/p>\n In this tutorial, you will discover how to apply the chain rule of calculus to challenging functions.\u00a0<\/span><\/p>\n After completing this tutorial, you will know:<\/p>\n Let\u2019s get started.\u00a0<\/span><\/p>\n The Chain Rule of Calculus \u2013 Even More Functions \u00a0<\/p>\n This tutorial is divided into two parts; they are:<\/p>\n For this tutorial, we assume that you already know what are:<\/p>\n You can review these concepts by clicking on the links given above.<\/p>\n We have already discovered the chain rule for univariate and multivariate functions, but we have only seen a few simple examples so far. Let\u2019s see a few more challenging ones here. We will be starting with univariate functions first, and then apply what we learn to multivariate functions.\u00a0<\/span><\/p>\n EXAMPLE 1<\/b>: Let\u2019s raise the bar a little by considering the following composite function:<\/p>\n We can separate the composite function into the inner function, f<\/i>(x<\/i>) = x<\/i>2<\/sup> \u2013 10, and the outer function, g<\/i>(x<\/i>) = \u221ax<\/i> = (x<\/i>)1\/2<\/sup>. The output of the inner function is denoted by the intermediate variable, u<\/i>, and its value will be fed into the input of the outer function.\u00a0<\/span><\/p>\n The first step is to find the derivative of the outer part of the composite function, while ignoring whatever is inside. For this purpose, we can apply the power rule:<\/p>\n dh \/ du<\/i> = (1\/2) (x<\/i>2<\/sup> \u2013 10)-1\/2<\/sup><\/p>\n The next step is to find the derivative of the inner part of the composite function, this time ignoring whatever is outside. We can apply the power rule here too:<\/p>\n du \/ dx<\/i> = 2x<\/i><\/p>\n Putting the two parts together and simplifying, we have:<\/p>\n EXAMPLE 2<\/b>: Let\u2019s repeat the procedure, this time with a different composite function:<\/p>\n We will again use, u<\/i>, the output of the inner function, as our intermediate variable.\u00a0<\/span><\/p>\n The outer function in this case is, cos x<\/i>. Finding its derivative, again ignoring the inside, gives us:\u00a0<\/span><\/p>\n dh<\/i> \/ du<\/i> = (cos(x<\/i>3<\/sup> \u2013 1))\u2019 = -sin(x<\/i>3<\/sup> \u2013 1)<\/p>\n The inner function is, x<\/i>3<\/sup> \u2013 1. Hence, its derivative becomes:<\/p>\n du<\/i> \/ dx<\/i> = (x<\/i>3<\/sup> \u2013 1)\u2019 = 3x<\/i>2<\/sup><\/p>\n Putting the two parts together, we obtain the derivative of the composite function:<\/p>\n EXAMPLE 3<\/b>: Let\u2019s now raise the bar a little further by considering a more challenging composite function:<\/p>\n If we observe this closely, we realize that not only do we have nested functions for which we will need to apply the chain rule multiple times, but we also have a product to which we will need to apply the product rule. \u00a0<\/span><\/p>\n We find that the outermost function is a cosine. In finding its derivative by the chain rule, we shall be using the intermediate variable, u<\/i>:<\/p>\n dh<\/i> \/ du<\/i> = (cos(x <\/i>\u221a(x<\/i>2<\/sup> \u2013 10) ))\u2019 = -sin(x <\/i>\u221a(x<\/i>2<\/sup> \u2013 10) )<\/p>\n Inside the cosine, we have the product, x <\/i>\u221a(x2<\/sup> \u2013 10), to which we will be applying the product rule to find its derivative (notice that we are always moving from the outside to the inside, in order to discover the operation that needs to be tackled next):<\/p>\n du<\/i> \/ dx<\/i> = (x <\/i>\u221a(x2<\/sup> \u2013 10) )\u2019 = \u221a(x2<\/sup> \u2013 10) + x<\/i> ( \u221a(x2<\/sup> \u2013 10) )\u2019<\/p>\n One of the components in the resulting term is, ( \u221a(x2<\/sup> \u2013 10) )\u2019, to which we shall be applying the chain rule again. Indeed, we have already done so above, and hence we can simply re-utilise the result:<\/p>\n ( \u221a(x2<\/sup> \u2013 10) )\u2019 = x<\/i> (x<\/i>2<\/sup> \u2013 10)-1\/2<\/sup><\/p>\n Putting all the parts together, we obtain the derivative of the composite function:<\/p>\n This can be simplified further into:<\/p>\n EXAMPLE 4<\/b>: Suppose that we are now presented by a multivariate function of two independent variables, s<\/i> and t<\/i>, with each of these variables being dependent on another two independent variables, x<\/i> and y<\/i>:<\/p>\n h<\/i> = g<\/i>(s<\/i>, t<\/i>) = s<\/i>2<\/sup> + t<\/i>3<\/sup><\/p>\n Where the functions, s <\/i>= xy<\/i>, and t<\/i> = 2x<\/i> \u2013 y<\/i>.\u00a0<\/span><\/p>\n Implementing the chain rule here requires the computation of partial derivatives, since we are working with multiple independent variables. Furthermore, s<\/i> and t<\/i> will also act as our intermediate variables. The formulae that we will be working with, defined with respect to each input, are the following:<\/p>\n From these formulae, we can see that we will need to find six different partial derivatives:<\/p>\n We can now proceed to substitute these terms in the formulae for \u2202h<\/i> \/ \u2202x <\/i>and <\/i>\u2202h<\/i> \/ \u2202y<\/i>:<\/p>\n And subsequently substitute for s<\/i> and t <\/i>to find the derivatives:<\/p>\n EXAMPLE 5<\/b>: Let\u2019s repeat this again, this time with a multivariate function of three independent variables, r<\/i>, s<\/i> and t<\/i>, with each of these variables being dependent on another two independent variables, x<\/i> and y<\/i>:<\/p>\n h<\/i> = g<\/i>(r, s<\/i>, t<\/i>) = r<\/i>2<\/sup> \u2013\u00a0rs<\/i> + t<\/i>3<\/sup><\/p>\n Where the functions, r<\/i> = x<\/i> cos y<\/i>, s <\/i>= x <\/i>ey<\/sup><\/i>, and t<\/i> = x<\/i> + y<\/i>.<\/p>\n This time round, r<\/i>, s<\/i> and t<\/i> will act as our intermediate variables. The formulae that we will be working with, defined with respect to each input, are the following:<\/p>\n From these formulae, we can see that we will now need to find nine different partial derivatives:<\/p>\n Again, we proceed to substitute these terms in the formulae for \u2202h<\/i> \/ \u2202x <\/i>and <\/i>\u2202h<\/i> \/ \u2202y<\/i>:<\/p>\n And subsequently substitute for r<\/i>, s<\/i> and t <\/i>to find the derivatives:<\/p>\n Which may be simplified a little further (hint: apply the trigonometric identity 2sin y<\/i> cos y<\/i> = sin 2y<\/i> to \u2202h<\/i> \/ \u2202y<\/i>):<\/p>\n No matter how complex the expression is, the procedure to follow remains similar:<\/p>\n Your last computation tells you the first thing to do.<\/i><\/p>\n \u2013 Page 143, Calculus for Dummies<\/a>, 2016.\u00a0<\/span><\/p>\n<\/blockquote>\n Hence, start by tackling the outer function first, then move inwards to the next one. You may need to apply other rules along the way, as we have seen for Example 3. Do not forget to take the partial derivatives if you are working with multivariate functions.\u00a0<\/span><\/p>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered\u00a0how to apply the chain rule of calculus to challenging functions.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions? The post The Chain Rule of Calculus \u2013 Even More Functions<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/07\/more_chain_rule_cover-scaled.jpg\")
<\/a><\/p>\n
Photo by Nan Ingraham<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/b><\/h2>\n
\n
Prerequisites<\/b><\/h2>\n
\n
The Chain Rule on Univariate Functions<\/b><\/h2>\n
<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\nThe Chain Rule on Multivariate Functions<\/b><\/h2>\n
<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n\n
Further Reading<\/b><\/h2>\n
Books<\/b><\/h3>\n
\n
Summary<\/b><\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n