Calculus is one of the core mathematical concepts in machine learning that permits us to understand the internal workings of different machine learning algorithms.\u00a0<\/span><\/p>\n One of the important applications of calculus in machine learning is the gradient descent algorithm, which, in tandem with backpropagation, allows us to train a neural network model.\u00a0<\/span><\/p>\n In this tutorial, you will discover the integral role of calculus in machine learning.\u00a0<\/span><\/p>\n After completing this tutorial, you will know:<\/p>\n Let\u2019s get started.\u00a0<\/span><\/p>\n Calculus in Machine Learning: Why it Works \u00a0<\/p>\n This tutorial is divided into two parts; they are:<\/p>\n A neural network model, whether shallow or deep, implements a function that maps a set of inputs to expected outputs.\u00a0<\/span><\/p>\n The function implemented by the neural network is learned through a training process, which iteratively searches for a set of weights that best enable the neural network to model the variations in the training data.<\/p>\n A very simple type of function is a linear mapping from a single input to a single output.\u00a0<\/span><\/i><\/p>\n <\/i>Page 187, Deep Learning, 2019.<\/p>\n<\/blockquote>\n Such a linear function can be represented by the equation of a line having a slope, m<\/i>, and a y-intercept, c<\/i>:<\/p>\n y<\/i> = mx<\/i> + c<\/i><\/p>\n Varying each of parameters, m<\/i> and c<\/i>, produces different linear models that define different input-output mappings.<\/p>\n \u00a0<\/p>\n Line Plot of Different Line Models Produced by Varying the Slope and Intercept \u00a0<\/p>\n The process of learning the mapping function, therefore, involves the approximation of these model parameters, or weights<\/i>, that result in the minimum error between the predicted and target outputs. This error is calculated by means of a loss function, cost function, or error function, as often used interchangeably, and the process of minimizing the loss is referred to as function optimization<\/i>.\u00a0<\/span><\/p>\n We can apply differential calculus to the process of function optimization. \u00a0<\/span><\/p>\n In order to understand better how differential calculus can be applied to function optimization, let us return to our specific example of having a linear mapping function.\u00a0<\/span><\/p>\n Say that we have some dataset of single input features, x<\/i>, and their corresponding target outputs, y<\/i>. In order to measure the error on the dataset, we shall be taking the sum of squared errors (SSE), computed between the predicted and target outputs, as our loss function.\u00a0<\/span><\/p>\n Carrying out a parameter sweep across different values for the model weights, w0<\/sub><\/em> = m<\/i> and w1<\/sub><\/em> = c<\/i>, generates individual error profiles that are convex in shape.<\/p>\n \u00a0<\/p>\n Line Plots of Error (SSE) Profiles Generated When Sweeping Across a Range of Values for the Slope and Intercept \u00a0<\/p>\n Combining the individual error profiles generates a three-dimensional error surface that is also convex in shape. This error surface is contained within a weight space, which is defined by the swept ranges of values for the model weights, w0<\/sub><\/em> and w1<\/sub><\/em>.<\/p>\n \u00a0<\/p>\n Three-Dimensional Plot of the Error (SSE) Surface Generated When Both Slope and Intercept are Varied \u00a0<\/p>\n Moving across this weight space is equivalent to moving between different linear models. Our objective is to identify the model that best fits the data among all possible alternatives. The best model is characterised by the lowest error on the dataset, which corresponds with the lowest point on the error surface.\u00a0<\/span><\/p>\n A convex or bowl-shaped error surface is incredibly useful for learning a linear function to model a dataset because it means that the learning process can be framed as a search for the lowest point on the error surface. The standard algorithm used to find this lowest point is known as gradient descent.\u00a0<\/span><\/i><\/p>\n <\/i>Page 194, Deep Learning, 2019.<\/p>\n<\/blockquote>\n The gradient descent algorithm, as the optimization algorithm, will seek to reach the lowest point on the error surface by following its gradient downhill. This descent is based upon the computation of the gradient, or slope, of the error surface.<\/p>\n This is where differential calculus comes into the picture.\u00a0<\/span><\/p>\n Calculus, and in particular differentiation, is the field of mathematics that deals with rates of change.<\/i>\u00a0<\/span><\/b><\/p>\n <\/i>Page 198, Deep Learning, 2019.<\/p>\n<\/blockquote>\n More formally, let us denote the function that we would like to optimize by:\u00a0<\/span><\/p>\n error = <\/i>f(weights<\/i>)<\/p>\n By computing the rate of change, or the slope, of the error with respect to the weights, the gradient descent algorithm can decide on how to change the weights in order to keep reducing the error.\u00a0<\/span><\/p>\n The error function that we have considered to optimize is relatively simple, because it is convex and characterised by a single global minimum.\u00a0<\/span><\/p>\n Nonetheless, in the context of machine learning, we often need to optimize more complex functions that can make the optimization task very challenging. Optimization can become even more challenging if the input to the function is also multidimensional.\u00a0<\/span><\/p>\n Calculus provides us with the necessary tools to address both challenges.\u00a0<\/span><\/p>\n Suppose that we have a more generic function that we wish to minimize, and which takes a real input, x<\/i>, to produce a real output, y<\/i>:<\/p>\n y<\/i> = f(x<\/i>)<\/p>\n Computing the rate of change at different values of x<\/i> is useful because it gives us an indication of the changes that we need to apply to x<\/i>, in order to obtain the corresponding changes in y<\/i>.\u00a0<\/span><\/p>\n Since we are minimizing the function, our goal is to reach a point that obtains as low a value of f(x<\/i>) as possible that is also characterised by zero rate of change; hence, a global minimum. Depending on the complexity of the function, this may not necessarily be possible since there may be many local minima or saddle points that the optimisation algorithm may remain caught into.\u00a0<\/span><\/p>\n In the context of deep learning, we optimize functions that may have many local minima that are not optimal, and many saddle points surrounded by very flat regions.\u00a0<\/span><\/i><\/p>\n <\/i>Page 84, Deep Learning, 2017.<\/p>\n<\/blockquote>\n Hence, within the context of deep learning, we often accept a suboptimal solution that may not necessarily correspond to a global minimum, so long as it corresponds to a very low value of f(x<\/i>).<\/p>\n \u00a0<\/p>\n Line Plot of Cost Function to Minimize Displaying Local and Global Minima \u00a0\u00a0<\/span><\/i><\/p>\n If the function we are working with takes multiple inputs, calculus also provides us with the concept of partial derivatives<\/i>; or in simpler terms, a method to calculate the rate of change of y<\/i> with respect to changes in each one of the inputs, x<\/i>i<\/sub><\/i>, while holding the remaining inputs constant.\u00a0<\/span><\/p>\n This is why each of the weights is updated independently in the gradient descent algorithm: the weight update rule is dependent on the partial derivative of the SSE for each weight, and because there is a different partial derivative for each weight, there is a separate weight update rule for each weight.\u00a0<\/span><\/i><\/p>\n <\/i>Page 200, Deep Learning, 2019.<\/p>\n<\/blockquote>\n Hence, if we consider again the minimization of an error function, calculating the partial derivative for the error with respect to each specific weight permits that each weight is updated independently of the others.\u00a0<\/span><\/p>\n This also means that the gradient descent algorithm may not follow a straight path down the error surface. Rather, each weight will be updated in proportion to the local gradient of the error curve. Hence, one weight may be updated by a larger amount than another, as much as needed for the gradient descent algorithm to reach the function minimum. \u00a0<\/span><\/p>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered the integral role of calculus in machine learning.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions? The post Calculus in Machine Learning: Why it Works<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/06\/calculus_in_machine_learning_cover-scaled.jpg\")
<\/a><\/p>\n
Photo by Hasmik Ghazaryan Olson<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/b><\/h2>\n
\n
Calculus in Machine Learning<\/b><\/h2>\n
\n
<\/a><\/p>\n
Taken from Deep Learning<\/p>\n<\/div>\n<\/a><\/p>\n
Taken from Deep Learning<\/p>\n<\/div>\n<\/a><\/p>\n
Taken from Deep Learning<\/p>\n<\/div>\n\n
\n
Why Calculus in Machine Learning Works<\/b><\/h2>\n
\n
<\/a><\/p>\n
Taken from Deep Learning<\/p>\n<\/div>\n\n
Further Reading<\/b><\/h2>\n
Books<\/b><\/h3>\n
\n
Summary<\/b><\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n