Derivatives are one of the most fundamental concepts in calculus. They describe how changes in the variable inputs affect the function outputs. The objective of this article is to provide a high-level introduction to calculating derivatives in PyTorch for those who are new to the framework. PyTorch offers a convenient way to calculate derivatives for user-defined functions.<\/p>\n
While we always have to deal with backpropagation (an algorithm known to be the backbone of a neural network) in neural networks, which optimizes the parameters to minimize the error in order to achieve higher classification accuracy; concept learned in this article will be used in later posts on deep learning for image processing and other computer vision problems.<\/p>\n
After going through this tutorial, you\u2019ll learn:<\/p>\n
- \n
- How to calculate derivatives in PyTorch.<\/li>\n
- How to use autograd in PyTorch to perform auto differentiation on tensors.<\/li>\n
- About the computation graph that involves different nodes and leaves, allowing you to calculate the gradients in a simple possible manner (using the chain rule).<\/li>\n
- How to calculate partial derivatives in PyTorch.<\/li>\n
- How to implement the derivative of functions with respect to multiple values.<\/li>\n<\/ul>\n
Let\u2019s get started.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/01\/jossuha-theophile-H-CZjCQfsFw-unsplash.jpg\")
<\/p>\nCalculating Derivatives in PyTorch
Picture by Jossuha Th\u00e9ophile<\/a>. Some rights reserved.<\/p>\n<\/div>\nDifferentiation in Autograd<\/strong><\/h2>\n
The autograd \u2013 an auto differentiation module in PyTorch \u2013 is used to calculate the derivatives and optimize the parameters in neural networks. It is intended primarily for gradient computations.<\/p>\n
Before we start, let\u2019s load up some necessary libraries we\u2019ll use in this tutorial.<\/p>\n
import matplotlib.pyplot as plt\r\nimport torch<\/pre>\n
Now, let\u2019s use a simple tensor and set the
requires_grad<\/code> parameter to true. This allows us to perform automatic differentiation and lets PyTorch evaluate the derivatives using the given value which, in this case, is 3.0.<\/p>\nx = torch.tensor(3.0, requires_grad = True)\r\nprint(\"creating a tensor x: \", x)<\/pre>\n<\/p>\n
creating a tensor x: tensor(3., requires_grad=True)<\/pre>\n
We\u2019ll use a simple equation $y=3x^2$ as an example and take the derivative with respect to variable
x<\/code>. So, let\u2019s create another tensor according to the given equation. Also, we\u2019ll apply a neat method.backward<\/code> on the variabley<\/code> that forms acyclic graph storing the computation history, and evaluate the result with.grad<\/code> for the given value.<\/p>\ny = 3 * x ** 2\r\nprint(\"Result of the equation is: \", y)\r\ny.backward()\r\nprint(\"Dervative of the equation at x = 3 is: \", x.grad)<\/pre>\n<\/p>\n
Result of the equation is: tensor(27., grad_fn=<MulBackward0>)\r\nDervative of the equation at x = 3 is: tensor(18.)<\/pre>\n
As you can see, we have obtained a value of 36, which is correct.<\/p>\n
Computational Graph<\/strong><\/h2>\n
PyTorch generates derivatives by building a backwards graph behind the scenes, while tensors and backwards functions are the graph\u2019s nodes. In a graph, PyTorch computes the derivative of a tensor depending on whether it is a leaf or not.<\/p>\n
PyTorch will not evaluate a tensor\u2019s derivative if its leaf attribute is set to True. We won\u2019t go into much detail about how the backwards graph is created and utilized, because the goal here is to give you a high-level knowledge of how PyTorch makes use of the graph to calculate derivatives.<\/p>\n
So, let\u2019s check how the tensors
x<\/code> andy<\/code> look internally once they are created. Forx<\/code>:<\/p>\nprint('data attribute of the tensor:',x.data)\r\nprint('grad attribute of the tensor::',x.grad)\r\nprint('grad_fn attribute of the tensor::',x.grad_fn)\r\nprint(\"is_leaf attribute of the tensor::\",x.is_leaf)\r\nprint(\"requires_grad attribute of the tensor::\",x.requires_grad)<\/pre>\n<\/p>\ndata attribute of the tensor: tensor(3.)\r\ngrad attribute of the tensor:: tensor(18.)\r\ngrad_fn attribute of the tensor:: None\r\nis_leaf attribute of the tensor:: True\r\nrequires_grad attribute of the tensor:: True<\/pre>\n
and for
y<\/code>:<\/p>\nprint('data attribute of the tensor:',y.data)\r\nprint('grad attribute of the tensor:',y.grad)\r\nprint('grad_fn attribute of the tensor:',y.grad_fn)\r\nprint(\"is_leaf attribute of the tensor:\",y.is_leaf)\r\nprint(\"requires_grad attribute of the tensor:\",y.requires_grad)<\/pre>\n<\/p>\nprint('data attribute of the tensor:',y.data)\r\nprint('grad attribute of the tensor:',y.grad)\r\nprint('grad_fn attribute of the tensor:',y.grad_fn)\r\nprint(\"is_leaf attribute of the tensor:\",y.is_leaf)\r\nprint(\"requires_grad attribute of the tensor:\",y.requires_grad)<\/pre>\nAs you can see, each tensor has been assigned with a particular set of attributes.<\/p>\n
The
data<\/code> attribute stores the tensor\u2019s data while thegrad_fn<\/code> attribute tells about the node in the graph. Likewise, the.grad<\/code> attribute holds the result of the derivative. Now that you have learnt some basics about the autograd and computational graph in PyTorch, let\u2019s take a little more complicated equation $y=6x^2+2x+4$ and calculate the derivative. The derivative of the equation is given by:<\/p>\n$$frac{dy}{dx} = 12x+2$$<\/p>\n
Evaluating the derivative at $x = 3$,<\/p>\n
$$left.frac{dy}{dx}rightvert_{x=3} = 12times 3+2 = 38$$<\/p>\n
Now, let\u2019s see how PyTorch does that,<\/p>\n
x = torch.tensor(3.0, requires_grad = True)\r\ny = 6 * x ** 2 + 2 * x + 4\r\nprint(\"Result of the equation is: \", y)\r\ny.backward()\r\nprint(\"Derivative of the equation at x = 3 is: \", x.grad)<\/pre>\n<\/p>\n
Result of the equation is: tensor(64., grad_fn=<AddBackward0>)\r\nDerivative of the equation at x = 3 is: tensor(38.)<\/pre>\n
The derivative of the equation is 38, which is correct.<\/p>\n
Implementing Partial Derivatives of Functions<\/strong><\/h2>\n
PyTorch also allows us to calculate partial derivatives of functions. For example, if we have to apply partial derivation to the following function,<\/p>\n
$$f(u,v) = u^3+v^2+4uv$$<\/p>\n
Its derivative with respect to $u$ is,<\/p>\n
$$frac{partial f}{partial u} = 3u^2 + 4v$$<\/p>\n
Similarly, the derivative with respect to $v$ will be,<\/p>\n
$$frac{partial f}{partial v} = 2v + 4u$$<\/p>\n
Now, let\u2019s do it the PyTorch way, where $u = 3$ and $v = 4$.<\/p>\n
We\u2019ll create
u<\/code>,v<\/code> andf<\/code> tensors and apply the.backward<\/code> attribute onf<\/code> in order to compute the derivative. Finally, we\u2019ll evaluate the derivative using the.grad<\/code> with respect to the values ofu<\/code> andv<\/code>.<\/p>\nu = torch.tensor(3., requires_grad=True)\r\nv = torch.tensor(4., requires_grad=True)\r\n\r\nf = u**3 + v**2 + 4*u*v\r\n\r\nprint(u)\r\nprint(v)\r\nprint(f)\r\n\r\nf.backward()\r\nprint(\"Partial derivative with respect to u: \", u.grad)\r\nprint(\"Partial derivative with respect to u: \", v.grad)<\/pre>\n<\/p>\n
tensor(3., requires_grad=True)\r\ntensor(4., requires_grad=True)\r\ntensor(91., grad_fn=<AddBackward0>)\r\nPartial derivative with respect to u: tensor(43.)\r\nPartial derivative with respect to u: tensor(20.)<\/pre>\n<\/p>\n
Derivative of Functions with Multiple Values<\/strong><\/h2>\n
What if we have a function with multiple values and we need to calculate the derivative with respect to its multiple values? For this, we\u2019ll make use of the sum attribute to (1) produce a scalar valued function, and then (2) take the derivative. This is how we can see the \u2018function vs. derivative\u2019 plot:<\/p>\n
# compute the derivative of the function with multiple values\r\nx = torch.linspace(-20, 20, 20, requires_grad = True)\r\nY = x ** 2\r\ny = torch.sum(Y)\r\ny.backward()\r\n\r\n# ploting the function and derivative\r\nfunction_line, = plt.plot(x.detach().numpy(), Y.detach().numpy(), label = 'Function')\r\nfunction_line.set_color(\"red\")\r\nderivative_line, = plt.plot(x.detach().numpy(), x.grad.detach().numpy(), label = 'Derivative')\r\nderivative_line.set_color(\"green\")\r\nplt.xlabel('x')\r\nplt.legend()\r\nplt.show()<\/pre>\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/01\/pytorch-deriv.png\")
<\/p>\nIn the two
plot()<\/code> function above, we extract the values from PyTorch tensors so we can visualize them. The.detach<\/code> method doesn\u2019t allow the graph to further track the operations. This makes it easy for us to convert a tensor to a numpy array.<\/p>\nSummary<\/strong><\/h2>\n
In this tutorial, you learned how to implement derivatives on various functions in PyTorch.<\/p>\n
Particularly, you learned:<\/p>\n
- \n
- How to calculate derivatives in PyTorch.<\/li>\n
- How to use autograd in PyTorch to perform auto differentiation on tensors.<\/li>\n
- About the computation graph that involves different nodes and leaves, allowing you to calculate the gradients in a simple possible manner (using the chain rule).<\/li>\n
- How to calculate partial derivatives in PyTorch.<\/li>\n
- How to implement the derivative of functions with respect to multiple values.<\/li>\n<\/ul>\n
The post Calculating Derivatives in PyTorch<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"