Calculus for Machine Learning Crash Course.
\nGet familiar with the calculus techniques in machine learning in 7 days.<\/h4>\n
Calculus is an important mathematics technique behind many machine learning algorithms. You don\u2019t always need to know it to use the algorithms. When you go deeper, you will see it is ubiquitous in every discussion on the theory behind a machine learning model.<\/p>\n
As a practitioner, we are most likely not going to encounter very hard calculus problems. If we need to do one, there are tools such as computer algebra systems to help, or at least, verify our solution. However, what is more important is understanding the idea behind calculus and relating the calculus terms to its use in our machine learning algorithms.<\/p>\n
In this crash course, you will discover some common calculus ideas used in machine learning. You will learn with exercises in Python in seven days.<\/p>\n
This is a big and important post. You might want to bookmark it.<\/p>\n
Let\u2019s get started.<\/p>\n
![\"Calculus](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/03\/1024px-Mechanical_integrator_CHM.agr_.jpg\")
<\/p>\nCalculus for Machine Learning (7-Day Mini-Course) Before we get started, let\u2019s make sure you are in the right place.<\/p>\n This course is for developers who may know some applied machine learning. Maybe you know how to work through a predictive modeling problem end to end, or at least most of the main steps, with popular tools.<\/p>\n The lessons in this course do assume a few things about you, such as:<\/p>\n You do NOT need to be:<\/p>\n This crash course will take you from a developer who knows a little machine learning to a developer who can effectively talk about the calculus concepts in machine learning algorithms.<\/p>\n Note: This crash course assumes you have a working Python 3.7 environment with some libraries such as SciPy and SymPy installed. If you need help with your environment, you can follow the step-by-step tutorial here:<\/p>\n This crash course is broken down into seven lessons.<\/p>\n You could complete one lesson per day (recommended) or complete all of the lessons in one day (hardcore). It really depends on the time you have available and your level of enthusiasm.<\/p>\n Below is a list of the seven lessons that will get you started and productive with data preparation in Python:<\/p>\n Each lesson could take you 5 minutes or up to 1 hour. Take your time and complete the lessons at your own pace. Ask questions, and even post results in the comments below.<\/p>\n The lessons might expect you to go off and find out how to do things. I will give you hints, but part of the point of each lesson is to force you to learn where to go to look for help with and about the algorithms and the best-of-breed tools in Python. (Hint<\/strong>: I have all of the answers on this blog; use the search box<\/em>.)<\/p>\n Post your results in the comments<\/strong>; I\u2019ll cheer you on!<\/p>\n Hang in there; don\u2019t give up.<\/p>\n In this lesson, you will discover what is differential calculus or differentiation.<\/p>\n Differentiation is the operation of transforming one mathematical function to another, called the derivative. The derivative tells the slope, or the rate of change, of the original function.<\/p>\n For example, if we have a function $f(x)=x^2$, its derivative is a function that tells us the rate of change of this function at $x$. The rate of change is defined as: $$f'(x) = frac{f(x+delta x)-f(x)}{delta x}$$ for a small quantity $delta x$.<\/p>\n Usually we will define the above in the form of a limit, i.e.,<\/p>\n $$f'(x) = lim_{delta xto 0} frac{f(x+delta x)-f(x)}{delta x}$$<\/p>\n to mean $delta x$ should be as close to zero as possible.<\/p>\n There are several rules of differentiation to help us find the derivative easier. One rule that fits the above example is $frac{d}{dx} x^n = nx^{n-1}$. Hence for $f(x)=x^2$, we have the derivative $f'(x)=2x$.<\/p>\n We can confirm this is the case by plotting the function $f'(x)$ computed according to the rate of change together with that computed according to the rule of differentiation. The following uses NumPy and matplotlib in Python:<\/p>\n In the plot above, we can see the derivative function found using the rate of change and then using the rule of differentiation coincide perfectly.<\/p>\n We can similarly do a differentiation of other functions. For example, $f(x)=x^3 \u2013 2x^2 + 1$. Find the derivative of this function using the rules of differentiation and compare your result with the result found using the rate of limits. Verify your result with the plot above. If you\u2019re doing it correctly, you should see the following graph:<\/p>\n In the next lesson, you will discover that integration is the reverse of differentiation.<\/p>\n In this lesson, you will discover integration is the reverse of differentiation.<\/p>\n If we consider a function $f(x)=2x$ and at intervals of $delta x$ each step (e.g., $delta x = 0.1$), we can compute, say, from $x=-10$ to $x=10$ as:<\/p>\n $$ Obviously, if we have a smaller step $delta x$, there are more terms in the above.<\/p>\n If we multiply each of the above with the step size and then add them up, i.e.,<\/p>\n $$ this sum is called the integral of $f(x)$. In essence, this sum is the area under the curve<\/strong> of $f(x)$, from $x=-10$ to $x=10$. A theorem in calculus says if we put the area under the curve as a function, its derivative is $f(x)$. Hence we can see the integration as a reverse operation of differentiation.<\/p>\n As we saw in Lesson 01, the differentiation of $f(x)=x^2$ is $f'(x)=2x$. This means for $f(x)=2x$, we can write $int f(x) dx = x^2$ or we can say the antiderivative of $f(x)=x$ is $x^2$. We can confirm this in Python by calculating the area directly:<\/p>\n This plot has the same shape as $f(x)$ in Lesson 01. Indeed, all functions differ by a constant (e.g., $f(x)$ and $f(x)+5$) that have the same derivative. Hence the plot of the antiderivative computed will be the original shifted vertically.<\/p>\n Consider $f(x)=3x^2-4x$, find the antiderivative of this function and plot it. Also, try to replace the Python code above with this function. If you plot both together, you should see the following:<\/p>\n Post your answer in the comments below. I would love to see what you come up with.<\/p>\n These two lessons are about functions with one variable. In the next lesson, you will discover how to apply differentiation to functions with multiple variables.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n In this lesson, you will learn the concept of gradient of a multivariate function.<\/p>\n If we have a function of not one variable but two or more, the differentiation is extended naturally to be the differentiation of the function with respect to each variable. For example, if we have the function $f(x,y) = x^2 + y^3$, we can write the differentiation in each variable as:<\/p>\n $$ Here we introduced the notation of a partial derivative, meaning to differentiate a function on one variable while assuming the other variables are constants. Hence in the above, when we compute $frac{partial f}{partial x}$, we ignored the $y^3$ part in the function $f(x,y)$.<\/p>\n A function with two variables can be visualized as a surface on a plane. The above function $f(x,y)$ can be visualized using matplotlib:<\/p>\n The gradient of this function is denoted as:<\/p>\n $$nabla f(x,y) = Big(frac{partial f}{partial x},; frac{partial f}{partial y}Big) = (2x,;3y^2)$$<\/p>\n Therefore, at each coordinate $(x,y)$, the gradient $nabla f(x,y)$ is a vector. This vector tells us two things:<\/p>\n One way to visualize the gradient is to consider it as a vector field<\/strong>:<\/p>\n The viridis color map in matplotlib will show a larger value in yellow and a lower value in purple. Hence we see the gradient is \u201csteeper\u201d at the edges than in the center in the above plot.<\/p>\n If we consider the coordinate (2,3), we can check which direction $f(x,y)$ will increase the fastest using the following:<\/p>\n Its output is:<\/p>\n The gradient vector according to the formula is (4,27), which the numerical result above is close enough.<\/p>\n Consider the function $f(x,y)=x^2+y^2$, what is the gradient vector at (1,1)? If you get the answer from partial differentiation, can you modify the above Python code to confirm it by checking the rate of change at different directions?<\/p>\n Post your answer in the comments below. I would love to see what you come up with.<\/p>\n In the next lesson, you will discover the differentiation of a function that takes vector input and produces vector output.<\/p>\n In this lesson, you will learn about Jacobian matrix.<\/p>\n The function $f(x,y)=(p(x,y), q(x,y))=(2xy, x^2y)$ is one with two input and two outputs. Sometimes we call this function taking vector arguments and returning a vector value. The differentiation of this function is a matrix called the Jacobian. The Jacobian of the above function is:<\/p>\n $$ In the Jacobian matrix, each row has the partial differentiation of each element of the output vector, and each column has the partial differentiation with respect to each element of the input vector.<\/p>\n We will see the use of Jacobian later. Since finding a Jacobian matrix involves a lot of partial differentiations, it would be great if we could let a computer check our math. In Python, we can verify the above result using SymPy:<\/p>\n Its output is:<\/p>\n We asked SymPy to define the symbols Consider the function where $p,q,r,s,t,u$ are constants. What is the Jacobian matrix of $f(x,y)$? Can you verify it with SymPy?<\/p>\n In the next lesson, you will discover the application of the Jacobian matrix in a neural network\u2019s backpropagation algorithm.<\/p>\n In this lesson, you will see how the backpropagation algorithm uses the Jacobian matrix.<\/p>\n If we consider a neural network with one hidden layer, we can represent it as a function:<\/p>\n $$ The input to the neural network is a vector $mathbf{x}=(x_1, x_2, cdots, x_N)$ and each $x_i$ will be multiplied with weight $w_{ik}$ and fed into the hidden layer. The output of neuron $k$ in the hidden layer will be multiplied with weight $u_k$ and fed into the output layer. The activation function of the hidden layer and output layer are $f$ and $g$, respectively.<\/p>\n If we consider<\/p>\n $$z_k = f_kbig(sum_{i=1}^N w_{ik}x_ibig)$$<\/p>\n then<\/p>\n $$ If we consider the entire layer at once, we have $mathbf{z}=(z_1, z_2, cdots, z_M)$ and then<\/p>\n $$ where $mathbf{W}$ is the $Mtimes N$ Jacobian matrix, where the element on row $k$ and column $i$ is $frac{partial z_k}{partial x_i}$.<\/p>\n This is how the backpropagation algorithm works in training a neural network! For a network with multiple hidden layers, we need to compute the Jacobian matrix for each layer.<\/p>\n The code below implements a neural network model that you can try yourself. It has two hidden layers and a classification network to separate points in 2-dimension into two classes. Try to look at the function If you play with this code, the class In the next lesson, you will discover the use of differentiation to find the optimal value of a function.<\/p>\n In this lesson, you will learn an important use of differentiation.<\/p>\n Because the differentiation of a function is the rate of change, we can make use of differentiation to find the optimal point of a function.<\/p>\n If a function attained its maximum, we would expect it to move from a lower point to the maximum, and if we move further, it falls to another lower point. Hence at the point of maximum, the rate of change of a function is zero. And vice versa for the minimum.<\/p>\n As an example, consider $f(x)=x^3-2x^2+1$. The derivative is $f'(x) = 3x^2-4x$ and $f'(x)=0$ at $x=0$ and $x=4\/3$. Hence these positions of $x$ are where $f(x)$ is at its maximum or minimum. We can visually confirm it by plotting $f(x)$ (see the plot in Lesson 01).<\/p>\n Consider the function $f(x)=log x$ and find its derivative. What will be the value of $x$ when $f'(x)=0$? What does it tell you about the maximum or minimum of the log function? Try to plot the function of $log x$ to visually confirm your answer.<\/p>\n In the next lesson, you will discover the application of this technique in finding the support vector.<\/p>\n In this lesson, you will learn how we can convert support vector machine into an optimization problem.<\/p>\n In a two-dimensional plane, any straight line can be represented by the equation:<\/p>\n $$ax+by+c=0$$<\/p>\n in the $xy$-coordinate system. A result from the study of coordinate geometry says that for any point $(x_0,y_0)$, its distance<\/strong> to the line $ax+by+c=0$ is:<\/p>\n $$ Consider the points (0,0), (1,2), and (2,1) in the $xy$-plane, in which the first point and the latter two points are in different classes. What is the line that best separates these two classes? This is the basis of a support vector machine classifier. The support vector is the line of maximum separation in this case.<\/p>\n To find such a line, we are looking for:<\/p>\n $$ The objective $a^2+b^2$ is to be minimized so that the distances from each data point to the line are maximized. The condition $-1(0a+0b+c)ge 1$ means the point (0,0) is of class $-1$; similarly for the other two points, they are of class $+1$. The straight line should put these two classes in different sides of the plane.<\/p>\n This is a constrained optimization<\/strong> problem, and the way to solve it is to use the Lagrange multiplier approach. The first step in using the Lagrange multiplier approach is to find the partial differentials of the following Lagrange function:<\/p>\n $$ and set the partial differentials to zero, then solve for $a$, $b$, and $c$. It would be too lengthy to demonstrate here, but we can use SciPy to find the solution to the above numerically:<\/p>\n It will print:<\/p>\n The above means the line to separate these three points is $0.67x + 0.67y \u2013 1 = 0$. Note that if you provided $N$ data points, there would be $N$ constraints to be defined.<\/p>\n Let\u2019s consider the points (-1,-1) and (-3,-1) to be the first class together with (0,0) and point (3,3) to be the second class together with points (1,2) and (2,1). In this problem of six points, can you modify the above program and find the line that separates the two classes? Don\u2019t be surprised to see the solution remain the same as above. There is a reason for it. Can you tell?<\/p>\n Post your answer in the comments below. I would love to see what you come up with.<\/p>\n This was the final lesson.<\/p>\n You made it. Well done!<\/p>\n Take a moment and look back at how far you have come.<\/p>\n You discovered:<\/p>\n How did you do with the mini-course?<\/strong> Do you have any questions? Were there any sticking points?<\/strong> The post Calculus for Machine Learning (7-day mini-course)<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Photo by ArnoldReinhold<\/a>, some rights reserved.<\/p>\n<\/div>\nWho Is This Crash-Course For?<\/h2>\n
\n
\n
\n
Crash-Course Overview<\/h2>\n
\n
Lesson 01: Differential Calculus<\/h2>\n
import numpy as np\r\nimport matplotlib.pyplot as plt\r\n\r\n# Define function f(x)\r\ndef f(x):\r\n return x**2\r\n\r\n# compute f(x) = x^2 for x=-10 to x=10\r\nx = np.linspace(-10,10,500)\r\ny = f(x)\r\n# Plot f(x) on left half of the figure\r\nfig = plt.figure(figsize=(12,5))\r\nax = fig.add_subplot(121)\r\nax.plot(x, y)\r\nax.set_title(\"y=f(x)\")\r\n\r\n# f'(x) using the rate of change\r\ndelta_x = 0.0001\r\ny1 = (f(x+delta_x) - f(x))\/delta_x\r\n# f'(x) using the rule\r\ny2 = 2 * x\r\n# Plot f'(x) on right half of the figure\r\nax = fig.add_subplot(122)\r\nax.plot(x, y1, c=\"r\", alpha=0.5, label=\"rate\")\r\nax.plot(x, y2, c=\"b\", alpha=0.5, label=\"rule\")\r\nax.set_title(\"y=f'(x)\")\r\nax.legend()\r\n\r\nplt.show()<\/pre>\n
![\"\"](\"https:\/\/commons.wikimedia.org\/wiki\/File:Mechanical_integrator_CHM.agr.jpg\")
<\/p>\nYour Task<\/h3>\n
<\/p>\nLesson 02: Integration<\/h2>\n
\nf(-10), f(-9.9), f(-9.8), cdots, f(9.8), f(9.9), f(10)
\n$$<\/p>\n
\nf(-10)times 0.1 + f(-9.9)times 0.1 + cdots + f(9.8)times 0.1 + f(9.9)times 0.1
\n$$<\/p>\nimport numpy as np\r\nimport matplotlib.pyplot as plt\r\n\r\ndef f(x):\r\n return 2*x\r\n\r\n# Set up x from -10 to 10 with small steps\r\ndelta_x = 0.1\r\nx = np.arange(-10, 10, delta_x)\r\n# Find f(x) * delta_x\r\nfx = f(x) * delta_x\r\n# Compute the running sum\r\ny = fx.cumsum()\r\n# Plot\r\nplt.plot(x, y)\r\nplt.show()<\/pre>\n
<\/p>\nYour Task<\/h3>\n
<\/h3>\nLesson 03: Gradient of a vector function<\/h2>\n
\nbegin{aligned}
\nfrac{partial f}{partial x} &= 2x \\
\nfrac{partial f}{partial y} &= 3y^2
\nend{aligned}
\n$$<\/p>\nimport numpy as np\r\nimport matplotlib.pyplot as plt\r\n\r\n# Define the range for x and y\r\nx = np.linspace(-10,10,1000)\r\nxv, yv = np.meshgrid(x, x, indexing='ij')\r\n\r\n# Compute f(x,y) = x^2 + y^3\r\nzv = xv**2 + yv**3\r\n\r\n# Plot the surface\r\nfig = plt.figure(figsize=(6,6))\r\nax = fig.add_subplot(projection='3d')\r\nax.plot_surface(xv, yv, zv, cmap=\"viridis\")\r\nplt.show()<\/pre>\n
<\/p>\n\n
import numpy as np\r\nimport matplotlib.pyplot as plt\r\n\r\n# Define the range for x and y\r\nx = np.linspace(-10,10,20)\r\nxv, yv = np.meshgrid(x, x, indexing='ij')\r\n\r\n# Compute the gradient of f(x,y)\r\nfx = 2*xv\r\nfy = 2*yv\r\n\r\n# Convert the vector (fx,fy) into size and direction\r\nsize = np.sqrt(fx**2 + fy**2)\r\ndir_x = fx\/size\r\ndir_y = fy\/size\r\n\r\n# Plot the surface\r\nplt.figure(figsize=(6,6))\r\nplt.quiver(xv, yv, dir_x, dir_y, size, cmap=\"viridis\")\r\nplt.show()<\/pre>\n
<\/p>\nimport numpy as np\r\n\r\ndef f(x, y):\r\n return x**2 + y**3\r\n\r\n# 0 to 360 degrees at 0.1-degree steps\r\nangles = np.arange(0, 360, 0.1)\r\n\r\n# coordinate to check\r\nx, y = 2, 3\r\n# step size for differentiation\r\nstep = 0.0001\r\n\r\n# To keep the size and direction of maximum rate of change\r\nmaxdf, maxangle = -np.inf, 0\r\nfor angle in angles:\r\n # convert degree to radian\r\n rad = angle * np.pi \/ 180\r\n # delta x and delta y for a fixed step size\r\n dx, dy = np.sin(rad)*step, np.cos(rad)*step\r\n # rate of change at a small step\r\n df = (f(x+dx, y+dy) - f(x,y))\/step\r\n # keep the maximum rate of change\r\n if df > maxdf:\r\n maxdf, maxangle = df, angle\r\n\r\n# Report the result\r\ndx, dy = np.sin(maxangle*np.pi\/180), np.cos(maxangle*np.pi\/180)\r\ngradx, grady = dx*maxdf, dy*maxdf\r\nprint(f\"Max rate of change at {maxangle} degrees\")\r\nprint(f\"Gradient vector at ({x},{y}) is ({dx*maxdf},{dy*maxdf})\")<\/pre>\nMax rate of change at 8.4 degrees\r\nGradient vector at (2,3) is (3.987419245872443,27.002750276227097)<\/pre>\n
Your Task<\/h3>\n
Lesson 04: Jacobian<\/h2>\n
\nmathbf{J} =
\nbegin{bmatrix}
\nfrac{partial p}{partial x} & frac{partial p}{partial y} \\
\nfrac{partial q}{partial x} & frac{partial q}{partial y}
\nend{bmatrix}
\n=
\nbegin{bmatrix}
\n2y & 2x \\
\n2xy & x^2
\nend{bmatrix}
\n$$<\/p>\nfrom sympy.abc import x, y\r\nfrom sympy import Matrix, pprint\r\n\r\nf = Matrix([2*x*y, x**2*y])\r\nvariables = Matrix([x,y])\r\npprint(f.jacobian(variables))<\/pre>\n
\u23a1 2\u22c5y 2\u22c5x\u23a4\r\n\u23a2 \u23a5\r\n\u23a2 2 \u23a5\r\n\u23a32\u22c5x\u22c5y x \u23a6<\/pre>\n
x<\/code> and y<\/code> and then defined the vector function f<\/code>. Afterward, the Jacobian can be found by calling the jacobian()<\/code> function.<\/p>\nYour Task<\/h3>\n
\n$$
\nf(x,y) = begin{bmatrix}
\nfrac{1}{1+e^{-(px+qy)}} & frac{1}{1+e^{-(rx+sy)}} & frac{1}{1+e^{-(tx+uy)}}
\nend{bmatrix}
\n$$<\/p>\nLesson 05: Backpropagation<\/h2>\n
\ny = gBig(sum_{k=1}^M u_k f_kbig(sum_{i=1}^N w_{ik}x_ibig)Big)
\n$$<\/p>\n
\nfrac{partial y}{partial x_i} = sum_{k=1}^M frac{partial y}{partial z_k}frac{partial z_k}{partial x_i}
\n$$<\/p>\n
\nfrac{partial y}{partial mathbf{x}} = mathbf{W}^topfrac{partial y}{partial mathbf{z}}
\n$$<\/p>\nYour Task<\/h3>\n
backward()<\/code> and identify which is the Jacobian matrix.<\/p>\nmlp<\/code> should not be modified, but you can change the parameters on how a model is created.<\/p>\nfrom sklearn.datasets import make_circles\r\nfrom sklearn.metrics import accuracy_score\r\nimport numpy as np\r\nnp.random.seed(0)\r\n\r\n# Find a small float to avoid division by zero\r\nepsilon = np.finfo(float).eps\r\n\r\n# Sigmoid function and its differentiation\r\ndef sigmoid(z):\r\n return 1\/(1+np.exp(-z.clip(-500, 500)))\r\ndef dsigmoid(z):\r\n s = sigmoid(z)\r\n return 2 * s * (1-s)\r\n\r\n# ReLU function and its differentiation\r\ndef relu(z):\r\n return np.maximum(0, z)\r\ndef drelu(z):\r\n return (z > 0).astype(float)\r\n\r\n# Loss function L(y, yhat) and its differentiation\r\ndef cross_entropy(y, yhat):\r\n \"\"\"Binary cross entropy function\r\n L = - y log yhat - (1-y) log (1-yhat)\r\n\r\n Args:\r\n y, yhat (np.array): nx1 matrices which n are the number of data instances\r\n Returns:\r\n average cross entropy value of shape 1x1, averaging over the n instances\r\n \"\"\"\r\n return ( -(y.T @ np.log(yhat.clip(epsilon)) +\r\n (1-y.T) @ np.log((1-yhat).clip(epsilon))\r\n ) \/ y.shape[1] )\r\n\r\ndef d_cross_entropy(y, yhat):\r\n \"\"\" dL\/dyhat \"\"\"\r\n return ( - np.divide(y, yhat.clip(epsilon))\r\n + np.divide(1-y, (1-yhat).clip(epsilon)) )\r\n\r\nclass mlp:\r\n '''Multilayer perceptron using numpy\r\n '''\r\n def __init__(self, layersizes, activations, derivatives, lossderiv):\r\n \"\"\"remember config, then initialize array to hold NN parameters\r\n without init\"\"\"\r\n # hold NN config\r\n self.layersizes = tuple(layersizes)\r\n self.activations = tuple(activations)\r\n self.derivatives = tuple(derivatives)\r\n self.lossderiv = lossderiv\r\n # parameters, each is a 2D numpy array\r\n L = len(self.layersizes)\r\n self.z = [None] * L\r\n self.W = [None] * L\r\n self.b = [None] * L\r\n self.a = [None] * L\r\n self.dz = [None] * L\r\n self.dW = [None] * L\r\n self.db = [None] * L\r\n self.da = [None] * L\r\n\r\n def initialize(self, seed=42):\r\n \"\"\"initialize the value of weight matrices and bias vectors with small\r\n random numbers.\"\"\"\r\n np.random.seed(seed)\r\n sigma = 0.1\r\n for l, (n_in, n_out) in enumerate(zip(self.layersizes, self.layersizes[1:]), 1):\r\n self.W[l] = np.random.randn(n_in, n_out) * sigma\r\n self.b[l] = np.random.randn(1, n_out) * sigma\r\n\r\n def forward(self, x):\r\n \"\"\"Feed forward using existing `W` and `b`, and overwrite the result\r\n variables `a` and `z`\r\n\r\n Args:\r\n x (numpy.ndarray): Input data to feed forward\r\n \"\"\"\r\n self.a[0] = x\r\n for l, func in enumerate(self.activations, 1):\r\n # z = W a + b, with `a` as output from previous layer\r\n # `W` is of size rxs and `a` the size sxn with n the number of data\r\n # instances, `z` the size rxn, `b` is rx1 and broadcast to each\r\n # column of `z`\r\n self.z[l] = (self.a[l-1] @ self.W[l]) + self.b[l]\r\n # a = g(z), with `a` as output of this layer, of size rxn\r\n self.a[l] = func(self.z[l])\r\n return self.a[-1]\r\n\r\n def backward(self, y, yhat):\r\n \"\"\"back propagation using NN output yhat and the reference output y,\r\n generates dW, dz, db, da\r\n \"\"\"\r\n # first `da`, at the output\r\n self.da[-1] = self.lossderiv(y, yhat)\r\n for l, func in reversed(list(enumerate(self.derivatives, 1))):\r\n # compute the differentials at this layer\r\n self.dz[l] = self.da[l] * func(self.z[l])\r\n self.dW[l] = self.a[l-1].T @ self.dz[l]\r\n self.db[l] = np.mean(self.dz[l], axis=0, keepdims=True)\r\n self.da[l-1] = self.dz[l] @ self.W[l].T\r\n\r\n def update(self, eta):\r\n \"\"\"Updates W and b\r\n\r\n Args:\r\n eta (float): Learning rate\r\n \"\"\"\r\n for l in range(1, len(self.W)):\r\n self.W[l] -= eta * self.dW[l]\r\n self.b[l] -= eta * self.db[l]\r\n\r\n# Make data: Two circles on x-y plane as a classification problem\r\nX, y = make_circles(n_samples=1000, factor=0.5, noise=0.1)\r\ny = y.reshape(-1,1) # our model expects a 2D array of (n_sample, n_dim)\r\n\r\n# Build a model\r\nmodel = mlp(layersizes=[2, 4, 3, 1],\r\n activations=[relu, relu, sigmoid],\r\n derivatives=[drelu, drelu, dsigmoid],\r\n lossderiv=d_cross_entropy)\r\nmodel.initialize()\r\nyhat = model.forward(X)\r\nloss = cross_entropy(y, yhat)\r\nscore = accuracy_score(y, (yhat > 0.5))\r\nprint(f\"Before training - loss value {loss} accuracy {score}\")\r\n\r\n# train for each epoch\r\nn_epochs = 150\r\nlearning_rate = 0.005\r\nfor n in range(n_epochs):\r\n model.forward(X)\r\n yhat = model.a[-1]\r\n model.backward(y, yhat)\r\n model.update(learning_rate)\r\n loss = cross_entropy(y, yhat)\r\n score = accuracy_score(y, (yhat > 0.5))\r\n print(f\"Iteration {n} - loss value {loss} accuracy {score}\")<\/pre>\nLesson 06: Optimization<\/h2>\n
Your task<\/h3>\n
Lesson 07: Support vector machine<\/h2>\n
\nfrac{vert ax_0+by_0+c vert}{sqrt{a^2+b^2}}
\n$$<\/p>\n
\nbegin{aligned}
\ntext{minimize} && a^2 + b^2 \\
\ntext{subject to} && -1(0a+0b+c) &ge 1 \\
\n&& +1(1a+2b+c) &ge 1 \\
\n&& +1(2a+1b+c) &ge 1
\nend{aligned}
\n$$<\/p>\n
\nL = a^2+b^2 + lambda_1(-c-1) + lambda_2 (a+2b+c-1) + lambda_3 (2a+b+c-1)
\n$$<\/p>\nimport numpy as np\r\nfrom scipy.optimize import minimize\r\n\r\ndef objective(w):\r\n return w[0]**2 + w[1]**2\r\n\r\ndef constraint1(w):\r\n \"Inequality for point (0,0)\"\r\n return -1*w[2] - 1\r\n\r\ndef constraint2(w):\r\n \"Inequality for point (1,2)\"\r\n return w[0] + 2*w[1] + w[2] - 1\r\n\r\ndef constraint3(w):\r\n \"Inequality for point (2,1)\"\r\n return 2*w[0] + w[1] + w[2] - 1\r\n\r\n# initial guess\r\nw0 = np.array([1, 1, 1])\r\n\r\n# optimize\r\nbounds = ((-10,10), (-10,10), (-10,10))\r\nconstraints = [\r\n {\"type\":\"ineq\", \"fun\":constraint1},\r\n {\"type\":\"ineq\", \"fun\":constraint2},\r\n {\"type\":\"ineq\", \"fun\":constraint3},\r\n]\r\nsolution = minimize(objective, w0, method=\"SLSQP\", bounds=bounds, constraints=constraints)\r\nw = solution.x\r\nprint(\"Objective:\", objective(w))\r\nprint(\"Solution:\", w)<\/pre>\nObjective: 0.8888888888888942\r\nSolution: [ 0.66666667 0.66666667 -1. ]<\/pre>\n
Your Task<\/h3>\n
The End!
\n(Look How Far You Have Come<\/em>)<\/h2>\n\n
Summary<\/h2>\n
\nDid you enjoy this crash course?<\/p>\n
\nLet me know. Leave a comment below.<\/p>\n