Gradient descent is an optimization algorithm that follows the negative gradient of an objective function in order to locate the minimum of the function.<\/p>\n
A limitation of gradient descent is that a single step size (learning rate) is used for all input variables. Extensions to gradient descent, like the Adaptive Movement Estimation (Adam) algorithm, use a separate step size for each input variable but may result in a step size that rapidly decreases to very small values.<\/p>\n
AdaMax<\/strong> is an extension to the Adam version of gradient descent that generalizes the approach to the infinite norm (max) and may result in a more effective optimization on some problems.<\/p>\n In this tutorial, you will discover how to develop gradient descent optimization with AdaMax from scratch.<\/p>\n After completing this tutorial, you will know:<\/p>\n Let\u2019s get started.<\/p>\n Gradient Descent Optimization With AdaMax From Scratch This tutorial is divided into three parts; they are:<\/p>\n Gradient<\/a> descent is an optimization algorithm.<\/p>\n It is technically referred to as a first-order optimization algorithm as it explicitly makes use of the first-order derivative of the target objective function.<\/p>\n First-order methods rely on gradient information to help direct the search for a minimum \u2026<\/p>\n<\/blockquote>\n \u2014 Page 69, Algorithms for Optimization<\/a>, 2019.<\/p>\n The first-order derivative, or simply the \u201cderivative<\/a>,\u201d is the rate of change or slope of the target function at a specific point, e.g. for a specific input.<\/p>\n If the target function takes multiple input variables, it is referred to as a multivariate function and the input variables can be thought of as a vector. In turn, the derivative of a multivariate target function may also be taken as a vector and is referred to generally as the gradient<\/a>.<\/p>\n The derivative or the gradient points in the direction of the steepest ascent of the target function for a specific input.<\/p>\n Gradient descent refers to a minimization optimization algorithm that follows the negative of the gradient downhill of the target function to locate the minimum of the function.<\/p>\n The gradient descent algorithm requires a target function that is being optimized and the derivative function for the objective function. The target function f() returns a score for a given set of inputs, and the derivative function f'() gives the derivative of the target function for a given set of inputs.<\/p>\n The gradient descent algorithm requires a starting point (x) in the problem, such as a randomly selected point in the input space.<\/p>\n The derivative is then calculated and a step is taken in the input space that is expected to result in a downhill movement in the target function, assuming we are minimizing the target function.<\/p>\n A downhill movement is made by first calculating how far to move in the input space, calculated as the step size (called alpha or the learning rate) multiplied by the gradient. This is then subtracted from the current point, ensuring we move against the gradient, or down the target function.<\/p>\n The steeper the objective function at a given point, the larger the magnitude of the gradient, and in turn, the larger the step taken in the search space. The size of the step taken is scaled using a step size hyperparameter.<\/p>\n If the step size is too small, the movement in the search space will be small and the search will take a long time. If the step size is too large, the search may bounce around the search space and skip over the optima.<\/p>\n Now that we are familiar with the gradient descent optimization algorithm, let\u2019s take a look at the AdaMax algorithm.<\/p>\n AdaMax algorithm is an extension to the Adaptive Movement Estimation (Adam) Optimization algorithm. More broadly, is an extension to the Gradient Descent Optimization algorithm.<\/p>\n The algorithm was described in the 2014 paper by Diederik Kingma and Jimmy Lei Ba titled \u201cAdam: A Method for Stochastic Optimization<\/a>.\u201d<\/p>\n Adam can be understood as updating weights inversely proportional to the scaled L2 norm (squared) of past gradients. AdaMax extends this to the so-called infinite norm (max) of past gradients.<\/p>\n In Adam, the update rule for individual weights is to scale their gradients inversely proportional to a (scaled) L^2 norm of their individual current and past gradients<\/p>\n<\/blockquote>\n \u2014 Adam: A Method for Stochastic Optimization<\/a>, 2014.<\/p>\n Generally, AdaMax automatically adapts a separate step size (learning rate) for each parameter in the optimization problem.<\/p>\n Let\u2019s step through each element of the algorithm.<\/p>\n First, we must maintain a moment vector and exponentially weighted infinity norm for each parameter being optimized as part of the search, referred to as m<\/em> and u<\/em> respectively.<\/p>\n They are initialized to 0.0 at the start of the search.<\/p>\n The algorithm is executed iteratively over time t starting at t=1, and each iteration involves calculating a new set of parameter values x, e.g. going from x(t-1)<\/em> to x(t)<\/em>.<\/p>\n It is perhaps easy to understand the algorithm if we focus on updating one parameter, which generalizes to updating all parameters via vector operations.<\/p>\n First, the gradient (partial derivatives) are calculated for the current time step.<\/p>\n Next, the moment vector is updated using the gradient and a hyperparameter beta1<\/em>.<\/p>\n The exponentially weighted infinity norm is updated using the beta2<\/em> hyperparameter.<\/p>\n Where max()<\/em> selects the maximum of the parameters and abs()<\/em> calculates the absolute value.<\/p>\n We can then update the parameter value. This can be broken down into three pieces; the first calculates the step size parameter, the second the gradient, and the third uses the step size and gradient to calculate the new parameter value.<\/p>\n Let\u2019s start with calculating the step size for the parameter using an initial step size hyperparameter called alpha<\/em> and a version of beta1<\/em> that is decaying over time with a specific value for this time step beta1(t)<\/em>:<\/p>\n The gradient used for updating the parameter is calculated as follows:<\/p>\n Finally, we can calculate the value for the parameter for this iteration.<\/p>\n Or the complete update equation can be stated as:<\/p>\n To review, there are three hyperparameters for the algorithm; they are:<\/p>\n The decay schedule for beta1(t) suggested in the paper is calculated using the initial beta1 value raised to the power t, although other decay schedules could be used such as holding the value constant or decaying more aggressively.<\/p>\n And that\u2019s it.<\/p>\n For the full derivation of the AdaMax algorithm in the context of the Adam algorithm, I recommend reading the paper:<\/p>\n Next, let\u2019s look at how we might implement the algorithm from scratch in Python.<\/p>\n In this section, we will explore how to implement the gradient descent optimization algorithm with AdaMax Momentum.<\/p>\n First, let\u2019s define an optimization function.<\/p>\n We will use a simple two-dimensional function that squares the input of each dimension and define the range of valid inputs from -1.0 to 1.0.<\/p>\n The objective()<\/em> function below implements this.<\/p>\n We can create a three-dimensional plot of the dataset to get a feeling for the curvature of the response surface.<\/p>\n The complete example of plotting the objective function is listed below.<\/p>\n Running the example creates a three-dimensional surface plot of the objective function.<\/p>\n We can see the familiar bowl shape with the global minima at f(0, 0) = 0.<\/p>\n Three-Dimensional Plot of the Test Objective Function<\/p>\n<\/div>\n We can also create a two-dimensional plot of the function. This will be helpful later when we want to plot the progress of the search.<\/p>\n The example below creates a contour plot of the objective function.<\/p>\n Running the example creates a two-dimensional contour plot of the objective function.<\/p>\n We can see the bowl shape compressed to contours shown with a color gradient. We will use this plot to plot the specific points explored during the progress of the search.<\/p>\n Two-Dimensional Contour Plot of the Test Objective Function<\/p>\n<\/div>\n Now that we have a test objective function, let\u2019s look at how we might implement the AdaMax optimization algorithm.<\/p>\n We can apply the gradient descent with AdaMax to the test problem.<\/p>\n First, we need a function that calculates the derivative for this function.<\/p>\n The derivative of x^2 is x * 2 in each dimension.<\/p>\n The derivative() function implements this below.<\/p>\n Next, we can implement gradient descent optimization with AdaMax.<\/p>\n First, we can select a random point in the bounds of the problem as a starting point for the search.<\/p>\n This assumes we have an array that defines the bounds of the search with one row for each dimension and the first column defines the minimum and the second column defines the maximum of the dimension.<\/p>\n Next, we need to initialize the moment vector and exponentially weighted infinity norm.<\/p>\n We then run a fixed number of iterations of the algorithm defined by the \u201cn_iter<\/em>\u201d hyperparameter.<\/p>\n The first step is to calculate the derivative for the current set of parameters.<\/p>\n Next, we need to perform the AdaMax update calculations. We will perform these calculations one variable at a time using an imperative programming style for readability.<\/p>\n In practice, I recommend using NumPy vector operations for efficiency.<\/p>\n First, we need to calculate the moment vector.<\/p>\n Next, we need to calculate the exponentially weighted infinity norm.<\/p>\n Then the step size used in the update.<\/p>\n And the change in variable.<\/p>\n Finally, we can calculate the new value for the variable.<\/p>\n This is then repeated for each parameter that is being optimized.<\/p>\n At the end of the iteration, we can evaluate the new parameter values and report the performance of the search.<\/p>\n We can tie all of this together into a function named adamax()<\/em> that takes the names of the objective and derivative functions as well as the algorithm hyperparameters and returns the best solution found at the end of the search and its evaluation.<\/p>\n We can then define the bounds of the function and the hyperparameters and call the function to perform the optimization.<\/p>\n In this case, we will run the algorithm for 60 iterations with an initial learning rate of 0.02, beta of 0.8, and a beta2 of 0.99, found after a little trial and error.<\/p>\n At the end of the run, we will report the best solution found.<\/p>\n Tying all of this together, the complete example of AdaMax gradient descent applied to our test problem is listed below.<\/p>\n Running the example applies the optimization algorithm with AdaMax to our test problem and reports the performance of the search for each iteration of the algorithm.<\/p>\n Note<\/strong>: Your results may vary<\/a> given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.<\/p>\n In this case, we can see that a near-optimal solution was found after perhaps 35 iterations of the search, with input values near 0.0 and 0.0, evaluating to 0.0.<\/p>\n We can plot the progress of the AdaMax search on a contour plot of the domain.<\/p>\n This can provide an intuition for the progress of the search over the iterations of the algorithm.<\/p>\n We must update the adamax() function to maintain a list of all solutions found during the search, then return this list at the end of the search.<\/p>\n The updated version of the function with these changes is listed below.<\/p>\n We can then execute the search as before, and this time retrieve the list of solutions instead of the best final solution.<\/p>\n We can then create a contour plot of the objective function, as before.<\/p>\n Finally, we can plot each solution found during the search as a white dot connected by a line.<\/p>\n Tying this all together, the complete example of performing the AdaMax optimization on the test problem and plotting the results on a contour plot is listed below.<\/p>\n Running the example performs the search as before, except in this case, the contour plot of the objective function is created.<\/p>\n In this case, we can see that a white dot is shown for each solution found during the search, starting above the optima and progressively getting closer to the optima at the center of the plot.<\/p>\n Contour Plot of the Test Objective Function With AdaMax Search Results Shown<\/p>\n<\/div>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered how to develop the gradient descent optimization with AdaMax from scratch.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions?<\/strong> The post Gradient Descent Optimization With AdaMax From Scratch<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
![\"Gradient](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/05\/Gradient-Descent-Optimization-With-AdaMax-From-Scratch.jpg\")
<\/p>\n
Photo by Don Graham<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
\n
\n
Gradient Descent<\/h2>\n
\n
\n
\n
\n
AdaMax Optimization Algorithm<\/h2>\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
Gradient Descent With AdaMax<\/h2>\n
Two-Dimensional Test Problem<\/h3>\n
# objective function\r\ndef objective(x, y):\r\n\treturn x**2.0 + y**2.0<\/pre>\n
# 3d plot of the test function\r\nfrom numpy import arange\r\nfrom numpy import meshgrid\r\nfrom matplotlib import pyplot\r\n\r\n# objective function\r\ndef objective(x, y):\r\n\treturn x**2.0 + y**2.0\r\n\r\n# define range for input\r\nr_min, r_max = -1.0, 1.0\r\n# sample input range uniformly at 0.1 increments\r\nxaxis = arange(r_min, r_max, 0.1)\r\nyaxis = arange(r_min, r_max, 0.1)\r\n# create a mesh from the axis\r\nx, y = meshgrid(xaxis, yaxis)\r\n# compute targets\r\nresults = objective(x, y)\r\n# create a surface plot with the jet color scheme\r\nfigure = pyplot.figure()\r\naxis = figure.gca(projection='3d')\r\naxis.plot_surface(x, y, results, cmap='jet')\r\n# show the plot\r\npyplot.show()<\/pre>\n
![\"Three-Dimensional](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/12\/Three-Dimensional-Plot-of-the-Test-Objective-Function.png\")
<\/p>\n# contour plot of the test function\r\nfrom numpy import asarray\r\nfrom numpy import arange\r\nfrom numpy import meshgrid\r\nfrom matplotlib import pyplot\r\n\r\n# objective function\r\ndef objective(x, y):\r\n\treturn x**2.0 + y**2.0\r\n\r\n# define range for input\r\nbounds = asarray([[-1.0, 1.0], [-1.0, 1.0]])\r\n# sample input range uniformly at 0.1 increments\r\nxaxis = arange(bounds[0,0], bounds[0,1], 0.1)\r\nyaxis = arange(bounds[1,0], bounds[1,1], 0.1)\r\n# create a mesh from the axis\r\nx, y = meshgrid(xaxis, yaxis)\r\n# compute targets\r\nresults = objective(x, y)\r\n# create a filled contour plot with 50 levels and jet color scheme\r\npyplot.contourf(x, y, results, levels=50, cmap='jet')\r\n# show the plot\r\npyplot.show()<\/pre>\n
![\"Two-Dimensional](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/12\/Two-Dimensional-Contour-Plot-of-the-Test-Objective-Function.png\")
<\/p>\nGradient Descent Optimization With AdaMax<\/h3>\n
\n
# derivative of objective function\r\ndef derivative(x, y):\r\n\treturn asarray([x * 2.0, y * 2.0])<\/pre>\n
...\r\n# generate an initial point\r\nx = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] - bounds[:, 0])<\/pre>\n
...\r\n# initialize moment vector and weighted infinity norm\r\nm = [0.0 for _ in range(bounds.shape[0])]\r\nu = [0.0 for _ in range(bounds.shape[0])]<\/pre>\n
...\r\n# run iterations of gradient descent\r\nfor t in range(n_iter):\r\n\t...<\/pre>\n
...\r\n# calculate gradient g(t)\r\ng = derivative(x[0], x[1])<\/pre>\n
...\r\n# build a solution one variable at a time\r\nfor i in range(x.shape[0]):\r\n\t...<\/pre>\n
...\r\n# m(t) = beta1 * m(t-1) + (1 - beta1) * g(t)\r\nm[i] = beta1 * m[i] + (1.0 - beta1) * g[i]<\/pre>\n
...\r\n# u(t) = max(beta2 * u(t-1), abs(g(t)))\r\nu[i] = max(beta2 * u[i], abs(g[i]))<\/pre>\n
...\r\n# step_size(t) = alpha \/ (1 - beta1(t))\r\nstep_size = alpha \/ (1.0 - beta1**(t+1))<\/pre>\n
...\r\n# delta(t) = m(t) \/ u(t)\r\ndelta = m[i] \/ u[i]<\/pre>\n
...\r\n# x(t) = x(t-1) - step_size(t) * delta(t)\r\nx[i] = x[i] - step_size * delta<\/pre>\n
...\r\n# evaluate candidate point\r\nscore = objective(x[0], x[1])\r\n# report progress\r\nprint('>%d f(%s) = %.5f' % (t, x, score))<\/pre>\n# gradient descent algorithm with adamax\r\ndef adamax(objective, derivative, bounds, n_iter, alpha, beta1, beta2):\r\n\t# generate an initial point\r\n\tx = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] - bounds[:, 0])\r\n\t# initialize moment vector and weighted infinity norm\r\n\tm = [0.0 for _ in range(bounds.shape[0])]\r\n\tu = [0.0 for _ in range(bounds.shape[0])]\r\n\t# run iterations of gradient descent\r\n\tfor t in range(n_iter):\r\n\t\t# calculate gradient g(t)\r\n\t\tg = derivative(x[0], x[1])\r\n\t\t# build a solution one variable at a time\r\n\t\tfor i in range(x.shape[0]):\r\n\t\t\t# m(t) = beta1 * m(t-1) + (1 - beta1) * g(t)\r\n\t\t\tm[i] = beta1 * m[i] + (1.0 - beta1) * g[i]\r\n\t\t\t# u(t) = max(beta2 * u(t-1), abs(g(t)))\r\n\t\t\tu[i] = max(beta2 * u[i], abs(g[i]))\r\n\t\t\t# step_size(t) = alpha \/ (1 - beta1(t))\r\n\t\t\tstep_size = alpha \/ (1.0 - beta1**(t+1))\r\n\t\t\t# delta(t) = m(t) \/ u(t)\r\n\t\t\tdelta = m[i] \/ u[i]\r\n\t\t\t# x(t) = x(t-1) - step_size(t) * delta(t)\r\n\t\t\tx[i] = x[i] - step_size * delta\r\n\t\t# evaluate candidate point\r\n\t\tscore = objective(x[0], x[1])\r\n\t\t# report progress\r\n\t\tprint('>%d f(%s) = %.5f' % (t, x, score))\r\n\treturn [x, score]<\/pre>\n...\r\n# seed the pseudo random number generator\r\nseed(1)\r\n# define range for input\r\nbounds = asarray([[-1.0, 1.0], [-1.0, 1.0]])\r\n# define the total iterations\r\nn_iter = 60\r\n# steps size\r\nalpha = 0.02\r\n# factor for average gradient\r\nbeta1 = 0.8\r\n# factor for average squared gradient\r\nbeta2 = 0.99\r\n# perform the gradient descent search with adamax\r\nbest, score = adamax(objective, derivative, bounds, n_iter, alpha, beta1, beta2)<\/pre>\n
...\r\n# summarize the result\r\nprint('Done!')\r\nprint('f(%s) = %f' % (best, score))<\/pre>\n# gradient descent optimization with adamax for a two-dimensional test function\r\nfrom numpy import asarray\r\nfrom numpy.random import rand\r\nfrom numpy.random import seed\r\n\r\n# objective function\r\ndef objective(x, y):\r\n\treturn x**2.0 + y**2.0\r\n\r\n# derivative of objective function\r\ndef derivative(x, y):\r\n\treturn asarray([x * 2.0, y * 2.0])\r\n\r\n# gradient descent algorithm with adamax\r\ndef adamax(objective, derivative, bounds, n_iter, alpha, beta1, beta2):\r\n\t# generate an initial point\r\n\tx = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] - bounds[:, 0])\r\n\t# initialize moment vector and weighted infinity norm\r\n\tm = [0.0 for _ in range(bounds.shape[0])]\r\n\tu = [0.0 for _ in range(bounds.shape[0])]\r\n\t# run iterations of gradient descent\r\n\tfor t in range(n_iter):\r\n\t\t# calculate gradient g(t)\r\n\t\tg = derivative(x[0], x[1])\r\n\t\t# build a solution one variable at a time\r\n\t\tfor i in range(x.shape[0]):\r\n\t\t\t# m(t) = beta1 * m(t-1) + (1 - beta1) * g(t)\r\n\t\t\tm[i] = beta1 * m[i] + (1.0 - beta1) * g[i]\r\n\t\t\t# u(t) = max(beta2 * u(t-1), abs(g(t)))\r\n\t\t\tu[i] = max(beta2 * u[i], abs(g[i]))\r\n\t\t\t# step_size(t) = alpha \/ (1 - beta1(t))\r\n\t\t\tstep_size = alpha \/ (1.0 - beta1**(t+1))\r\n\t\t\t# delta(t) = m(t) \/ u(t)\r\n\t\t\tdelta = m[i] \/ u[i]\r\n\t\t\t# x(t) = x(t-1) - step_size(t) * delta(t)\r\n\t\t\tx[i] = x[i] - step_size * delta\r\n\t\t# evaluate candidate point\r\n\t\tscore = objective(x[0], x[1])\r\n\t\t# report progress\r\n\t\tprint('>%d f(%s) = %.5f' % (t, x, score))\r\n\treturn [x, score]\r\n\r\n# seed the pseudo random number generator\r\nseed(1)\r\n# define range for input\r\nbounds = asarray([[-1.0, 1.0], [-1.0, 1.0]])\r\n# define the total iterations\r\nn_iter = 60\r\n# steps size\r\nalpha = 0.02\r\n# factor for average gradient\r\nbeta1 = 0.8\r\n# factor for average squared gradient\r\nbeta2 = 0.99\r\n# perform the gradient descent search with adamax\r\nbest, score = adamax(objective, derivative, bounds, n_iter, alpha, beta1, beta2)\r\n# summarize the result\r\nprint('Done!')\r\nprint('f(%s) = %f' % (best, score))<\/pre>\n...\r\n>33 f([-0.00122185 0.00427944]) = 0.00002\r\n>34 f([-0.00045147 0.00289913]) = 0.00001\r\n>35 f([0.00022176 0.00165754]) = 0.00000\r\n>36 f([0.00073314 0.00058534]) = 0.00000\r\n>37 f([ 0.00105092 -0.00030082]) = 0.00000\r\n>38 f([ 0.00117382 -0.00099624]) = 0.00000\r\n>39 f([ 0.00112512 -0.00150609]) = 0.00000\r\n>40 f([ 0.00094497 -0.00184321]) = 0.00000\r\n>41 f([ 0.00068206 -0.002026 ]) = 0.00000\r\n>42 f([ 0.00038579 -0.00207647]) = 0.00000\r\n>43 f([ 9.99977780e-05 -2.01849176e-03]) = 0.00000\r\n>44 f([-0.00014145 -0.00187632]) = 0.00000\r\n>45 f([-0.00031698 -0.00167338]) = 0.00000\r\n>46 f([-0.00041753 -0.00143134]) = 0.00000\r\n>47 f([-0.00044531 -0.00116942]) = 0.00000\r\n>48 f([-0.00041125 -0.00090399]) = 0.00000\r\n>49 f([-0.00033193 -0.00064834]) = 0.00000\r\nDone!\r\nf([-0.00033193 -0.00064834]) = 0.000001<\/pre>\n<\/p>\n
Visualization of AdaMax Optimization<\/h3>\n
# gradient descent algorithm with adamax\r\ndef adamax(objective, derivative, bounds, n_iter, alpha, beta1, beta2):\r\n\tsolutions = list()\r\n\t# generate an initial point\r\n\tx = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] - bounds[:, 0])\r\n\t# initialize moment vector and weighted infinity norm\r\n\tm = [0.0 for _ in range(bounds.shape[0])]\r\n\tu = [0.0 for _ in range(bounds.shape[0])]\r\n\t# run iterations of gradient descent\r\n\tfor t in range(n_iter):\r\n\t\t# calculate gradient g(t)\r\n\t\tg = derivative(x[0], x[1])\r\n\t\t# build a solution one variable at a time\r\n\t\tfor i in range(x.shape[0]):\r\n\t\t\t# m(t) = beta1 * m(t-1) + (1 - beta1) * g(t)\r\n\t\t\tm[i] = beta1 * m[i] + (1.0 - beta1) * g[i]\r\n\t\t\t# u(t) = max(beta2 * u(t-1), abs(g(t)))\r\n\t\t\tu[i] = max(beta2 * u[i], abs(g[i]))\r\n\t\t\t# step_size(t) = alpha \/ (1 - beta1(t))\r\n\t\t\tstep_size = alpha \/ (1.0 - beta1**(t+1))\r\n\t\t\t# delta(t) = m(t) \/ u(t)\r\n\t\t\tdelta = m[i] \/ u[i]\r\n\t\t\t# x(t) = x(t-1) - step_size(t) * delta(t)\r\n\t\t\tx[i] = x[i] - step_size * delta\r\n\t\t# evaluate candidate point\r\n\t\tscore = objective(x[0], x[1])\r\n\t\tsolutions.append(x.copy())\r\n\t\t# report progress\r\n\t\tprint('>%d f(%s) = %.5f' % (t, x, score))\r\n\treturn solutions<\/pre>\n...\r\n# seed the pseudo random number generator\r\nseed(1)\r\n# define range for input\r\nbounds = asarray([[-1.0, 1.0], [-1.0, 1.0]])\r\n# define the total iterations\r\nn_iter = 60\r\n# steps size\r\nalpha = 0.02\r\n# factor for average gradient\r\nbeta1 = 0.8\r\n# factor for average squared gradient\r\nbeta2 = 0.99\r\n# perform the gradient descent search with adamax\r\nsolutions = adamax(objective, derivative, bounds, n_iter, alpha, beta1, beta2)<\/pre>\n
...\r\n# sample input range uniformly at 0.1 increments\r\nxaxis = arange(bounds[0,0], bounds[0,1], 0.1)\r\nyaxis = arange(bounds[1,0], bounds[1,1], 0.1)\r\n# create a mesh from the axis\r\nx, y = meshgrid(xaxis, yaxis)\r\n# compute targets\r\nresults = objective(x, y)\r\n# create a filled contour plot with 50 levels and jet color scheme\r\npyplot.contourf(x, y, results, levels=50, cmap='jet')<\/pre>\n
...\r\n# plot the sample as black circles\r\nsolutions = asarray(solutions)\r\npyplot.plot(solutions[:, 0], solutions[:, 1], '.-', color='w')<\/pre>\n
# example of plotting the adamax search on a contour plot of the test function\r\nfrom numpy import asarray\r\nfrom numpy import arange\r\nfrom numpy.random import rand\r\nfrom numpy.random import seed\r\nfrom numpy import meshgrid\r\nfrom matplotlib import pyplot\r\nfrom mpl_toolkits.mplot3d import Axes3D\r\n\r\n# objective function\r\ndef objective(x, y):\r\n\treturn x**2.0 + y**2.0\r\n\r\n# derivative of objective function\r\ndef derivative(x, y):\r\n\treturn asarray([x * 2.0, y * 2.0])\r\n\r\n# gradient descent algorithm with adamax\r\ndef adamax(objective, derivative, bounds, n_iter, alpha, beta1, beta2):\r\n\tsolutions = list()\r\n\t# generate an initial point\r\n\tx = bounds[:, 0] + rand(len(bounds)) * (bounds[:, 1] - bounds[:, 0])\r\n\t# initialize moment vector and weighted infinity norm\r\n\tm = [0.0 for _ in range(bounds.shape[0])]\r\n\tu = [0.0 for _ in range(bounds.shape[0])]\r\n\t# run iterations of gradient descent\r\n\tfor t in range(n_iter):\r\n\t\t# calculate gradient g(t)\r\n\t\tg = derivative(x[0], x[1])\r\n\t\t# build a solution one variable at a time\r\n\t\tfor i in range(x.shape[0]):\r\n\t\t\t# m(t) = beta1 * m(t-1) + (1 - beta1) * g(t)\r\n\t\t\tm[i] = beta1 * m[i] + (1.0 - beta1) * g[i]\r\n\t\t\t# u(t) = max(beta2 * u(t-1), abs(g(t)))\r\n\t\t\tu[i] = max(beta2 * u[i], abs(g[i]))\r\n\t\t\t# step_size(t) = alpha \/ (1 - beta1(t))\r\n\t\t\tstep_size = alpha \/ (1.0 - beta1**(t+1))\r\n\t\t\t# delta(t) = m(t) \/ u(t)\r\n\t\t\tdelta = m[i] \/ u[i]\r\n\t\t\t# x(t) = x(t-1) - step_size(t) * delta(t)\r\n\t\t\tx[i] = x[i] - step_size * delta\r\n\t\t# evaluate candidate point\r\n\t\tscore = objective(x[0], x[1])\r\n\t\tsolutions.append(x.copy())\r\n\t\t# report progress\r\n\t\tprint('>%d f(%s) = %.5f' % (t, x, score))\r\n\treturn solutions\r\n\r\n# seed the pseudo random number generator\r\nseed(1)\r\n# define range for input\r\nbounds = asarray([[-1.0, 1.0], [-1.0, 1.0]])\r\n# define the total iterations\r\nn_iter = 60\r\n# steps size\r\nalpha = 0.02\r\n# factor for average gradient\r\nbeta1 = 0.8\r\n# factor for average squared gradient\r\nbeta2 = 0.99\r\n# perform the gradient descent search with adamax\r\nsolutions = adamax(objective, derivative, bounds, n_iter, alpha, beta1, beta2)\r\n# sample input range uniformly at 0.1 increments\r\nxaxis = arange(bounds[0,0], bounds[0,1], 0.1)\r\nyaxis = arange(bounds[1,0], bounds[1,1], 0.1)\r\n# create a mesh from the axis\r\nx, y = meshgrid(xaxis, yaxis)\r\n# compute targets\r\nresults = objective(x, y)\r\n# create a filled contour plot with 50 levels and jet color scheme\r\npyplot.contourf(x, y, results, levels=50, cmap='jet')\r\n# plot the sample as black circles\r\nsolutions = asarray(solutions)\r\npyplot.plot(solutions[:, 0], solutions[:, 1], '.-', color='w')\r\n# show the plot\r\npyplot.show()<\/pre>\n![\"Contour](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/01\/Contour-Plot-of-the-Test-Objective-Function-With-AdaMax-Search-Results-Shown.png\")
<\/p>\nFurther Reading<\/h2>\n
Papers<\/h3>\n
\n
Books<\/h3>\n
\n
APIs<\/h3>\n
\n
Articles<\/h3>\n
\n
Summary<\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n