Optimization involves finding the inputs to an objective function that result in the minimum or maximum output of the function.<\/p>\n
The open-source Python library for scientific computing called SciPy provides a suite of optimization algorithms. Many of the algorithms are used as a building block in other algorithms, most notably machine learning algorithms in the scikit-learn library.<\/p>\n
These optimization algorithms<\/strong> can be used directly in a standalone manner to optimize a function. Most notably, algorithms for local search and algorithms for global search, the two main types of optimization you may encounter on a machine learning project.<\/p>\n In this tutorial, you will discover optimization algorithms provided by the SciPy library.<\/p>\n After completing this tutorial, you will know:<\/p>\n Let\u2019s get started.<\/p>\n Function Optimization With SciPy This tutorial is divided into three parts; they are:<\/p>\n The Python SciPy open-source library for scientific computing provides a suite of optimization techniques.<\/p>\n Many of the algorithms are used as building blocks for other algorithms within the SciPy library, as well as machine learning libraries such as scikit-learn.<\/p>\n Before we review specific techniques, let\u2019s look at the types of algorithms provided by the library.<\/p>\n They are:<\/p>\n All algorithms assume the objective function that is being optimized is a minimization function. If your function is maximizing, it can be converted to minimizing by adding a negative sign to values returned from your objective function.<\/p>\n In addition to the above list, the library also provides utility functions used by some of the algorithms, as well as the Rosenbrock test problem.<\/p>\n For a good overview of the capabilities of the SciPy library for optimization, see:<\/p>\n Now that we have a high-level idea of the types of optimization techniques supported by the library, let\u2019s take a closer look at two groups of algorithms we are more likely to use in applied machine learning. They are local search and global search.<\/p>\n Local search<\/a>, or local function optimization, refers to algorithms that seek the input to a function that results in the minimum or maximum output where the function or constrained region being searched is assumed to have a single optima, e.g. unimodal.<\/p>\n The function that is being optimized may or may not be convex, and may have one or more than one input variable.<\/p>\n A local search optimization may be applied directly to optimize a function if the function is believed or known to be unimodal; otherwise, the local search algorithm may be applied to fine-tune the result of a global search algorithm.<\/p>\n The SciPy library provides local search via the minimize() function<\/a>.<\/p>\n The minimize()<\/em> function takes as input the name of the objective function that is being minimized and the initial point from which to start the search and returns an OptimizeResult<\/a> that summarizes the success or failure of the search and the details of the solution if found.<\/p>\n Additional information about the objective function can be provided if known, such as the bounds on the input variables, a function for computing the first derivative of the function (gradient or Jacobian matrix<\/a>), a function for computing the second derivative of the function (Hessian matrix<\/a>), and any constraints on the inputs.<\/p>\n Importantly, the function provides the \u201cmethod<\/em>\u201d argument that allows the specific optimization used in the local search to be specified.<\/p>\n A suite of popular local search algorithms are available, such as:<\/p>\n The example below demonstrates how to solve a two-dimensional convex function using the L-BFGS-B local search algorithm.<\/p>\n Running the example performs the optimization and reports the success or failure of the search, the number of function evaluations performed, and the input that resulted in the optima of the function.<\/p>\n Now that we are familiar with using a local search algorithm with SciPy, let\u2019s look at global search.<\/p>\n Global search<\/a> or global function optimization refers to algorithms that seek the input to a function that results in the minimum or maximum output where the function or constrained region being searched is assumed to have multiple local optima, e.g. multimodal.<\/p>\n The function that is being optimized is typically nonlinear, nonconvex, and may have one or more than one input variable.<\/p>\n Global search algorithms are typically stochastic, meaning that they make use of randomness in the search process and may or may not manage a population of candidate solutions as part of the search.<\/p>\n The SciPy library provides a number of stochastic global optimization algorithms, each via different functions. They are:<\/p>\n The library also provides the shgo()<\/a> function for sequence optimization and the brute()<\/a> for grid search optimization.<\/p>\n Each algorithm returns an OptimizeResult<\/a> object that summarizes the success or failure of the search and the details of the solution if found.<\/p>\n The example below demonstrates how to solve a two-dimensional multimodal function using simulated annealing.<\/p>\n Running the example performs the optimization and reports the success or failure of the search, the number of function evaluations performed, and the input that resulted in the optima of the function.<\/p>\n This section provides more resources on the topic if you are looking to go deeper.<\/p>\n In this tutorial, you discovered optimization algorithms provided by the SciPy library.<\/p>\n Specifically, you learned:<\/p>\n Do you have any questions?<\/strong> The post Function Optimization With SciPy<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"\n
![\"Function](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/02\/Function-Optimization-With-SciPy.jpg\")
<\/p>\n
Photo by Manoel Lemos<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
\n
Optimization With SciPy<\/h2>\n
\n
\n
Local Search With SciPy<\/h2>\n
...\r\n# minimize an objective function\r\nresult = minimize(objective, point)<\/pre>\n
\n
# l-bfgs-b algorithm local optimization of a convex function\r\nfrom scipy.optimize import minimize\r\nfrom numpy.random import rand\r\n\r\n# objective function\r\ndef objective(x):\r\n\treturn x[0]**2.0 + x[1]**2.0\r\n\r\n# define range for input\r\nr_min, r_max = -5.0, 5.0\r\n# define the starting point as a random sample from the domain\r\npt = r_min + rand(2) * (r_max - r_min)\r\n# perform the l-bfgs-b algorithm search\r\nresult = minimize(objective, pt, method='L-BFGS-B')\r\n# summarize the result\r\nprint('Status : %s' % result['message'])\r\nprint('Total Evaluations: %d' % result['nfev'])\r\n# evaluate solution\r\nsolution = result['x']\r\nevaluation = objective(solution)\r\nprint('Solution: f(%s) = %.5f' % (solution, evaluation))<\/pre>\nStatus : b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'\r\nTotal Evaluations: 9\r\nSolution: f([3.38059583e-07 3.70089258e-07]) = 0.00000<\/pre>\n
Global Search With SciPy<\/h2>\n
\n
# simulated annealing global optimization for a multimodal objective function\r\nfrom scipy.optimize import dual_annealing\r\n\r\n# objective function\r\ndef objective(v):\r\n\tx, y = v\r\n\treturn (x**2 + y - 11)**2 + (x + y**2 -7)**2\r\n\r\n# define range for input\r\nr_min, r_max = -5.0, 5.0\r\n# define the bounds on the search\r\nbounds = [[r_min, r_max], [r_min, r_max]]\r\n# perform the simulated annealing search\r\nresult = dual_annealing(objective, bounds)\r\n# summarize the result\r\nprint('Status : %s' % result['message'])\r\nprint('Total Evaluations: %d' % result['nfev'])\r\n# evaluate solution\r\nsolution = result['x']\r\nevaluation = objective(solution)\r\nprint('Solution: f(%s) = %.5f' % (solution, evaluation))<\/pre>\nStatus : ['Maximum number of iteration reached']\r\nTotal Evaluations: 4028\r\nSolution: f([-3.77931027 -3.283186 ]) = 0.00000<\/pre>\n<\/p>\n
Further Reading<\/h2>\n
APIs<\/h3>\n
\n
Articles<\/h3>\n
\n
Summary<\/h2>\n
\n
\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n