Optimization<\/strong> is a field of mathematics concerned with finding a good or best solution among many candidates.<\/p>\n It is an important foundational topic required in machine learning as most machine learning algorithms are fit on historical data using an optimization algorithm. Additionally, broader problems, such as model selection and hyperparameter tuning, can also be framed as an optimization problem.<\/p>\n Although having some background in optimization is critical for machine learning practitioners, it can be a daunting topic given that it is often described using highly mathematical language.<\/p>\n In this post, you will discover top books on optimization that will be helpful to machine learning practitioners.<\/p>\n Let’s get started.<\/p>\n Books on Optimization for Machine Learning The field of optimization is enormous as it touches many other fields of study.<\/p>\n As such, there are hundreds of books on the topic, and most are textbooks filed with math and proofs. This is fair enough given that it is a highly mathematical subject.<\/p>\n Nevertheless, there are books that provide a more approachable description of optimization algorithms.<\/p>\n Not all optimization algorithms are relevant to machine learning; instead, it is useful to focus on a small subset of algorithms.<\/p>\n Frankly, it is hard to group optimization<\/a> algorithms as there are many concerns. Nevertheless, it is important to have some idea of the optimization that underlies simpler algorithms, such as linear regression and logistic regression (e.g. convex optimization, least squares, newton methods, etc.), and neural networks (first-order methods, gradient descent, etc.).<\/p>\n These are foundational optimization algorithms covered in most optimization textbooks.<\/p>\n Not all optimization problems in machine learning are well behaved, such as optimization used in AutoML and hyperparameter tuning. Therefore, knowledge of stochastic optimization algorithms is required (simulated annealing, genetic algorithms, particle swarm, etc.). Although these are optimization algorithms, they are also a type of learning algorithm referred to as biologically inspired computation<\/a> or computational intelligence<\/a>.<\/p>\n Therefore, we will take a look at both books that cover classical optimization algorithms as well as books on alternate optimization algorithms.<\/p>\n In fact, the first book we will look at covers both types of algorithms, and much more.<\/p>\n This book was written by Mykel Kochenderfer<\/a> and Tim Wheeler<\/a> and was published in 2019.<\/p>\n Algorithms for Optimization<\/p>\n<\/div>\n This book might be one of the very few textbooks that I’ve seen that broadly covers the field of optimization techniques relevant to modern machine learning.<\/p>\n This book provides a broad introduction to optimization with a focus on practical algorithms for the design of engineering systems. We cover a wide variety of optimization topics, introducing the underlying mathematical problem formulations and the algorithms for solving them. Figures, examples, and exercises are provided to convey the intuition behind the various approaches.<\/p>\n<\/blockquote>\n — Page xiiix, Algorithms for Optimization<\/a>, 2019.<\/p>\n Importantly the algorithms range from univariate methods (bisection, line search, etc.) to first-order methods (gradient descent), second-order methods (Newton’s method), direct methods (pattern search), stochastic methods (simulated annealing), and population methods (genetic algorithms, particle swarm), and so much more.<\/p>\n It includes both technical descriptions of algorithms with references and worked examples of algorithms in Julia. It’s a shame the examples are not in Python as this would make the book near perfect in my eyes.<\/p>\n The complete table of contents for the book is listed below.<\/p>\n I like this book a lot; it is full of valuable practical advice. I highly recommend it!<\/p>\n This book was written by Jorge Nocedal<\/a> and Stephen Wright<\/a> and was published in 2006.<\/p>\n Numerical Optimization<\/p>\n<\/div>\n This book is focused on the math and theory of the optimization algorithms presented and does cover many of the foundational techniques used by common machine learning algorithms. It may be a little too heavy for the average practitioner.<\/p>\n The book is intended as a textbook for graduate students in mathematical subjects.<\/p>\n We intend that this book will be used in graduate-level courses in optimization, as offered in engineering, operations research, computer science, and mathematics departments.<\/p>\n<\/blockquote>\n — Page xviii, Numerical Optimization<\/a>, 2006.<\/p>\n Even though it is highly mathematical, the descriptions of the algorithms are precise and may provide a useful alternative description to complement the other books listed.<\/p>\n The complete table of contents for the book is listed below.<\/p>\n It’s a solid textbook on optimization.<\/p>\n If you do prefer the theoretical approach to the subject, another widely used mathematical book on optimization is “Convex Optimization<\/a>” written by Stephen Boyd<\/a> and Lieven Vandenberghe<\/a> and published in 2004.<\/p>\n This book was written by Andries Engelbrecht<\/a> and published in 2007.<\/p>\n Computational Intelligence: An Introduction<\/p>\n<\/div>\n This book provides an excellent overview of the field of nature-inspired optimization algorithms, also referred to as computational intelligence<\/a>. This includes fields such as evolutionary computation<\/a> and swarm intelligence<\/a>.<\/p>\n This book is far less mathematical than the previous textbooks and is more focused on the metaphor of the inspired system and how to configure and use the specific algorithms with lots of pseudocode explanations.<\/p>\n While the material is introductory in nature, it does not shy away from details, and does present the mathematical foundations to the interested reader. The intention of the book is not to provide thorough attention to all computational intelligence paradigms and algorithms, but to give an overview of the most popular and frequently used models.<\/p>\n<\/blockquote>\n — Page xxix, Computational Intelligence: An Introduction<\/a>, 2007.<\/p>\n Algorithms like genetic algorithms, genetic programming, evolutionary strategies, differential evolution, and particle swarm optimization are useful to know for machine learning model hyperparameter tuning and perhaps even model selection. They also form the core of many modern AutoML systems.<\/p>\n The complete table of contents for the book is listed below.<\/p>\n I’m a fan of this book and recommend it.<\/p>\n In this post, you discovered books on optimization algorithms that are helpful to know for applied machine learning.<\/p>\n Did I miss a good book on optimization?<\/strong> Have you read any of the books listed?<\/strong> The post 3 Books on Optimization for Machine Learning<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"![\"Books](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2021\/01\/Books-on-Optimization-for-Machine-Learning.jpg\")
<\/p>\n
Photo by Patrick Alexander<\/a>, some rights reserved.<\/p>\n<\/div>\nOverview<\/h2>\n
Algorithms for Optimization<\/a><\/h2>\n
![\"Algorithms](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/09\/Algorithms-for-Optimization.jpg\")
<\/a><\/p>\n\n
\n
Learn More:<\/h3>\n
\n
Numerical Optimization<\/a><\/h2>\n
![\"Numerical](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/09\/Numerical-Optimization.jpeg\")
<\/a><\/p>\n\n
\n
Learn More:<\/h3>\n
\n
Computational Intelligence: An Introduction<\/a><\/h2>\n
![\"Computational](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2020\/09\/Computational-Intelligence-An-Introduction.jpg\")
<\/a><\/p>\n\n
\n
\n
\n
\n
\n
\n
\n
Learn More:<\/h3>\n
\n
Summary<\/h2>\n
\nLet me know in the comments below.<\/p>\n
\nLet me know what you think of it in the comments.<\/p>\n