This article was written by Alberto Quesada<\/a>.<\/span><\/em><\/p>\n \u00a0<\/span><\/em><\/p>\n The procedure used to carry out the learning process in a neural network is called the optimization algorithm (or optimizer).\u00a0<\/span>There are many different optimization algorithms. All have different characteristics and performance in terms of memory requirements, speed and precision.<\/span><\/p>\n \u00a0<\/span><\/p>\n Problem formulation<\/strong><\/span><\/p>\n The learning problem is formulated in terms of the minimization of a\u00a0loss index,\u00a0f. This is a function which measures the performance of a\u00a0neural network\u00a0on a\u00a0data set.<\/span><\/p>\n The loss index is, in general, composed of an error and a regularization terms. The\u00a0error term\u00a0evaluates how a neural network fits the data set. The\u00a0regularization term\u00a0is used to prevent overfitting, by controlling the effective complexity of the neural network.<\/span><\/p>\n The loss function depends on the adaptive\u00a0parameters (biases and synaptic weights) in the neural network. We can conveniently group them together into a single n-dimensional weight vector\u00a0w.<\/span><\/p>\n The problem of minimizing continuous and differentiable functions of many variables has been widely studied. Many of the conventional approaches to this problem are directly applicable to that of training neural networks.<\/span><\/p>\n One-dimensional optimization<\/strong><\/span><\/p>\n Although the loss function depends on many parameters, one-dimensional optimization methods are of great importance here. Indeed, they are very often used in the training process of a neural network.<\/span><\/p>\n Certainly, many training algorithms first compute a training direction\u00a0d\u00a0and then a training rate\u00a0<\/span>\u03b7<\/span>; that minimizes the loss in that direction,\u00a0f(<\/span>\u03b7<\/span>).<\/span><\/p>\n In this regard, one-dimensional optimization methods search for the minimum of a given one-dimensional function. Some of the algorithms which are widely used are the golden section method and Brent’s method. Both reduce the bracket of a minimum until the distance between the two outer points in the bracket is less than a defined tolerance.<\/span><\/p>\n \u00a0<\/span><\/strong><\/p>\n Multidimensional optimization<\/strong><\/span><\/p>\n The learning problem for neural networks is formulated as searching of a parameter vector\u00a0w<\/span>\u2217\u00a0at which the loss function\u00a0f takes a minimum value. The necessary condition states that if the neural network is at a minimum of the loss function, then the gradient is the zero vector.<\/span><\/p>\n The loss function is, in general, a non-linear function of the parameters. As a consequence, it is not possible to find closed training algorithms for the minima. Instead, we consider a search through the parameter space consisting of a succession of steps. At each step, the loss will decrease by adjusting the neural network parameters.<\/span><\/p>\n In this way, to train a neural network we start with some parameter vector (often chosen at random). Then, we generate a sequence of parameters, so that the loss function is reduced at each iteration of the algorithm. The change of loss between two steps is called the loss decrement. The training algorithm stops when a specified condition, or stopping criterion, is satisfied.<\/span><\/p>\n \u00a0<\/span><\/p>\n These are the main training algorithms for neural networks:<\/span><\/p>\n \u00a0<\/span><\/em><\/p>\n To read the whole article, with formulas and illustrations, click here<\/a>.<\/em><\/p>\n \u00a0<\/span><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<\/a><\/span><\/p>\n
\n