{"id":2091,"date":"2019-05-03T06:31:41","date_gmt":"2019-05-03T06:31:41","guid":{"rendered":"https:\/\/www.aiproblog.com\/index.php\/2019\/05\/03\/the-approximation-power-of-neural-networks-with-python-codes\/"},"modified":"2019-05-03T06:31:41","modified_gmt":"2019-05-03T06:31:41","slug":"the-approximation-power-of-neural-networks-with-python-codes","status":"publish","type":"post","link":"https:\/\/www.aiproblog.com\/index.php\/2019\/05\/03\/the-approximation-power-of-neural-networks-with-python-codes\/","title":{"rendered":"The Approximation Power of Neural Networks (with Python\u00a0codes)"},"content":{"rendered":"

Author: Marco Tavora<\/p>\n

\n

Introduction<\/h3>\n

It is a well-known fact that\u00a0<\/span>neural networks can approximate the output of any continuous mathematical function<\/span>, no matter how complicated it might be. Take for instance the function below:<\/p>\n<\/p>\n

\n
<\/div>\n
<\/div>\n<\/div>\n

<\/p>\n

Though it has a pretty complicated shape, the theorems we will discuss shortly guarantee that one can build some neural network that can approximate\u00a0<\/span>f(x)<\/em>\u00a0<\/span>as accurately as we want. Neural networks, therefore, display a type of universal behavior.<\/p>\n

One of the reasons neural networks have received so much attention is that in addition to these rather remarkable universal properties,\u00a0<\/span>they possess many powerful algorithms for learning functions<\/a>.<\/p>\n

Universality and the underlying mathematics<\/h3>\n

This piece will give an informal overview of some of the fundamental mathematical results (theorems) underlying these approximation capabilities of artificial neural networks.<\/p>\n

\u201cAlmost any process you can imagine can be thought of as function computation\u2026 [Examples include] naming a piece of music based on a short sample of the piece [\u2026], translating a Chinese text into English [\u2026], taking an mp4 movie file and generating a description of the plot of the movie, and a discussion of the quality of the\u00a0acting.\u201d<\/p><\/blockquote>\n

\u2014 Michael\u00a0Nielsen<\/p><\/blockquote>\n

A motivation for using neural nets as approximators: Kolmogorov\u2019s Theorem<\/h3>\n

In 1957, the Russian mathematician\u00a0<\/span>Andrey Kolmogorov<\/a> (picture below) known for his contributions to a wide range of mathematical topics (such as probability theory, topology, turbulence, computational complexity, and many others),\u00a0<\/span>proved<\/a>\u00a0<\/span>an important theorem about the representation of real functions of many variables. According to Kolmogorov\u2019s theorem,\u00a0<\/span>multivariate functions can be expressed via a combination of sums and compositions of (a finite number of) univariate functions<\/span>.<\/p>\n<\/p>\n

\n
<\/div>\n
<\/div>\n<\/div>\n

A little more formally, the theorem states that a continuous function\u00a0<\/span>f<\/em>\u00a0<\/span>of real variables defined in the\u00a0<\/span>n<\/em>-dimensional hypercube (with\u00a0<\/span>n<\/em>\u00a0<\/span>\u2265 2) can be represented as follows:<\/span><\/p>\n<\/p>\n

\n
<\/div>\n
<\/div>\n<\/div>\n

<\/p>\n

In this expression, the\u00a0<\/span>g<\/em>s are univariate functions and the \u03d5s are continuous, entirely (monotonically) increasing functions (as shown in the figure below) that do not depend on the choice of\u00a0<\/span>f.<\/em><\/span><\/p>\n<\/p>\n

\n
<\/div>\n
<\/div>\n<\/div>\n

Universal Approximation Theorem\u00a0(UAT)<\/h3>\n

The UAT states that feed-forward neural networks containing a single hidden layer with a finite number of nodes can be used to approximate any continuous function provided rather mild assumptions about the form of the activation function are satisfied. Now,\u00a0<\/span>since almost any process we can imagine can be described by some mathematical function, neural networks can, at least in principle, predict the outcome of nearly every process.<\/span><\/p>\n

There are several rigorous proofs of the universality of feed-forward artificial neural nets using different activation functions. Let us, for the sake of brevity, restrict ourselves to the sigmoid function. Sigmoids are \u201cS\u201d-shaped and include as special cases the\u00a0<\/span>logistic function<\/a>, the\u00a0<\/span>Gompertz curve<\/a>, and the\u00a0<\/span>ogee curve<\/a>.<\/p>\n

Python code for the\u00a0sigmoid<\/h3>\n

A quick Python code to build and plot a sigmoid function is:<\/p>\n

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<\/pre>\n
upper, lower = 6, -6
num = 100<\/pre>\n
def sigmoid_activation(x):
if x > upper:
return 1
\nelif x < lower:
\nreturn 0
\nreturn 1\/(1+np.exp(-x))<\/pre>\n
vals = [sigmoid_activation(x) for 
x in np.linspace(lower, upper, num=num)]
plt.plot(np.linspace(lower, 
\nupper, 
\nnum=num), vals);
\nplt.title('Sigmoid');<\/pre>\n<\/p>\n
\n
<\/div>\n
<\/div>\n<\/div>\n
George Cybenko\u2019s Proof<\/h3>\n
The proof given by\u00a0<\/span>Cybenko (1989)<\/a>\u00a0<\/span>is known for its elegance, simplicity, and conciseness. In his article, he proves the following statement. Let \u03d5 be any continuous function of the sigmoid type (see discussion above). Given any multivariate continuous function<\/p>\n<\/p>\n
\n
<\/div>\n
<\/div>\n<\/div>\n
inside a\u00a0<\/span>compact<\/a>\u00a0<\/span>subset of the\u00a0<\/span>N<\/em>-dimensional real space and any positive \u03f5, there are vectors<\/p>\n<\/p>\n
\n
<\/div>\n
<\/div>\n<\/div>\n
(the\u00a0<\/span>weights<\/strong>), constants<\/p>\n<\/p>\n
\n
<\/div>\n
<\/div>\n<\/p>\n
(the\u00a0<\/span>bias\u00a0<\/span><\/strong>terms) and<\/p>\n<\/p>\n
\n
<\/div>\n
<\/div>\n<\/p>\n
such that<\/p>\n<\/p>\n
\n
<\/div>\n
<\/div>\n<\/p>\n
for any\u00a0<\/span>x\u00a0<\/span><\/em>(the NN inputs) inside the compact subset, where the function\u00a0<\/span>G<\/em>\u00a0<\/span>is given by:<\/p>\n<\/p>\n
\n
<\/div>\n
<\/div>\n<\/div>\n
Choosing the appropriate parameters, neural nets can be used to represent functions with a wide range of different forms.<\/p>\n
An example using\u00a0Python<\/h3>\n
To make these statements less abstract, let us build a simple Python code to illustrate what has been discussed until now. The following analysis is based on Michael Nielsen\u2019s\u00a0<\/span>great online book<\/a>\u00a0<\/span>and\u00a0<\/span>on the exceptional lectures byMatt Brems<\/a>\u00a0<\/span>and\u00a0<\/span>Justin Pounders<\/a>\u00a0<\/span>at the\u00a0<\/span>General Assembly Data Science Immersive (DSI)<\/a>.<\/span><\/p>\n
We will build a neural network to approximate the following simple function:<\/p>\n<\/p>\n
\n
<\/div>\n
<\/div>\n<\/div>\n
A few remarks are needed before diving into the code:<\/p>\n