We usually use TensorFlow to build a neural network. However, TensorFlow is not limited to this. Behind the scene, TensorFlow is a tensor library with automatic differentiation capability. Hence we can easily use it to solve a numerical optimization problem with gradient descent. In this post, we are going to show how TensorFlow\u2019s automatic differentiation engine, autograd, works.<\/p>\n
After finishing this tutorial, you will learn:<\/p>\n
- \n
- What is autograd in TensorFlow<\/li>\n
- How to make use of autograd and an optimizer to solve an optimization problem<\/li>\n<\/ul>\n
Let\u2019s get started.<\/p>\n
\n![\"\"](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2022\/06\/lukas-tennie-3dyDozzCORw-unsplash-scaled.jpg\")
<\/p>\nUsing autograd in TensorFlow to Solve a Regression Problem
Photo by Lukas Tennie<\/a>. Some rights reserved.<\/p>\n<\/div>\nOverview<\/h2>\n
This tutorial is in three parts; they are:<\/p>\n
- \n
- part 1<\/li>\n
- part 2<\/li>\n
- part 3<\/li>\n<\/ul>\n
Autograd in TensorFlow<\/h2>\n
In TensorFlow 2.x, we can define variables and constants as TensorFlow objects and build an expression with them. The expression is essentially a function of the variables. Hence we may derive its derivative function, i.e., the differentiation or the gradient. This feature is one of the many fundamental features in TensorFlow. The deep learning model would make use of this in the training loop.<\/p>\n
It is easier to explain autograd with an example. In TensorFlow 2.x, we can create a constant matrix as follows:<\/p>\n
import tensorflow as tf\r\n\r\nx = tf.constant([1, 2, 3])\r\nprint(x)\r\nprint(x.shape)\r\nprint(x.dtype)<\/pre>\n
The above prints:<\/p>\n
tf.Tensor([1 2 3], shape=(3,), dtype=int32)\r\n(3,)\r\n<dtype: 'int32'><\/pre>\n
Which means we created an integer vector (in the form of Tensor object). This vector can work like a NumPy vector in most of the cases. For example, we can do
x+x<\/code> or2*x<\/code> and the result is just as what we would expect. TensorFlow comes with many functions for array manipulation that match NumPy, such astf.transpose<\/code> ortf.concat<\/code>.<\/p>\nCreating variables in TensorFlow is just the same, for example:<\/p>\n
import tensorflow as tf\r\n\r\nx = tf.Variable([1, 2, 3])\r\nprint(x)\r\nprint(x.shape)\r\nprint(x.dtype)<\/pre>\n
This would print:<\/p>\n
<tf.Variable 'Variable:0' shape=(3,) dtype=int32, numpy=array([1, 2, 3], dtype=int32)>\r\n(3,)\r\n<dtype: 'int32'><\/pre>\n
and the operations (such as
x+x<\/code> and2*x<\/code>) that we can apply to Tensor objects can also be applied to variables. The only difference between variables and constants is the former allows the value to change while the latter is immutable. This distinction is important when we run a gradient tape<\/strong>, as follows:<\/p>\nimport tensorflow as tf\r\n\r\nx = tf.Variable(3.6)\r\n\r\nwith tf.GradientTape() as tape:\r\n y = x*x\r\n\r\ndy = tape.gradient(y, x)\r\nprint(dy)<\/pre>\n
This prints:<\/p>\n
tf.Tensor(7.2, shape=(), dtype=float32)<\/pre>\n
What it does is the following: We defined a variable
x<\/code> (with value 3.6) and then created a gradient tape. While the gradient tape is working, we computey=x*x<\/code> or $$y=x^2$$. The gradient tape monitored how the variables are manipulated. Afterwards, we ask the gradient tape to find the derivative $$dfrac{dy}{dx}$$. We know $$y=x^2$$ means $$y\u2019=2x$$. Hence the output would give us a value of $$3.6times 2=7.2$$.<\/p>\nUsing autograd for Polynomial Regression<\/h2>\n
How this feature in TensorFlow helpful? Let\u2019s consider a case that we have a polynomial in the form of $$y=f(x)$$ and we are given several $$(x,y)$$ samples. How can we recover the polynomial $$f(x)$$? One way to do it is to assume random coefficient for the polynomial and feed in the samples $$(x,y)$$. If the polynomial is found, we should see the value of $$y$$ matches $$f(x)$$. The closer they are, the closer our estimate is to the correct polynomial.<\/p>\n
This is indeed a numerical optimization problem such that we want to minimize the difference between $$y$$ and $$f(x)$$. We can use gradient descent to solve it.<\/p>\n
Let\u2019s consider an example. We can build a polynomial $$f(x)=x^2 + 2x + 3$$ in NumPy as follows:<\/p>\n
import numpy as np\r\n\r\npolynomial = np.poly1d([1, 2, 3])\r\nprint(polynomial)<\/pre>\n
This prints:<\/p>\n
2\r\n1 x + 2 x + 3<\/pre>\n
We may use the polynomial as a function, such as:<\/p>\n
print(polynomial(1.5))<\/pre>\n
And this prints
8.25<\/code>, for $$(1.5)^2+2times(1.5)+3 = 8.25$$.<\/p>\nNow we may generate a number of samples from this function, using NumPy:<\/p>\n
N = 20 # number of samples\r\n\r\n# Generate random samples roughly between -10 to +10\r\nX = np.random.randn(N,1) * 5\r\nY = polynomial(X)<\/pre>\n
In the above, both
X<\/code> andY<\/code> are NumPy arrays of shape(20,1)<\/code> and they are related as $$y=f(x)$$ for the polynomial $$f(x)$$.<\/p>\nNow assume we do not know what is our polynomial except it is quadratic. And we would like to recover the coefficients. Since a quadratic polynomial is in the form of $$Ax^2+Bx+C$$, we have three unknowns to find. We can find them using gradient descent algorithm that we implement, or using an existing gradient descent optimizer. The following demonstrates how it works:<\/p>\n
import tensorflow as tf\r\n\r\n# Assume samples X and Y are prepared elsewhere\r\n\r\nXX = np.hstack([X*X, X, np.ones_like(X)])\r\n\r\nw = tf.Variable(tf.random.normal((3,1))) # the 3 coefficients\r\nx = tf.constant(XX, dtype=tf.float32) # input sample\r\ny = tf.constant(Y, dtype=tf.float32) # output sample\r\noptimizer = tf.keras.optimizers.Nadam(lr=0.01)\r\nprint(w)\r\n\r\nfor _ in range(1000):\r\n with tf.GradientTape() as tape:\r\n y_pred = x @ w\r\n mse = tf.reduce_sum(tf.square(y - y_pred))\r\n grad = tape.gradient(mse, w)\r\n optimizer.apply_gradients([(grad, w)])\r\n\r\nprint(w)<\/pre>\n
The
print<\/code> statement before the for loop gives three random number, such as<\/p>\n<tf.Variable 'Variable:0' shape=(3, 1) dtype=float32, numpy=\r\narray([[-2.1450958 ],\r\n [-1.1278448 ],\r\n [ 0.31241694]], dtype=float32)><\/pre>\n
but the one after the for loop gives us the coefficients very close to that in our polynomial:<\/p>\n
<tf.Variable 'Variable:0' shape=(3, 1) dtype=float32, numpy=\r\narray([[1.0000628],\r\n [2.0002015],\r\n [2.996219 ]], dtype=float32)><\/pre>\n
What the above code does is the following: First we create a variable vector
w<\/code> of 3 values, namely the coefficients $$A,B,C$$. Then we create an array of shape $$(N,3)$$, which $$N$$ is the number of samples in our arrayX<\/code>. This array has 3 columns, which are respectively the value of $$x^2$$, $$x$$, and 1. We build such an array from the vectorX<\/code> usingnp.hstack()<\/code> function. Similarly, we build the TensorFlow constanty<\/code> from NumPy arrayY<\/code>.<\/p>\nAfterwards, we use a for loop to run gradient descent in 1000 iterations. In each iteration, we compute $$x times w$$ in matrix form to find $$Ax^2+Bx+C$$ and assign it to the variable
y_pred<\/code>. Then we comparey<\/code> andy_pred<\/code> and find the mean square error. Next, we derive the gradient, i.e., the rate of change of the mean square error with respect to the coefficientsw<\/code>. And based on this gradient, we use gradient descent to updatew<\/code>.<\/p>\nIn essence, the above code is to find the coefficients
w<\/code> that minimizes the mean square error.<\/p>\nPutting everything together, the following is the complete code:<\/p>\n
import numpy as np\r\nimport tensorflow as tf\r\n\r\nN = 20 # number of samples\r\n\r\n# Generate random samples roughly between -10 to +10\r\npolynomial = np.poly1d([1, 2, 3])\r\nX = np.random.randn(N,1) * 5\r\nY = polynomial(X)\r\n\r\n# Prepare input as an array of shape (N,3)\r\nXX = np.hstack([X*X, X, np.ones_like(X)])\r\n\r\n# Prepare TensorFlow objects\r\nw = tf.Variable(tf.random.normal((3,1))) # the 3 coefficients\r\nx = tf.constant(XX, dtype=tf.float32) # input sample\r\ny = tf.constant(Y, dtype=tf.float32) # output sample\r\noptimizer = tf.keras.optimizers.Nadam(lr=0.01)\r\nprint(w)\r\n\r\n# Run optimizer\r\nfor _ in range(1000):\r\n with tf.GradientTape() as tape:\r\n y_pred = x @ w\r\n mse = tf.reduce_sum(tf.square(y - y_pred))\r\n grad = tape.gradient(mse, w)\r\n optimizer.apply_gradients([(grad, w)])\r\n\r\nprint(w)<\/pre>\n<\/p>\n
Using autograd to Solve a Math Puzzle<\/h2>\n
In the above, we used 20 samples and it is more than enough to fit a quadratic equation. We may use gradient descent to solve some math puzzle as well. For example, the following problem:<\/p>\n
[ A ] + [ B ] = 9\r\n + -\r\n[ C ] - [ D ] = 1\r\n = =\r\n 8 2<\/pre>\n
In other words, we would like to find the values of $$A,B,C,D$$ such that:<\/p>\n
$$begin{aligned}
\nA + B &= 9 \\
\nC \u2013 D &= 1 \\
\nA + C &= 8 \\
\nB \u2013 D &= 2
\nend{aligned}$$<\/p>\nThis can also be solved using autograd, as follows:<\/p>\n
import tensorflow as tf\r\nimport random\r\n\r\nA = tf.Variable(random.random())\r\nB = tf.Variable(random.random())\r\nC = tf.Variable(random.random())\r\nD = tf.Variable(random.random())\r\n\r\n# Gradient descent loop\r\nEPOCHS = 1000\r\noptimizer = tf.keras.optimizers.Nadam(lr=0.1)\r\nfor _ in range(EPOCHS):\r\n with tf.GradientTape() as tape:\r\n y1 = A + B - 9\r\n y2 = C - D - 1\r\n y3 = A + C - 8\r\n y4 = B - D - 2\r\n sqerr = y1*y1 + y2*y2 + y3*y3 + y4*y4\r\n gradA, gradB, gradC, gradD = tape.gradient(sqerr, [A, B, C, D])\r\n optimizer.apply_gradients([(gradA, A), (gradB, B), (gradC, C), (gradD, D)])\r\n\r\nprint(A)\r\nprint(B)\r\nprint(C)\r\nprint(D)<\/pre>\n
There can be multiple solutions to this problem. One solution is the following:<\/p>\n
<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=4.6777573>\r\n<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=4.3222437>\r\n<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=3.3222427>\r\n<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=2.3222432><\/pre>\n
Which means $$A=4.68$$, $$B=4.32$$, $$C=3.32$$, and $$D=2.32$$. We can verify this solution fits the problem.<\/p>\n
What in the above code does is to define the four unknown as variables with a random initial value. Then we compute the result of the four equations and compare it to the expected answer. We then sum up the squared error and ask TensorFlow to minimize it. The minimum possible square error is zero attained when our solution fits exactly the problem.<\/p>\n
Note in the way we ask the gradient tape to produce the gradient: We ask the gradient of
sqerr<\/code> respective toA<\/code>,B<\/code>,C<\/code>, andD<\/code>. Hence four gradients are found. We then apply each gradient to the respective variables in each iteration. Rather than looking for the gradient in four different calls totape.gradient()<\/code>, this is required in TensorFlow because the gradient ofsqerr<\/code> can only be recalled once by default.<\/p>\nFurther Reading<\/h2>\n
This section provides more resources on the topic if you are looking to go deeper.<\/p>\n
Articles:<\/strong><\/p>\n
- \n
- Introduction to gradients and automatic differentiation<\/a><\/li>\n
- Advanced automatic differentiation<\/a><\/li>\n<\/ul>\n
Summary<\/h2>\n
In this post, we demonstrated how TensorFlow\u2019s automatic differentiation works. This is the building block for carrying out deep learning training. Specifically, you learned:<\/p>\n
- \n
- What is automatic differentiation in TensorFlow<\/li>\n
- How we can use gradient tape to carry out automatic differentiation<\/li>\n
- How we can use automatic differentiation to solve a optimization problem<\/li>\n<\/ul>\n
The post Using autograd in TensorFlow to Solve a Regression Problem<\/a> appeared first on Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n
Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
- Advanced automatic differentiation<\/a><\/li>\n<\/ul>\n
- Introduction to gradients and automatic differentiation<\/a><\/li>\n