{"id":1931,"date":"2019-03-26T18:00:26","date_gmt":"2019-03-26T18:00:26","guid":{"rendered":"https:\/\/www.aiproblog.com\/index.php\/2019\/03\/26\/how-to-evaluate-pixel-scaling-methods-for-image-classification-with-convolutional-neural-networks\/"},"modified":"2019-03-26T18:00:26","modified_gmt":"2019-03-26T18:00:26","slug":"how-to-evaluate-pixel-scaling-methods-for-image-classification-with-convolutional-neural-networks","status":"publish","type":"post","link":"https:\/\/www.aiproblog.com\/index.php\/2019\/03\/26\/how-to-evaluate-pixel-scaling-methods-for-image-classification-with-convolutional-neural-networks\/","title":{"rendered":"How to Evaluate Pixel Scaling Methods for Image Classification With Convolutional Neural Networks"},"content":{"rendered":"<p>Author: Jason Brownlee<\/p>\n<div>\n<p>Image data must be prepared before it can be used as the basis for modeling in image classification tasks.<\/p>\n<p>One aspect of preparing image data is scaling pixel values, such as normalizing the values to the range 0-1, centering, standardization, and more.<\/p>\n<blockquote>\n<p>How do you choose a good, or even best, pixel scaling method for your image classification or <a href=\"https:\/\/machinelearningmastery.com\/what-is-computer-vision\/\">computer vision<\/a> modeling task?<\/p>\n<\/blockquote>\n<p>In this tutorial, you will discover how to choose a pixel scaling method for image classification with deep learning methods.<\/p>\n<p>After completing this tutorial, you will know:<\/p>\n<ul>\n<li>A procedure for choosing a pixel scaling method using experimentation and empirical results on a specific dataset.<\/li>\n<li>How to implement standard pixel scaling methods for preparing image data for modeling.<\/li>\n<li>How to work through a case study for choosing a pixel scaling method for a standard image classification problem.<\/li>\n<\/ul>\n<p>Let\u2019s get started.<\/p>\n<div id=\"attachment_7368\" style=\"width: 650px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-7368\" class=\"size-full wp-image-7368\" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/03\/How-to-Evaluate-Pixel-Scaling-Methods-for-Image-Classification-With-Convolutional-Neural-Networks.jpg\" alt=\"How to Evaluate Pixel Scaling Methods for Image Classification With Convolutional Neural Networks\" width=\"640\" height=\"426\" srcset=\"http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2019\/03\/How-to-Evaluate-Pixel-Scaling-Methods-for-Image-Classification-With-Convolutional-Neural-Networks.jpg 640w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2019\/03\/How-to-Evaluate-Pixel-Scaling-Methods-for-Image-Classification-With-Convolutional-Neural-Networks-300x200.jpg 300w\" sizes=\"(max-width: 640px) 100vw, 640px\"><\/p>\n<p id=\"caption-attachment-7368\" class=\"wp-caption-text\">How to Evaluate Pixel Scaling Methods for Image Classification With Convolutional Neural Networks<br \/>Photo by <a href=\"https:\/\/www.flickr.com\/photos\/104082864@N04\/46758894811\/\">Andres Alvarado<\/a>, some rights reserved.<\/p>\n<\/div>\n<h2>Tutorial Overview<\/h2>\n<p>This tutorial is divided into 6 parts; they are:<\/p>\n<ol>\n<li>Procedure for Choosing a Pixel Scaling Method<\/li>\n<li>Choose Dataset: MNIST Image Classification<\/li>\n<li>Choose Model: Convolutional Neural Network<\/li>\n<li>Choose Pixel Scaling Methods<\/li>\n<li>Run Experiment<\/li>\n<li>Analyze Results<\/li>\n<\/ol>\n<h2>Procedure for Choosing a Pixel Scaling Method<\/h2>\n<p>Given a new image classification task, what pixel scaling methods should be used?<\/p>\n<p>There are many ways to answer this question; for example:<\/p>\n<ul>\n<li>Use techniques reportedly used for similar problems in research papers.<\/li>\n<li>Use heuristics from blog posts, courses, or books.<\/li>\n<li>Use your favorite technique.<\/li>\n<li>Use the simplest technique.<\/li>\n<li>\u2026<\/li>\n<\/ul>\n<p>Instead, I recommend using experimentation in order to discover what works best for your specific dataset.<\/p>\n<p>This can be achieved using the following process:<\/p>\n<ul>\n<li><strong>Step 1: Choose Dataset<\/strong>. This may be the entire training dataset or a small subset. The idea is to complete the experiments quickly and get a result.<\/li>\n<li><strong>Step 2: Choose Model<\/strong>. Design a model that is skillful, but not necessarily the best model for the problem. Some parallel prototyping of models may be required.<\/li>\n<li><strong>Step 3: Choose Pixel Scaling Methods<\/strong>. List 3-5 data preparation schemes for evaluation of your problem.<\/li>\n<li><strong>Step 4: Run Experiment<\/strong>. Run the experiments in such a way that the results are robust and representative, ideally repeat each experiment multiple times.<\/li>\n<li><strong>Step 5: Analyze Results<\/strong>. Compare methods both in terms of the speed of learning and mean performance across repeated experiments.<\/li>\n<\/ul>\n<p>The experimental approach will use a non-optimized model and perhaps a subset of training data, both of which may add noise to the decision you must make.<\/p>\n<p>Therefore, you are looking for a signal that one data preparation scheme for your images is clearly better than the others; if this is not the case for your dataset, then the simplest (least computationally complex) technique should be used, such as pixel normalization.<\/p>\n<p>A clear signal of a superior pixel scaling method may be seen in one of two ways:<\/p>\n<ul>\n<li><strong>Faster Learning<\/strong>. Learning curves clearly show that a model learns faster with a given data preparation scheme.<\/li>\n<li><strong>Better Accuracy<\/strong>. Mean model performance clearly shows better accuracy with a given data preparation scheme.<\/li>\n<\/ul>\n<p>Now that we have a procedure for choosing a pixel scaling method for image data, let\u2019s look at an example. We will use the MNIST image classification task fit with a CNN and evaluate a range of standard pixel scaling methods.<\/p>\n<h2>Step 1. Choose Dataset: MNIST Image Classification<\/h2>\n<p>The <a href=\"https:\/\/en.wikipedia.org\/wiki\/MNIST_database\">MNIST problem<\/a>, or MNIST for short, is an image classification problem comprised of 70,000 images of handwritten digits.<\/p>\n<p>The goal of the problem is to classify a given image of a handwritten digit as an integer from 0 to 9. As such, it is a multiclass image classification problem.<\/p>\n<p>It is a standard dataset for evaluating machine learning and deep learning algorithms. <a href=\"http:\/\/rodrigob.github.io\/are_we_there_yet\/build\/classification_datasets_results.html#4d4e495354\">Best results<\/a> for the dataset are about 99.79% accurate, or an error rate of about 0.21% (e.g. less than 1%).<\/p>\n<p>This dataset is provided as part of the Keras library and can be automatically downloaded (if needed) and loaded into memory by a call to the <a href=\"https:\/\/keras.io\/datasets\/#mnist-database-of-handwritten-digits\">keras.datasets.mnist.load_data() function<\/a>.<\/p>\n<p>The function returns two tuples: one for the training inputs and outputs and one for the test inputs and outputs. For example:<\/p>\n<pre class=\"crayon-plain-tag\"># example of loading the MNIST dataset\r\nfrom keras.datasets import mnist\r\n(x_train, y_train), (x_test, y_test) = mnist.load_data()<\/pre>\n<p>We can load the MNIST dataset and summarize it.<\/p>\n<p>The complete example is listed below.<\/p>\n<pre class=\"crayon-plain-tag\"># load and summarize the MNIST dataset\r\nfrom keras.datasets import mnist\r\n# load dataset\r\n(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\r\n# summarize dataset shape\r\nprint('Train', train_images.shape, train_labels.shape)\r\nprint('Test', (test_images.shape, test_labels.shape))\r\n# summarize pixel values\r\nprint('Train', train_images.min(), train_images.max(), train_images.mean(), train_images.std())\r\nprint('Train', test_images.min(), test_images.max(), test_images.mean(), test_images.std())<\/pre>\n<p>Running the example first loads the dataset into memory. Then the shape of the training and test datasets is reported.<\/p>\n<p>We can see that all images are 28 by 28 pixels with a single channel for grayscale images. There are 60,000 images for the training dataset and 10,000 for the test dataset.<\/p>\n<p>We can also see that pixel values are integer values between 0 and 255 and that the mean and standard deviation of the pixel values are similar between the two datasets.<\/p>\n<pre class=\"crayon-plain-tag\">Train (60000, 28, 28) (60000,)\r\nTest ((10000, 28, 28), (10000,))\r\nTrain 0 255 33.318421449829934 78.56748998339798\r\nTrain 0 255 33.791224489795916 79.17246322228644<\/pre>\n<p>The dataset is relatively small; we will use the entire train and test dataset<\/p>\n<p>Now that we are familiar with MNIST and how to load the dataset, let\u2019s review some pixel scaling methods.<\/p>\n<h2>Step 2. Choose Model: Convolutional Neural Network<\/h2>\n<p>We will use a convolutional neural network model to evaluate the different pixel scaling methods.<\/p>\n<p>A CNN is expected to perform very well on this problem, although the model chosen for this experiment does not have to perform well or best for the problem. Instead, it must be skillful (better than random) and must allow the impact of different data preparation schemes to be differentiated in terms of speed of learning and\/or model performance.<\/p>\n<p>As such, the model must have sufficient capacity to learn the problem.<\/p>\n<p>We will demonstrate the baseline model on the MNIST problem.<\/p>\n<p>First, the dataset must be loaded and the shape of the train and test dataset expanded to add a channel dimension, set to one as we only have a single black and white channel.<\/p>\n<pre class=\"crayon-plain-tag\"># load dataset\r\n(trainX, trainY), (testX, testY) = mnist.load_data()\r\n# reshape dataset to have a single channel\r\nwidth, height, channels = trainX.shape[1], trainX.shape[2], 1\r\ntrainX = trainX.reshape((trainX.shape[0], width, height, channels))\r\ntestX = testX.reshape((testX.shape[0], width, height, channels))<\/pre>\n<p>Next, we will normalize the pixel values for this example and one hot encode the target values, required for multiclass classification.<\/p>\n<pre class=\"crayon-plain-tag\"># normalize pixel values\r\ntrainX = trainX.astype('float32') \/ 255\r\ntestX = testX.astype('float32') \/ 255\r\n# one hot encode target values\r\ntrainY = to_categorical(trainY)\r\ntestY = to_categorical(testY)<\/pre>\n<p>The model is defined as a convolutional layer followed by a max pooling layer; this combination is repeated again, then the filter maps are flattened, interpreted by a fully connected layer and followed by an output layer.<\/p>\n<p>The ReLU activation function is used for hidden layers and the softmax activation function is used for the output layer. Enough filter maps and nodes are specified to provide sufficient capacity to learn the problem.<\/p>\n<pre class=\"crayon-plain-tag\"># define model\r\nmodel = Sequential()\r\nmodel.add(Conv2D(32, (3, 3), activation='relu', input_shape=(width, height, channels)))\r\nmodel.add(MaxPooling2D((2, 2)))\r\nmodel.add(Conv2D(64, (3, 3), activation='relu'))\r\nmodel.add(MaxPooling2D((2, 2)))\r\nmodel.add(Flatten())\r\nmodel.add(Dense(64, activation='relu'))\r\nmodel.add(Dense(10, activation='softmax'))<\/pre>\n<p>The <a href=\"https:\/\/machinelearningmastery.com\/adam-optimization-algorithm-for-deep-learning\/\">Adam<\/a> variation of stochastic gradient descent is used to find the model weights. The categorical cross entropy loss function is used, required for multi-class classification, and classification accuracy is monitored during training.<\/p>\n<pre class=\"crayon-plain-tag\"># compile model\r\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])<\/pre>\n<p>The model is fit for five training epochs and a large batch size of 128 images is used.<\/p>\n<pre class=\"crayon-plain-tag\"># fit model\r\nmodel.fit(trainX, trainY, epochs=5, batch_size=128)<\/pre>\n<p>Once fit, the model is evaluated on the test dataset.<\/p>\n<pre class=\"crayon-plain-tag\"># evaluate model\r\n_, acc = model.evaluate(testX, testY, verbose=0)\r\nprint(acc)<\/pre>\n<p>The complete example is listed below and will easily run on the CPU in about a minute.<\/p>\n<pre class=\"crayon-plain-tag\"># baseline cnn model for the mnist problem\r\nfrom keras.datasets import mnist\r\nfrom keras.utils import to_categorical\r\nfrom keras.models import Sequential\r\nfrom keras.layers import Conv2D\r\nfrom keras.layers import MaxPooling2D\r\nfrom keras.layers import Dense\r\nfrom keras.layers import Flatten\r\n# load dataset\r\n(trainX, trainY), (testX, testY) = mnist.load_data()\r\n# reshape dataset to have a single channel\r\nwidth, height, channels = trainX.shape[1], trainX.shape[2], 1\r\ntrainX = trainX.reshape((trainX.shape[0], width, height, channels))\r\ntestX = testX.reshape((testX.shape[0], width, height, channels))\r\n# normalize pixel values\r\ntrainX = trainX.astype('float32') \/ 255\r\ntestX = testX.astype('float32') \/ 255\r\n# one hot encode target values\r\ntrainY = to_categorical(trainY)\r\ntestY = to_categorical(testY)\r\n# define model\r\nmodel = Sequential()\r\nmodel.add(Conv2D(32, (3, 3), activation='relu', input_shape=(width, height, channels)))\r\nmodel.add(MaxPooling2D((2, 2)))\r\nmodel.add(Conv2D(64, (3, 3), activation='relu'))\r\nmodel.add(MaxPooling2D((2, 2)))\r\nmodel.add(Flatten())\r\nmodel.add(Dense(64, activation='relu'))\r\nmodel.add(Dense(10, activation='softmax'))\r\n# compile model\r\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\r\n# fit model\r\nmodel.fit(trainX, trainY, epochs=5, batch_size=128)\r\n# evaluate model\r\n_, acc = model.evaluate(testX, testY, verbose=0)\r\nprint(acc)<\/pre>\n<p>Running the example shows that the model is capable of learning the problem well and quickly.<\/p>\n<p>In fact, the performance of the model on the test dataset on this run is 99%, or a 1% error rate. This is not state of the art (by design), but is not terribly far from state of the art either.<\/p>\n<pre class=\"crayon-plain-tag\">60000\/60000 [==============================] - 13s 220us\/step - loss: 0.2321 - acc: 0.9323\r\nEpoch 2\/5\r\n60000\/60000 [==============================] - 12s 204us\/step - loss: 0.0628 - acc: 0.9810\r\nEpoch 3\/5\r\n60000\/60000 [==============================] - 13s 208us\/step - loss: 0.0446 - acc: 0.9861\r\nEpoch 4\/5\r\n60000\/60000 [==============================] - 13s 209us\/step - loss: 0.0340 - acc: 0.9895\r\nEpoch 5\/5\r\n60000\/60000 [==============================] - 12s 208us\/step - loss: 0.0287 - acc: 0.9908\r\n0.99<\/pre>\n<\/p>\n<h2>Step 3. Choose Pixel Scaling Methods<\/h2>\n<p>Neural network models often cannot be trained on raw pixel values, such as pixel values in the range of 0 to 255.<\/p>\n<p>The reason is that the network uses a weighted sum of inputs, and for the network to both be stable and train effectively, weights should be kept small.<\/p>\n<p>Instead, the pixel values must be scaled prior to training. There are perhaps three main approaches to scaling pixel values; they are:<\/p>\n<ul>\n<li><strong>Normalization<\/strong>: pixel values are scaled to the range 0-1.<\/li>\n<li><strong>Centering<\/strong>: the mean pixel value is subtracted from each pixel value resulting in a distribution of pixel values centered on a mean of zero.<\/li>\n<li><strong>Standardization<\/strong>: the pixel values are scaled to a standard Gaussian with a mean of zero and a standard deviation of one.<\/li>\n<\/ul>\n<p>Traditionally, sigmoid activation functions were used and inputs that sum to 0 (zero mean) were preferred. This may or may not still be the case with the wide adoption of ReLU and similar activation functions.<\/p>\n<p>Further, in centering and standardization, the mean or mean and standard deviation can be calculated across a channel, an image, a mini-batch, or the entire training dataset. This may add additional variations on a chosen scaling method that may be evaluated.<\/p>\n<p>Normalization is often the default approach as we can assume pixel values are always in the range 0-255, making the procedure very simple and efficient to implement.<\/p>\n<p>Centering is often promoted as the preferred approach as it was used in many popular papers, although the mean can be calculated per image (global) or channel (local) and across the batch of images or the entire training dataset, and often the procedure described in a paper does not specify exactly which variation was used.<\/p>\n<p>We will experiment with the three approaches listed above, namely normalization, centering, and standardization. The mean for centering and the mean and standard deviation for standardization will be calculated across the entire training dataset.<\/p>\n<p>Other variations you could explore include:<\/p>\n<ul>\n<li>Calculating statistics for each channel (for color images).<\/li>\n<li>Calculating statistics for each image.<\/li>\n<li>Calculating statistics for each batch.<\/li>\n<li>Normalizing after centering or standardizing.<\/li>\n<\/ul>\n<p>The example below implements the three chosen pixel scaling methods and demonstrate their effect on the MNIST dataset.<\/p>\n<pre class=\"crayon-plain-tag\"># demonstrate pixel scaling methods on mnist dataset\r\nfrom keras.datasets import mnist\r\n\r\n# normalize images\r\ndef prep_normalize(train, test):\r\n\t# convert from integers to floats\r\n\ttrain_norm = train.astype('float32')\r\n\ttest_norm = test.astype('float32')\r\n\t# normalize to range 0-1\r\n\ttrain_norm = train_norm \/ 255.0\r\n\ttest_norm = test_norm \/ 255.0\r\n\t# return normalized images\r\n\treturn train_norm, test_norm\r\n\r\n# center images\r\ndef prep_center(train, test):\r\n\t# convert from integers to floats\r\n\ttrain_cent = train.astype('float32')\r\n\ttest_cent = test.astype('float32')\r\n\t# calculate statistics\r\n\tm = train_cent.mean()\r\n\t# center datasets\r\n\ttrain_cent = train_cent - m\r\n\ttest_cent = test_cent - m\r\n\t# return normalized images\r\n\treturn train_cent, test_cent\r\n\r\n# standardize images\r\ndef prep_standardize(train, test):\r\n\t# convert from integers to floats\r\n\ttrain_stan = train.astype('float32')\r\n\ttest_stan = test.astype('float32')\r\n\t# calculate statistics\r\n\tm = train_stan.mean()\r\n\ts = train_stan.std()\r\n\t# center datasets\r\n\ttrain_stan = (train_stan - m) \/ s\r\n\ttest_stan = (test_stan - m) \/ s\r\n\t# return normalized images\r\n\treturn train_stan, test_stan\r\n\r\n# load dataset\r\n(train_images, train_labels), (test_images, test_labels) = mnist.load_data()\r\n# normalize\r\ntrainX, testX = prep_normalize(train_images, test_images)\r\nprint('normalization')\r\nprint('Train', trainX.min(), trainX.max(), trainX.mean(), trainX.std())\r\nprint('Test', testX.min(), testX.max(), testX.mean(), testX.std())\r\n# center\r\ntrainX, testX = prep_center(train_images, test_images)\r\nprint('center')\r\nprint('Train', trainX.min(), trainX.max(), trainX.mean(), trainX.std())\r\nprint('Test', testX.min(), testX.max(), testX.mean(), testX.std())\r\n# standardize\r\ntrainX, testX = prep_standardize(train_images, test_images)\r\nprint('standardize')\r\nprint('Train', trainX.min(), trainX.max(), trainX.mean(), trainX.std())\r\nprint('Test', testX.min(), testX.max(), testX.mean(), testX.std())<\/pre>\n<p>Running the example first normalizes the dataset and reports the min, max, mean, and standard deviation for the train and test dataset.<\/p>\n<p>This is then repeated for the centering and standardization data preparation schemes. The results provide evidence that the scaling procedures are indeed implemented correctly.<\/p>\n<pre class=\"crayon-plain-tag\">normalization\r\nTrain 0.0 1.0 0.13066062 0.30810776\r\nTest 0.0 1.0 0.13251467 0.31048027\r\n\r\ncenter\r\nTrain -33.318447 221.68155 -1.9512918e-05 78.567444\r\nTest -33.318447 221.68155 0.47278798 79.17245\r\n\r\nstandardize\r\nTrain -0.42407447 2.8215446 -3.4560264e-07 0.9999998\r\nTest -0.42407447 2.8215446 0.0060174568 1.0077008<\/pre>\n<\/p>\n<h2>Step 4. Run Experiment<\/h2>\n<p>Now that we have defined the dataset, the model, and the data preparation schemes to evaluate, we are ready to define and run the experiment.<\/p>\n<p>Each model takes about one minute to run on the CPU, so we don\u2019t want to the experiment to take too long. We will evaluate each of the three data preparation schemes and each scheme will be evaluated 10 times, meaning that about 30 minutes will be required to complete the experiment on modern hardware.<\/p>\n<p>We can define a function to load the dataset afresh when needed.<\/p>\n<pre class=\"crayon-plain-tag\"># load train and test dataset\r\ndef load_dataset():\r\n\t# load dataset\r\n\t(trainX, trainY), (testX, testY) = mnist.load_data()\r\n\t# reshape dataset to have a single channel\r\n\twidth, height, channels = trainX.shape[1], trainX.shape[2], 1\r\n\ttrainX = trainX.reshape((trainX.shape[0], width, height, channels))\r\n\ttestX = testX.reshape((testX.shape[0], width, height, channels))\r\n\t# one hot encode target values\r\n\ttrainY = to_categorical(trainY)\r\n\ttestY = to_categorical(testY)\r\n\treturn trainX, trainY, testX, testY<\/pre>\n<p>We can also define a function to define and compile our model ready to fit on the problem.<\/p>\n<pre class=\"crayon-plain-tag\"># define cnn model\r\ndef define_model():\r\n\tmodel = Sequential()\r\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', input_shape=(width, height, channels)))\r\n\tmodel.add(MaxPooling2D((2, 2)))\r\n\tmodel.add(Conv2D(64, (3, 3), activation='relu'))\r\n\tmodel.add(MaxPooling2D((2, 2)))\r\n\tmodel.add(Flatten())\r\n\tmodel.add(Dense(64, activation='relu'))\r\n\tmodel.add(Dense(10, activation='softmax'))\r\n\t# compile model\r\n\tmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\r\n\treturn model<\/pre>\n<p>We already have functions for preparing the pixel data for the train and test datasets.<\/p>\n<p>Finally, we can define a function called <em>repeated_evaluation()<\/em> that takes the name of the data preparation function to call to prepare the data and will load the dataset and repeatedly define the model, prepare the dataset, fit, and evaluate the model. It will return a list of accuracy scores that can be used to summarize the performance of the model under the chosen data preparation scheme.<\/p>\n<pre class=\"crayon-plain-tag\"># repeated evaluation of model with data prep scheme\r\ndef repeated_evaluation(datapre_func, n_repeats=10):\r\n\t# prepare data\r\n\ttrainX, trainY, testX, testY = load_dataset()\r\n\t# repeated evaluation\r\n\tscores = list()\r\n\tfor i in range(n_repeats):\r\n\t\t# define model\r\n\t\tmodel = define_model()\r\n\t\t# prepare data\r\n\t\tprep_trainX, prep_testX = datapre_func(trainX, testX)\r\n\t\t# fit model\r\n\t\tmodel.fit(prep_trainX, trainY, epochs=5, batch_size=64, verbose=0)\r\n\t\t# evaluate model\r\n\t\t_, acc = model.evaluate(prep_testX, testY, verbose=0)\r\n\t\t# store result\r\n\t\tscores.append(acc)\r\n\t\tprint('> %d: %.3f' % (i, acc * 100.0))\r\n\treturn scores<\/pre>\n<p>The <em>repeated_evaluation()<\/em> function can then be called for each of the three data preparation schemes and the mean and standard deviation of model performance under the scheme can be reported.<\/p>\n<p>We can also create a box and whisker plot to summarize and compare the distribution of accuracy scores for each scheme.<\/p>\n<pre class=\"crayon-plain-tag\">all_scores = list()\r\n# normalization\r\nscores = repeated_evaluation(prep_normalize)\r\nprint('Normalization: %.3f (%.3f)' % (mean(scores), std(scores)))\r\nall_scores.append(scores)\r\n# center\r\nscores = repeated_evaluation(prep_center)\r\nprint('Centered: %.3f (%.3f)' % (mean(scores), std(scores)))\r\nall_scores.append(scores)\r\n# standardize\r\nscores = repeated_evaluation(prep_standardize)\r\nprint('Standardized: %.3f (%.3f)' % (mean(scores), std(scores)))\r\nall_scores.append(scores)\r\n# box and whisker plots of results\r\npyplot.boxplot(all_scores, labels=['norm', 'cent', 'stan'])\r\npyplot.show()<\/pre>\n<p>Tying all of this together, the complete example of running the experiment to compare pixel scaling methods on the MNIST dataset is listed below.<\/p>\n<pre class=\"crayon-plain-tag\"># comparison of training-set based pixel scaling methods on MNIST\r\nfrom numpy import mean\r\nfrom numpy import std\r\nfrom matplotlib import pyplot\r\nfrom keras.datasets import mnist\r\nfrom keras.utils import to_categorical\r\nfrom keras.models import Sequential\r\nfrom keras.layers import Conv2D\r\nfrom keras.layers import MaxPooling2D\r\nfrom keras.layers import Dense\r\nfrom keras.layers import Flatten\r\n\r\n# load train and test dataset\r\ndef load_dataset():\r\n\t# load dataset\r\n\t(trainX, trainY), (testX, testY) = mnist.load_data()\r\n\t# reshape dataset to have a single channel\r\n\twidth, height, channels = trainX.shape[1], trainX.shape[2], 1\r\n\ttrainX = trainX.reshape((trainX.shape[0], width, height, channels))\r\n\ttestX = testX.reshape((testX.shape[0], width, height, channels))\r\n\t# one hot encode target values\r\n\ttrainY = to_categorical(trainY)\r\n\ttestY = to_categorical(testY)\r\n\treturn trainX, trainY, testX, testY\r\n\r\n# define cnn model\r\ndef define_model():\r\n\tmodel = Sequential()\r\n\tmodel.add(Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\r\n\tmodel.add(MaxPooling2D((2, 2)))\r\n\tmodel.add(Conv2D(64, (3, 3), activation='relu'))\r\n\tmodel.add(MaxPooling2D((2, 2)))\r\n\tmodel.add(Flatten())\r\n\tmodel.add(Dense(64, activation='relu'))\r\n\tmodel.add(Dense(10, activation='softmax'))\r\n\t# compile model\r\n\tmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\r\n\treturn model\r\n\r\n# normalize images\r\ndef prep_normalize(train, test):\r\n\t# convert from integers to floats\r\n\ttrain_norm = train.astype('float32')\r\n\ttest_norm = test.astype('float32')\r\n\t# normalize to range 0-1\r\n\ttrain_norm = train_norm \/ 255.0\r\n\ttest_norm = test_norm \/ 255.0\r\n\t# return normalized images\r\n\treturn train_norm, test_norm\r\n\r\n# center images\r\ndef prep_center(train, test):\r\n\t# convert from integers to floats\r\n\ttrain_cent = train.astype('float32')\r\n\ttest_cent = test.astype('float32')\r\n\t# calculate statistics\r\n\tm = train_cent.mean()\r\n\t# center datasets\r\n\ttrain_cent = train_cent - m\r\n\ttest_cent = test_cent - m\r\n\t# return normalized images\r\n\treturn train_cent, test_cent\r\n\r\n# standardize images\r\ndef prep_standardize(train, test):\r\n\t# convert from integers to floats\r\n\ttrain_stan = train.astype('float32')\r\n\ttest_stan = test.astype('float32')\r\n\t# calculate statistics\r\n\tm = train_stan.mean()\r\n\ts = train_stan.std()\r\n\t# center datasets\r\n\ttrain_stan = (train_stan - m) \/ s\r\n\ttest_stan = (test_stan - m) \/ s\r\n\t# return normalized images\r\n\treturn train_stan, test_stan\r\n\r\n# repeated evaluation of model with data prep scheme\r\ndef repeated_evaluation(datapre_func, n_repeats=10):\r\n\t# prepare data\r\n\ttrainX, trainY, testX, testY = load_dataset()\r\n\t# repeated evaluation\r\n\tscores = list()\r\n\tfor i in range(n_repeats):\r\n\t\t# define model\r\n\t\tmodel = define_model()\r\n\t\t# prepare data\r\n\t\tprep_trainX, prep_testX = datapre_func(trainX, testX)\r\n\t\t# fit model\r\n\t\tmodel.fit(prep_trainX, trainY, epochs=5, batch_size=64, verbose=0)\r\n\t\t# evaluate model\r\n\t\t_, acc = model.evaluate(prep_testX, testY, verbose=0)\r\n\t\t# store result\r\n\t\tscores.append(acc)\r\n\t\tprint('> %d: %.3f' % (i, acc * 100.0))\r\n\treturn scores\r\n\r\nall_scores = list()\r\n# normalization\r\nscores = repeated_evaluation(prep_normalize)\r\nprint('Normalization: %.3f (%.3f)' % (mean(scores), std(scores)))\r\nall_scores.append(scores)\r\n# center\r\nscores = repeated_evaluation(prep_center)\r\nprint('Centered: %.3f (%.3f)' % (mean(scores), std(scores)))\r\nall_scores.append(scores)\r\n# standardize\r\nscores = repeated_evaluation(prep_standardize)\r\nprint('Standardized: %.3f (%.3f)' % (mean(scores), std(scores)))\r\nall_scores.append(scores)\r\n# box and whisker plots of results\r\npyplot.boxplot(all_scores, labels=['norm', 'cent', 'stan'])\r\npyplot.show()<\/pre>\n<p>Running the example may take about 30 minutes on the CPU and your results may vary given the stochastic nature of the training algorithm.<\/p>\n<p>The accuracy is reported for each repeated evaluation of the model and the mean and standard deviation of accuracy scores are repeated at the end of each run.<\/p>\n<pre class=\"crayon-plain-tag\">> 0: 98.930\r\n> 1: 98.960\r\n> 2: 98.910\r\n> 3: 99.050\r\n> 4: 99.040\r\n> 5: 98.800\r\n> 6: 98.880\r\n> 7: 99.020\r\n> 8: 99.100\r\n> 9: 99.050\r\nNormalization: 0.990 (0.001)\r\n> 0: 98.570\r\n> 1: 98.530\r\n> 2: 98.230\r\n> 3: 98.110\r\n> 4: 98.840\r\n> 5: 98.720\r\n> 6: 9.800\r\n> 7: 98.170\r\n> 8: 98.710\r\n> 9: 10.320\r\nCentered: 0.808 (0.354)\r\n> 0: 99.150\r\n> 1: 98.820\r\n> 2: 99.000\r\n> 3: 98.850\r\n> 4: 99.140\r\n> 5: 99.050\r\n> 6: 99.120\r\n> 7: 99.100\r\n> 8: 98.940\r\n> 9: 99.110\r\nStandardized: 0.990 (0.001<\/pre>\n<\/p>\n<div id=\"attachment_7367\" style=\"width: 1290px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-7367\" class=\"size-full wp-image-7367\" src=\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/01\/Box-and-Whisker-Plot-of-CNN-Performance-on-MNIST-with-Different-Pixel-Scaling-Methods.png\" alt=\"Box and Whisker Plot of CNN Performance on MNIST With Different Pixel Scaling Methods\" width=\"1280\" height=\"960\" srcset=\"http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2019\/01\/Box-and-Whisker-Plot-of-CNN-Performance-on-MNIST-with-Different-Pixel-Scaling-Methods.png 1280w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2019\/01\/Box-and-Whisker-Plot-of-CNN-Performance-on-MNIST-with-Different-Pixel-Scaling-Methods-300x225.png 300w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2019\/01\/Box-and-Whisker-Plot-of-CNN-Performance-on-MNIST-with-Different-Pixel-Scaling-Methods-768x576.png 768w, http:\/\/3qeqpr26caki16dnhd19sv6by6v.wpengine.netdna-cdn.com\/wp-content\/uploads\/2019\/01\/Box-and-Whisker-Plot-of-CNN-Performance-on-MNIST-with-Different-Pixel-Scaling-Methods-1024x768.png 1024w\" sizes=\"(max-width: 1280px) 100vw, 1280px\"><\/p>\n<p id=\"caption-attachment-7367\" class=\"wp-caption-text\">Box and Whisker Plot of CNN Performance on MNIST With Different Pixel Scaling Methods<\/p>\n<\/div>\n<h2>Step 5. Analyze Results<\/h2>\n<p>For brevity, we will only look at model performance in the comparison of data preparation schemes. An extension to this study would also look at learning rates under each pixel scaling method.<\/p>\n<p>The results of the experiments show that there is little or no difference (at the chosen precision) between pixel normalization and standardization with the chosen model on the MNIST dataset.<\/p>\n<p>From these results, I would use normalization over standardization on this dataset with this model because of the good results and because of the simplicity of normalization as compared to standardization.<\/p>\n<p>These are useful results in that they show that the default heuristic to center pixel values prior to modeling would not be good advice for this dataset.<\/p>\n<p>Sadly, the box and whisker plot does not make a comparison between the spread of accuracy scores easy as some terrible outlier scores for the centering scaling method squash the distributions.<\/p>\n<h2>Extensions<\/h2>\n<p>This section lists some ideas for extending the tutorial that you may wish to explore.<\/p>\n<ul>\n<li><strong>Batch-Wise Scaling<\/strong>. Update the study to calculate scaling statistics per batch instead of across the entire training dataset and see if that makes a difference to the choice of scaling method.<\/li>\n<li><strong>Learning Curves<\/strong>. Update the study to collect a few learning curves for each data scaling method and compare the speed of learning.<\/li>\n<li><strong>CIFAR<\/strong>. Repeat the study on the CIFAR-10 dataset and add pixel scaling methods that support global (scale across all channels) and local (scaling per channel) approaches.<\/li>\n<\/ul>\n<p>If you explore any of these extensions, I\u2019d love to know.<br \/>\nPost your findings in the comments below.<\/p>\n<h2>Further Reading<\/h2>\n<p>This section provides more resources on the topic if you are looking to go deeper.<\/p>\n<ul>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/MNIST_database\">MNIST database, Wikipedia.<\/a><\/li>\n<li><a href=\"http:\/\/rodrigob.github.io\/are_we_there_yet\/build\/classification_datasets_results.html#4d4e495354\">MNIST, Classification datasets results.<\/a><\/li>\n<li><a href=\"https:\/\/keras.io\/datasets\/#mnist-database-of-handwritten-digits\">MNIST database of handwritten digits, Keras API.<\/a><\/li>\n<\/ul>\n<h2>Summary<\/h2>\n<p>In this tutorial, you discovered how to choose a pixel scaling method for image classification with deep learning methods.<\/p>\n<p>Specifically, you learned:<\/p>\n<ul>\n<li>A procedure for choosing a pixel scaling method using experimentation and empirical results on a specific dataset.<\/li>\n<li>How to implement standard pixel scaling methods for preparing image data for modeling.<\/li>\n<li>How to work through a case study for choosing a pixel scaling method for a standard image classification problem.<\/li>\n<\/ul>\n<p>Do you have any questions?<br \/>\nAsk your questions in the comments below and I will do my best to answer.<\/p>\n<p>The post <a rel=\"nofollow\" href=\"https:\/\/machinelearningmastery.com\/how-to-evaluate-pixel-scaling-methods-for-image-classification\/\">How to Evaluate Pixel Scaling Methods for Image Classification With Convolutional Neural Networks<\/a> appeared first on <a rel=\"nofollow\" href=\"https:\/\/machinelearningmastery.com\/\">Machine Learning Mastery<\/a>.<\/p>\n<\/div>\n<p><a href=\"https:\/\/machinelearningmastery.com\/how-to-evaluate-pixel-scaling-methods-for-image-classification\/\">Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Author: Jason Brownlee Image data must be prepared before it can be used as the basis for modeling in image classification tasks. One aspect of [&hellip;] <span class=\"read-more-link\"><a class=\"read-more\" href=\"https:\/\/www.aiproblog.com\/index.php\/2019\/03\/26\/how-to-evaluate-pixel-scaling-methods-for-image-classification-with-convolutional-neural-networks\/\">Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":1932,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[24],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/1931"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=1931"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/1931\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/1932"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=1931"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=1931"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=1931"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}