Generative Adversarial Networks<\/a>, or GANs, are an architecture for training generative models, such as deep convolutional neural networks for generating images.<\/p>\n Developing a GAN for generating images requires both a discriminator convolutional neural network model for classifying whether a given image is real or generated and a generator model that uses inverse convolutional layers to transform an input to a full two-dimensional image of pixel values.<\/p>\n It can be challenging to understand both how GANs work and how deep convolutional neural network models can be trained in a GAN architecture for image generation. A good starting point for beginners is to practice developing and using GANs on standard image datasets used in the field of computer vision, such as the MNIST handwritten digit dataset. Using small and well-understood datasets means that smaller models can be developed and trained quickly, allowing the focus to be put on the model architecture and image generation process itself.<\/p>\n In this tutorial, you will discover how to develop a generative adversarial network with deep convolutional networks for generating handwritten digits.<\/p>\n After completing this tutorial, you will know:<\/p>\n Let\u2019s get started.<\/p>\n How to Develop a Generative Adversarial Network for an MNIST Handwritten Digits From Scratch in Keras This tutorial is divided into seven parts; they are:<\/p>\n The MNIST dataset<\/a> is an acronym that stands for the Modified National Institute of Standards and Technology dataset.<\/p>\n It is a dataset of 70,000 small square 28\u00d728 pixel grayscale images of handwritten single digits between 0 and 9.<\/p>\n The task is to classify a given image of a handwritten digit into one of 10 classes representing integer values from 0 to 9, inclusively.<\/p>\n Keras provides access to the MNIST dataset via the mnist.load_dataset() function<\/a>. It returns two tuples, one with the input and output elements for the standard training dataset, and another with the input and output elements for the standard test dataset.<\/p>\n The example below loads the dataset and summarizes the shape of the loaded dataset.<\/p>\n Note<\/strong>: the first time you load the dataset, Keras will automatically download a compressed version of the images and save them under your home directory in ~\/.keras\/datasets\/<\/em>. The download is fast as the dataset is only about eleven megabytes in its compressed form.<\/p>\n Running the example loads the dataset and prints the shape of the input and output components of the train and test splits of images.<\/p>\n We can see that there are 60K examples in the training set and 10K in the test set and that each image is a square of 28 by 28 pixels.<\/p>\n The images are grayscale with a black background (0 pixel value) and the handwritten digits in white (pixel values near 255). This means if the images were plotted, they would be mostly black with a white digit in the middle.<\/p>\n We can plot some of the images from the training dataset using the matplotlib library using the imshow() function<\/a> and specify the color map via the \u2018cmap<\/em>\u2018 argument as \u2018gray<\/em>\u2018 to show the pixel values correctly.<\/p>\n Alternately, the images are easier to review when we reverse the colors and plot the background as white and the handwritten digits in black.<\/p>\n They are easier to view as most of the image is now white with the area of interest in black. This can be achieved using a reverse grayscale color map, as follows:<\/p>\n The example below plots the first 25 images from the training dataset in a 5 by 5 square.<\/p>\n Running the example creates a plot of 25 images from the MNIST training dataset, arranged in a 5\u00d75 square.<\/p>\n Plot of the First 25 Handwritten Digits From the MNIST Dataset.<\/p>\n<\/div>\n We will use the images in the training dataset as the basis for training a Generative Adversarial Network.<\/p>\n Specifically, the generator model will learn how to generate new plausible handwritten digits between 0 and 9, using a discriminator that will try to distinguish between real images from the MNIST training dataset and new images output by the generator model.<\/p>\n This is a relatively simple problem that does not require a sophisticated generator or discriminator model, although it does require the generation of a grayscale output image.<\/p>\n The first step is to define the discriminator model.<\/p>\n The model must take a sample image from our dataset as input and output a classification prediction as to whether the sample is real or fake.<\/p>\n This is a binary classification problem:<\/p>\n The discriminator model has two convolutional layers<\/a> with 64 filters each, a small kernel size of 3, and larger than normal stride of 2. The model has no pooling layers and a single node in the output layer with the sigmoid activation function to predict whether the input sample is real or fake. The model is trained to minimize the binary cross entropy loss function<\/a>, appropriate for binary classification.<\/p>\n We will use some best practices in defining the discriminator model, such as the use of LeakyReLU instead of ReLU<\/a>, using Dropout<\/a>, and using the Adam version of stochastic gradient descent<\/a> with a learning rate of 0.0002 and a momentum of 0.5.<\/p>\n The function define_discriminator()<\/em> below defines the discriminator model and parametrizes the size of the input image.<\/p>\n We can use this function to define the discriminator model and summarize it.<\/p>\n The complete example is listed below.<\/p>\n Running the example first summarizes the model architecture, showing the input and output from each layer.<\/p>\n We can see that the aggressive 2\u00d72 stride acts to down-sample the input image<\/a>, first from 28\u00d728 to 14\u00d714, then to 7\u00d77, before the model makes an output prediction.<\/p>\n This pattern is by design as we do not use pooling layers and use the large stride as achieve a similar downsampling effect. We will see a similar pattern, but in reverse, in the generator model in the next section.<\/p>\n A plot of the model is also created and we can see that the model expects two inputs and will predict a single output.<\/p>\n Note<\/strong>: creating this plot assumes that the pydot and graphviz libraries are installed. If this is a problem, you can comment out the import statement for the plot_model<\/em> function and the call to the plot_model()<\/em> function.<\/p>\n Plot of the Discriminator Model in the MNIST GAN<\/p>\n<\/div>\n We could start training this model now with real examples with a class label of one, and randomly generated samples with a class label of zero.<\/p>\n The development of these elements will be useful later, and it helps to see that the discriminator is just a normal neural network model for binary classification.<\/p>\n First, we need a function to load and prepare the dataset of real images.<\/p>\n We will use the mnist.load_data()<\/em> function to load the MNIST dataset and just use the input part of the training dataset as the real images.<\/p>\n The images are 2D arrays of pixels and convolutional neural networks expect 3D arrays of images as input, where each image has one or more channels.<\/p>\n We must update the images to have an additional dimension for the grayscale channel. We can do this using the expand_dims() NumPy function<\/a> and specify the final dimension for the channels-last image format.<\/p>\n Finally, we must scale the pixel values<\/a> from the range of unsigned integers in [0,255] to the normalized range of [0,1].<\/p>\n The load_real_samples()<\/em> function below implements this.<\/p>\n The model will be updated in batches, specifically with a collection of real samples and a collection of generated samples. On training, epoch is defined as one pass through the entire training dataset.<\/p>\n We could systematically enumerate all samples in the training dataset, and that is a good approach, but good training via stochastic gradient descent requires that the training dataset be shuffled prior to each epoch. A simpler approach is to select random samples of images from the training dataset.<\/p>\n The generate_real_samples()<\/em> function below will take the training dataset as an argument and will select a random subsample of images; it will also return class labels for the sample, specifically a class label of 1, to indicate real images.<\/p>\n Now, we need a source of fake images.<\/p>\n We don\u2019t have a generator model yet, so instead, we can generate images comprised of random pixel values, specifically random pixel values<\/a> in the range [0,1] like our scaled real images.<\/p>\n The generate_fake_samples()<\/em> function below implements this behavior and generates images of random pixel values and their associated class label of 0, for fake.<\/p>\n Finally, we need to train the discriminator model.<\/p>\n This involves repeatedly retrieving samples of real images and samples of generated images and updating the model for a fixed number of iterations.<\/p>\n We will ignore the idea of epochs for now (e.g. complete passes through the training dataset) and fit the discriminator model for a fixed number of batches. The model will learn to discriminate between real and fake (randomly generated) images rapidly, therefore, not many batches will be required before it learns to discriminate perfectly.<\/p>\n The train_discriminator()<\/em> function implements this, using a batch size of 256 images where 128 are real and 128 are fake each iteration.<\/p>\n We update the discriminator separately for real and fake examples so that we can calculate the accuracy of the model on each sample prior to the update. This gives insight into how the discriminator model is performing over time.<\/p>\n Tying all of this together, the complete example of training an instance of the discriminator model on real and randomly generated (fake) images is listed below.<\/p>\n Running the example first defines the model, loads the MNIST dataset, then trains the discriminator model.<\/p>\n Note<\/strong>: your specific results may vary given the stochastic nature of the learning algorithm. Consider running the example a few times.<\/p>\n In this case, the discriminator model learns to tell the difference between real and randomly generated MNIST images very quickly, in about 50 batches.<\/p>\n Now that we know how to define and train the discriminator model, we need to look at developing the generator model.<\/p>\n The generator model is responsible for creating new, fake but plausible images of handwritten digits.<\/p>\n It does this by taking a point from the latent space as input and outputting a square grayscale image.<\/p>\n The latent space is an arbitrarily defined vector space of Gaussian-distributed values<\/a>, e.g. 100 dimensions. It has no meaning, but by drawing points from this space randomly and providing them to the generator model during training, the generator model will assign meaning to the latent points and, in turn, the latent space, until, at the end of training, the latent vector space represents a compressed representation of the output space, MNIST images, that only the generator knows how to turn into plausible MNIST images.<\/p>\n Note: we don\u2019t have to use a 100 element vector as input; it is a round number and widely used, but I would expect that 10, 50, or 500 would work just as well.<\/p>\n Developing a generator model requires that we transform a vector from the latent space with, 100 dimensions to a 2D array with 28\u00d728 or 784 values.<\/p>\n There are a number of ways to achieve this but there is one approach that has proven effective at deep convolutional generative adversarial networks. It involves two main elements.<\/p>\n The first is a Dense layer as the first hidden layer that has enough nodes to represent a low-resolution version of the output image. Specifically, an image half the size (one quarter the area) of the output image would be 14\u00d714 or 196 nodes, and an image one quarter the size (one eighth the area) would be 7\u00d77 or 49 nodes.<\/p>\n We don\u2019t just want one low-resolution version of the image; we want many parallel versions or interpretations of the input. This is a pattern in convolutional neural networks<\/a> where we have many parallel filters resulting in multiple parallel activation maps, called feature maps, with different interpretations of the input. We want the same thing in reverse: many parallel versions of our output with different learned features that can be collapsed in the output layer into a final image. The model needs space to invent, create, or generate.<\/p>\n Therefore, the first hidden layer, the Dense, needs enough nodes for multiple low-resolution versions of our output image, such as 128.<\/p>\n The activations from these nodes can then be reshaped into something image-like to pass into a convolutional layer, such as 128 different 7\u00d77 feature maps.<\/p>\n The next major architectural innovation involves upsampling the low-resolution image to a higher resolution version of the image.<\/p>\n There are two common ways to do this upsampling process, sometimes called deconvolution.<\/p>\n One way is to use an UpSampling2D<\/em> layer (like a reverse pooling layer) followed by a normal Conv2D<\/em> layer. The other and perhaps more modern way is to combine these two operations into a single layer, called a Conv2DTranspose<\/em>. We will use this latter approach for our generator.<\/p>\n The Conv2DTranspose<\/em> layer can be configured with a stride of (2\u00d72) that will quadruple the area of the input feature maps (double their width and height dimensions). It is also good practice to use a kernel size that is a factor of the stride (e.g. double) to avoid a checkerboard pattern<\/a> that can be observed when upsampling.<\/p>\n This can be repeated to arrive at our 28\u00d728 output image.<\/p>\n Again, we will use the LeakyReLU with a default slope of 0.2, reported as a best practice when training GAN models.<\/p>\n The output layer of the model is a Conv2D<\/em> with one filter and a kernel size of 7\u00d77 and \u2018same\u2019 padding<\/a>, designed to create a single feature map and preserve its dimensions at 28\u00d728 pixels. A sigmoid activation is used to ensure output values are in the desired range of [0,1].<\/p>\n The define_generator()<\/em> function below implements this and defines the generator model.<\/p>\n Note<\/strong>: the generator model is not compiled and does not specify a loss function or optimization algorithm. This is because the generator is not trained directly. We will learn more about this in the next section.<\/p>\n We can summarize the model to help better understand the input and output shapes.<\/p>\n The complete example is listed below.<\/p>\n Running the example summarizes the layers of the model and their output shape.<\/p>\n We can see that, as designed, the first hidden layer has 6,272 parameters or 128 * 7 * 7, the activations of which are reshaped into 128 7\u00d77 feature maps. The feature maps are then upscaled via the two Conv2DTranspose<\/em> layers to the desired output shape of 28\u00d728, until the output layer, where a single activation map is output.<\/p>\n A plot of the model is also created and we can see that the model expects a 100-element point from the latent space as input and will predict a two-element vector as output.<\/p>\n Note<\/strong>: creating this plot assumes that the pydot and graphviz libraries are installed. If this is a problem, you can comment out the import statement for the plot_model function and the call to the plot_model function.<\/p>\n Plot of the Generator Model in the MNIST GAN<\/p>\n<\/div>\n This model cannot do much at the moment.<\/p>\n Nevertheless, we can demonstrate how to use it to generate samples. This is a helpful demonstration to understand the generator as just another model, and some of these elements will be useful later.<\/p>\n The first step is to generate new points in the latent space. We can achieve this by calling the randn() NumPy function<\/a> for generating arrays of random numbers drawn from a standard Gaussian<\/a>.<\/p>\n The array of random numbers can then be reshaped into samples, that is n rows with 100 elements per row. The generate_latent_points()<\/em> function below implements this and generates the desired number of points in the latent space that can be used as input to the generator model.<\/p>\n Next, we can use the generated points as input to the generator model to generate new samples, then plot the samples.<\/p>\n We can update the generate_fake_samples()<\/em> function from the previous section to take the generator model as an argument and use it to generate the desired number of samples by first calling the generate_latent_points()<\/em> function to generate the required number of points in latent space as input to the model.<\/p>\n The updated generate_fake_samples()<\/em> function is listed below and returns both the generated samples and the associated class labels.<\/p>\n We can then plot the generated samples as we did the real MNIST examples in the first section by calling the imshow() function with the reversed grayscale color map.<\/p>\n The complete example of generating new MNIST images with the untrained generator model is listed below.<\/p>\n Running the example generates 25 examples of fake MNIST images and visualizes them on a single plot of 5 by 5 images.<\/p>\n As the model is not trained, the generated images are completely random pixel values in [0, 1].<\/p>\n Example of 25 MNIST Images Output by the Untrained Generator Model<\/p>\n<\/div>\n Now that we know how to define and use the generator model, the next step is to train the model.<\/p>\n The weights in the generator model are updated based on the performance of the discriminator model.<\/p>\n When the discriminator is good at detecting fake samples, the generator is updated more, and when the discriminator model is relatively poor or confused when detecting fake samples, the generator model is updated less.<\/p>\n This defines the zero-sum or adversarial relationship between these two models.<\/p>\n There may be many ways to implement this using the Keras API, but perhaps the simplest approach is to create a new model that combines the generator and discriminator models.<\/p>\n Specifically, a new GAN model can be defined that stacks the generator and discriminator such that the generator receives as input random points in the latent space and generates samples that are fed into the discriminator model directly, classified, and the output of this larger model can be used to update the model weights of the generator.<\/p>\n To be clear, we are not talking about a new third model, just a new logical model that uses the already-defined layers and weights from the standalone generator and discriminator models.<\/p>\n Only the discriminator is concerned with distinguishing between real and fake examples, therefore the discriminator model can be trained in a standalone manner on examples of each, as we did in the section on the discriminator model above.<\/p>\n The generator model is only concerned with the discriminator\u2019s performance on fake examples. Therefore, we will mark all of the layers in the discriminator as not trainable when it is part of the GAN model so that they can not be updated and overtrained on fake examples.<\/p>\n When training the generator via this logical GAN model, there is one more important change. We want the discriminator to think that the samples output by the generator are real, not fake. Therefore, when the generator is trained as part of the GAN model, we will mark the generated samples as real (class 1).<\/p>\n Why would we want to do this?<\/strong><\/em><\/p>\n We can imagine that the discriminator will then classify the generated samples as not real (class 0) or a low probability of being real (0.3 or 0.5). The backpropagation process used to update the model weights will see this as a large error and will update the model weights (i.e. only the weights in the generator) to correct for this error, in turn making the generator better at generating good fake samples.<\/p>\n Let\u2019s make this concrete.<\/p>\n The define_gan()<\/em> function below takes as arguments the already-defined generator and discriminator models and creates the new logical third model subsuming these two models. The weights in the discriminator are marked as not trainable, which only affects the weights as seen by the GAN model and not the standalone discriminator model.<\/p>\n The GAN model then uses the same binary cross entropy loss function as the discriminator and the efficient Adam version of stochastic gradient descent with the learning rate of 0.0002 and momentum 0.5, recommended when training deep convolutional GANs.<\/p>\n Making the discriminator not trainable is a clever trick in the Keras API.<\/p>\n The trainable property impacts the model after it is compiled. The discriminator model was compiled with trainable layers, therefore the model weights in those layers will be updated when the standalone model is updated via calls to the train_on_batch()<\/em> function.<\/p>\n The discriminator model was then marked as not trainable, added to the GAN model, and compiled. In this model, the model weights of the discriminator model are not trainable and cannot be changed when the GAN model is updated via calls to the train_on_batch()<\/em> function. This change in the trainable property does not impact the training of standalone discriminator model.<\/p>\n This behavior is described in the Keras API documentation here:<\/p>\n The complete example of creating the discriminator, generator, and composite model is listed below.<\/p>\n Running the example first creates a summary of the composite model.<\/p>\n We can see that the model expects MNIST images as input and predict a single value as output.<\/p>\n A plot of the model is also created and we can see that the model expects a 100-element point in latent space as input and will predict a single output classification label.<\/p>\n Note<\/strong>: creating this plot assumes that the pydot and graphviz libraries are installed. If this is a problem, you can comment out the import statement for the plot_model<\/em> function and the call to the plot_model()<\/em> function.<\/p>\n Plot of the Composite Generator and Discriminator Model in the MNIST GAN<\/p>\n<\/div>\n Training the composite model involves generating a batch worth of points in the latent space via the generate_latent_points()<\/em> function in the previous section, and class=1<\/em> labels and calling the train_on_batch()<\/em> function.<\/p>\n The train_gan()<\/em> function below demonstrates this, although is pretty simple as only the generator will be updated each epoch, leaving the discriminator with default model weights.<\/p>\n Instead, what is required is that we first update the discriminator model with real and fake samples, then update the generator via the composite model.<\/p>\n This requires combining elements from the train_discriminator()<\/em> function defined in the discriminator section above and the train_gan()<\/em> function defined above. It also requires that we enumerate over both epochs and batches within in an epoch.<\/p>\n The complete train function for updating the discriminator model and the generator (via the composite model) is listed below.<\/p>\n There are a few things to note in this model training function.<\/p>\n First, the number of batches within an epoch is defined by how many times the batch size divides into the training dataset. We have a dataset size of 60K samples, so with rounding down, there are 234 batches per epoch.<\/p>\n The discriminator model is updated once per batch by combining one half a batch of fake and real examples into a single batch via the vstack() NumPy function<\/a>. You could update the discriminator with each half batch separately (recommended for more complex datasets) but combining the samples into a single batch will be faster over a long run, especially when training on GPU hardware.<\/p>\n Finally, we report the loss each batch. It is critical to keep an eye on the loss over batches. The reason for this is that a crash in the discriminator loss indicates that the generator model has started generating rubbish examples that the discriminator can easily discriminate.<\/p>\n Monitor the discriminator loss and expect it to hover around 0.5 to 0.8 per batch on this dataset. The generator loss is less critical and may hover between 0.5 and 2 or higher on this dataset. A clever programmer might even attempt to detect the crashing loss of the discriminator, halt, and then restart the training process.<\/p>\n We almost have everything we need to develop a GAN for the MNIST handwritten digits dataset.<\/p>\n One remaining aspect is the evaluation of the model.<\/p>\n Generally, there are no objective ways to evaluate the performance of a GAN model.<\/p>\n We cannot calculate this objective error score for generated images. It might be possible in the case of MNIST images because the images are so well constrained, but in general, it is not possible (yet).<\/p>\n Instead, images must be subjectively evaluated for quality by a human operator. This means that we cannot know when to stop training without looking at examples of generated images. In turn, the adversarial nature of the training process means that the generator is changing after every batch, meaning that once \u201cgood enough<\/em>\u201d images can be generated, the subjective quality of the images may then begin to vary, improve, or even degrade with subsequent updates.<\/p>\n There are three ways to handle this complex training situation.<\/p>\n All three of these actions can be performed at the same time for a given training epoch, such as every five or 10 training epochs. The result will be a saved generator model for which we have a way of subjectively assessing the quality of its output and objectively knowing how well the discriminator was fooled at the time the model was saved.<\/p>\n Training the GAN over many epochs, such as hundreds or thousands of epochs, will result in many snapshots of the model that can be inspected and from which specific outputs and models can be cherry-picked for later use.<\/p>\n First, we can define a function called summarize_performance()<\/em> function that will summarize the performance of the discriminator model. It does this by retrieving a sample of real MNIST images, as well as generating the same number of fake MNIST images with the generator model, then evaluating the classification accuracy of the discriminator model on each sample and reporting these scores.<\/p>\n This function can be called from the train()<\/em> function based on the current epoch number, such as every 10 epochs.<\/p>\n Next, we can update the summarize_performance()<\/em> function to both save the model and to create and save a plot generated examples.<\/p>\n The generator model can be saved by calling the save()<\/em> function on the generator model and providing a unique filename based on the training epoch number.<\/p>\n We can develop a function to create a plot of the generated samples.<\/p>\n As we are evaluating the discriminator on 100 generated MNIST images, we can plot all 100 images as a 10 by 10 grid. The save_plot()<\/em> function below implements this, again saving the resulting plot with a unique filename based on the epoch number.<\/p>\n The updated summarize_performance()<\/em> function with these additions is listed below.<\/p>\n We now have everything we need to train and evaluate a GAN on the MNIST handwritten digit dataset.<\/p>\n The complete example is listed below.<\/p>\n Note: this example can run on a CPU but may take a number of hours. The example can run on a GPU, such as the Amazon EC2 p3 instances, and will complete in a few minutes.<\/p>\n For help on setting up an AWS EC2 instance to run this code, see the tutorial:<\/p>\n The chosen configuration results in the stable training of both the generative and discriminative model.<\/p>\n The model performance is reported every batch, including the loss of both the discriminative (d<\/em>) and generative (g<\/em>) models.<\/p>\n Note<\/strong>: your specific results may vary given the stochastic nature of the training algorithm. Try running the example a few times.<\/p>\n In this case, the loss remains stable over the course of training.<\/p>\n The generator is evaluated every 20 epochs, resulting in 10 evaluations, 10 plots of generated images, and 10 saved models.<\/p>\n In this case, we can see that the accuracy fluctuates over training. When viewing the discriminator model\u2019s accuracy score in concert with generated images, we can see that the accuracy on fake examples does not correlate well with the subjective quality of images, but the accuracy for real examples may.<\/p>\n It is crude and possibly unreliable metric of GAN performance, along with loss.<\/p>\n More training, beyond some point, does not mean better quality generated images.<\/p>\n In this case, the results after 10 epochs are low quality, although we can see that the generator has learned to generate centered figures in white on a back background (recall we have inverted the grayscale in the plot).<\/p>\n Plot of 100 GAN Generated MNIST Figures After 10 Epochs<\/p>\n<\/div>\n After 20 or 30 more epochs, the model begins to generate very plausible MNIST figures, suggesting that 100 epochs are probably not required for the chosen model configurations.<\/p>\n Plot of 100 GAN Generated MNIST Figures After 40 Epochs<\/p>\n<\/div>\n The generated images after 100 epochs are not greatly different, but I believe I can detect less blocky-ness in the curves.<\/p>\n Plot of 100 GAN Generated MNIST Figures After 100 Epochs<\/p>\n<\/div>\n Once a final generator model is selected, it can be used in a standalone manner for your application.<\/p>\n This involves first loading the model from file, then using it to generate images. The generation of each image requires a point in the latent space as input.<\/p>\n The complete example of loading the saved model and generating images is listed below. In this case, we will use the model saved after 100 training epochs, but the model saved after 40 or 50 epochs would work just as well.<\/p>\n Running the example first loads the model, samples 25 random points in the latent space, generates 25 images, then plots the results as a single image.<\/p>\n We can see that most of the images are plausible, or plausible pieces of handwritten digits.<\/p>\n Example of 25 GAN Generated MNIST Handwritten Images<\/p>\n<\/div>\n The latent space now defines a compressed representation of MNIST handwritten digits.<\/p>\n You can experiment with generating different points in this space and see what types of numbers they generate.<\/p>\n The example below generates a single handwritten digit using a vector of all 0.0 values.<\/p>\n Note<\/strong>: Your specific results may vary given the stochastic nature of the model and the learning algorithm.<\/p>\n In this case, a vector of all zeros results in a handwritten 9 or maybe an 8. You can then try navigating the space and see if you can generate a range of similar, but different handwritten digits.<\/p>\n Example of a GAN Generated MNIST Handwritten Digit for a Vector of Zeros<\/p>\n<\/div>\n This section lists some ideas for extending the tutorial that you may wish to explore.<\/p>\n If you explore any of these extensions, I\u2019d love to know. This section provides more resources on the topic if you are looking to go deeper.<\/p>\n\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/06\/How-to-Develop-a-Generative-Adversarial-Network-for-an-MNIST-Handwritten-Digits-From-Scratch-in-Keras.jpg\")
<\/p>\n
Photo by jcookfisher<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
\n
MNIST Handwritten Digit Dataset<\/h2>\n
# example of loading the mnist dataset\r\nfrom keras.datasets.mnist import load_data\r\n# load the images into memory\r\n(trainX, trainy), (testX, testy) = load_data()\r\n# summarize the shape of the dataset\r\nprint('Train', trainX.shape, trainy.shape)\r\nprint('Test', testX.shape, testy.shape)<\/pre>\nTrain (60000, 28, 28) (60000,)\r\nTest (10000, 28, 28) (10000,)<\/pre>\n
# plot raw pixel data\r\npyplot.imshow(trainX[i], cmap='gray')<\/pre>\n
# plot raw pixel data\r\npyplot.imshow(trainX[i], cmap='gray_r')<\/pre>\n
# example of loading the mnist dataset\r\nfrom keras.datasets.mnist import load_data\r\nfrom matplotlib import pyplot\r\n# load the images into memory\r\n(trainX, trainy), (testX, testy) = load_data()\r\n# plot images from the training dataset\r\nfor i in range(25):\r\n\t# define subplot\r\n\tpyplot.subplot(5, 5, 1 + i)\r\n\t# turn off axis\r\n\tpyplot.axis('off')\r\n\t# plot raw pixel data\r\n\tpyplot.imshow(trainX[i], cmap='gray_r')\r\npyplot.show()<\/pre>\n![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/04\/Plot-of-the-First-25-Handwritten-Digits-From-the-MNIST-Dataset-1024x768.png\")
<\/p>\nHow to Define and Train the Discriminator Model<\/h2>\n
\n
# define the standalone discriminator model\r\ndef define_discriminator(in_shape=(28,28,1)):\r\n\tmodel = Sequential()\r\n\tmodel.add(Conv2D(64, (3,3), strides=(2, 2), padding='same', input_shape=in_shape))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Dropout(0.4))\r\n\tmodel.add(Conv2D(64, (3,3), strides=(2, 2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Dropout(0.4))\r\n\tmodel.add(Flatten())\r\n\tmodel.add(Dense(1, activation='sigmoid'))\r\n\t# compile model\r\n\topt = Adam(lr=0.0002, beta_1=0.5)\r\n\tmodel.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])\r\n\treturn model<\/pre>\n
# example of defining the discriminator model\r\nfrom keras.models import Sequential\r\nfrom keras.optimizers import Adam\r\nfrom keras.layers import Dense\r\nfrom keras.layers import Conv2D\r\nfrom keras.layers import Flatten\r\nfrom keras.layers import Dropout\r\nfrom keras.layers import LeakyReLU\r\nfrom keras.utils.vis_utils import plot_model\r\n\r\n# define the standalone discriminator model\r\ndef define_discriminator(in_shape=(28,28,1)):\r\n\tmodel = Sequential()\r\n\tmodel.add(Conv2D(64, (3,3), strides=(2, 2), padding='same', input_shape=in_shape))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Dropout(0.4))\r\n\tmodel.add(Conv2D(64, (3,3), strides=(2, 2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Dropout(0.4))\r\n\tmodel.add(Flatten())\r\n\tmodel.add(Dense(1, activation='sigmoid'))\r\n\t# compile model\r\n\topt = Adam(lr=0.0002, beta_1=0.5)\r\n\tmodel.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])\r\n\treturn model\r\n\r\n# define model\r\nmodel = define_discriminator()\r\n# summarize the model\r\nmodel.summary()\r\n# plot the model\r\nplot_model(model, to_file='discriminator_plot.png', show_shapes=True, show_layer_names=True)<\/pre>\n
_________________________________________________________________\r\nLayer (type) Output Shape Param #\r\n=================================================================\r\nconv2d_1 (Conv2D) (None, 14, 14, 64) 640\r\n_________________________________________________________________\r\nleaky_re_lu_1 (LeakyReLU) (None, 14, 14, 64) 0\r\n_________________________________________________________________\r\ndropout_1 (Dropout) (None, 14, 14, 64) 0\r\n_________________________________________________________________\r\nconv2d_2 (Conv2D) (None, 7, 7, 64) 36928\r\n_________________________________________________________________\r\nleaky_re_lu_2 (LeakyReLU) (None, 7, 7, 64) 0\r\n_________________________________________________________________\r\ndropout_2 (Dropout) (None, 7, 7, 64) 0\r\n_________________________________________________________________\r\nflatten_1 (Flatten) (None, 3136) 0\r\n_________________________________________________________________\r\ndense_1 (Dense) (None, 1) 3137\r\n=================================================================\r\nTotal params: 40,705\r\nTrainable params: 40,705\r\nNon-trainable params: 0\r\n_________________________________________________________________<\/pre>\n
![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/04\/Plot-of-the-Discriminator-Model-in-the-MNIST-GAN.png\")
<\/p>\n# load mnist dataset\r\n(trainX, _), (_, _) = load_data()<\/pre>\n
# expand to 3d, e.g. add channels dimension\r\nX = expand_dims(trainX, axis=-1)<\/pre>\n
# convert from unsigned ints to floats\r\nX = X.astype('float32')\r\n# scale from [0,255] to [0,1]\r\nX = X \/ 255.0<\/pre>\n# load and prepare mnist training images\r\ndef load_real_samples():\r\n\t# load mnist dataset\r\n\t(trainX, _), (_, _) = load_data()\r\n\t# expand to 3d, e.g. add channels dimension\r\n\tX = expand_dims(trainX, axis=-1)\r\n\t# convert from unsigned ints to floats\r\n\tX = X.astype('float32')\r\n\t# scale from [0,255] to [0,1]\r\n\tX = X \/ 255.0\r\n\treturn X<\/pre>\n# select real samples\r\ndef generate_real_samples(dataset, n_samples):\r\n\t# choose random instances\r\n\tix = randint(0, dataset.shape[0], n_samples)\r\n\t# retrieve selected images\r\n\tX = dataset[ix]\r\n\t# generate 'real' class labels (1)\r\n\ty = ones((n_samples, 1))\r\n\treturn X, y<\/pre>\n
# generate n fake samples with class labels\r\ndef generate_fake_samples(n_samples):\r\n\t# generate uniform random numbers in [0,1]\r\n\tX = rand(28 * 28 * n_samples)\r\n\t# reshape into a batch of grayscale images\r\n\tX = X.reshape((n_samples, 28, 28, 1))\r\n\t# generate 'fake' class labels (0)\r\n\ty = zeros((n_samples, 1))\r\n\treturn X, y<\/pre>\n
# train the discriminator model\r\ndef train_discriminator(model, dataset, n_iter=100, n_batch=256):\r\n\thalf_batch = int(n_batch \/ 2)\r\n\t# manually enumerate epochs\r\n\tfor i in range(n_iter):\r\n\t\t# get randomly selected 'real' samples\r\n\t\tX_real, y_real = generate_real_samples(dataset, half_batch)\r\n\t\t# update discriminator on real samples\r\n\t\t_, real_acc = model.train_on_batch(X_real, y_real)\r\n\t\t# generate 'fake' examples\r\n\t\tX_fake, y_fake = generate_fake_samples(half_batch)\r\n\t\t# update discriminator on fake samples\r\n\t\t_, fake_acc = model.train_on_batch(X_fake, y_fake)\r\n\t\t# summarize performance\r\n\t\tprint('>%d real=%.0f%% fake=%.0f%%' % (i+1, real_acc*100, fake_acc*100))<\/pre>\n# example of training the discriminator model on real and random mnist images\r\nfrom numpy import expand_dims\r\nfrom numpy import ones\r\nfrom numpy import zeros\r\nfrom numpy.random import rand\r\nfrom numpy.random import randint\r\nfrom keras.datasets.mnist import load_data\r\nfrom keras.optimizers import Adam\r\nfrom keras.models import Sequential\r\nfrom keras.layers import Dense\r\nfrom keras.layers import Conv2D\r\nfrom keras.layers import Flatten\r\nfrom keras.layers import Dropout\r\nfrom keras.layers import LeakyReLU\r\n\r\n# define the standalone discriminator model\r\ndef define_discriminator(in_shape=(28,28,1)):\r\n\tmodel = Sequential()\r\n\tmodel.add(Conv2D(64, (3,3), strides=(2, 2), padding='same', input_shape=in_shape))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Dropout(0.4))\r\n\tmodel.add(Conv2D(64, (3,3), strides=(2, 2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Dropout(0.4))\r\n\tmodel.add(Flatten())\r\n\tmodel.add(Dense(1, activation='sigmoid'))\r\n\t# compile model\r\n\topt = Adam(lr=0.0002, beta_1=0.5)\r\n\tmodel.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])\r\n\treturn model\r\n\r\n# load and prepare mnist training images\r\ndef load_real_samples():\r\n\t# load mnist dataset\r\n\t(trainX, _), (_, _) = load_data()\r\n\t# expand to 3d, e.g. add channels dimension\r\n\tX = expand_dims(trainX, axis=-1)\r\n\t# convert from unsigned ints to floats\r\n\tX = X.astype('float32')\r\n\t# scale from [0,255] to [0,1]\r\n\tX = X \/ 255.0\r\n\treturn X\r\n\r\n# select real samples\r\ndef generate_real_samples(dataset, n_samples):\r\n\t# choose random instances\r\n\tix = randint(0, dataset.shape[0], n_samples)\r\n\t# retrieve selected images\r\n\tX = dataset[ix]\r\n\t# generate 'real' class labels (1)\r\n\ty = ones((n_samples, 1))\r\n\treturn X, y\r\n\r\n# generate n fake samples with class labels\r\ndef generate_fake_samples(n_samples):\r\n\t# generate uniform random numbers in [0,1]\r\n\tX = rand(28 * 28 * n_samples)\r\n\t# reshape into a batch of grayscale images\r\n\tX = X.reshape((n_samples, 28, 28, 1))\r\n\t# generate 'fake' class labels (0)\r\n\ty = zeros((n_samples, 1))\r\n\treturn X, y\r\n\r\n# train the discriminator model\r\ndef train_discriminator(model, dataset, n_iter=100, n_batch=256):\r\n\thalf_batch = int(n_batch \/ 2)\r\n\t# manually enumerate epochs\r\n\tfor i in range(n_iter):\r\n\t\t# get randomly selected 'real' samples\r\n\t\tX_real, y_real = generate_real_samples(dataset, half_batch)\r\n\t\t# update discriminator on real samples\r\n\t\t_, real_acc = model.train_on_batch(X_real, y_real)\r\n\t\t# generate 'fake' examples\r\n\t\tX_fake, y_fake = generate_fake_samples(half_batch)\r\n\t\t# update discriminator on fake samples\r\n\t\t_, fake_acc = model.train_on_batch(X_fake, y_fake)\r\n\t\t# summarize performance\r\n\t\tprint('>%d real=%.0f%% fake=%.0f%%' % (i+1, real_acc*100, fake_acc*100))\r\n\r\n# define the discriminator model\r\nmodel = define_discriminator()\r\n# load image data\r\ndataset = load_real_samples()\r\n# fit the model\r\ntrain_discriminator(model, dataset)<\/pre>\n...\r\n>96 real=100% fake=100%\r\n>97 real=100% fake=100%\r\n>98 real=100% fake=100%\r\n>99 real=100% fake=100%\r\n>100 real=100% fake=100%<\/pre>\n
How to Define and Use the Generator Model<\/h2>\n
\n
# foundation for 7x7 image\r\nmodel.add(Dense(128 * 7 * 7, input_dim=100))<\/pre>\n
model.add(Reshape((7, 7, 128)))<\/pre>\n
# upsample to 14x14\r\nmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))<\/pre>\n
# define the standalone generator model\r\ndef define_generator(latent_dim):\r\n\tmodel = Sequential()\r\n\t# foundation for 7x7 image\r\n\tn_nodes = 128 * 7 * 7\r\n\tmodel.add(Dense(n_nodes, input_dim=latent_dim))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Reshape((7, 7, 128)))\r\n\t# upsample to 14x14\r\n\tmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\t# upsample to 28x28\r\n\tmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Conv2D(1, (7,7), activation='sigmoid', padding='same'))\r\n\treturn model<\/pre>\n
# example of defining the generator model\r\nfrom keras.models import Sequential\r\nfrom keras.layers import Dense\r\nfrom keras.layers import Reshape\r\nfrom keras.layers import Conv2D\r\nfrom keras.layers import Conv2DTranspose\r\nfrom keras.layers import LeakyReLU\r\nfrom keras.utils.vis_utils import plot_model\r\n\r\n# define the standalone generator model\r\ndef define_generator(latent_dim):\r\n\tmodel = Sequential()\r\n\t# foundation for 7x7 image\r\n\tn_nodes = 128 * 7 * 7\r\n\tmodel.add(Dense(n_nodes, input_dim=latent_dim))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Reshape((7, 7, 128)))\r\n\t# upsample to 14x14\r\n\tmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\t# upsample to 28x28\r\n\tmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Conv2D(1, (7,7), activation='sigmoid', padding='same'))\r\n\treturn model\r\n\r\n# define the size of the latent space\r\nlatent_dim = 100\r\n# define the generator model\r\nmodel = define_generator(latent_dim)\r\n# summarize the model\r\nmodel.summary()\r\n# plot the model\r\nplot_model(model, to_file='generator_plot.png', show_shapes=True, show_layer_names=True)<\/pre>\n
_________________________________________________________________\r\nLayer (type) Output Shape Param #\r\n=================================================================\r\ndense_1 (Dense) (None, 6272) 633472\r\n_________________________________________________________________\r\nleaky_re_lu_1 (LeakyReLU) (None, 6272) 0\r\n_________________________________________________________________\r\nreshape_1 (Reshape) (None, 7, 7, 128) 0\r\n_________________________________________________________________\r\nconv2d_transpose_1 (Conv2DTr (None, 14, 14, 128) 262272\r\n_________________________________________________________________\r\nleaky_re_lu_2 (LeakyReLU) (None, 14, 14, 128) 0\r\n_________________________________________________________________\r\nconv2d_transpose_2 (Conv2DTr (None, 28, 28, 128) 262272\r\n_________________________________________________________________\r\nleaky_re_lu_3 (LeakyReLU) (None, 28, 28, 128) 0\r\n_________________________________________________________________\r\nconv2d_1 (Conv2D) (None, 28, 28, 1) 6273\r\n=================================================================\r\nTotal params: 1,164,289\r\nTrainable params: 1,164,289\r\nNon-trainable params: 0\r\n_________________________________________________________________<\/pre>\n
![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/04\/Plot-of-the-Generator-Model-in-the-MNIST-GAN.png\")
<\/p>\n# generate points in latent space as input for the generator\r\ndef generate_latent_points(latent_dim, n_samples):\r\n\t# generate points in the latent space\r\n\tx_input = randn(latent_dim * n_samples)\r\n\t# reshape into a batch of inputs for the network\r\n\tx_input = x_input.reshape(n_samples, latent_dim)\r\n\treturn x_input<\/pre>\n
# use the generator to generate n fake examples, with class labels\r\ndef generate_fake_samples(g_model, latent_dim, n_samples):\r\n\t# generate points in latent space\r\n\tx_input = generate_latent_points(latent_dim, n_samples)\r\n\t# predict outputs\r\n\tX = g_model.predict(x_input)\r\n\t# create 'fake' class labels (0)\r\n\ty = zeros((n_samples, 1))\r\n\treturn X, y<\/pre>\n
# example of defining and using the generator model\r\nfrom numpy import zeros\r\nfrom numpy.random import randn\r\nfrom keras.models import Sequential\r\nfrom keras.layers import Dense\r\nfrom keras.layers import Reshape\r\nfrom keras.layers import Conv2D\r\nfrom keras.layers import Conv2DTranspose\r\nfrom keras.layers import LeakyReLU\r\nfrom matplotlib import pyplot\r\n\r\n# define the standalone generator model\r\ndef define_generator(latent_dim):\r\n\tmodel = Sequential()\r\n\t# foundation for 7x7 image\r\n\tn_nodes = 128 * 7 * 7\r\n\tmodel.add(Dense(n_nodes, input_dim=latent_dim))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Reshape((7, 7, 128)))\r\n\t# upsample to 14x14\r\n\tmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\t# upsample to 28x28\r\n\tmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Conv2D(1, (7,7), activation='sigmoid', padding='same'))\r\n\treturn model\r\n\r\n# generate points in latent space as input for the generator\r\ndef generate_latent_points(latent_dim, n_samples):\r\n\t# generate points in the latent space\r\n\tx_input = randn(latent_dim * n_samples)\r\n\t# reshape into a batch of inputs for the network\r\n\tx_input = x_input.reshape(n_samples, latent_dim)\r\n\treturn x_input\r\n\r\n# use the generator to generate n fake examples, with class labels\r\ndef generate_fake_samples(g_model, latent_dim, n_samples):\r\n\t# generate points in latent space\r\n\tx_input = generate_latent_points(latent_dim, n_samples)\r\n\t# predict outputs\r\n\tX = g_model.predict(x_input)\r\n\t# create 'fake' class labels (0)\r\n\ty = zeros((n_samples, 1))\r\n\treturn X, y\r\n\r\n# size of the latent space\r\nlatent_dim = 100\r\n# define the discriminator model\r\nmodel = define_generator(latent_dim)\r\n# generate samples\r\nn_samples = 25\r\nX, _ = generate_fake_samples(model, latent_dim, n_samples)\r\n# plot the generated samples\r\nfor i in range(n_samples):\r\n\t# define subplot\r\n\tpyplot.subplot(5, 5, 1 + i)\r\n\t# turn off axis labels\r\n\tpyplot.axis('off')\r\n\t# plot single image\r\n\tpyplot.imshow(X[i, :, :, 0], cmap='gray_r')\r\n# show the figure\r\npyplot.show()<\/pre>\n![\"Example](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/04\/Example-of-25-MNIST-Images-Output-by-the-Untrained-Generator-Model-1024x768.png\")
<\/p>\nHow to Train the Generator Model<\/h2>\n
\n
# define the combined generator and discriminator model, for updating the generator\r\ndef define_gan(g_model, d_model):\r\n\t# make weights in the discriminator not trainable\r\n\td_model.trainable = False\r\n\t# connect them\r\n\tmodel = Sequential()\r\n\t# add generator\r\n\tmodel.add(g_model)\r\n\t# add the discriminator\r\n\tmodel.add(d_model)\r\n\t# compile model\r\n\topt = Adam(lr=0.0002, beta_1=0.5)\r\n\tmodel.compile(loss='binary_crossentropy', optimizer=opt)\r\n\treturn model<\/pre>\n
\n
# demonstrate creating the three models in the gan\r\nfrom keras.optimizers import Adam\r\nfrom keras.models import Sequential\r\nfrom keras.layers import Dense\r\nfrom keras.layers import Reshape\r\nfrom keras.layers import Flatten\r\nfrom keras.layers import Conv2D\r\nfrom keras.layers import Conv2DTranspose\r\nfrom keras.layers import LeakyReLU\r\nfrom keras.layers import Dropout\r\nfrom keras.utils.vis_utils import plot_model\r\n\r\n# define the standalone discriminator model\r\ndef define_discriminator(in_shape=(28,28,1)):\r\n\tmodel = Sequential()\r\n\tmodel.add(Conv2D(64, (3,3), strides=(2, 2), padding='same', input_shape=in_shape))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Dropout(0.4))\r\n\tmodel.add(Conv2D(64, (3,3), strides=(2, 2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Dropout(0.4))\r\n\tmodel.add(Flatten())\r\n\tmodel.add(Dense(1, activation='sigmoid'))\r\n\t# compile model\r\n\topt = Adam(lr=0.0002, beta_1=0.5)\r\n\tmodel.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])\r\n\treturn model\r\n\r\n# define the standalone generator model\r\ndef define_generator(latent_dim):\r\n\tmodel = Sequential()\r\n\t# foundation for 7x7 image\r\n\tn_nodes = 128 * 7 * 7\r\n\tmodel.add(Dense(n_nodes, input_dim=latent_dim))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Reshape((7, 7, 128)))\r\n\t# upsample to 14x14\r\n\tmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\t# upsample to 28x28\r\n\tmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Conv2D(1, (7,7), activation='sigmoid', padding='same'))\r\n\treturn model\r\n\r\n# define the combined generator and discriminator model, for updating the generator\r\ndef define_gan(g_model, d_model):\r\n\t# make weights in the discriminator not trainable\r\n\td_model.trainable = False\r\n\t# connect them\r\n\tmodel = Sequential()\r\n\t# add generator\r\n\tmodel.add(g_model)\r\n\t# add the discriminator\r\n\tmodel.add(d_model)\r\n\t# compile model\r\n\topt = Adam(lr=0.0002, beta_1=0.5)\r\n\tmodel.compile(loss='binary_crossentropy', optimizer=opt)\r\n\treturn model\r\n\r\n# size of the latent space\r\nlatent_dim = 100\r\n# create the discriminator\r\nd_model = define_discriminator()\r\n# create the generator\r\ng_model = define_generator(latent_dim)\r\n# create the gan\r\ngan_model = define_gan(g_model, d_model)\r\n# summarize gan model\r\ngan_model.summary()\r\n# plot gan model\r\nplot_model(gan_model, to_file='gan_plot.png', show_shapes=True, show_layer_names=True)<\/pre>\n
_________________________________________________________________\r\nLayer (type) Output Shape Param #\r\n=================================================================\r\nsequential_2 (Sequential) (None, 28, 28, 1) 1164289\r\n_________________________________________________________________\r\nsequential_1 (Sequential) (None, 1) 40705\r\n=================================================================\r\nTotal params: 1,204,994\r\nTrainable params: 1,164,289\r\nNon-trainable params: 40,705\r\n_________________________________________________________________<\/pre>\n
![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/04\/Plot-of-the-Composite-Generator-and-Discriminator-Model-in-the-MNIST-GAN.png\")
<\/p>\n# train the composite model\r\ndef train_gan(gan_model, latent_dim, n_epochs=100, n_batch=256):\r\n\t# manually enumerate epochs\r\n\tfor i in range(n_epochs):\r\n\t\t# prepare points in latent space as input for the generator\r\n\t\tx_gan = generate_latent_points(latent_dim, n_batch)\r\n\t\t# create inverted labels for the fake samples\r\n\t\ty_gan = ones((n_batch, 1))\r\n\t\t# update the generator via the discriminator's error\r\n\t\tgan_model.train_on_batch(x_gan, y_gan)<\/pre>\n
# train the generator and discriminator\r\ndef train(g_model, d_model, gan_model, dataset, latent_dim, n_epochs=100, n_batch=256):\r\n\tbat_per_epo = int(dataset.shape[0] \/ n_batch)\r\n\thalf_batch = int(n_batch \/ 2)\r\n\t# manually enumerate epochs\r\n\tfor i in range(n_epochs):\r\n\t\t# enumerate batches over the training set\r\n\t\tfor j in range(bat_per_epo):\r\n\t\t\t# get randomly selected 'real' samples\r\n\t\t\tX_real, y_real = generate_real_samples(dataset, half_batch)\r\n\t\t\t# generate 'fake' examples\r\n\t\t\tX_fake, y_fake = generate_fake_samples(g_model, latent_dim, half_batch)\r\n\t\t\t# create training set for the discriminator\r\n\t\t\tX, y = vstack((X_real, X_fake)), vstack((y_real, y_fake))\r\n\t\t\t# update discriminator model weights\r\n\t\t\td_loss, _ = d_model.train_on_batch(X, y)\r\n\t\t\t# prepare points in latent space as input for the generator\r\n\t\t\tX_gan = generate_latent_points(latent_dim, n_batch)\r\n\t\t\t# create inverted labels for the fake samples\r\n\t\t\ty_gan = ones((n_batch, 1))\r\n\t\t\t# update the generator via the discriminator's error\r\n\t\t\tg_loss = gan_model.train_on_batch(X_gan, y_gan)\r\n\t\t\t# summarize loss on this batch\r\n\t\t\tprint('>%d, %d\/%d, d=%.3f, g=%.3f' % (i+1, j+1, bat_per_epo, d_loss, g_loss))<\/pre>\nHow to Evaluate GAN Model Performance<\/h2>\n
\n
# evaluate the discriminator, plot generated images, save generator model\r\ndef summarize_performance(epoch, g_model, d_model, dataset, latent_dim, n_samples=100):\r\n\t# prepare real samples\r\n\tX_real, y_real = generate_real_samples(dataset, n_samples)\r\n\t# evaluate discriminator on real examples\r\n\t_, acc_real = d_model.evaluate(X_real, y_real, verbose=0)\r\n\t# prepare fake examples\r\n\tx_fake, y_fake = generate_fake_samples(g_model, latent_dim, n_samples)\r\n\t# evaluate discriminator on fake examples\r\n\t_, acc_fake = d_model.evaluate(x_fake, y_fake, verbose=0)\r\n\t# summarize discriminator performance\r\n\tprint('>Accuracy real: %.0f%%, fake: %.0f%%' % (acc_real*100, acc_fake*100))<\/pre>\n# train the generator and discriminator\r\ndef train(g_model, d_model, gan_model, dataset, latent_dim, n_epochs=100, n_batch=256):\r\n\tbat_per_epo = int(dataset.shape[0] \/ n_batch)\r\n\thalf_batch = int(n_batch \/ 2)\r\n\t# manually enumerate epochs\r\n\tfor i in range(n_epochs):\r\n\t...\r\n\t# evaluate the model performance, sometimes\r\n\tif (i+1) % 10 == 0:\r\n\t\tsummarize_performance(i, g_model, d_model, dataset, latent_dim)<\/pre>\n
...\r\n# save the generator model tile file\r\nfilename = 'generator_model_%03d.h5' % (epoch + 1)\r\ng_model.save(filename)<\/pre>\n
# create and save a plot of generated images (reversed grayscale)\r\ndef save_plot(examples, epoch, n=10):\r\n\t# plot images\r\n\tfor i in range(n * n):\r\n\t\t# define subplot\r\n\t\tpyplot.subplot(n, n, 1 + i)\r\n\t\t# turn off axis\r\n\t\tpyplot.axis('off')\r\n\t\t# plot raw pixel data\r\n\t\tpyplot.imshow(examples[i, :, :, 0], cmap='gray_r')\r\n\t# save plot to file\r\n\tfilename = 'generated_plot_e%03d.png' % (epoch+1)\r\n\tpyplot.savefig(filename)\r\n\tpyplot.close()<\/pre>\n# evaluate the discriminator, plot generated images, save generator model\r\ndef summarize_performance(epoch, g_model, d_model, dataset, latent_dim, n_samples=100):\r\n\t# prepare real samples\r\n\tX_real, y_real = generate_real_samples(dataset, n_samples)\r\n\t# evaluate discriminator on real examples\r\n\t_, acc_real = d_model.evaluate(X_real, y_real, verbose=0)\r\n\t# prepare fake examples\r\n\tx_fake, y_fake = generate_fake_samples(g_model, latent_dim, n_samples)\r\n\t# evaluate discriminator on fake examples\r\n\t_, acc_fake = d_model.evaluate(x_fake, y_fake, verbose=0)\r\n\t# summarize discriminator performance\r\n\tprint('>Accuracy real: %.0f%%, fake: %.0f%%' % (acc_real*100, acc_fake*100))\r\n\t# save plot\r\n\tsave_plot(x_fake, epoch)\r\n\t# save the generator model tile file\r\n\tfilename = 'generator_model_%03d.h5' % (epoch + 1)\r\n\tg_model.save(filename)<\/pre>\n<\/p>\nComplete Example of GAN for MNIST<\/h2>\n
\n
# example of training a gan on mnist\r\nfrom numpy import expand_dims\r\nfrom numpy import zeros\r\nfrom numpy import ones\r\nfrom numpy import vstack\r\nfrom numpy.random import randn\r\nfrom numpy.random import randint\r\nfrom keras.datasets.mnist import load_data\r\nfrom keras.optimizers import Adam\r\nfrom keras.models import Sequential\r\nfrom keras.layers import Dense\r\nfrom keras.layers import Reshape\r\nfrom keras.layers import Flatten\r\nfrom keras.layers import Conv2D\r\nfrom keras.layers import Conv2DTranspose\r\nfrom keras.layers import LeakyReLU\r\nfrom keras.layers import Dropout\r\nfrom matplotlib import pyplot\r\n\r\n# define the standalone discriminator model\r\ndef define_discriminator(in_shape=(28,28,1)):\r\n\tmodel = Sequential()\r\n\tmodel.add(Conv2D(64, (3,3), strides=(2, 2), padding='same', input_shape=in_shape))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Dropout(0.4))\r\n\tmodel.add(Conv2D(64, (3,3), strides=(2, 2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Dropout(0.4))\r\n\tmodel.add(Flatten())\r\n\tmodel.add(Dense(1, activation='sigmoid'))\r\n\t# compile model\r\n\topt = Adam(lr=0.0002, beta_1=0.5)\r\n\tmodel.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])\r\n\treturn model\r\n\r\n# define the standalone generator model\r\ndef define_generator(latent_dim):\r\n\tmodel = Sequential()\r\n\t# foundation for 7x7 image\r\n\tn_nodes = 128 * 7 * 7\r\n\tmodel.add(Dense(n_nodes, input_dim=latent_dim))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Reshape((7, 7, 128)))\r\n\t# upsample to 14x14\r\n\tmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\t# upsample to 28x28\r\n\tmodel.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))\r\n\tmodel.add(LeakyReLU(alpha=0.2))\r\n\tmodel.add(Conv2D(1, (7,7), activation='sigmoid', padding='same'))\r\n\treturn model\r\n\r\n# define the combined generator and discriminator model, for updating the generator\r\ndef define_gan(g_model, d_model):\r\n\t# make weights in the discriminator not trainable\r\n\td_model.trainable = False\r\n\t# connect them\r\n\tmodel = Sequential()\r\n\t# add generator\r\n\tmodel.add(g_model)\r\n\t# add the discriminator\r\n\tmodel.add(d_model)\r\n\t# compile model\r\n\topt = Adam(lr=0.0002, beta_1=0.5)\r\n\tmodel.compile(loss='binary_crossentropy', optimizer=opt)\r\n\treturn model\r\n\r\n# load and prepare mnist training images\r\ndef load_real_samples():\r\n\t# load mnist dataset\r\n\t(trainX, _), (_, _) = load_data()\r\n\t# expand to 3d, e.g. add channels dimension\r\n\tX = expand_dims(trainX, axis=-1)\r\n\t# convert from unsigned ints to floats\r\n\tX = X.astype('float32')\r\n\t# scale from [0,255] to [0,1]\r\n\tX = X \/ 255.0\r\n\treturn X\r\n\r\n# select real samples\r\ndef generate_real_samples(dataset, n_samples):\r\n\t# choose random instances\r\n\tix = randint(0, dataset.shape[0], n_samples)\r\n\t# retrieve selected images\r\n\tX = dataset[ix]\r\n\t# generate 'real' class labels (1)\r\n\ty = ones((n_samples, 1))\r\n\treturn X, y\r\n\r\n# generate points in latent space as input for the generator\r\ndef generate_latent_points(latent_dim, n_samples):\r\n\t# generate points in the latent space\r\n\tx_input = randn(latent_dim * n_samples)\r\n\t# reshape into a batch of inputs for the network\r\n\tx_input = x_input.reshape(n_samples, latent_dim)\r\n\treturn x_input\r\n\r\n# use the generator to generate n fake examples, with class labels\r\ndef generate_fake_samples(g_model, latent_dim, n_samples):\r\n\t# generate points in latent space\r\n\tx_input = generate_latent_points(latent_dim, n_samples)\r\n\t# predict outputs\r\n\tX = g_model.predict(x_input)\r\n\t# create 'fake' class labels (0)\r\n\ty = zeros((n_samples, 1))\r\n\treturn X, y\r\n\r\n# create and save a plot of generated images (reversed grayscale)\r\ndef save_plot(examples, epoch, n=10):\r\n\t# plot images\r\n\tfor i in range(n * n):\r\n\t\t# define subplot\r\n\t\tpyplot.subplot(n, n, 1 + i)\r\n\t\t# turn off axis\r\n\t\tpyplot.axis('off')\r\n\t\t# plot raw pixel data\r\n\t\tpyplot.imshow(examples[i, :, :, 0], cmap='gray_r')\r\n\t# save plot to file\r\n\tfilename = 'generated_plot_e%03d.png' % (epoch+1)\r\n\tpyplot.savefig(filename)\r\n\tpyplot.close()\r\n\r\n# evaluate the discriminator, plot generated images, save generator model\r\ndef summarize_performance(epoch, g_model, d_model, dataset, latent_dim, n_samples=100):\r\n\t# prepare real samples\r\n\tX_real, y_real = generate_real_samples(dataset, n_samples)\r\n\t# evaluate discriminator on real examples\r\n\t_, acc_real = d_model.evaluate(X_real, y_real, verbose=0)\r\n\t# prepare fake examples\r\n\tx_fake, y_fake = generate_fake_samples(g_model, latent_dim, n_samples)\r\n\t# evaluate discriminator on fake examples\r\n\t_, acc_fake = d_model.evaluate(x_fake, y_fake, verbose=0)\r\n\t# summarize discriminator performance\r\n\tprint('>Accuracy real: %.0f%%, fake: %.0f%%' % (acc_real*100, acc_fake*100))\r\n\t# save plot\r\n\tsave_plot(x_fake, epoch)\r\n\t# save the generator model tile file\r\n\tfilename = 'generator_model_%03d.h5' % (epoch + 1)\r\n\tg_model.save(filename)\r\n\r\n# train the generator and discriminator\r\ndef train(g_model, d_model, gan_model, dataset, latent_dim, n_epochs=100, n_batch=256):\r\n\tbat_per_epo = int(dataset.shape[0] \/ n_batch)\r\n\thalf_batch = int(n_batch \/ 2)\r\n\t# manually enumerate epochs\r\n\tfor i in range(n_epochs):\r\n\t\t# enumerate batches over the training set\r\n\t\tfor j in range(bat_per_epo):\r\n\t\t\t# get randomly selected 'real' samples\r\n\t\t\tX_real, y_real = generate_real_samples(dataset, half_batch)\r\n\t\t\t# generate 'fake' examples\r\n\t\t\tX_fake, y_fake = generate_fake_samples(g_model, latent_dim, half_batch)\r\n\t\t\t# create training set for the discriminator\r\n\t\t\tX, y = vstack((X_real, X_fake)), vstack((y_real, y_fake))\r\n\t\t\t# update discriminator model weights\r\n\t\t\td_loss, _ = d_model.train_on_batch(X, y)\r\n\t\t\t# prepare points in latent space as input for the generator\r\n\t\t\tX_gan = generate_latent_points(latent_dim, n_batch)\r\n\t\t\t# create inverted labels for the fake samples\r\n\t\t\ty_gan = ones((n_batch, 1))\r\n\t\t\t# update the generator via the discriminator's error\r\n\t\t\tg_loss = gan_model.train_on_batch(X_gan, y_gan)\r\n\t\t\t# summarize loss on this batch\r\n\t\t\tprint('>%d, %d\/%d, d=%.3f, g=%.3f' % (i+1, j+1, bat_per_epo, d_loss, g_loss))\r\n\t\t# evaluate the model performance, sometimes\r\n\t\tif (i+1) % 10 == 0:\r\n\t\t\tsummarize_performance(i, g_model, d_model, dataset, latent_dim)\r\n\r\n# size of the latent space\r\nlatent_dim = 100\r\n# create the discriminator\r\nd_model = define_discriminator()\r\n# create the generator\r\ng_model = define_generator(latent_dim)\r\n# create the gan\r\ngan_model = define_gan(g_model, d_model)\r\n# load image data\r\ndataset = load_real_samples()\r\n# train model\r\ntrain(g_model, d_model, gan_model, dataset, latent_dim)<\/pre>\n>1, 1\/234, d=0.711, g=0.678\r\n>1, 2\/234, d=0.703, g=0.698\r\n>1, 3\/234, d=0.694, g=0.717\r\n>1, 4\/234, d=0.684, g=0.740\r\n>1, 5\/234, d=0.679, g=0.757\r\n>1, 6\/234, d=0.668, g=0.777\r\n...\r\n>100, 230\/234, d=0.690, g=0.710\r\n>100, 231\/234, d=0.692, g=0.705\r\n>100, 232\/234, d=0.698, g=0.701\r\n>100, 233\/234, d=0.697, g=0.688\r\n>100, 234\/234, d=0.693, g=0.698<\/pre>\n
>Accuracy real: 51%, fake: 78%\r\n>Accuracy real: 30%, fake: 95%\r\n>Accuracy real: 75%, fake: 59%\r\n>Accuracy real: 98%, fake: 11%\r\n>Accuracy real: 27%, fake: 92%\r\n>Accuracy real: 21%, fake: 92%\r\n>Accuracy real: 29%, fake: 96%\r\n>Accuracy real: 4%, fake: 99%\r\n>Accuracy real: 18%, fake: 97%\r\n>Accuracy real: 28%, fake: 89%<\/pre>\n
![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/04\/Plot-of-100-GAN-Generated-MNIST-Figures-After-10-Epochs.png\")
<\/p>\n![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/04\/Plot-of-100-GAN-Generated-MNIST-Figures-After-40-Epochs.png\")
<\/p>\n![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/04\/Plot-of-100-GAN-Generated-MNIST-Figures-After-100-Epochs.png\")
<\/p>\nHow to Use the Final Generator Model to Generate Images<\/h2>\n
# example of loading the generator model and generating images\r\nfrom keras.models import load_model\r\nfrom numpy.random import randn\r\nfrom matplotlib import pyplot\r\n\r\n# generate points in latent space as input for the generator\r\ndef generate_latent_points(latent_dim, n_samples):\r\n\t# generate points in the latent space\r\n\tx_input = randn(latent_dim * n_samples)\r\n\t# reshape into a batch of inputs for the network\r\n\tx_input = x_input.reshape(n_samples, latent_dim)\r\n\treturn x_input\r\n\r\n# create and save a plot of generated images (reversed grayscale)\r\ndef save_plot(examples, n):\r\n\t# plot images\r\n\tfor i in range(n * n):\r\n\t\t# define subplot\r\n\t\tpyplot.subplot(n, n, 1 + i)\r\n\t\t# turn off axis\r\n\t\tpyplot.axis('off')\r\n\t\t# plot raw pixel data\r\n\t\tpyplot.imshow(examples[i, :, :, 0], cmap='gray_r')\r\n\tpyplot.show()\r\n\r\n# load model\r\nmodel = load_model('generator_model_100.h5')\r\n# generate images\r\nlatent_points = generate_latent_points(100, 25)\r\n# generate images\r\nX = model.predict(latent_points)\r\n# plot the result\r\nsave_plot(X, 5)<\/pre>\n![\"Example](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/04\/Example-of-25-GAN-Generated-MNIST-Handwritten-Images-1024x768.png\")
<\/p>\n# example of generating an image for a specific point in the latent space\r\nfrom keras.models import load_model\r\nfrom numpy import asarray\r\nfrom matplotlib import pyplot\r\n# load model\r\nmodel = load_model('generator_model_100.h5')\r\n# all 0s\r\nvector = asarray([[0.0 for _ in range(100)]])\r\n# generate image\r\nX = model.predict(vector)\r\n# plot the result\r\npyplot.imshow(X[0, :, :, 0], cmap='gray_r')\r\npyplot.show()<\/pre>\n![\"Example](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/04\/Example-of-a-GAN-Generated-MNIST-Handwritten-Digit-for-a-Vector-of-Zeros-300x225.png\")
<\/p>\nExtensions<\/h2>\n
\n
\nPost your findings in the comments below.<\/p>\nFurther Reading<\/h2>\n
Books<\/h3>\n
\n
Papers<\/h3>\n
\n
API<\/h3>\n
\n
Posts<\/h3>\n