The Least Squares Generative Adversarial Network, or LSGAN for short, is an extension to the GAN architecture that addresses the problem of vanishing gradients and loss saturation.<\/p>\n
It is motivated by the desire to provide a signal to the generator about fake samples that are far from the discriminator model\u2019s decision boundary for classifying them as real or fake. The further the generated images are from the decision boundary, the larger the error signal provided to the generator, encouraging the generation of more realistic images.<\/p>\n
The LSGAN can be implemented with a minor change to the output layer of the discriminator layer and the adoption of the least squares, or L2, loss function.<\/p>\n
In this tutorial, you will discover how to develop a least squares generative adversarial network.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- The LSGAN addresses vanishing gradients and loss saturation of the deep convolutional GAN.<\/li>\n
- The LSGAN can be implemented by a mean squared error or L2 loss function for the discriminator model.<\/li>\n
- How to implement the LSGAN model for generating handwritten digits for the MNIST dataset.<\/li>\n<\/ul>\n
Discover how to develop DCGANs, conditional GANs, Pix2Pix, CycleGANs, and more with Keras in my new GANs book<\/a>, with 29 step-by-step tutorials and full source code.<\/p>\n
Let\u2019s get started.<\/p>\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/07\/How-to-Develop-a-Least-Squares-Generative-Adversarial-Network-LSGAN-for-Image-Generation.jpg\")
<\/p>\nHow to Develop a Least Squares Generative Adversarial Network (LSGAN) for Image Generation
Photo by alyssa BLACK<\/a>., some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into three parts; they are:<\/p>\n
- \n
- What Is Least Squares GAN<\/li>\n
- How to Develop an LSGAN for MNIST Handwritten Digits<\/li>\n
- How to Generate Images With LSGAN<\/li>\n<\/ol>\n
What Is Least Squares GAN<\/h2>\n
The standard Generative Adversarial Network, or GAN for short, is an effective architecture for training an unsupervised generative model.<\/p>\n
The architecture involves training a discriminator model to tell the difference between real (from the dataset) and fake (generated) images, and using the discriminator, in turn, to train the generator model. The generator is updated in such a way that it is encouraged to generate images that are more likely to fool the discriminator.<\/p>\n
The discriminator is a binary classifier and is trained using binary cross-entropy loss function. A limitation of this loss function is that it is primarily concerned with whether the predictions are correct or not, and less so with how correct or incorrect they might be.<\/p>\n
\n
\u2026 when we use the fake samples to update the generator by making the discriminator believe they are from real data, it will cause almost no error because they are on the correct side, i.e., the real data side, of the decision boundary<\/p>\n<\/blockquote>\n
\u2014 Least Squares Generative Adversarial Networks<\/a>, 2016.<\/p>\n
This can be conceptualized in two dimensions as a line or decision boundary separating dots that represent real and fake images. The discriminator is responsible for devising the decision boundary to best separate real and fake images and the generator is responsible for creating new points that look like real points, confusing the discriminator.<\/p>\n
The choice of cross-entropy loss means that points generated far from the boundary are right or wrong, but provide very little gradient information to the generator on how to generate better images.<\/p>\n
This small gradient for generated images far from the decision boundary is referred to as a vanishing gradient problem or a loss saturation. The loss function is unable to give a strong signal as to how to best update the model.<\/p>\n
The Least Squares Generative Adversarial Network, or LSGAN for short, is an extension to the GAN architecture proposed by Xudong Mao<\/a>, et al. in their 2016 paper titled \u201cLeast Squares Generative Adversarial Networks<\/a>.\u201d The LSGAN is a modification to the GAN architecture that changes the loss function for the discriminator from binary cross entropy to a least squares loss.<\/p>\n
The motivation for this change is that the least squares loss will penalize generated images based on their distance from the decision boundary. This will provide a strong gradient signal for generated images that are very different or far from the existing data and address the problem of saturated loss.<\/p>\n
\n
\u2026 minimizing the objective function of regular GAN suffers from vanishing gradients, which makes it hard to update the generator. LSGANs can relieve this problem because LSGANs penalize samples based on their distances to the decision boundary, which generates more gradients to update the generator.<\/p>\n<\/blockquote>\n
\u2014 Least Squares Generative Adversarial Networks<\/a>, 2016.<\/p>\n
This can be conceptualized with a plot, below, taken from the paper, that shows on the left the sigmoid decision boundary (blue) and generated fake points far from the decision boundary (pink), and on the right the least squares decision boundary (red) and the points far from the boundary (pink) given a gradient that moves them closer to the boundary.<\/p>\n
![\"Plot](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/05\/Plot-of-the-Sigmoid-Decision-Boundary-vs-the-Least-Squared-Decision-Boundary-for-Updating-the-Generator.png\")
<\/p>\nPlot of the Sigmoid Decision Boundary vs. the Least Squared Decision Boundary for Updating the Generator.
Taken from: Least Squares Generative Adversarial Networks.<\/p>\n<\/div>\nIn addition to avoiding loss saturation, the LSGAN also results in a more stable training process and the generation of higher quality and larger images than the traditional deep convolutional GAN.<\/p>\n
\n
First, LSGANs are able to generate higher quality images than regular GANs. Second, LSGANs perform more stable during the learning process.<\/p>\n<\/blockquote>\n
![\"Summary](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/05\/Summary-of-the-Generator-left-and-Discriminator-right-Model-Architectures-used-in-LSGAN-Experiments.png\")
<\/p>\n