Author: Kim Martineau | MIT Quest for Intelligence<\/p>\n
If you\u2019ve ever wondered what a loaf of bread would look like as a cat<\/a>,\u00a0edges2cats<\/a>\u00a0is for you. The program that turns\u00a0sketches into images of cats is one of many whimsical creations inspired\u00a0by\u00a0Phillip Isola<\/a>\u2019s image-to-image translation software released in the early days of generative adversarial networks, or GANs. In a 2016 paper, Isola and his colleagues showed how a\u00a0new type of GAN\u00a0could transform<\/a> a hand-drawn shoe into\u00a0its fashion-photo equivalent, or turn an aerial photo into a grayscale map. Later, the researchers\u00a0showed<\/a>\u00a0how\u00a0landscape photos\u00a0could be reimagined in the impressionist brushstrokes of Monet or Van Gogh. Now an assistant professor in MIT\u2019s\u00a0Department of Electrical Engineering and Computer Science, Isola continues to explore what GANs can do.\u00a0<\/em><\/p>\n GANs work by pairing two neural networks, trained on a large set of images. One network, the generator, outputs an image patterned after the training examples. The other network, the discriminator, rates how well the generator\u2019s output image resembles the training data. If the discriminator can tell it\u2019s a fake, the generator tries again and again until its output images are indistinguishable from the examples. When Isola first heard of GANs, he was experimenting with nearest-neighbor algorithms to try to\u00a0infer the underlying structure of objects and scenes. <\/em><\/p>\n GANs have an uncanny ability to get at the essential structure of a place, face, or object, making structured prediction easier. Introduced five years ago, GANs have been used to visualize the\u00a0ravages of climate change<\/a>, produce more realistic\u00a0computer simulations<\/a>, and\u00a0protect\u00a0sensitive data<\/a>, among other applications. <\/em><\/p>\n To connect the growing number of GAN enthusiasts at MIT and beyond, Isola has recently helped to organize\u00a0GANocracy<\/a>, a day of talks, tutorials, and posters being held at MIT on May 31 that is co-sponsored by the\u00a0MIT Quest for Intelligence<\/a>\u00a0and\u00a0MIT-IBM Watson AI Lab<\/a>. Isola recently spoke about the future of GANs.<\/em><\/p>\n Q:\u00a0<\/strong>Your\u00a0image-to-image translation paper\u00a0has more than 2,000 citations. What made it so popular?<\/p>\n A:<\/strong> It was one of the earlier papers<\/a> to show that GANs are useful for predicting visual data. We showed that this setting is very general, and can be thought of as translating between different visualizations of the world, which we called image-to-image translation. GANs were originally proposed as a model for producing realistic images from scratch. But the most useful application may be structured prediction, which is what GANs are mostly being used for these days.<\/p>\n Q: <\/strong>GANs are easily customized and shared on social media. Any favorites among these projects?<\/p>\n A:<\/strong> #Edges2cats<\/a> is probably my favorite, and it helped to popularize the framework early on. Architect Nono Mart\u00ednez Alonso has used pix2pix<\/a> for exploring interesting tools for\u00a0sketch-based design<\/a>.\u00a0I like everything by Mario Klingemann;\u00a0Alternative Face<\/a>\u00a0is especially thought-provoking. It puts one person\u2019s words into someone else\u2019s mouth, hinting at a potential future of \u201calternative facts.\u201d\u00a0Scott Eaton<\/a>\u00a0is pushing the limits of GANs by translating sketches into 3-D sculptures.\u00a0<\/p>\n Q: <\/strong>What other GAN art grabs you?<\/p>\n A:<\/strong> I really like all of it. One remarkable example is\u00a0GANbreeder<\/a>. It\u2019s a human-curated evolution of GAN-generated images. The crowd chooses which images to breed or kill off. Over many generations, we end up with beautiful and unexpected images.<\/p>\n Q: <\/strong>How are GANs being used beyond art?\u00a0<\/strong><\/p>\n A:<\/strong> In medical imaging, they\u2019re being used to generate CT scans from MRIs. There\u2019s potential there, but it can be easy to misinterpret the results: GANs are\u00a0making predictions, not revealing the truth. We don’t yet have good ways to measure the uncertainty of their predictions. I’m also excited about the use of GANs for simulations. Robots are often trained in simulators to reduce costs, creating complications when we deploy them in the real world. GANs can help bridge the gap between simulation and reality.<\/p>\n Q: <\/strong>Will GANs redefine what it means to be an artist?<\/p>\n A:<\/strong> I don’t know, but it’s a super-interesting question. Several of our\u00a0GANocracy speakers<\/a>\u00a0are artists, and I hope will touch on this. GANs and other generative models are different than other kinds of algorithmic art. They are trained to imitate, so the people being imitated probably deserve some credit. The art collective, Obvious,\u00a0recently sold\u00a0a GAN image<\/a> at Christie’s for $432,500. Obvious selected the image, signed and framed it, but the code was derived from work by\u00a0then-17-year-old\u00a0Robbie Barrat<\/a>. Ian Goodfellow helped develop the underlying algorithm.\u00a0<\/p>\n Q: <\/strong>Where is the field heading?<\/p>\n A:<\/strong> As amazing as GANs are, they are just one type of generative model. GANs might eventually fade in popularity, but generative models are here to stay. As models of high-dimensional structured data,\u00a0generative models get\u00a0close to what we mean when we say \u201ccreate,\u201d \u201cvisualize,\u201d and \u201cimagine.\u201d I think they will be used more and more to approximate capabilities that still seem uniquely human. But GANs do have some unique properties. For one, they solve the generative modeling problem via a two-player competition, creating a generator-discriminator arms race that leads to emergent complexity. Arms races show up across machine learning, including in the AI that achieved superhuman abilities in the game Go.<\/p>\n Q: <\/strong>Are you worried about the potential abuse of GANs?<\/p>\n A:<\/strong> I\u2019m definitely concerned about the use of GANs to generate and spread misleading content, or so-called fake news. GANs make it a lot easier to create doctored photos and videos, where you no longer have to be a video editing expert to make it look like a politician is saying something they never actually said.<\/p>\n Q: <\/strong>You and the other GANocracy organizers are advocating for so-called GANtidotes. Why?<\/p>\n A:<\/strong> We would like to inoculate society against the misuse of GANs. Everyone could just stop trusting what we see online, but then we\u2019d risk losing touch with reality. I\u2019d like to preserve a future in which \u201cseeing is believing.\u201d Luckily, many people are working on technical antidotes that range from detectors that seek out the telltale artifacts in a GAN-manipulated image to cryptographic signatures that verify that a photo has not been edited since it was taken. There are a lot of ideas out there, so I\u2019m optimistic it can be solved.<\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"